GCP
Highflame Deployment Docs for Google Cloud Platform
Deployment on GKE
⚠️ Reference Only (If you are using Terraform)
The highflame-iac repository contains reference Terraform configurations for managing Highflame cloud resources. Do not copy these directly into your IaC code or run them against your cloud environment without first reviewing and patching them to match your account, networking, and security requirements.
Prerequisites
Ensure that the following tools and resources are installed and available:
Access to the GCP console
Set of domain names for Highflame services (Highflame Team will share the list of services that require the ingress)
GKE cluster with at least 6 worker nodes
Create a Global static IP and use it in the GCE ingress
Enable logging for GKE
GKE addons to be installed
Horizontal Pod Autoscaling
Http Load Balancing
GCE Persistent Disk CSI driver
Network Policy Config
Custom GCP service account and assign it to the GKE node pool with the following permissions
logging.logEntries.createlogging.logEntries.routemonitoring.metricDescriptors.createmonitoring.metricDescriptors.getmonitoring.metricDescriptors.listmonitoring.monitoredResourceDescriptors.getmonitoring.monitoredResourceDescriptors.listmonitoring.timeSeries.create
Create a default node pool for system nodes in the GKE cluster
Postgres Server from Cloud SQL
Memorystore for Redis
Helm v3
Kubectl utility
All the cloud resources (managed services such as Postgres, Redis etc) should be in the same VPC, or those should be accessible from the Kubernetes Cluster
Cloud Resources and Sizing
Highflame services
Deploying the services into your GKE cluster with help of the helm charts
CPU nodes: Setup atleast 3 nodes and the worker node type can be
c2d-highcpu-8GPU nodes: Setup atleast 4 nodes and the worker node type can be
c2d-highcpu-8with gpunvidia-tesla-t4enabled
Trascational Database
Cloud SQL Postgres server
The Postgres server can be single node or primary - secondary cluster and the db tier can be db-custom-4-8192
Analytical Database
Clickhouse database server
Deploying into the GKE
Cache
Memorystore for Redis
Redis Memory size can be 4
Object Store
GCS bucket
Highflame services required to store files and data in object store, such as Store the clickhouse db backup in the GCS bucket for DR as clickhouse is deploying into the GKE cluster
Logging
Cloud logging
The GKE service logs can be pushed to the logging space with help of GKE addons
Authentication
Clerk
External - Managed by Highflame
Highflame GCP Service Account
Prerequisites
Ensure that the following tools and resources are installed and available:
Access to the GKE cluster setup above
gcloud CLI
Service Account
Setting up the environment vars
Create a GCP service account
GCP Workload Identity
Resource Access - GCP GCS
There will be a few GCS bucket requirements for HighFlame, and those were mentioned in the highflame service variables list.
Grant the GCP service account access to GCS
Logging Setup
GCP provides a fully managed logging solution natively integrated into GKE via the GCP Ops Agent (powered by an optimized, Google-managed Fluent Bit daemonset). It is automatically configured to:
Fetch container logs from
/var/log/containers/*.logfor each microservice deployed in GKE.Automatically parse and enrich logs with Kubernetes metadata (namespace, pod name, container name, and cluster location).
Exclude system noise and filter logs to focus strictly on your workload namespaces.
Stream logs automatically to Google Cloud Logging (formerly Stackdriver) with zero manual agent management.
Note: In GCP, logs are natively grouped by the Resource Type k8s_container. Instead of creating separate physical tables per service, you filter your unified log stream dynamically using resource.labels.container_name.
Analytical Database setup - Clickhouse on GKE
This document is the long-form runbook for installing ClickHouse on a GKE cluster using the Altinity ClickHouse Operator, and a GCS bucket-backed clickhouse-backup sidecar.
What gets deployed
The full stack lives entirely inside a dedicated clickhouse namespace and consists of five logical components:
Altinity operator
Helm release ch-operator
Upstream chart
Watches ClickHouseInstallation (CHI) and ClickHouseKeeperInstallation (CHK) CRDs
Service account, ConfigMap, Secret
k8s resources
highflame-clickhouse-deps.yml
IRSA-annotated clickhouse-sa, clickhouse-cm, and clickhouse-secrets
ClickHouse Keeper
ClickHouseKeeperInstallation/ch
highflame-clickhouse-values.yml
1 replica, 20Gi PVC. Replaces ZooKeeper
ClickHouse cluster
ClickHouseInstallation/ch
highflame-clickhouse-values.yml
1 shard × 1 replica, 4 CPU / 12 Gi RAM, 512 Gi PVC, clickhouse-backup sidecar
Backup config
k8s resources
highflame-clickhouse-backup-config.yml
Backup config for GCS bucket integration
CronJob + Job
k8s resources
highflame-clickhouse-addons-values.yml
12-hourly remote backups, 60-backup retention; setup Job applies 3-day TTL on system.*_log tables
Pinned versions (verify before installing):
Operator chart:0.25.5clickhouse/clickhouse-server:25.10.2clickhouse/clickhouse-keeper:25.10.2altinity/clickhouse-backup:2.6.39
After installation, the in-cluster service endpoints are:
HTTP:
clickhouse-ch.clickhouse.svc.cluster.local:8123Native TCP:
clickhouse-ch.clickhouse.svc.cluster.local:9000Keeper:
keeper-ch.clickhouse.svc.cluster.local:2181
These match the CLICKHOUSE_HOST ConfigMap value and the zookeeper.nodes block in the CHI spec
Prerequisites
Before you start, you need:
A GKE cluster with
kubectlcontext pointing at it (kubectl config current-context).A storage class that supports
ReadWriteOncePVCs —pd-ssdis recommended. Both the Keeper and the ClickHouse server use the cluster default unless you uncommentstorageClassNamein the CHI/CHK templates.A GCS bucket for remote backups, in the same GCP project as the cluster. Versioning + lifecycle rules are recommended but not required.
A Service account for the backup Service Account (IRSA) with permission to the GCS bucket
roles/storage.objectViewerandroles/iam.serviceAccountTokenCreator. The SA must allow the OIDC provider of the cluster to assume it forsystem:serviceaccount:clickhouse:clickhouse-sa.helm≥ 3.10
Config files
highflame-clickhouse-deps.yml
Defines:
ServiceAccount/clickhouse-sa— annotationiam.gke.io/gcp-service-accountmust be set to the SA created in the prerequisites.ConfigMap/clickhouse-cm—CLICKHOUSE_HOST,BACKUP_KEEP=60,CREATE_DB_LIST(comma-separated DBs to create on first boot),BACKUP_DB_LIST=highflame(comma-separated DBs to back up).Secret/clickhouse-secrets— admin / readonly / backup usernames + passwords.
You must edit:
iam.gke.io/gcp-service-account
IRSA Service account (SA)
CH_ADMIN_PASSWORD
strong password for admin user highflame_admin
CH_READONLY_PASSWORD
strong password for readonly user highflame_readonly
CH_BACKUP_PASSWORD
strong password for backup user highflame_backup
highflame-clickhouse-values.yml
Defines the ClickHouseKeeperInstallation/ch and ClickHouseInstallation/ch custom resources.
Other useful knobs in this file (edit if your environment differs):
templates.podTemplates[clickhouse].spec.containers[clickhouse].resources— default request and limit are bothcpu: 4,memory: 12Gi.templates.volumeClaimTemplates[clickhouse-data].resources.requests.storage— default512Gi.templates.podTemplates[clickhouse].spec.nodeSelector— uncomment to pin to a specific nodegroup.
highflame-clickhouse-backup-config.yml
Renders into Secret/clickhouse-backup-config (mounted into the sidecar at /opt/backup-config.yml). Update:
SA_EMAIL— GCP service account emailGCS_BUCKET— GCS bucket nameclickhouse.username/clickhouse.passwordto matchCH_BACKUP_USERNAME/CH_BACKUP_PASSWORDfrom the previous stepThe default
path: highflame-clickhouseis a key prefix inside the GCS bucket — change it per environment if you share a GCS bucket between environments.
highflame-clickhouse-addons-values.yml
Defines:
CronJob/clickhouse-backup— runs every 12h (0 */12 * * *), iteratesBACKUP_DB_LISTfrom the ConfigMap, callsclickhouse-backup create_remoteper DB, and prunes old remote backups beyondBACKUP_KEEP.Job/clickhouse-setup— one-shot job that waits for ClickHouse to answer/ping, creates databases listed inCREATE_DB_LIST, and appliesMODIFY TTL event_date + INTERVAL 3 DAYto everysystem.*_logtable.
Install — step by step
Run from the repo root unless noted otherwise.
Namespace
Operator
Verify the operator pod is running, and the CRDs are installed:
You should see
clickhouseinstallations.clickhouse.altinity.comclickhouseinstallationtemplates.clickhouse.altinity.comclickhousekeeperinstallations.clickhouse-keeper.altinity.com
Dependencies (SA + ConfigMap + Secret)
Keeper + ClickHouse cluster
Verify the deployment:
Backup secret
The setup Job runs once and exits Completed. The CronJob fires every 12h
Clickhouse GCS bucket backup management (From the backup containers)
List down the remote backup
Restore from remote backup (For DR)
Highflame service deployment
Follow this documentation to deploy Highflame services to your Kubernetes cluster using Helm charts.
Whether you're standing up a fresh environment or upgrading an existing one, this guide has you covered end to end.
Highflame ServicesLast updated