GCP

Highflame Deployment Docs for Google Cloud Platform

Deployment on GKE

⚠️ Reference Only (If you are using Terraform)

The highflame-iac repository contains reference Terraform configurations for managing Highflame cloud resources. Do not copy these directly into your IaC code or run them against your cloud environment without first reviewing and patching them to match your account, networking, and security requirements.

Prerequisites

Ensure that the following tools and resources are installed and available:

  • Access to the GCP console

  • Set of domain names for Highflame services (Highflame Team will share the list of services that require the ingress)

  • GKE cluster with at least 6 worker nodes

  • Create a Global static IP and use it in the GCE ingress

  • Enable logging for GKE

  • GKE addons to be installed

    • Horizontal Pod Autoscaling

    • Http Load Balancing

    • GCE Persistent Disk CSI driver

    • Network Policy Config

  • Custom GCP service account and assign it to the GKE node pool with the following permissions

    • logging.logEntries.create

    • logging.logEntries.route

    • monitoring.metricDescriptors.create

    • monitoring.metricDescriptors.get

    • monitoring.metricDescriptors.list

    • monitoring.monitoredResourceDescriptors.get

    • monitoring.monitoredResourceDescriptors.list

    • monitoring.timeSeries.create

  • Create a default node pool for system nodes in the GKE cluster

  • Postgres Server from Cloud SQL

  • Memorystore for Redis

  • Helm v3

  • Kubectl utility

  • All the cloud resources (managed services such as Postgres, Redis etc) should be in the same VPC, or those should be accessible from the Kubernetes Cluster

Cloud Resources and Sizing

Highflame components
Cloud Resources
Size

Highflame services

Deploying the services into your GKE cluster with help of the helm charts

  • CPU nodes: Setup atleast 3 nodes and the worker node type can be c2d-highcpu-8

  • GPU nodes: Setup atleast 4 nodes and the worker node type can be c2d-highcpu-8 with gpu nvidia-tesla-t4 enabled

Trascational Database

Cloud SQL Postgres server

The Postgres server can be single node or primary - secondary cluster and the db tier can be db-custom-4-8192

Analytical Database

Clickhouse database server

Deploying into the GKE

Cache

Memorystore for Redis

Redis Memory size can be 4

Object Store

GCS bucket

Highflame services required to store files and data in object store, such as Store the clickhouse db backup in the GCS bucket for DR as clickhouse is deploying into the GKE cluster

Logging

Cloud logging

The GKE service logs can be pushed to the logging space with help of GKE addons

Authentication

Clerk

External - Managed by Highflame

Highflame GCP Service Account

Prerequisites

Ensure that the following tools and resources are installed and available:

  • Access to the GKE cluster setup above

  • gcloud CLI

Service Account

  1. Setting up the environment vars

  1. Create a GCP service account

  1. GCP Workload Identity

Resource Access - GCP GCS

There will be a few GCS bucket requirements for HighFlame, and those were mentioned in the highflame service variables list.

  1. Grant the GCP service account access to GCS

Logging Setup

GCP provides a fully managed logging solution natively integrated into GKE via the GCP Ops Agent (powered by an optimized, Google-managed Fluent Bit daemonset). It is automatically configured to:

  • Fetch container logs from /var/log/containers/*.log for each microservice deployed in GKE.

  • Automatically parse and enrich logs with Kubernetes metadata (namespace, pod name, container name, and cluster location).

  • Exclude system noise and filter logs to focus strictly on your workload namespaces.

  • Stream logs automatically to Google Cloud Logging (formerly Stackdriver) with zero manual agent management.

Note: In GCP, logs are natively grouped by the Resource Type k8s_container. Instead of creating separate physical tables per service, you filter your unified log stream dynamically using resource.labels.container_name.

Analytical Database setup - Clickhouse on GKE

This document is the long-form runbook for installing ClickHouse on a GKE cluster using the Altinity ClickHouse Operator, and a GCS bucket-backed clickhouse-backup sidecar.

What gets deployed

The full stack lives entirely inside a dedicated clickhouse namespace and consists of five logical components:

Component
Kind
Source
Notes

Altinity operator

Helm release ch-operator

Upstream chart

Watches ClickHouseInstallation (CHI) and ClickHouseKeeperInstallation (CHK) CRDs

Service account, ConfigMap, Secret

k8s resources

highflame-clickhouse-deps.yml

IRSA-annotated clickhouse-sa, clickhouse-cm, and clickhouse-secrets

ClickHouse Keeper

ClickHouseKeeperInstallation/ch

highflame-clickhouse-values.yml

1 replica, 20Gi PVC. Replaces ZooKeeper

ClickHouse cluster

ClickHouseInstallation/ch

highflame-clickhouse-values.yml

1 shard × 1 replica, 4 CPU / 12 Gi RAM, 512 Gi PVC, clickhouse-backup sidecar

Backup config

k8s resources

highflame-clickhouse-backup-config.yml

Backup config for GCS bucket integration

CronJob + Job

k8s resources

highflame-clickhouse-addons-values.yml

12-hourly remote backups, 60-backup retention; setup Job applies 3-day TTL on system.*_log tables

Pinned versions (verify before installing):

  • Operator chart: 0.25.5

  • clickhouse/clickhouse-server: 25.10.2

  • clickhouse/clickhouse-keeper: 25.10.2

  • altinity/clickhouse-backup: 2.6.39

After installation, the in-cluster service endpoints are:

  • HTTP: clickhouse-ch.clickhouse.svc.cluster.local:8123

  • Native TCP: clickhouse-ch.clickhouse.svc.cluster.local:9000

  • Keeper: keeper-ch.clickhouse.svc.cluster.local:2181

These match the CLICKHOUSE_HOST ConfigMap value and the zookeeper.nodes block in the CHI spec

Prerequisites

Before you start, you need:

  1. A GKE cluster with kubectl context pointing at it (kubectl config current-context).

  2. A storage class that supports ReadWriteOnce PVCs — pd-ssd is recommended. Both the Keeper and the ClickHouse server use the cluster default unless you uncomment storageClassName in the CHI/CHK templates.

  3. A GCS bucket for remote backups, in the same GCP project as the cluster. Versioning + lifecycle rules are recommended but not required.

  4. A Service account for the backup Service Account (IRSA) with permission to the GCS bucket roles/storage.objectViewer and roles/iam.serviceAccountTokenCreator. The SA must allow the OIDC provider of the cluster to assume it for system:serviceaccount:clickhouse:clickhouse-sa.

  5. helm ≥ 3.10

Config files

highflame-clickhouse-deps.yml

Defines:

  • ServiceAccount/clickhouse-sa — annotation iam.gke.io/gcp-service-account must be set to the SA created in the prerequisites.

  • ConfigMap/clickhouse-cmCLICKHOUSE_HOST, BACKUP_KEEP=60, CREATE_DB_LIST (comma-separated DBs to create on first boot), BACKUP_DB_LIST=highflame (comma-separated DBs to back up).

  • Secret/clickhouse-secrets — admin / readonly / backup usernames + passwords.

You must edit:

Field
Required value

iam.gke.io/gcp-service-account

IRSA Service account (SA)

CH_ADMIN_PASSWORD

strong password for admin user highflame_admin

CH_READONLY_PASSWORD

strong password for readonly user highflame_readonly

CH_BACKUP_PASSWORD

strong password for backup user highflame_backup

highflame-clickhouse-values.yml

Defines the ClickHouseKeeperInstallation/ch and ClickHouseInstallation/ch custom resources.

Other useful knobs in this file (edit if your environment differs):

  • templates.podTemplates[clickhouse].spec.containers[clickhouse].resources — default request and limit are both cpu: 4, memory: 12Gi.

  • templates.volumeClaimTemplates[clickhouse-data].resources.requests.storage — default 512Gi.

  • templates.podTemplates[clickhouse].spec.nodeSelector — uncomment to pin to a specific nodegroup.

highflame-clickhouse-backup-config.yml

Renders into Secret/clickhouse-backup-config (mounted into the sidecar at /opt/backup-config.yml). Update:

  • SA_EMAIL — GCP service account email

  • GCS_BUCKET — GCS bucket name

  • clickhouse.username / clickhouse.password to match CH_BACKUP_USERNAME / CH_BACKUP_PASSWORD from the previous step

  • The default path: highflame-clickhouse is a key prefix inside the GCS bucket — change it per environment if you share a GCS bucket between environments.

highflame-clickhouse-addons-values.yml

Defines:

  • CronJob/clickhouse-backup — runs every 12h (0 */12 * * *), iterates BACKUP_DB_LIST from the ConfigMap, calls clickhouse-backup create_remote per DB, and prunes old remote backups beyond BACKUP_KEEP.

  • Job/clickhouse-setup — one-shot job that waits for ClickHouse to answer /ping, creates databases listed in CREATE_DB_LIST, and applies MODIFY TTL event_date + INTERVAL 3 DAY to every system.*_log table.

Install — step by step

Run from the repo root unless noted otherwise.

Namespace

Operator

Verify the operator pod is running, and the CRDs are installed:

You should see

  • clickhouseinstallations.clickhouse.altinity.com

  • clickhouseinstallationtemplates.clickhouse.altinity.com

  • clickhousekeeperinstallations.clickhouse-keeper.altinity.com

Dependencies (SA + ConfigMap + Secret)

Keeper + ClickHouse cluster

Verify the deployment:

Backup secret

The setup Job runs once and exits Completed. The CronJob fires every 12h

Clickhouse GCS bucket backup management (From the backup containers)

List down the remote backup

Restore from remote backup (For DR)

Highflame service deployment

Follow this documentation to deploy Highflame services to your Kubernetes cluster using Helm charts.

Whether you're standing up a fresh environment or upgrading an existing one, this guide has you covered end to end.

Highflame Services

Last updated