Single Region

Single region deployment in AWS consolidates all infrastructure components within one selected region.

Single Region Deployment on EKS

⚠️ Reference Only (If you are using Terraform)

The highflame-iac repository contains reference Terraform configurations for managing Highflame cloud resources. Do not copy these directly into your IaC code or run them against your cloud environment without first reviewing and patching them to match your account, networking, and security requirements.

Benefits of Single-Region Deployment

• Simplified Management: Easier oversight and maintenance due to a single, centralized location for all resources.

• Cost Efficiency: Potentially lower data transfer costs between services within the same region.

• Consistency: Ensures uniformity of services and offers within a specific AWS region.

Prerequisites

Ensure that the following tools and resources are installed and available:

• Access to AWS Management Console

• Set of domain names for Highflame services (Highflame Team will share the list of services that require the ingress)

• EKS cluster with at least 6 worker nodes (Best Practice: 3 zones for the VPC and use 6 nodes, with 2 nodes in each Availability Zone, to ensure high availability)

• EKS worker nodes should have the following IAM permissions

  • arn:aws:iam::aws:policy/CloudWatchFullAccess

  • arn:aws:iam::aws:policy/CloudWatchLogsFullAccess

  • arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy

• EKS addons to be installed

  • coredns

  • kube-proxy

  • vpc-cni

  • aws-ebs-csi-driver

  • amazon-cloudwatch-observability

• EKS can be configured with cluster-autoscaler for managing the worker node group scaling

• EKS should be configured with metrics-server for managing the Highflame service HPA

• Aurora Postgres cluster from RDS

• Redis cluster from Elasticache

• ALB ingress controller for EKS and SSL certificates in the ACM

• Helm v3

• Kubectl utility

• All the cloud resources (managed services such as Postgres, Redis etc) should be in the same VPC, or those should be accessible from the Kubernetes Cluster

Cloud Resources and Sizing

Highflame components	Cloud Resources	Size
Highflame services	Deploying the services into your EKS cluster with help of the helm charts	CPU nodes: Setup atleast 3 nodes across different zones and the worker node type can be c6a.2xlarge with ami type AL2023_x86_64_STANDARD GPU nodes: Setup atleast 4 nodes across different zones and the worker node type can be g4dn.xlarge with ami type AL2023_x86_64_NVIDIA
Trascational Database	Aurora database compatible with postgres	The Aurora server can be single node or cluster and the node type can be db.r6g.large
Analytical Database	Clickhouse database server	Deploying into the EKS
Cache	ElastiCache for Redis OSS	Redis cluster of min 2 nodes with node type cache.m5.large
Object Store	S3 bucket	Highflame services required to store files and data in object store, such as Store the clickhouse db backup in the S3 for DR as clickhouse is deploying into the EKS cluster
Logging	Cloudwatch	The EKS service logs can be pushed to the Cloudwatch with help of fluent-bit
Authentication	Clerk	External - Managed by Highflame

Highflame AWS IAM Role

Prerequisites

Ensure that the following tools and resources are installed and available:

• Access to the EKS cluster setup above

• AWS CLI

• eksctl

IAM Role

1. Setting up the environment vars

export K8S_CLUSTER_NAME="highflame-poc-eks"    ## AWS EKS cluster name
export K8S_NAMESPACE="highflame-poc"           ## Highflame k8s namespace
export IAM_ROLE_NAME="highflame-svc-role"      ## Name given for the AWS IAM role
export K8S_SA_NAME="highflame-k8s-sa"          ## Name for K8S service account

1. Get the AWS Account ID

AWS_ACC_ID=$(aws sts get-caller-identity --query Account --output text)

1. Fetch the EKS cluster's OIDC issuer

OIDC_ISSUER=$(aws eks describe-cluster --name ${K8S_CLUSTER_NAME} --query "cluster.identity.oidc.issuer" --output text | sed -e "s~https://~~")

1. Create an IAM OIDC provider if it does not exist

# Confirm whether the OIDC provider is available
aws iam list-open-id-connect-providers | grep ${OIDC_ISSUER}
# If no OIDC provider is present, then create an IAM OIDC provider
eksctl utils associate-iam-oidc-provider \
    --cluster ${K8S_CLUSTER_NAME} --approve

1. Create a Trust Relationship

cat >iam_trust.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${AWS_ACC_ID}:oidc-provider/${OIDC_ISSUER}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_ISSUER}:aud": "sts.amazonaws.com",
          "${OIDC_ISSUER}:sub": "system:serviceaccount:${K8S_NAMESPACE}:${K8S_SA_NAME}"
        }
      }
    }
  ]
}
EOF

aws iam create-role \
  --role-name ${IAM_ROLE_NAME} \
  --assume-role-policy-document file://iam_trust.json

1. Get the ARN of the IAM role

IAM_ROLE_ARN=$(aws iam get-role --role-name ${IAM_ROLE_NAME} --query "Role.Arn" --output text)
echo "${IAM_ROLE_ARN}"

Resource Access - Amazon S3

There will be a few S3 bucket requirements for HighFlame, and those were mentioned in the highflame service variables list.

For each S3 bucket, this policy needs to be added to the IAM role created above

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [ "s3:*" ],
      "Resource": [
        "arn:aws:s3:::${S3_BUCKET_NAME}/*",
        "arn:aws:s3:::${S3_BUCKET_NAME}" 
        ]
    }
  ]
}

Logging Setup

Fluent Bit is deployed as a DaemonSet in the EKS cluster to collect logs from each node. It is configured to:

• Tail logs from /var/log/containers/*.log for each microservice deployed in the EKS

• Parse Kubernetes logs using the Kubernetes filter and create an individual log group with the highflame service name

• Forward logs to Amazon CloudWatch Logs under the respective log stream name, the name matches the highflame service name

IAM Permissions for Fluent Bit (via IRSA) are configured to allow writing to CloudWatch Logs.

A production-ready guide to deploying Fluent Bit as a DaemonSet on an Amazon EKS cluster using Helm 3, with logs shipped to Amazon CloudWatch Logs

Overview

Fluent Bit is a lightweight log processor and forwarder built for containerized environments. On EKS, we deploy it as a DaemonSet, so a single pod runs on every node and tails:

• /var/log/containers/*.log — all container stdout/stderr

• /var/log/messages — node systemd / kubelet logs (optional)

• /var/log/kube-apiserver-audit.log — audit logs (optional)

Logs are enriched with Kubernetes metadata (pod, namespace, labels) and shipped to a configured output — CloudWatch Logs by default.

IAM Setup (IRSA)

Fluent Bit needs permission to write to CloudWatch Logs. We grant this via IAM Roles for Service Accounts (IRSA) — no static keys. Either you can use the EKS worker nodes permission (mentioned in the Prerequisites or create a custom IAM policy and role, then pass it to the fluent-bit helm deployment)

Add the Helm Repository

helm repo add fluent https://fluent.github.io/helm-charts
helm repo update fluent

# See available chart versions
helm search repo fluent/fluent-bit --versions

Install Fluent Bit

A complete, annotated values.yaml for shipping EKS logs to CloudWatch:

Change the following according to your setup in the Helm values file

logNamespace: prod
projectName: highflame
region: us-east-1
logRetention: 365
k8sCluster: highflame-prod-eks

## Change it for each namespace
logNamespace: prod
projectName: highflame
region: us-east-1
logRetention: 365
k8sCluster: highflame-prod-eks

## Actual Helm chart Value file https://github.com/fluent/helm-charts/blob/main/charts/fluent-bit/values.yaml
## https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/configuration-file
config:
  service: |
    [SERVICE]
        Daemon            Off
        Flush             {{ .Values.flush }}
        Log_Level         {{ .Values.logLevel }}
        Parsers_File      /fluent-bit/etc/parsers.conf
        Parsers_File      /fluent-bit/etc/conf/custom_parsers.conf
        HTTP_Server       On
        HTTP_Listen       0.0.0.0
        HTTP_Port         {{ .Values.metricsPort }}
        Health_Check      On

  ## https://docs.fluentbit.io/manual/pipeline/inputs
  inputs: |
    [INPUT]
        Name              tail
        Path              /var/log/containers/*_{{ .Values.logNamespace }}*.log
        Parser            docker
        Tag               {{ .Values.projectName }}.*
        multiline.parser  docker, cri
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On

  ## https://docs.fluentbit.io/manual/pipeline/filters
  filters: |
    [FILTER]
        Name                kubernetes
        Match               {{ .Values.projectName }}.*
        Kube_Tag_Prefix     {{ .Values.projectName }}.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        Keep_Log            Off
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On

    [FILTER]
        Name                grep
        Match               {{ .Values.projectName }}.*
        Exclude             log /.*/health*./
        Exclude             msg /.*/health*./

  ## https://docs.fluentbit.io/manual/pipeline/outputs
  outputs: |
    [OUTPUT]
        Name                cloudwatch_logs
        Match               {{ .Values.projectName }}.*
        region              {{ .Values.region }}
        log_group_name      {{ .Values.k8sCluster }}-service-log
        auto_create_group   On
        log_retention_days  {{ .Values.logRetention }}
        log_stream_name     {{ .Values.projectName }}

  ## https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/upstream-servers
  ## This configuration is deprecated, please use `extraFiles` instead.
  upstream: {}

  ## https://docs.fluentbit.io/manual/pipeline/parsers
  customParsers: |
    [PARSER]
        Name docker_no_time
        Format json
        Time_Keep Off
        Time_Key time
        Time_Format %Y-%m-%dT%H:%M:%S.%L

Deploy the fluent bit

helm install fluent-bit fluent/fluent-bit \
  --namespace logging \
  --create-namespace \
  --version 0.46.2 \
  --values values.yaml \
  --wait --timeout 5m

References

• Fluent Bit Helm chart: https://github.com/fluent/helm-charts/tree/main/charts/fluent-bit

• Fluent Bit docs: https://docs.fluentbit.io

• EKS logging best practices: https://docs.aws.amazon.com/eks/latest/best-practices/logging.html

• CloudWatch output plugin: https://docs.fluentbit.io/manual/pipeline/outputs/cloudwatch

• IRSA: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html

Analytical Database setup - Clickhouse on EKS

This document is the long-form runbook for installing ClickHouse on an EKS cluster using the Altinity ClickHouse Operator, and an S3-backed clickhouse-backup sidecar.

What gets deployed

The full stack lives entirely inside a dedicated clickhouse namespace and consists of five logical components:

Component	Kind	Source	Notes
Altinity operator	Helm release ch-operator	Upstream chart	Watches ClickHouseInstallation (CHI) and ClickHouseKeeperInstallation (CHK) CRDs
Service account, ConfigMap, Secret	k8s resources	highflame-clickhouse-deps.yml	IRSA-annotated clickhouse-sa, clickhouse-cm, and clickhouse-secrets
ClickHouse Keeper	ClickHouseKeeperInstallation/ch	highflame-clickhouse-values.yml	1 replica, 20Gi PVC. Replaces ZooKeeper
ClickHouse cluster	ClickHouseInstallation/ch	highflame-clickhouse-values.yml	1 shard × 1 replica, 4 CPU / 12 Gi RAM, 512 Gi PVC, clickhouse-backup sidecar
Backup config	k8s resources	highflame-clickhouse-backup-config.yml	Backup config for S3 integration
CronJob + Job	k8s resources	highflame-clickhouse-addons-values.yml	12-hourly remote backups, 60-backup retention; setup Job applies 3-day TTL on system.*_log tables

Pinned versions (verify before installing):

• Operator chart: 0.25.5

• clickhouse/clickhouse-server: 25.10.2

• clickhouse/clickhouse-keeper: 25.10.2

• altinity/clickhouse-backup: 2.6.39

After installation, the in-cluster service endpoints are:

• HTTP: clickhouse-ch.clickhouse.svc.cluster.local:8123

• Native TCP: clickhouse-ch.clickhouse.svc.cluster.local:9000

• Keeper: keeper-ch.clickhouse.svc.cluster.local:2181

These match the CLICKHOUSE_HOST ConfigMap value and the zookeeper.nodes block in the CHI spec

Prerequisites

Before you start, you need:

1. An EKS cluster with kubectl context pointing at it (kubectl config current-context).

2. A storage class that supports ReadWriteOnce PVCs — gp3 is recommended. Both the Keeper and the ClickHouse server use the cluster default unless you uncomment storageClassName in the CHI/CHK templates.

3. An S3 bucket for remote backups, in the same AWS account as the cluster. Versioning + lifecycle rules are recommended but not required.

4. An IAM role for the backup Service Account (IRSA) with permission to s3:GetObject, s3:PutObject, s3:DeleteObject, s3:ListBucket on that bucket. The role's trust policy must allow the OIDC provider of the cluster to assume it for system:serviceaccount:clickhouse:clickhouse-sa.

5. helm ≥ 3.10

Config files

highflame-clickhouse-deps.yml

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: ""
  name: clickhouse-sa
  namespace: clickhouse
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: clickhouse-cm
  namespace: clickhouse
data:
  CLICKHOUSE_HOST: "clickhouse-ch.clickhouse.svc.cluster.local"
  BACKUP_KEEP: "60"
  CREATE_DB_LIST: ""
  BACKUP_DB_LIST: "highflame"
---
apiVersion: v1
kind: Secret
metadata:
  name: clickhouse-secrets
  namespace: clickhouse
type: Opaque
stringData:
  CH_ADMIN_USERNAME: "highflame_admin"
  CH_ADMIN_PASSWORD: ""
  CH_READONLY_USERNAME: "highflame_readonly"
  CH_READONLY_PASSWORD: ""
  CH_BACKUP_USERNAME: "highflame_backup"
  CH_BACKUP_PASSWORD: ""

Defines:

• ServiceAccount/clickhouse-sa — annotation eks.amazonaws.com/role-arn must be set to the IRSA role ARN created in the prerequisites.

• ConfigMap/clickhouse-cm — CLICKHOUSE_HOST, BACKUP_KEEP=60, CREATE_DB_LIST (comma-separated DBs to create on first boot), BACKUP_DB_LIST=highflame (comma-separated DBs to back up).

• Secret/clickhouse-secrets — admin / readonly / backup usernames + passwords.

You must edit:

Field	Required value
eks.amazonaws.com/role-arn	IRSA role ARN
CH_ADMIN_PASSWORD	strong password for admin user highflame_admin
CH_READONLY_PASSWORD	strong password for readonly user highflame_readonly
CH_BACKUP_PASSWORD	strong password for backup user highflame_backup

highflame-clickhouse-values.yml

Defines the ClickHouseKeeperInstallation/ch and ClickHouseInstallation/ch custom resources.

apiVersion: "clickhouse-keeper.altinity.com/v1"
kind: "ClickHouseKeeperInstallation"

metadata:
  name: "ch"
  namespace: clickhouse

spec:
  configuration:
    clusters:
      - name: cluster
        layout:
          replicasCount: 1
    settings:
      keeper_server/tcp_port: "2181"
      keeper_server/raft_port: "9444"

  defaults:
    templates:
      podTemplate: keeper
      volumeClaimTemplate: keeper

  templates:
    podTemplates:
      - name: keeper
        metadata:
          labels:
            app: clickhouse-keeper
        spec:
          affinity:
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                - labelSelector:
                    matchExpressions:
                      - key: "app"
                        operator: In
                        values:
                          - clickhouse-keeper
                  topologyKey: "kubernetes.io/hostname"
          containers:
            - name: keeper
              image: "clickhouse/clickhouse-keeper:25.10.2"
              command: [ "/usr/bin/clickhouse-keeper", "start" ]
              resources:
                requests:
                  memory: "512Mi"
                  cpu: "500m"
                limits:
                  memory: "1024Mi"
                  cpu: "1000m"
          securityContext:
            fsGroup: 101

    volumeClaimTemplates:
      - name: keeper
        spec:
          # storageClassName: storage_classname
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 20Gi
---
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"

metadata:
  name: "ch"
  namespace: clickhouse

spec: 
  configuration:
    zookeeper:
      nodes:
        - host: "keeper-ch.clickhouse.svc.cluster.local"
          port: 2181
    clusters:
      - name: cluster
        layout:
          shardsCount: 1
          replicasCount: 1
    users:
      highflame_admin/password:
        valueFrom:
          secretKeyRef:
            name: clickhouse-secrets
            key: CH_ADMIN_PASSWORD
      highflame_admin/networks/ip:
        - "0.0.0.0/0"
        - "::/0"
      highflame_admin/profile: default
      highflame_admin/quota: default
      highflame_readonly/password:
        valueFrom:
          secretKeyRef:
            name: clickhouse-secrets
            key: CH_READONLY_PASSWORD
      highflame_readonly/networks/ip:
        - "0.0.0.0/0"
        - "::/0"
      highflame_readonly/profile: readonly
      highflame_readonly/quota: readonly
      highflame_backup/password:
        valueFrom:
          secretKeyRef:
            name: clickhouse-secrets
            key: CH_BACKUP_PASSWORD
      highflame_backup/networks/ip:
        - "0.0.0.0/0"
        - "::/0"
      highflame_backup/profile: backup
      highflame_backup/quota: backup      
    profiles:
      readonly/readonly: 2
      readonly/max_execution_time: 120
      readonly/max_memory_usage: 1000000000
      readonly/max_rows_to_read: 100000000
      readonly/max_result_rows: 10000
      readonly/read_overflow_mode: throw
      readonly/result_overflow_mode: throw
      readonly/timeout_overflow_mode: throw
      backup/readonly: 0
    quotas:
      readonly/interval/duration: 3600
      backup/interval/duration: 3600

  defaults:
    templates:
      podTemplate: clickhouse
      dataVolumeClaimTemplate: clickhouse-data
      # logVolumeClaimTemplate: clickhouse-log

  templates:
    podTemplates:
      - name: clickhouse
        spec:
          serviceAccountName: clickhouse-sa
          containers:
            - name: clickhouse
              image: clickhouse/clickhouse-server:25.10.2
              resources:
                requests:
                  cpu: "4"
                  memory: "12Gi"
                limits:
                  cpu: "4"
                  memory: "12Gi"
              volumeMounts:
                - name: clickhouse-data
                  mountPath: /var/lib/clickhouse
                # - name: clickhouse-log
                #   mountPath: /var/log/clickhouse-server
            - name: clickhouse-backup
              image: altinity/clickhouse-backup:2.6.39
              command:
                - /bin/bash
                - -c
                - "/bin/clickhouse-backup -c /opt/backup-config.yml server"
              volumeMounts:
                - name: clickhouse-data
                  mountPath: /var/lib/clickhouse
                - name: clickhouse-backup-config
                  mountPath: /opt/backup-config.yml
                  subPath: backup-config.yml
                  readOnly: true
          volumes:
            - name: clickhouse-backup-config
              secret:
                secretName: clickhouse-backup-config
                optional: false
          # nodeSelector:
          #   kube/nodegroup: "general"

    volumeClaimTemplates:
      - name: clickhouse-data
        spec:
          # storageClassName: storage_classname
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 512Gi
      # - name: clickhouse-log
      #   spec:
      #     storageClassName: storage_classname
      #     accessModes:
      #       - ReadWriteOnce
      #     resources:
      #       requests:
      #         storage: 100Gi

Other useful knobs in this file (edit if your environment differs):

• templates.podTemplates[clickhouse].spec.containers[clickhouse].resources — default request and limit are both cpu: 4, memory: 12Gi.

• templates.volumeClaimTemplates[clickhouse-data].resources.requests.storage — default 512Gi.

• templates.podTemplates[clickhouse].spec.nodeSelector — uncomment to pin to a specific nodegroup.

highflame-clickhouse-backup-config.yml

general:
  remote_storage: s3
  disable_progress_bar: false
  backups_to_keep_local: 1
  backups_to_keep_remote: 10
  log_level: info
  allow_empty_backups: false
clickhouse:
  username: highflame_backup
  password: ""
  host: clickhouse-ch.clickhouse.svc.cluster.local
  port: 9000
  disk_mapping: {}
  skip_tables:
    - system.*
  timeout: 5m
  freeze_by_part: false
  secure: false
  skip_verify: false
  sync_replicated_tables: true
  log_sql_queries: false
s3:
  bucket: "S3_BUCKET"
  endpoint: "https://s3.S3_REGION.amazonaws.com"
  region: S3_REGION
  path: highflame-clickhouse
  force_path_style: true
  compression_level: 1
  compression_format: tar
  disable_cert_verification: false
  storage_class: STANDARD

Renders into Secret/clickhouse-backup-config (mounted into the sidecar at /opt/backup-config.yml). Update:

• S3_BUCKET — bucket name.

• S3_REGION — AWS region (drives both region and the endpoint URL).

• clickhouse.username / clickhouse.password to match CH_BACKUP_USERNAME / CH_BACKUP_PASSWORD from the previous step

• The default path: highflame-clickhouse is a key prefix inside the bucket — change it per environment if you share a bucket between environments.

highflame-clickhouse-addons-values.yml

apiVersion: batch/v1
kind: CronJob
metadata:
  name: clickhouse-backup
  namespace: clickhouse
spec:
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 1
  failedJobsHistoryLimit: 1
  schedule: "0 */12 * * *"
  jobTemplate:
    spec:
      activeDeadlineSeconds: 14400
      backoffLimit: 1
      template:
        spec:
          serviceAccountName: clickhouse-sa
          restartPolicy: Never
          containers:
            - name: clickhouse-backup
              image: altinity/clickhouse-backup:2.6.39
              envFrom:
                - configMapRef:
                    name: clickhouse-cm
              command:
                - /bin/bash
                - -c
                - |
                  set -e
                  BACKUP_CFG="/opt/backup-config.yml"
                  BACKUP_TIME="$(date +%F-%H-%M-%S)"

                  if [[ "$${BACKUP_DB_LIST}" != "" ]] ; then
                    IFS=',' read -r -a DBS <<< "$${BACKUP_DB_LIST}"
                    for db in "$${DBS[@]}" ; do
                      BACKUP_NAME="$${db}--$${BACKUP_TIME}"
                      echo "Creating backup $${BACKUP_NAME}"
                      /bin/clickhouse-backup -c $${BACKUP_CFG} create_remote --tables="$${db}.*" $${BACKUP_NAME}

                      echo "Deleting all local backups"
                      rm -rf /var/lib/clickhouse/backup/$${BACKUP_NAME}

                      echo "Rotating old remote backups - keeping last $${BACKUP_KEEP}"
                      BACKUPS=$$(clickhouse-backup -c $${BACKUP_CFG} list remote | grep -i "$${db}--" | awk '{print $$1}')
                      BKP_COUNT=$$(echo "$${BACKUPS}" | wc -l)

                      if [[ "$${BKP_COUNT}" -le "$${BACKUP_KEEP}" ]] ; then
                        echo "Nothing to delete...($${BKP_COUNT} backups, keep $${BACKUP_KEEP})"
                      else
                        DELETE_COUNT=$$((BKP_COUNT - BACKUP_KEEP))
                        DELETE_LIST=$$(echo "$${BACKUPS}" | head -n $${DELETE_COUNT})

                        for bkp in $${DELETE_LIST} ; do
                          echo "Deleting remote backup: $${bkp}"
                          clickhouse-backup -c $${BACKUP_CFG} delete remote "$${bkp}"
                        done
                      fi
                      echo "Backup job completed for database $${db}"
                    done
                  else
                    echo "Backup is disabled. Current BACKUP_DB_LIST is empty"
                  fi
              volumeMounts:
                - name: clickhouse-data
                  mountPath: /var/lib/clickhouse
                - name: clickhouse-backup-config
                  mountPath: /opt/backup-config.yml
                  subPath: backup-config.yml
                  readOnly: true
          volumes:
            - name: clickhouse-data
              persistentVolumeClaim:
                claimName: clickhouse-data
            - name: clickhouse-backup-config
              secret:
                secretName: clickhouse-backup-config
                optional: false
          affinity:
            podAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                - labelSelector:
                    matchExpressions:
                      - key: clickhouse.altinity.com/chi
                        operator: In
                        values:
                          - ch
                  topologyKey: "kubernetes.io/hostname"
---
apiVersion: batch/v1
kind: Job
metadata:
  name: clickhouse-setup
  namespace: clickhouse
spec:
  completions: 1
  parallelism: 1
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      initContainers:
        - name: wait
          image: curlimages/curl:8.17.0
          command:
            - /bin/sh
            - -c
            - |
              echo "Waiting for ClickHouse to be ready..."

              until curl -s "http://$${CLICKHOUSE_HOST}:8123/ping" | grep -q "Ok" ; do
                echo "ClickHouse not ready yet. Waiting..."
                sleep 5
              done

              echo "ClickHouse is ready!"
          envFrom:
            - configMapRef:
                name: clickhouse-cm
      containers:
        - name: setup
          image: clickhouse/clickhouse-server:25.10.2
          command:
            - /bin/bash
            - -c
            - |
              set -e

              export SYS_DB_NAME="system"
              export LOG_TABLES=($(clickhouse-client \
                  --host="$${CLICKHOUSE_HOST}" \
                  --user="$${CLICKHOUSE_USER}" \
                  --password="$${CLICKHOUSE_PASSWORD}" \
                  --query="SELECT name FROM $${SYS_DB_NAME}.tables WHERE database = '$${SYS_DB_NAME}' AND (engine LIKE '%Log' OR name LIKE '%_log')"))

              if [[ "$${CREATE_DB_LIST}" != "" ]] ; then
                IFS=',' read -r -a DBS <<< "$${CREATE_DB_LIST}"
                for db in "$${DBS[@]}" ; do
                  echo "Creating DB: $${db}"
                  clickhouse-client \
                    --host="$${CLICKHOUSE_HOST}" \
                    --user="$${CLICKHOUSE_USER}" \
                    --password="$${CLICKHOUSE_PASSWORD}" \
                    --query="CREATE DATABASE IF NOT EXISTS $${db}"
                done
              else
                echo "Database creation is disabled. Current CREATE_DB_LIST is empty"
              fi

              echo "DB creation is completed...!"

              for log_tbl in $${LOG_TABLES[@]} ; do
                echo "Applying $${SYS_DB_NAME} log TTL to $${log_tbl}"

                table_name="$${log_tbl##*.}"

                exists=$$(clickhouse-client \
                          --host="$${CLICKHOUSE_HOST}" \
                          --user="$${CLICKHOUSE_USER}" \
                          --password="$${CLICKHOUSE_PASSWORD}" \
                          --query="SELECT count() FROM $${SYS_DB_NAME}.tables WHERE database='$${SYS_DB_NAME}' AND name='$${table_name}'")

                if [ "$${exists}" -eq 1 ]; then
                  clickhouse-client \
                    --host="$${CLICKHOUSE_HOST}" \
                    --user="$${CLICKHOUSE_USER}" \
                    --password="$${CLICKHOUSE_PASSWORD}" \
                    --query "
                              SET alter_sync = 0;
                              ALTER TABLE $${SYS_DB_NAME}.$${log_tbl} MODIFY TTL event_date + INTERVAL 3 DAY;
                              ALTER TABLE $${SYS_DB_NAME}.$${log_tbl} MATERIALIZE TTL;
                            "
                else
                  echo "Table $${log_tbl} does not exist yet. Skipping."
                fi
              done

              for log_tbl in $${LOG_TABLES[@]} ; do
                echo "Validating the table size of the table $${log_tbl}"
                table_name="$${log_tbl##*.}"

                clickhouse-client \
                  --host="$${CLICKHOUSE_HOST}" \
                  --user="$${CLICKHOUSE_USER}" \
                  --password="$${CLICKHOUSE_PASSWORD}" \
                  --query "
                            SELECT name, engine_full
                            FROM $${SYS_DB_NAME}.tables
                            WHERE database='$${SYS_DB_NAME}' AND name IN ('$${table_name}');
                          "
              done
          envFrom:
            - configMapRef:
                name: clickhouse-cm
          env:
            - name: CLICKHOUSE_USER
              valueFrom:
                secretKeyRef:
                  name: clickhouse-secrets
                  key: CH_ADMIN_USERNAME
            - name: CLICKHOUSE_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: clickhouse-secrets
                  key: CH_ADMIN_PASSWORD

Defines:

• CronJob/clickhouse-backup — runs every 12h (0 */12 * * *), iterates BACKUP_DB_LIST from the ConfigMap, calls clickhouse-backup create_remote per DB, and prunes old remote backups beyond BACKUP_KEEP.

• Job/clickhouse-setup — one-shot job that waits for ClickHouse to answer /ping, creates databases listed in CREATE_DB_LIST, and applies MODIFY TTL event_date + INTERVAL 3 DAY to every system.*_log table.

Install — step by step

Run from the repo root unless noted otherwise.

Namespace

kubectl create ns clickhouse

Operator

helm repo add altinity https://helm.altinity.com
helm repo update

helm install ch-operator altinity/altinity-clickhouse-operator \
  --namespace clickhouse \
  --version 0.25.5

Verify the operator pod is running, and the CRDs are installed:

kubectl -n clickhouse get pods -l app=clickhouse-operator
kubectl get crds | grep -E 'clickhouseinstallation|clickhousekeeper'

You should see

• clickhouseinstallations.clickhouse.altinity.com

• clickhouseinstallationtemplates.clickhouse.altinity.com

• clickhousekeeperinstallations.clickhouse-keeper.altinity.com

Dependencies (SA + ConfigMap + Secret)

kubectl apply -f highflame-clickhouse-deps.yml

Keeper + ClickHouse cluster

kubectl apply -f highflame-clickhouse-values.yml

Verify the deployment:

kubectl -n clickhouse get pods

Backup secret

kubectl apply -f highflame-clickhouse-addons-values.yml

The setup Job runs once and exits Completed. The CronJob fires every 12h

Clickhouse S3 backup management (From the backup containers)

List down the remote backup

kpe chi-ch-cluster-0-0-0 /bin/bash -n clickhouse -c clickhouse-backup
/bin/clickhouse-backup -c /opt/backup-config.yml list remote

Restore from remote backup (For DR)

/bin/clickhouse-backup -c /opt/backup-config.yml restore_remote BACKUP_NAME

Highflame service deployment

Follow this documentation to deploy Highflame services to your Kubernetes cluster using Helm charts.

Whether you're standing up a fresh environment or upgrading an existing one, this guide has you covered end to end.

Highflame Services