# Azure

## Deployment on AKS

### **⚠️ Reference Only** (If you are using Terraform)

The [`highflame-iac`](https://github.com/highflame-ai/highflame-iac/blob/main/terraform/README.md) repository contains reference Terraform configurations for managing Highflame cloud resources. **Do not** copy these directly into your IaC code or run them against your cloud environment without first reviewing and patching them to match your account, networking, and security requirements.

### Prerequisites <a href="#prerequisites" id="prerequisites"></a>

Ensure that the following tools and resources are installed and available:

* Access to Azure Portal
* Set of domain names for Highflame services (Highflame Team will share the list of services that require the ingress)
* AKS cluster with at least 6 worker nodes for user mode nodepool (**Best Practice:** 3 - 4 nodes for System mode nodepool)
* Create a log analytics workspace that can be used in AKS logging
* Application gateway with SSL certs for AKS ingress integration
* AKS addons to be installed
  * Storage profile for disk driver
  * OMS with the log analytics workspace created above
  * Ingress addon with AGIC
  * Autoscaler profile needs to be enabled
  * Enable Workload autoscaler
* Create a default node pool for system nodes in the AKS cluster
* Postgres Flexible Server with private endpoint
* Azure Cache for Redis + private endpoint
* Helm v3
* Kubectl utility
* All the cloud resources (managed services such as Postgres, Redis etc) should be in the same VNet, or those should be accessible from the Kubernetes Cluster

### Cloud Resources and Sizing

| Highflame components  | Cloud Resources                                                           | Size                                                                                                                                                                                                                                                                                                                                                    |
| --------------------- | ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Highflame services    | Deploying the services into your AKS cluster with help of the helm charts | <ul><li><strong>CPU nodes</strong>: Setup atleast 3 nodes in the user mode nodepool and the worker node SKU can be <code>Standard\_F8s\_v2</code> or <code>Standard\_D8as\_v6</code> </li><li><strong>GPU nodes</strong>: Setup atleast 4 nodes in the user mode nodepool and the worker node SKU can be <code>Standard\_NC4as\_T4\_v3</code></li></ul> |
| Trascational Database | Postgres Flexible Server                                                  | The Postgres server can be single node or primary - secondary cluster and the SKU can be `GP_Standard_D4ads_v5`                                                                                                                                                                                                                                         |
| Analytical Database   | Clickhouse database server                                                | Deploying into the AKS                                                                                                                                                                                                                                                                                                                                  |
| Cache                 | Azure Cache for Redis                                                     | Redis SKU can be `Standard`                                                                                                                                                                                                                                                                                                                             |
| Object Store          | Azure blob store                                                          | Highflame services required to store files and data in object store, such as Store the clickhouse db backup in the Azure blob store for DR as clickhouse is deploying into the AKS cluster                                                                                                                                                              |
| Logging               | Log analytics workspace                                                   | The AKS service logs can be pushed to the Workspace with help of AKS addons                                                                                                                                                                                                                                                                             |
| Authentication        | Clerk                                                                     | External - Managed by Highflame                                                                                                                                                                                                                                                                                                                         |

### Highflame Azure Identity

#### Prerequisites <a href="#prerequisites" id="prerequisites"></a>

Ensure that the following tools and resources are installed and available:

* Access to the AKS cluster setup above
* Ensure OIDC issuer is enabled on your AKS cluster
* Enable Workload Identity on your AKS cluster
* Azure CLI

#### User-Assigned Managed Identity (UAMI)

1. Setting up the environment vars

```bash
export K8S_CLUSTER_NAME="highflame-poc-eks"           ## AKS cluster name
export K8S_NAMESPACE="highflame-poc"                  ## Highflame k8s namespace
export MANAGED_IDENTITY_NAME="highflame-svc-identity" ## A name for UAMI
export K8S_SA_NAME="highflame-k8s-sa"                 ## Name for K8S service account
export RESOURCE_GROUP="highflame-rg"                  ## Name of the resource group
```

2. Create a User-Assigned Managed Identity

```bash
az identity create \
  --name ${MANAGED_IDENTITY_NAME} \
  --resource-group ${RESOURCE_GROUP}

MANAGED_IDENTITY_CLIENT_ID=$(az identity show \
  --name ${MANAGED_IDENTITY_NAME} \
  --resource-group ${RESOURCE_GROUP} \
  --query clientId -o tsv)

echo ${MANAGED_IDENTITY_CLIENT_ID}
```

3. Create a federated identity credentials

```bash
OIDC_ISSUER=$(az aks show \
  --name ${K8S_CLUSTER_NAME} \
  --resource-group ${RESOURCE_GROUP} \
  --query "oidcIssuerProfile.issuerUrl" -o tsv)

az identity federated-credential create \
  --name highflame-federated-identity \
  --identity-name ${MANAGED_IDENTITY_NAME} \
  --resource-group ${RESOURCE_GROUP} \
  --issuer ${OIDC_ISSUER} \
  --subject system:serviceaccount:${K8S_NAMESPACE}:${K8S_SA_NAME} \
  --audiences api://AzureADTokenExchange
```

#### Resource Access - Azure Blob store

There will be a few Azure blob store requirements for HighFlame, and those were mentioned in the [highflame service variables](https://github.com/highflame-ai/highflame-iac/blob/main/docs/service-vars.md) list.

For each Azure Blob store, this command needs to be run

1. Setting up some additional environment vars

```bash
export STORAGE_ACCOUNT_NAME="highflame" ## Name of the storage account
export STORAGE_CONTAINER_NAME="highflame" ## Name of the storage container
```

2. Grant the managed identity a role that allows it to access Azure Blob Storage and its container

```bash
STORAGE_ID=$(az storage account show \
  --name ${STORAGE_ACCOUNT_NAME} \
  --resource-group ${RESOURCE_GROUP} --query id -o tsv)

az role assignment create \
  --assignee ${MANAGED_IDENTITY_CLIENT_ID} \
  --role "Storage Blob Data Contributor" \
  --scope "${STORAGE_ID}/blobServices/default/containers/${STORAGE_CONTAINER_NAME}"
```

### Analytical Database setup - Clickhouse on AKS

This document is the long-form runbook for installing ClickHouse on an AKS cluster using the [Altinity ClickHouse Operator](https://github.com/Altinity/clickhouse-operator), and an Azure blob store-backed [clickhouse-backup](https://github.com/Altinity/clickhouse-backup) sidecar.

#### What gets deployed <a href="#id-1-what-gets-deployed" id="id-1-what-gets-deployed"></a>

The full stack lives entirely inside a dedicated `clickhouse` namespace and consists of five logical components:

<table><thead><tr><th width="187">Component</th><th>Kind</th><th>Source</th><th>Notes</th></tr></thead><tbody><tr><td>Altinity operator</td><td>Helm release <code>ch-operator</code></td><td>Upstream chart</td><td>Watches <code>ClickHouseInstallation</code> (CHI) and <code>ClickHouseKeeperInstallation</code> (CHK) CRDs</td></tr><tr><td>Service account, ConfigMap, Secret</td><td>k8s resources</td><td><code>highflame-clickhouse-deps.yml</code></td><td>IRSA-annotated <code>clickhouse-sa</code>, <code>clickhouse-cm</code>, and <code>clickhouse-secrets</code></td></tr><tr><td>ClickHouse Keeper</td><td><code>ClickHouseKeeperInstallation/ch</code></td><td><code>highflame-clickhouse-values.yml</code></td><td>1 replica, 20Gi PVC. Replaces ZooKeeper</td></tr><tr><td>ClickHouse cluster</td><td><code>ClickHouseInstallation/ch</code></td><td><code>highflame-clickhouse-values.yml</code></td><td>1 shard × 1 replica, 4 CPU / 12 Gi RAM, 512 Gi PVC, <code>clickhouse-backup</code> sidecar</td></tr><tr><td>Backup config</td><td>k8s resources</td><td><code>highflame-clickhouse-backup-config.yml</code></td><td>Backup config for Azure blob store integration</td></tr><tr><td>CronJob + Job</td><td>k8s resources</td><td><code>highflame-clickhouse-addons-values.yml</code></td><td>12-hourly remote backups, 60-backup retention; setup Job applies 3-day TTL on <code>system.*_log</code> tables</td></tr></tbody></table>

Pinned versions (verify before installing):

* `Operator chart`: `0.25.5`
* `clickhouse/clickhouse-server`: `25.10.2`
* `clickhouse/clickhouse-keeper`: `25.10.2`
* `altinity/clickhouse-backup`: `2.6.39`

After installation, the in-cluster service endpoints are:

* HTTP: `clickhouse-ch.clickhouse.svc.cluster.local:8123`
* Native TCP: `clickhouse-ch.clickhouse.svc.cluster.local:9000`
* Keeper: `keeper-ch.clickhouse.svc.cluster.local:2181`

These match the `CLICKHOUSE_HOST` ConfigMap value and the `zookeeper.nodes` block in the CHI spec

#### Prerequisites

Before you start, you need:

1. **An AKS cluster** with `kubectl` context pointing at it (`kubectl config current-context`).
2. **A storage class** that supports `ReadWriteOnce` PVCs — `Premium_LRS` is recommended. Both the Keeper and the ClickHouse server use the cluster default unless you uncomment `storageClassName` in the CHI/CHK templates.
3. **An Azure blob store and container** for remote backups, in the same Azure subscription as the cluster. Versioning + lifecycle rules are recommended but not required.
4. **An User-Assigned Managed Identity (UAMI) for the backup Service Account (IRSA)** with permission to the Azure blob store and the container `Storage Blob Data Contributor` on that blob store. The UAMI must allow the OIDC provider of the cluster to assume it for `system:serviceaccount:clickhouse:clickhouse-sa`.
5. **`helm` ≥ 3.10**

#### Config files

{% hint style="info" %}
highflame-clickhouse-deps.yml
{% endhint %}

```bash
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    azure.workload.identity/client-id: ""
  name: clickhouse-sa
  namespace: clickhouse
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: clickhouse-cm
  namespace: clickhouse
data:
  CLICKHOUSE_HOST: "clickhouse-ch.clickhouse.svc.cluster.local"
  BACKUP_KEEP: "60"
  CREATE_DB_LIST: ""
  BACKUP_DB_LIST: "highflame"
---
apiVersion: v1
kind: Secret
metadata:
  name: clickhouse-secrets
  namespace: clickhouse
type: Opaque
stringData:
  CH_ADMIN_USERNAME: "highflame_admin"
  CH_ADMIN_PASSWORD: ""
  CH_READONLY_USERNAME: "highflame_readonly"
  CH_READONLY_PASSWORD: ""
  CH_BACKUP_USERNAME: "highflame_backup"
  CH_BACKUP_PASSWORD: ""
```

Defines:

* `ServiceAccount/clickhouse-sa` — annotation `azure.workload.identity/client-id` must be set to the UAMI created in the prerequisites.
* `ConfigMap/clickhouse-cm` — `CLICKHOUSE_HOST`, `BACKUP_KEEP=60`, `CREATE_DB_LIST` (comma-separated DBs to create on first boot), `BACKUP_DB_LIST=highflame` (comma-separated DBs to back up).
* `Secret/clickhouse-secrets` — admin / readonly / backup usernames + passwords.

You must edit:

| Field                               | Required value                                         |
| ----------------------------------- | ------------------------------------------------------ |
| `azure.workload.identity/client-id` | IRSA UAMI                                              |
| `CH_ADMIN_PASSWORD`                 | strong password for admin user `highflame_admin`       |
| `CH_READONLY_PASSWORD`              | strong password for readonly user `highflame_readonly` |
| `CH_BACKUP_PASSWORD`                | strong password for backup user `highflame_backup`     |

{% hint style="info" %}
highflame-clickhouse-values.yml
{% endhint %}

Defines the `ClickHouseKeeperInstallation/ch` and `ClickHouseInstallation/ch` custom resources.

```bash
apiVersion: "clickhouse-keeper.altinity.com/v1"
kind: "ClickHouseKeeperInstallation"

metadata:
  name: "ch"
  namespace: clickhouse

spec:
  configuration:
    clusters:
      - name: cluster
        layout:
          replicasCount: 1
    settings:
      keeper_server/tcp_port: "2181"
      keeper_server/raft_port: "9444"

  defaults:
    templates:
      podTemplate: keeper
      volumeClaimTemplate: keeper

  templates:
    podTemplates:
      - name: keeper
        metadata:
          labels:
            app: clickhouse-keeper
        spec:
          affinity:
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                - labelSelector:
                    matchExpressions:
                      - key: "app"
                        operator: In
                        values:
                          - clickhouse-keeper
                  topologyKey: "kubernetes.io/hostname"
          containers:
            - name: keeper
              image: "clickhouse/clickhouse-keeper:25.10.2"
              command: [ "/usr/bin/clickhouse-keeper", "start" ]
              resources:
                requests:
                  memory: "512Mi"
                  cpu: "500m"
                limits:
                  memory: "1024Mi"
                  cpu: "1000m"
          securityContext:
            fsGroup: 101

    volumeClaimTemplates:
      - name: keeper
        spec:
          # storageClassName: storage_classname
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 20Gi
---
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"

metadata:
  name: "ch"
  namespace: clickhouse

spec: 
  configuration:
    zookeeper:
      nodes:
        - host: "keeper-ch.clickhouse.svc.cluster.local"
          port: 2181
    clusters:
      - name: cluster
        layout:
          shardsCount: 1
          replicasCount: 1
    users:
      highflame_admin/password:
        valueFrom:
          secretKeyRef:
            name: clickhouse-secrets
            key: CH_ADMIN_PASSWORD
      highflame_admin/networks/ip:
        - "0.0.0.0/0"
        - "::/0"
      highflame_admin/profile: default
      highflame_admin/quota: default
      highflame_readonly/password:
        valueFrom:
          secretKeyRef:
            name: clickhouse-secrets
            key: CH_READONLY_PASSWORD
      highflame_readonly/networks/ip:
        - "0.0.0.0/0"
        - "::/0"
      highflame_readonly/profile: readonly
      highflame_readonly/quota: readonly
      highflame_backup/password:
        valueFrom:
          secretKeyRef:
            name: clickhouse-secrets
            key: CH_BACKUP_PASSWORD
      highflame_backup/networks/ip:
        - "0.0.0.0/0"
        - "::/0"
      highflame_backup/profile: backup
      highflame_backup/quota: backup      
    profiles:
      readonly/readonly: 2
      readonly/max_execution_time: 120
      readonly/max_memory_usage: 1000000000
      readonly/max_rows_to_read: 100000000
      readonly/max_result_rows: 10000
      readonly/read_overflow_mode: throw
      readonly/result_overflow_mode: throw
      readonly/timeout_overflow_mode: throw
      backup/readonly: 0
    quotas:
      readonly/interval/duration: 3600
      backup/interval/duration: 3600

  defaults:
    templates:
      podTemplate: clickhouse
      dataVolumeClaimTemplate: clickhouse-data
      # logVolumeClaimTemplate: clickhouse-log

  templates:
    podTemplates:
      - name: clickhouse
        spec:
          serviceAccountName: clickhouse-sa
          containers:
            - name: clickhouse
              image: clickhouse/clickhouse-server:25.10.2
              resources:
                requests:
                  cpu: "4"
                  memory: "12Gi"
                limits:
                  cpu: "4"
                  memory: "12Gi"
              volumeMounts:
                - name: clickhouse-data
                  mountPath: /var/lib/clickhouse
                # - name: clickhouse-log
                #   mountPath: /var/log/clickhouse-server
            - name: clickhouse-backup
              image: altinity/clickhouse-backup:2.6.39
              command:
                - /bin/bash
                - -c
                - "/bin/clickhouse-backup -c /opt/backup-config.yml server"
              volumeMounts:
                - name: clickhouse-data
                  mountPath: /var/lib/clickhouse
                - name: clickhouse-backup-config
                  mountPath: /opt/backup-config.yml
                  subPath: backup-config.yml
                  readOnly: true
          volumes:
            - name: clickhouse-backup-config
              secret:
                secretName: clickhouse-backup-config
                optional: false
          # nodeSelector:
          #   kube/nodegroup: "general"

    volumeClaimTemplates:
      - name: clickhouse-data
        spec:
          # storageClassName: storage_classname
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 512Gi
      # - name: clickhouse-log
      #   spec:
      #     storageClassName: storage_classname
      #     accessModes:
      #       - ReadWriteOnce
      #     resources:
      #       requests:
      #         storage: 100Gi
```

Other useful knobs in this file (edit if your environment differs):

* `templates.podTemplates[clickhouse].spec.containers[clickhouse].resources` — default request and limit are both `cpu: 4`, `memory: 12Gi`.
* `templates.volumeClaimTemplates[clickhouse-data].resources.requests.storage` — default `512Gi`.
* `templates.podTemplates[clickhouse].spec.nodeSelector` — uncomment to pin to a specific nodegroup.

{% hint style="info" %}
highflame-clickhouse-backup-config.yml
{% endhint %}

```bash
general:
  remote_storage: azblob
  disable_progress_bar: false
  backups_to_keep_local: 1
  backups_to_keep_remote: 10
  log_level: info
  allow_empty_backups: false
clickhouse:
  username: highflame_backup
  password: ""
  host: clickhouse-ch.clickhouse.svc.cluster.local
  port: 9000
  disk_mapping: {}
  skip_tables:
    - system.*
  timeout: 5m
  freeze_by_part: false
  secure: false
  skip_verify: false
  sync_replicated_tables: true
  log_sql_queries: false
azblob:
  endpoint_suffix: "core.windows.net"
  account_name: "BLOB_STORE"
  use_managed_identity: true
  container: "BLOB_CONTAINER"
  assume_container_exists: true
  path: highflame-clickhouse
  compression_format: tar
  compression_level: 1
```

Renders into `Secret/clickhouse-backup-config` (mounted into the sidecar at `/opt/backup-config.yml`). Update:

* `BLOB_STORE` — Azure blob store name
* `BLOB_CONTAINER` — Azure blob container name
* `clickhouse.username` / `clickhouse.password` to match `CH_BACKUP_USERNAME` / `CH_BACKUP_PASSWORD` from the previous step
* The default `path: highflame-clickhouse` is a key prefix inside the azure blob container — change it per environment if you share a azure blob store between environments.

{% hint style="info" %}
highflame-clickhouse-addons-values.yml
{% endhint %}

```bash
apiVersion: batch/v1
kind: CronJob
metadata:
  name: clickhouse-backup
  namespace: clickhouse
spec:
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 1
  failedJobsHistoryLimit: 1
  schedule: "0 */12 * * *"
  jobTemplate:
    spec:
      activeDeadlineSeconds: 14400
      backoffLimit: 1
      template:
        spec:
          serviceAccountName: clickhouse-sa
          restartPolicy: Never
          containers:
            - name: clickhouse-backup
              image: altinity/clickhouse-backup:2.6.39
              envFrom:
                - configMapRef:
                    name: clickhouse-cm
              command:
                - /bin/bash
                - -c
                - |
                  set -e
                  BACKUP_CFG="/opt/backup-config.yml"
                  BACKUP_TIME="$(date +%F-%H-%M-%S)"

                  if [[ "$${BACKUP_DB_LIST}" != "" ]] ; then
                    IFS=',' read -r -a DBS <<< "$${BACKUP_DB_LIST}"
                    for db in "$${DBS[@]}" ; do
                      BACKUP_NAME="$${db}--$${BACKUP_TIME}"
                      echo "Creating backup $${BACKUP_NAME}"
                      /bin/clickhouse-backup -c $${BACKUP_CFG} create_remote --tables="$${db}.*" $${BACKUP_NAME}

                      echo "Deleting all local backups"
                      rm -rf /var/lib/clickhouse/backup/$${BACKUP_NAME}

                      echo "Rotating old remote backups - keeping last $${BACKUP_KEEP}"
                      BACKUPS=$$(clickhouse-backup -c $${BACKUP_CFG} list remote | grep -i "$${db}--" | awk '{print $$1}')
                      BKP_COUNT=$$(echo "$${BACKUPS}" | wc -l)

                      if [[ "$${BKP_COUNT}" -le "$${BACKUP_KEEP}" ]] ; then
                        echo "Nothing to delete...($${BKP_COUNT} backups, keep $${BACKUP_KEEP})"
                      else
                        DELETE_COUNT=$$((BKP_COUNT - BACKUP_KEEP))
                        DELETE_LIST=$$(echo "$${BACKUPS}" | head -n $${DELETE_COUNT})

                        for bkp in $${DELETE_LIST} ; do
                          echo "Deleting remote backup: $${bkp}"
                          clickhouse-backup -c $${BACKUP_CFG} delete remote "$${bkp}"
                        done
                      fi
                      echo "Backup job completed for database $${db}"
                    done
                  else
                    echo "Backup is disabled. Current BACKUP_DB_LIST is empty"
                  fi
              volumeMounts:
                - name: clickhouse-data
                  mountPath: /var/lib/clickhouse
                - name: clickhouse-backup-config
                  mountPath: /opt/backup-config.yml
                  subPath: backup-config.yml
                  readOnly: true
          volumes:
            - name: clickhouse-data
              persistentVolumeClaim:
                claimName: clickhouse-data
            - name: clickhouse-backup-config
              secret:
                secretName: clickhouse-backup-config
                optional: false
          affinity:
            podAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                - labelSelector:
                    matchExpressions:
                      - key: clickhouse.altinity.com/chi
                        operator: In
                        values:
                          - ch
                  topologyKey: "kubernetes.io/hostname"
---
apiVersion: batch/v1
kind: Job
metadata:
  name: clickhouse-setup
  namespace: clickhouse
spec:
  completions: 1
  parallelism: 1
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      initContainers:
        - name: wait
          image: curlimages/curl:8.17.0
          command:
            - /bin/sh
            - -c
            - |
              echo "Waiting for ClickHouse to be ready..."

              until curl -s "http://$${CLICKHOUSE_HOST}:8123/ping" | grep -q "Ok" ; do
                echo "ClickHouse not ready yet. Waiting..."
                sleep 5
              done

              echo "ClickHouse is ready!"
          envFrom:
            - configMapRef:
                name: clickhouse-cm
      containers:
        - name: setup
          image: clickhouse/clickhouse-server:25.10.2
          command:
            - /bin/bash
            - -c
            - |
              set -e

              export LOG_TABLES=(
                "system.trace_log"
                "system.text_log"
                "system.processors_profile_log"
                "system.query_log"
                "system.metric_log"
                "system.part_log"
                "system.asynchronous_metric_log"
              )

              if [[ "$${CREATE_DB_LIST}" != "" ]] ; then
                IFS=',' read -r -a DBS <<< "$${CREATE_DB_LIST}"
                for db in "$${DBS[@]}" ; do
                  echo "Creating DB: $${db}"
                  clickhouse-client \
                    --host="$${CLICKHOUSE_HOST}" \
                    --user="$${CLICKHOUSE_USER}" \
                    --password="$${CLICKHOUSE_PASSWORD}" \
                    --query="CREATE DATABASE IF NOT EXISTS $${db}"
                done
              else
                echo "Database creation is disabled. Current CREATE_DB_LIST is empty"
              fi

              echo "DB creation is completed...!"

              for log_tbl in $${LOG_TABLES[@]} ; do
                echo "Applying system log TTL to $${log_tbl}"

                db_name="$${log_tbl%%.*}"
                table_name="$${log_tbl##*.}"

                exists=$$(clickhouse-client \
                          --host="$${CLICKHOUSE_HOST}" \
                          --user="$${CLICKHOUSE_USER}" \
                          --password="$${CLICKHOUSE_PASSWORD}" \
                          --query="SELECT count() FROM $${db_name}.tables WHERE database='$${db_name}' AND name='$${table_name}'")

                if [ "$${exists}" -eq 1 ]; then
                  clickhouse-client \
                    --host="$${CLICKHOUSE_HOST}" \
                    --user="$${CLICKHOUSE_USER}" \
                    --password="$${CLICKHOUSE_PASSWORD}" \
                    --query "
                              SET alter_sync = 0;
                              ALTER TABLE $${log_tbl}
                              MODIFY TTL event_date + INTERVAL 3 DAY;
                            "
                else
                  echo "Table $${log_tbl} does not exist yet. Skipping."
                fi
              done

              for log_tbl in $${LOG_TABLES[@]} ; do
                echo "Validating the table size of the table $${log_tbl}"
                db_name="$${log_tbl%%.*}"
                table_name="$${log_tbl##*.}"

                clickhouse-client \
                  --host="$${CLICKHOUSE_HOST}" \
                  --user="$${CLICKHOUSE_USER}" \
                  --password="$${CLICKHOUSE_PASSWORD}" \
                  --query "
                            SELECT name, engine_full
                            FROM $${db_name}.tables
                            WHERE database='$${db_name}' AND name IN ('$${table_name}');
                          "
              done
          envFrom:
            - configMapRef:
                name: clickhouse-cm
          env:
            - name: CLICKHOUSE_USER
              valueFrom:
                secretKeyRef:
                  name: clickhouse-secrets
                  key: CH_ADMIN_USERNAME
            - name: CLICKHOUSE_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: clickhouse-secrets
                  key: CH_ADMIN_PASSWORD
```

Defines:

* `CronJob/clickhouse-backup` — runs every 12h (`0 */12 * * *`), iterates `BACKUP_DB_LIST` from the ConfigMap, calls `clickhouse-backup create_remote` per DB, and prunes old remote backups beyond `BACKUP_KEEP`.
* `Job/clickhouse-setup` — one-shot job that waits for ClickHouse to answer `/ping`, creates databases listed in `CREATE_DB_LIST`, and applies `MODIFY TTL event_date + INTERVAL 3 DAY` to every `system.*_log` table.

#### Install — step by step

Run from the repo root unless noted otherwise.

Namespace

```bash
kubectl create ns clickhouse
```

Operator

```bash
helm repo add altinity https://helm.altinity.com
helm repo update

helm install ch-operator altinity/altinity-clickhouse-operator \
  --namespace clickhouse \
  --version 0.25.5
```

Verify the operator pod is running, and the CRDs are installed:

```bash
kubectl -n clickhouse get pods -l app=clickhouse-operator
kubectl get crds | grep -E 'clickhouseinstallation|clickhousekeeper'
```

You should see&#x20;

* `clickhouseinstallations.clickhouse.altinity.com`
* `clickhouseinstallationtemplates.clickhouse.altinity.com`
* `clickhousekeeperinstallations.clickhouse-keeper.altinity.com`

Dependencies (SA + ConfigMap + Secret)

```bash
kubectl apply -f highflame-clickhouse-deps.yml
```

Keeper + ClickHouse cluster

```bash
kubectl apply -f highflame-clickhouse-values.yml
```

Verify the deployment:

```bash
kubectl -n clickhouse get pods
```

Backup secret

```bash
kubectl apply -f highflame-clickhouse-addons-values.yml
```

The setup Job runs once and exits `Completed`. The CronJob fires every 12h

#### Clickhouse Azure Blob Store backup management (From the backup containers)

List down the remote backup

```bash
kpe chi-ch-cluster-0-0-0 /bin/bash -n clickhouse -c clickhouse-backup
/bin/clickhouse-backup -c /opt/backup-config.yml list remote
```

Restore from remote backup (For DR)

```bash
/bin/clickhouse-backup -c /opt/backup-config.yml restore_remote BACKUP_NAME
```

### Highflame service deployment

Follow this documentation to deploy Highflame services to your Kubernetes cluster using Helm charts.

Whether you're standing up a fresh environment or upgrading an existing one, this guide has you covered end to end.

{% content-ref url="/pages/SOHIdzebu16rS55b9pZg" %}
[Highflame Services](/deployment-guides/highflame-services.md)
{% endcontent-ref %}


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.highflame.ai/deployment-guides/azure.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.