Support

Highflame Support Access - Optional

BYOC Customer Access via Tailscale

Operating kubectl against private customer clusters across AWS, GCP, and Azure — without VPNs, public endpoints, or per-customer firewall rules.

Why this exists

Highflame is deployed into customer-owned VPCs across AWS, GCP, and Azure (BYOC). The K8s API server and Studio UI are private — reachable only from inside the customer's network. Our support and SRE teams still need:

• kubectl against each customer's private cluster

• Browser access to each customer's Studio UI - Optional for debugging

The Tailscale Kubernetes operator solves all. The customer installs one Helm chart. From there, every Highflame-internal service (k8s resources) can be exposed by a single Service annotation — no new NLBs, no PrivateLink endpoints, no inbound firewall rules.

Architecture

How it works

The Tailscale operator runs as a pod inside the customer's cluster. It dials outbound to Tailscale (UDP 41641 or TCP 443 DERP relay) and registers as a node on our tailnet, tagged with a per-customer tag. With apiServerProxyConfig.mode="true", it also exposes the cluster's K8s API at a stable tailnet hostname like customer-kube.tail-scale.ts.net and proxies authenticated requests into the API server using our tailnet identity. Customer revokes by removing the tag; we revoke by deleting the operator. No inbound firewall rules ever touch the customer's network.

Security properties — what a customer's security team needs to know

Customer network — perimeter stays untouched

• No inbound firewall rule is ever added on the customer side. The operator pod dials outbound only; there is no listener on the customer's VPC reachable from the public internet.

• No public IP, NLB, or PrivateLink endpoint is provisioned for the K8s API server or Studio. They remain on private cluster addresses.

• Egress is narrowly scoped — only *.tailscale.com:443/tcp (plus optional UDP for direct peer-to-peer). See Customer egress requirements. The operator does not make arbitrary internet calls.

• End-to-end WireGuard encryption. Tailscale's coordination and DERP servers route ciphertext only; they cannot decrypt the kubectl.

• No third-party data plane. Traffic flows through Tailscale's coordination/DERP infrastructure only. No CDN, no inspection proxy, no shared bastion.

Access — minimum-privilege, identity-bound

• Only named Highflame SRE engineers can reach the cluster, authenticated via their Tailscale identity. No shared bearer tokens, no static API keys the customer has to rotate.

• Read-only by default. Highflame SRE is bound to a custom highflame-sre ClusterRole that excludes secrets, RBAC objects, and pods/exec / pods/portforward. The scope is configurable per cluster.

• Per-customer isolation. Each customer has its own OAuth clients and per-customer tag chain (tag:k8s-operator-<customer>, tag:k8s-<customer>, tag:cust-<customer>) — shared across that customer's environments (dev/staging/prod), distinct between customers.

• Short-lived auth keys. The operator's OAuth client mints ephemeral, single-use auth keys per device registration. No long-lived credential is ever stored on a registered tailnet node — only the OAuth client secret (held in a single in-cluster Kubernetes Secret).

• All access is logged in the K8s audit log as the impersonated group highflame-sre. Customer's existing audit pipeline captures every request without changes.

Visibility — opt-in service exposure

• Nothing is exposed by default. A Service inside the cluster is only reachable from Highflame's tailnet if you opt it in with either an tailscale.com/expose: "true" annotation on the Service or a Tailscale Ingress resource pointing at it. Removing the annotation / deleting the Ingress removes the exposure.

• Authenticated K8s API access only. The operator's API-server proxy rejects connections without a valid Tailscale identity and a tailscale.com/cap/kubernetes impersonation grant. Anonymous TCP from the tailnet is dropped, not just unauthorized API calls.

Revocation — either side can cut access independently

• Customer kill switch: helm uninstall tailscale-operator -n tailscale. Every Highflame device on the tailnet loses routing to the cluster within seconds. No Highflame coordination required.

• Highflame kill switch: deleting the customer's tag entries from Highflame's tailnet ACL revokes access for all SRE engineers in one edit. No customer coordination required.

• Per-engineer revocation: removing an SRE's email from group:highflame-sre in the tailnet ACL revokes their access to every customer cluster simultaneously, in one edit, without touching any customer.

Audit & monitoring

• K8s API server audit log — every kubectl call shows up under impersonated identity highflame-sre on the customer's existing audit pipeline.

• Tailscale audit log — every Highflame-side connection (source identity → destination tag, timestamp) is recorded in Highflame's tailnet (shareable on request).

• Tailscale device registration log — every operator/proxy pod registration is logged with its OAuth client ID, so an unexpected device join can be traced to a specific credential.

Supply chain

• Open-source operator. Image source: https://github.com/tailscale/tailscale/tree/main/cmd/k8s-operator. Auditable, reproducible.

• Pinned image tag. The helm install pins both operator and proxy images to a specific Tailscale release (currently v1.96.5). No latest, no auto-upgrade — upgrades require an explicit helm upgrade with a new tag.

• Single namespace footprint. All operator and proxy pods run in the tailscale namespace. The customer can apply a NetworkPolicy to restrict operators' in-cluster reach further, if desired (e.g., limit to specific namespaces).

• Single chart, single Service annotation. No CRDs the customer has to maintain, no sidecars injected into customer workloads.

Cryptographic & identity properties

• WireGuard (Noise IK) for the data plane between tailnet peers.

• TLS 1.3 with publicly-trusted (Let's Encrypt) certificates for the API-server proxy hostname. The certificate is renewed automatically by the operator; no manual rotation.

Customer egress requirements

There are no inbound firewall rules to add on the customer side, but the operator pod must reach Tailscale outbound. Most BYOC customers run default-deny egress, so they need to allowlist the destinations below before the helm install will succeed.

Minimum allowlist (TCP only — works in every environment):

Destination	Port	Protocol	Purpose
controlplane.tailscale.com	443	TCP	Coordination server (key exchange, peer discovery)
login.tailscale.com	443	TCP	OAuth auth during operator startup
derp1.tailscale.com … derp30.tailscale.com	443	TCP	DERP relay (encrypted WebSocket fallback)
pkgs.tailscale.com	443	TCP	Operator image pulls (skip if image is mirrored to customer registry)

A single wildcard rule — *.tailscale.com:443/tcp — covers all four and is what most customers ultimately write.

One-time setup (Highflame side)

The Highflame team will manage it and can refer to this doc

Install the operator

Prerequisites

You will need the following before starting.

Tooling on your workstation

Tool	Minimum version	Notes
kubectl	1.32	Configured against the target cluster
helm	3.x	For helm charts deployment

Credentials and licenses

You will need the following from your Highflame representative:

• Highflame Tailscale Credentials — OAUTH_CLIENT_ID, OAUTH_CLIENT_SECRET

Helm deployment (per environment)

All values are passed inline — no value files to manage on the customer side. Customer runs into their target kube-context:

helm repo add tailscale https://pkgs.tailscale.com/helmcharts
helm repo update

CUSTOMER=customer                                          # ← customer name (used in TAGS, same for all envs)
DEPLOY_ENV=prod                                            # ← per-env identifier (used in HOSTNAMES, unique per env)
OAUTH_CLIENT_ID="<customer-prod oauth client id>"
OAUTH_CLIENT_SECRET="<customer-prod oauth client secret>"

helm upgrade --install tailscale-operator tailscale/tailscale-operator \
  --namespace tailscale --create-namespace \
  --version 1.96.5 \
  --set-string oauth.clientId="${OAUTH_CLIENT_ID}" \
  --set-string oauth.clientSecret="${OAUTH_CLIENT_SECRET}" \
  --set apiServerProxyConfig.mode="true" \
  --set operatorConfig.hostname="${CUSTOMER}-${DEPLOY_ENV}-kube" \
  --set operatorConfig.defaultTags[0]="tag:cust-${CUSTOMER}" \
  --set-string proxyConfig.defaultTags="tag:k8s-${CUSTOMER}" \
  --set operatorConfig.image.tag="v1.96.5" \
  --set proxyConfig.image.tag="v1.96.5" \
  --wait

Variable	Used in	Same across envs of the customer?
CUSTOMER	tag:cust-…, tag:k8s-… (Tailscale tags)	✅ Yes — customer for dev/prod/test
DEPLOY_ENV	operatorConfig.hostname (<env>-kube)	❌ No — each env gets a distinct tailnet hostname so SRE can target it specifically
OAUTH_CLIENT_ID/SECRET	helm oauth.* values	❌ No — one per env (rotate independently)

Verify the operator pod is Running:

kubectl get pods -n tailscale
# operator-7c5b8f8b6c-xxxxx   1/1   Running

Verify in the Tailscale machine dashboard as well - the Highflame representative will do this for you.

Apply K8s RBAC for the tailnet identity

Customer applies (one file, identical across all customers):

# rbac-highflame-sre.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: highflame-sre-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind:     ClusterRole
  name:     cluster-admin          # "cluster-admin" or "view"
subjects:
  - kind:    Group
    name:    highflame-sre
    apiGroup: rbac.authorization.k8s.io

Choose the role according to the support requirements - cluster-admin : Full permission to the K8s cluster - view : Readonly permission to the K8s cluster

kubectl apply -f rbac-highflame-sre.yaml

This is what lets the K8s API server recognize a request from the operator's proxy as the group highflame-sre, which we mapped to our SRE team in the tailnet ACL grant.