← TTF AKS Smoke Test

Azure Kubernetes Service · Platform Architecture

Private AKS platform: dev, test and production

Proposed target architecture for the Testwise Kubernetes platform. Two private AKS clusters per stage, shared platform services in a single resource group, and one fleet to keep the clusters on the same version. Non-production sits on twcoredevmainbecvnet 10.56.0.0/14. No public API server, no public listener on any cluster and no publicly reachable PaaS endpoint — public traffic arrives only through Azure Front Door Premium over Private Link, while the ingress load balancer stays internal (live origin 10.56.1.122).

Region: Belgium Central Non-production spoke: 10.56.0.0/14 Non-production: dev + test Production: 2 clusters (next phase) Private clusters only Public entry: Front Door Premium Terraform, Azure DevOps pipeline
Diagram 1

Non-production target architecture

Two private AKS clusters, dev and test, each in its own resource group and its own Terraform workspace. Everything they share — the private DNS zones, one container registry, and the Key Vault and App Configuration pair per environment — lives in a single platform resource group alongside the fleet. dev and test keep separate configuration because they are meant to differ; they share the registry because an image should be built once and promoted, distinguished by tag rather than by which registry it sits in.

5 × Private Endpointnode + pod IP spacenode + pod IP spaceprivate resolutionNon-Production Target ArchitectureTwo private AKS clusters (dev, test) with shared platform services — Azure, Belgium CentralPHASE 1Azure Subscription — Testwise0f66eb97-2901-4bfa-8896-9644e541b861Resource group: tw-main-plat-bec-rgShared platform services — deployed once, consumed by both clustersPrivate DNS zonesprivatelink.vaultcore.azure.netprivatelink.azurecr.io · privatelink.azconfig.ioOne VNet link per zone — single owner, no per-env duplication3 zonesAKS Fleet ManagerMembers: aks-dev, aks-test · update orchestration onlyStaged node-image upgrades: dev → wait → testSee diagram 4 for the update flowno hubContainer registry1 × Premium (shared)Public access disableddev / test imagesseparated by tagKey Vault — devkv (dev secrets)RBAC, purge protectRead by AKS devvia Secrets Store CSIApp Config — devappcs (dev settings)Public access disabledKey Vault referencesfor secret valuesKey Vault — testkv (test secrets)RBAC, purge protectRead by AKS testvia Secrets Store CSIApp Config — testappcs (test settings)Public access disabledKey Vault referencesfor secret valuesVirtual network: twcoredevmainbecvnet — 10.56.0.0/14Non-prod spoke, RG tw-core-dev-bec-rg — linked to the privatelink.* zones abovedev subnetssnet-aks-system10.56.0.128/25system node poolsnet-aks-apps10.56.1.0/25apps pool + ILB 10.56.1.122snet-aks-pods10.56.2.0/23Azure CNI pod IPs (512)shared platform subnetscoreacrpe10.56.0.0/29snet-pe10.56.4.0/27ACR PE plus KV / App Config PEs andPLS NAT. Policies disabled on snet-pe.No public endpoint on any PaaS service.test subnets — plannedsnet-aks-systemappendedsame sizes as devsnet-aks-appsin 10.56.0.0/14internal LB at applysnet-aks-pods/23Azure CNI pod IPsResource group: tw-main-dev-bec-rgCluster environment — Terraform workspace “dev”AKS — devprivate clusterPrivate API server (Azure-managed DNS)Azure CNI · pod subnet · no overlaysystem pool (critical addons only) + apps poolIstio add-on · Key Vault CSI driveringress-nginx on internal LB 10.56.1.122PRIVATEObservabilityper clusterLog Analytics workspaceAzure Monitor workspace (Prometheus)Azure Managed GrafanaNode RG: tw-aksnodes-dev-bec-rg(VMSS nodes + internal load balancer)Resource group: tw-main-test-bec-rgCluster environment — Terraform workspace “test”AKS — testprivate clusterPrivate API server (Azure-managed DNS)Azure CNI · pod subnet · no overlaysystem pool (critical addons only) + apps poolIstio add-on · Key Vault CSI driveringress-nginx on internal LB plannedPRIVATEObservabilityper clusterLog Analytics workspaceAzure Monitor workspace (Prometheus)Azure Managed GrafanaNode RG: tw-aksnodes-test-bec-rg(VMSS nodes + internal load balancer)Public entry — Azure Front Door Premium over Private LinkDetail in diagram 2. Neither cluster exposes a public listener of its own.ClientHTTPS 443Front Door PremiumTLS terminates · managed certPrivate EndpointWest Europe · approved by handPrivate Link Serviceone per cluster · made by HelmInternal LB → nginx10.56.1.122 · test plannedBoth clusters are private by design — no public API server, no public listener on the cluster, and no public PaaS endpoint. The only public front end is the Front Door profile.Non-prod address space is twcoredevmainbecvnet 10.56.0.0/14. Ingress ILB is 10.56.1.122 (cidrhost of snet-aks-apps, −6). A second cluster appends subnets in the same /14.Log Analytics, Azure Monitor workspace and Managed Grafana are provisioned in West Europe: those resource types are not yet available in Belgium Central.Out of scope for this phase: the automated certificate-renewal Function App (kept in code, not deployed). snet-automation stays reserved for it.

Standalone vector file: 01-nonprod-architecture.svg

Diagram 2

Edge — Front Door Premium

Both clusters are private and neither has a public listener, so public traffic arrives through one Azure Front Door Premium profile and reaches the internal ingress load balancer over Private Link. Premium matters here: the Standard tier cannot use a Private Link origin. TLS terminates at the edge on an Azure-managed certificate and the hop to nginx is plain HTTP inside the Microsoft backbone, which is why the origin Host header is set on each origin rather than once on the route. The second cluster joins as a second origin in the same origin group, with its own Private Link Service — equal priority load-balances between the two, a different priority turns it into failover.

Edge — Front Door Premium over Private LinkOne public entry point in front of both private clusters; the ingress load balancer stays internalPUBLIC ENTRYClientHTTPS 443Browser or API consumeron the public internet.Finds the endpoint byCNAME on the customdomain.Azure Front Door Premium — Microsoft global edgePublic TLS terminates here; origins are reached privately, never over the internetFront Door profilePremium_AzureFrontDoorPremium is required — theStandard tier cannot reachan internal load balancer.WAF and rule sets availableon this tier.Custom domain + TLSAzure-managed certificateMinimum TLS 1.2.CNAME to the endpoint plusTXT at _dnsauth.<host> forvalidation. Public DNS isnot created by Terraform.Route + origin groupone group, two originspatterns_to_match /*supported Http, Httpshttps_redirect trueforwarding HttpOnlyprobe GET /afd-health HTTP:80Private EndpointWest Europe · AFD-managedBelgium Central is not onthe Front Door Private Linkregion list.The endpoint sits in aMicrosoft subscription.MANUALNo public listener exists on either cluster — the only reachable front end is this profile.Azure Subscription — Testwise0f66eb97-2901-4bfa-8896-9644e541b861Resource group: tw-main-dev-bec-rgCluster environment — Terraform workspace “dev”Private Link Servicetw-core-dev-bec-pls-ingressCreated by the ingress-nginx Service annotation azure-pls-name — Helm, not Terraform.NAT addresses come from snet-pe, which must have Private Link network policies disabled.azure-pls-visibility lets the Front Door subscription request a connection; approval is manual.Internal load balancer10.56.1.122kubernetes-internal — no public IP, no internet-facing frontend.Front Door reaches nginx here over plain HTTP on port 80.AKS — devprivate cluster · ingress-nginxnginx routes by Ingress Host header, which is why the header is set per origin.Kiali is not published here — it stays on the VPN path to this same load balancer.Resource group: tw-main-test-bec-rgCluster environment — Terraform workspace “test”PLANNEDPrivate Link Servicecreated with the test clusterCreated by the ingress-nginx Service annotation azure-pls-name — Helm, not Terraform.NAT addresses come from snet-pe, which must have Private Link network policies disabled.azure-pls-visibility lets the Front Door subscription request a connection; approval is manual.Internal load balancerplannedkubernetes-internal — no public IP, no internet-facing frontend.Front Door reaches nginx here over plain HTTP on port 80.AKS — testprivate cluster · ingress-nginxnginx routes by Ingress Host header, which is why the header is set per origin.Kiali is not published here — it stays on the VPN path to this same load balancer.origin 1 — dev · priority 1 · weight 1000origin 2 — test · second Private Link Service, plannedWhy Private Link instead of public ingressA public ingress would mean a public IP on the cluster and a certificate to renew inside it. Here the edge owns the certificate and the cluster owns nothing publicly reachable.TLS ends at the edge and the hop to nginx is plain HTTP across the Microsoft backbone, so each origin carries its own Host header rather than the route carrying one for everybody.The Private Endpoint is created in West Europe because Belgium Central is not on the Front Door Private Link region list — one extra backbone hop, worth measuring with curl -w.Every origin needs its own Private Link Service, so the second cluster joins as a second origin in the same group: equal priority load-balances, different priority fails over.The live origin is 10.56.1.122 on twcoredevmainbecvnet (10.56.0.0/14). Test is a second origin in the same /14 once its PLS exists.Certificates inside the cluster come from Key Vault through the Secrets Store CSI driver. The nodes have no internet egress, so an in-cluster ACME client could not reach Let’s Encrypt.

Standalone vector file: 02-edge-front-door.svg

Diagram 3

Address plan

Non-production uses one spoke, twcoredevmainbecvnet, with address space 10.56.0.0/14. Subnets are allocated in list order from the front of that block: ACR PE, AKS system, apps (ingress ILB 10.56.1.122), pods, snet-pe, then a reserved automation subnet. A second cluster does not get a second VNet — it appends from 10.56.4.48. The bar zooms the first /21, where every address in use today sits; the rest of the /14 is free.

Address Plan — Spoke VNet 10.56.0.0/14twcoredevmainbecvnet as allocated today; the bar is the first /21, where every live subnet sits262144 ADDRESSESsnet-aks-pods10.56.2.0/23unallocatedtest appends here10.56.0.010.56.2.010.56.4.010.56.6.010.56.8.0dev cluster (live)shared platformreserved (not deployed)free — rest of 10.56.0.0/14Subnet detail — allocated in list order by hashicorp/subnets/cidrVerified in Azure on twcoredevmainbecvnet. A second cluster appends new names; it does not reuse these prefixes.SUBNETPREFIXUSABLEOWNERDELEGATION / POLICYPURPOSEcoreacrpe10.56.0.0/293platformACR private endpointsnet-aks-system10.56.0.128/25123devSystem node pool (critical addons only)snet-aks-apps10.56.1.0/25123devApps node pool + internal LB 10.56.1.122snet-aks-pods10.56.2.0/23507devMicrosoft.ContainerServiceAzure CNI pod IPssnet-pe10.56.4.0/2727platformPE network policies disabledKV / App Config PEs + PLS NATsnet-automation10.56.4.32/2811devMicrosoft.Web/serverFarmsReserved — cert-renewal Function App (not deployed)(test cluster)appended in /14testPlanned; next free prefix is 10.56.4.48Capacity checkWhy /23 pod subnetsMax nodes per cluster: 3 system + 3 apps = 6max_pods = 30 → worst case 180 pod IPs per clusterA /23 gives 507 usable IPs — roughly 2.8× headroom.The /14 is the environment block, not a packed map.Dev occupies the first ~1.1k addresses. Test, and anythingelse non-prod, is appended from 10.56.4.48 onward.Operator and CI access to private resourcesNo public endpoints — access is network-scopedEngineers reach the private API server and private endpoints over apoint-to-site VPN into the spoke, with an in-VNet DNS forwarder soprivatelink.* names resolve to the private endpoint IPs.CI/CD needs the same path: either a self-hosted agent inside theVNet, or the AKS “command invoke” API for cluster operations.Managed DevOps Pools use a separate 10.54.0.0/21 VNet.Usable counts follow Azure's reservation of 5 addresses per subnet. Ingress ILB is cidrhost(snet-aks-apps, −6) = 10.56.1.122.The bar shows 10.56.0.0/21 of a /14. Everything from 10.56.8.0 through 10.59.255.255 is also free.

Standalone vector file: 03-network-layout.svg

Diagram 4

Fleet update flow

Fleet Manager is used purely to orchestrate upgrades — there is no hub cluster, so no additional control plane to pay for or secure. A single update run walks the clusters in a fixed order: dev first, a 300 second soak, then test. Runs are created in a NotStarted state and must be triggered explicitly, so an infrastructure deployment can never upgrade a running cluster as a side effect.

Fleet Manager — Staged Update FlowOne place to orchestrate node-image and Kubernetes upgrades across both clusters, in a fixed orderUPDATE ORCHESTRATION ONLYtw-main-plat-bec-rgFleet resource — no hub cluster, so no extra control plane cost and no workload placementKubernetes Fleet Managertw-main-plat-bec-fleetMembers joined as groups: · group “dev” → aks-dev · group “test” → aks-testClusters stay owned by their ownTerraform workspace.Update strategystaged, sequentialStage 1: group dev ↓ after_stage_wait = 300 sStage 2: group testA stage must finish before the nextone starts — dev is the canary.Update runNodeImageOnlyNode image upgrade, consistentimage version across the fleet.Created in state NotStarted:applying Terraform never touchesa running cluster by itself.Triggerexplicit, human or CIaz fleet updaterun startor the Azure portal.Deliberate gate: upgrades arenever a side effect of aninfrastructure deployment.Execution order of a single update runRun startedNotStarted → RunningStage 1 — devaks-dev node images upgraded,surge 10%, one pool at a timeSoak 300 sFixed wait — smoke tests andGrafana dashboards observedStage 2 — testaks-test upgraded only ifstage 1 succeededRun completeBoth clusters on the samenode image versionWhy this matters for the two-cluster modelWithout a fleet, each cluster is upgraded by hand and the two drift apart — different node images, different Kubernetes patch levels, different behaviour under test.With the staged strategy, dev always absorbs an upgrade first and test follows only after a successful soak, so test stays a trustworthy gate for production.

Standalone vector file: 04-fleet-update-flow.svg

Diagram 5

Production — next phase

Production reuses the same Terraform modules and the same private-only posture, with two clusters behind one fleet. It runs on its own spoke VNet: no non-production deployment can reach production networking, and each side owns a complete set of private DNS zones — a VNet can only be linked once per privatelink namespace, so separate VNets remove that constraint instead of working around it. Configuration is the second difference: production runs a single Key Vault and a single App Configuration shared by both clusters, because both must behave identically and a second copy would only introduce drift. In code this is one list variable, not a forked module.

3 × Private EndpointProduction Architecture — Next PhaseSame modules and the same private-only posture; one configuration set shared by both clustersPHASE 2 — PROPOSEDAzure Subscription — productionProduction runs on its own spoke VNet, fully separated from non-productionResource group: tw-main-prd-bec-rgShared production platform servicesPrivate DNS zones3 privatelink zonesvaultcore · azurecr · azconfigLinked to the production spokeVNet only.Owned by the platform stack.Container registry1 × PremiumProduction images only —promoted from the non-prodregistry, never rebuilt.Public access disabled.PEKey Vault1 set for both clustersBoth clusters read identicalconfiguration, so a secondvault would only add drift.RBAC + purge protection.SHAREDApp Configuration1 set for both clustersSingle source of truth forapplication settings.Key Vault references forsecret values.SHAREDVirtual network: dedicated production spokeOwn address space, planned from scratch — must not overlap the non-production spokesnet-peprivate endpoints for ACR, Key Vault, App Configurationsnet-aks-system / -apps / -pods × 2one symmetric set per cluster, sized for 3–5 nodesResource group: tw-main-prd1-bec-rgProduction cluster 1 — own Terraform workspaceAKS — prd1private clusterSKU tier Standard (SLA-backed API)system + apps pools, 3–5 nodesStandard_D4s_v5 · autoscalingIstio add-on · Key Vault CSI driveringress-nginx on internal LBPRIVATEObservabilityper clusterLog Analytics — 90 day retentionAzure Monitor workspace (Prometheus)Azure Managed GrafanaNode RG: tw-aksnodes-prd1-bec-rgResource group: tw-main-prd2-bec-rgProduction cluster 2 — own Terraform workspaceAKS — prd2private clusterSKU tier Standard (SLA-backed API)system + apps pools, 3–5 nodesStandard_D4s_v5 · autoscalingIstio add-on · Key Vault CSI driveringress-nginx on internal LBPRIVATEObservabilityper clusterLog Analytics — 90 day retentionAzure Monitor workspace (Prometheus)Azure Managed GrafanaNode RG: tw-aksnodes-prd2-bec-rgHow production differs from non-productionTwo differences, both deliberate. Its own spoke VNet, so no non-production change can reach production and each side owns a full set of private DNS zones —which removes the shared-zone ownership workaround entirely. And one configuration set instead of two, because both clusters must behave identically.

Standalone vector file: 05-prod-architecture.svg

Diagram 6

Delivery sequence

The current pvt and sbx environments are removed before the new build starts. They are not redundant infrastructure that can be left running in parallel: they occupy the address ranges, the private endpoint subnet and the private DNS zone links that the new environments need. Steps 2 and 3 carry no risk to anything running; step 1 is the only destructive one, and it applies to non-production only.

Delivery SequenceThe existing pvt/sbx pair is removed first: it holds the addresses, the private endpoints and the DNS zones the new build needsGREENFIELD REBUILD1DecommissionDestroy the current pvt and sbxstacks — clusters, PaaS, subnets,private endpoints, DNS zones.Export anything still needed first:Key Vault secrets, container images.destructive2RefactorMove DNS zones and PaaS into theplatform stack; drop the public/private switch from every module.No Azure change in this step —code and review only.no downtime risk3PlatformApply the platform stack: DNSzones, shared subnet, registry,Key Vaults, App Configs, fleet.Deployed once. Everything theclusters depend on now exists.no downtime risk4dev clusterApply workspace dev: subnets,private AKS, ingress, observability.Join it to the fleet.First proof that the private-onlymodel works end to end.additive5test clusterApply workspace test — identicalcode path, different variables.Join it to the fleet.Confirms the model is reusable,not hand-tuned per cluster.additiveWhy not migrate in placeThe two builds cannot coexist: they share one spoke VNet, the same address ranges and the same privatelink.* zones, which Azure allows to be linked to a VNet only once per namespace.Moving the existing resources instead would mean an Azure resource move plus Terraform state surgery on every object — more moving parts, and more ways to fail, than rebuilding a non-production environment.

Standalone vector file: 06-cutover-sequence.svg