Contents Non-production target architecture Edge — Front Door Premium Address plan Fleet update flow Production — next phase Delivery sequence
Diagram 1 Non-production target architecture Two private AKS clusters, dev and test , each in its own resource group and its own Terraform workspace. Everything they share — the private DNS zones, one container registry, and the Key Vault and App Configuration pair per environment — lives in a single platform resource group alongside the fleet. dev and test keep separate configuration because they are meant to differ; they share the registry because an image should be built once and promoted, distinguished by tag rather than by which registry it sits in.
5 × Private Endpoint node + pod IP space node + pod IP space private resolution Non-Production Target Architecture Two private AKS clusters (dev, test) with shared platform services — Azure, Belgium Central PHASE 1 Azure Subscription — Testwise 0f66eb97-2901-4bfa-8896-9644e541b861 Resource group: tw-main-plat-bec-rg Shared platform services — deployed once, consumed by both clusters Private DNS zones privatelink.vaultcore.azure.net privatelink.azurecr.io · privatelink.azconfig.io One VNet link per zone — single owner, no per-env duplication 3 zones AKS Fleet Manager Members: aks-dev, aks-test · update orchestration only Staged node-image upgrades: dev → wait → test See diagram 4 for the update flow no hub Container registry 1 × Premium (shared) Public access disabled dev / test images separated by tag Key Vault — dev kv (dev secrets) RBAC, purge protect Read by AKS dev via Secrets Store CSI App Config — dev appcs (dev settings) Public access disabled Key Vault references for secret values Key Vault — test kv (test secrets) RBAC, purge protect Read by AKS test via Secrets Store CSI App Config — test appcs (test settings) Public access disabled Key Vault references for secret values Virtual network: twcoredevmainbecvnet — 10.56.0.0/14 Non-prod spoke, RG tw-core-dev-bec-rg — linked to the privatelink.* zones above dev subnets snet-aks-system 10.56.0.128/25 system node pool snet-aks-apps 10.56.1.0/25 apps pool + ILB 10.56.1.122 snet-aks-pods 10.56.2.0/23 Azure CNI pod IPs (512) shared platform subnets coreacrpe 10.56.0.0/29 snet-pe 10.56.4.0/27 ACR PE plus KV / App Config PEs and PLS NAT. Policies disabled on snet-pe. No public endpoint on any PaaS service. test subnets — planned snet-aks-system appended same sizes as dev snet-aks-apps in 10.56.0.0/14 internal LB at apply snet-aks-pods /23 Azure CNI pod IPs Resource group: tw-main-dev-bec-rg Cluster environment — Terraform workspace “dev” AKS — dev private cluster Private API server (Azure-managed DNS) Azure CNI · pod subnet · no overlay system pool (critical addons only) + apps pool Istio add-on · Key Vault CSI driver ingress-nginx on internal LB 10.56.1.122 PRIVATE Observability per cluster Log Analytics workspace Azure Monitor workspace (Prometheus) Azure Managed Grafana Node RG: tw-aksnodes-dev-bec-rg (VMSS nodes + internal load balancer) Resource group: tw-main-test-bec-rg Cluster environment — Terraform workspace “test” AKS — test private cluster Private API server (Azure-managed DNS) Azure CNI · pod subnet · no overlay system pool (critical addons only) + apps pool Istio add-on · Key Vault CSI driver ingress-nginx on internal LB planned PRIVATE Observability per cluster Log Analytics workspace Azure Monitor workspace (Prometheus) Azure Managed Grafana Node RG: tw-aksnodes-test-bec-rg (VMSS nodes + internal load balancer) Public entry — Azure Front Door Premium over Private Link Detail in diagram 2. Neither cluster exposes a public listener of its own. Client HTTPS 443 Front Door Premium TLS terminates · managed cert Private Endpoint West Europe · approved by hand Private Link Service one per cluster · made by Helm Internal LB → nginx 10.56.1.122 · test planned Both clusters are private by design — no public API server, no public listener on the cluster, and no public PaaS endpoint. The only public front end is the Front Door profile. Non-prod address space is twcoredevmainbecvnet 10.56.0.0/14. Ingress ILB is 10.56.1.122 (cidrhost of snet-aks-apps, −6). A second cluster appends subnets in the same /14. Log Analytics, Azure Monitor workspace and Managed Grafana are provisioned in West Europe: those resource types are not yet available in Belgium Central. Out of scope for this phase: the automated certificate-renewal Function App (kept in code, not deployed). snet-automation stays reserved for it. Standalone vector file: 01-nonprod-architecture.svg
Diagram 2 Edge — Front Door Premium Both clusters are private and neither has a public listener, so public traffic arrives through one Azure Front Door Premium profile and reaches the internal ingress load balancer over Private Link . Premium matters here: the Standard tier cannot use a Private Link origin. TLS terminates at the edge on an Azure-managed certificate and the hop to nginx is plain HTTP inside the Microsoft backbone, which is why the origin Host header is set on each origin rather than once on the route. The second cluster joins as a second origin in the same origin group, with its own Private Link Service — equal priority load-balances between the two, a different priority turns it into failover.
Edge — Front Door Premium over Private Link One public entry point in front of both private clusters; the ingress load balancer stays internal PUBLIC ENTRY Client HTTPS 443 Browser or API consumer on the public internet. Finds the endpoint by CNAME on the custom domain. Azure Front Door Premium — Microsoft global edge Public TLS terminates here; origins are reached privately, never over the internet Front Door profile Premium_AzureFrontDoor Premium is required — the Standard tier cannot reach an internal load balancer. WAF and rule sets available on this tier. Custom domain + TLS Azure-managed certificate Minimum TLS 1.2. CNAME to the endpoint plus TXT at _dnsauth.<host> for validation. Public DNS is not created by Terraform. Route + origin group one group, two origins patterns_to_match /* supported Http, Https https_redirect true forwarding HttpOnly probe GET /afd-health HTTP:80 Private Endpoint West Europe · AFD-managed Belgium Central is not on the Front Door Private Link region list. The endpoint sits in a Microsoft subscription. MANUAL No public listener exists on either cluster — the only reachable front end is this profile. Azure Subscription — Testwise 0f66eb97-2901-4bfa-8896-9644e541b861 Resource group: tw-main-dev-bec-rg Cluster environment — Terraform workspace “dev” Private Link Service tw-core-dev-bec-pls-ingress Created by the ingress-nginx Service annotation azure-pls-name — Helm, not Terraform. NAT addresses come from snet-pe, which must have Private Link network policies disabled. azure-pls-visibility lets the Front Door subscription request a connection; approval is manual. Internal load balancer 10.56.1.122 kubernetes-internal — no public IP, no internet-facing frontend. Front Door reaches nginx here over plain HTTP on port 80. AKS — dev private cluster · ingress-nginx nginx routes by Ingress Host header, which is why the header is set per origin. Kiali is not published here — it stays on the VPN path to this same load balancer. Resource group: tw-main-test-bec-rg Cluster environment — Terraform workspace “test” PLANNED Private Link Service created with the test cluster Created by the ingress-nginx Service annotation azure-pls-name — Helm, not Terraform. NAT addresses come from snet-pe, which must have Private Link network policies disabled. azure-pls-visibility lets the Front Door subscription request a connection; approval is manual. Internal load balancer planned kubernetes-internal — no public IP, no internet-facing frontend. Front Door reaches nginx here over plain HTTP on port 80. AKS — test private cluster · ingress-nginx nginx routes by Ingress Host header, which is why the header is set per origin. Kiali is not published here — it stays on the VPN path to this same load balancer. origin 1 — dev · priority 1 · weight 1000 origin 2 — test · second Private Link Service, planned Why Private Link instead of public ingress A public ingress would mean a public IP on the cluster and a certificate to renew inside it. Here the edge owns the certificate and the cluster owns nothing publicly reachable. TLS ends at the edge and the hop to nginx is plain HTTP across the Microsoft backbone, so each origin carries its own Host header rather than the route carrying one for everybody. The Private Endpoint is created in West Europe because Belgium Central is not on the Front Door Private Link region list — one extra backbone hop, worth measuring with curl -w. Every origin needs its own Private Link Service, so the second cluster joins as a second origin in the same group: equal priority load-balances, different priority fails over. The live origin is 10.56.1.122 on twcoredevmainbecvnet (10.56.0.0/14). Test is a second origin in the same /14 once its PLS exists. Certificates inside the cluster come from Key Vault through the Secrets Store CSI driver. The nodes have no internet egress, so an in-cluster ACME client could not reach Let’s Encrypt. Standalone vector file: 02-edge-front-door.svg
Diagram 3 Address plan Non-production uses one spoke, twcoredevmainbecvnet , with address space 10.56.0.0/14 . Subnets are allocated in list order from the front of that block: ACR PE, AKS system, apps (ingress ILB 10.56.1.122), pods, snet-pe, then a reserved automation subnet. A second cluster does not get a second VNet — it appends from 10.56.4.48. The bar zooms the first /21, where every address in use today sits; the rest of the /14 is free.
Address Plan — Spoke VNet 10.56.0.0/14 twcoredevmainbecvnet as allocated today; the bar is the first /21, where every live subnet sits 262144 ADDRESSES snet-aks-pods 10.56.2.0/23 unallocated test appends here 10.56.0.0 10.56.2.0 10.56.4.0 10.56.6.0 10.56.8.0 dev cluster (live) shared platform reserved (not deployed) free — rest of 10.56.0.0/14 Subnet detail — allocated in list order by hashicorp/subnets/cidr Verified in Azure on twcoredevmainbecvnet. A second cluster appends new names; it does not reuse these prefixes. SUBNET PREFIX USABLE OWNER DELEGATION / POLICY PURPOSE coreacrpe 10.56.0.0/29 3 platform — ACR private endpoint snet-aks-system 10.56.0.128/25 123 dev — System node pool (critical addons only) snet-aks-apps 10.56.1.0/25 123 dev — Apps node pool + internal LB 10.56.1.122 snet-aks-pods 10.56.2.0/23 507 dev Microsoft.ContainerService Azure CNI pod IPs snet-pe 10.56.4.0/27 27 platform PE network policies disabled KV / App Config PEs + PLS NAT snet-automation 10.56.4.32/28 11 dev Microsoft.Web/serverFarms Reserved — cert-renewal Function App (not deployed) (test cluster) appended in /14 — test — Planned; next free prefix is 10.56.4.48 Capacity check Why /23 pod subnets Max nodes per cluster: 3 system + 3 apps = 6 max_pods = 30 → worst case 180 pod IPs per cluster A /23 gives 507 usable IPs — roughly 2.8× headroom. The /14 is the environment block, not a packed map. Dev occupies the first ~1.1k addresses. Test, and anything else non-prod, is appended from 10.56.4.48 onward. Operator and CI access to private resources No public endpoints — access is network-scoped Engineers reach the private API server and private endpoints over a point-to-site VPN into the spoke, with an in-VNet DNS forwarder so privatelink.* names resolve to the private endpoint IPs. CI/CD needs the same path: either a self-hosted agent inside the VNet, or the AKS “command invoke” API for cluster operations. Managed DevOps Pools use a separate 10.54.0.0/21 VNet. Usable counts follow Azure's reservation of 5 addresses per subnet. Ingress ILB is cidrhost(snet-aks-apps, −6) = 10.56.1.122. The bar shows 10.56.0.0/21 of a /14. Everything from 10.56.8.0 through 10.59.255.255 is also free. Standalone vector file: 03-network-layout.svg
Diagram 4 Fleet update flow Fleet Manager is used purely to orchestrate upgrades — there is no hub cluster, so no additional control plane to pay for or secure. A single update run walks the clusters in a fixed order: dev first, a 300 second soak, then test. Runs are created in a NotStarted state and must be triggered explicitly, so an infrastructure deployment can never upgrade a running cluster as a side effect.
Fleet Manager — Staged Update Flow One place to orchestrate node-image and Kubernetes upgrades across both clusters, in a fixed order UPDATE ORCHESTRATION ONLY tw-main-plat-bec-rg Fleet resource — no hub cluster, so no extra control plane cost and no workload placement Kubernetes Fleet Manager tw-main-plat-bec-fleet Members joined as groups: · group “dev” → aks-dev · group “test” → aks-test Clusters stay owned by their own Terraform workspace. Update strategy staged, sequential Stage 1: group dev ↓ after_stage_wait = 300 s Stage 2: group test A stage must finish before the next one starts — dev is the canary. Update run NodeImageOnly Node image upgrade, consistent image version across the fleet. Created in state NotStarted: applying Terraform never touches a running cluster by itself. Trigger explicit, human or CI az fleet updaterun start or the Azure portal. Deliberate gate: upgrades are never a side effect of an infrastructure deployment. Execution order of a single update run Run started NotStarted → Running Stage 1 — dev aks-dev node images upgraded, surge 10%, one pool at a time Soak 300 s Fixed wait — smoke tests and Grafana dashboards observed Stage 2 — test aks-test upgraded only if stage 1 succeeded Run complete Both clusters on the same node image version Why this matters for the two-cluster model Without a fleet, each cluster is upgraded by hand and the two drift apart — different node images, different Kubernetes patch levels, different behaviour under test. With the staged strategy, dev always absorbs an upgrade first and test follows only after a successful soak, so test stays a trustworthy gate for production. Standalone vector file: 04-fleet-update-flow.svg
Diagram 5 Production — next phase Production reuses the same Terraform modules and the same private-only posture, with two clusters behind one fleet. It runs on its own spoke VNet : no non-production deployment can reach production networking, and each side owns a complete set of private DNS zones — a VNet can only be linked once per privatelink namespace, so separate VNets remove that constraint instead of working around it. Configuration is the second difference: production runs a single Key Vault and a single App Configuration shared by both clusters, because both must behave identically and a second copy would only introduce drift. In code this is one list variable, not a forked module.
3 × Private Endpoint Production Architecture — Next Phase Same modules and the same private-only posture; one configuration set shared by both clusters PHASE 2 — PROPOSED Azure Subscription — production Production runs on its own spoke VNet, fully separated from non-production Resource group: tw-main-prd-bec-rg Shared production platform services Private DNS zones 3 privatelink zones vaultcore · azurecr · azconfig Linked to the production spoke VNet only. Owned by the platform stack. Container registry 1 × Premium Production images only — promoted from the non-prod registry, never rebuilt. Public access disabled. PE Key Vault 1 set for both clusters Both clusters read identical configuration, so a second vault would only add drift. RBAC + purge protection. SHARED App Configuration 1 set for both clusters Single source of truth for application settings. Key Vault references for secret values. SHARED Virtual network: dedicated production spoke Own address space, planned from scratch — must not overlap the non-production spoke snet-pe private endpoints for ACR, Key Vault, App Configuration snet-aks-system / -apps / -pods × 2 one symmetric set per cluster, sized for 3–5 nodes Resource group: tw-main-prd1-bec-rg Production cluster 1 — own Terraform workspace AKS — prd1 private cluster SKU tier Standard (SLA-backed API) system + apps pools, 3–5 nodes Standard_D4s_v5 · autoscaling Istio add-on · Key Vault CSI driver ingress-nginx on internal LB PRIVATE Observability per cluster Log Analytics — 90 day retention Azure Monitor workspace (Prometheus) Azure Managed Grafana Node RG: tw-aksnodes-prd1-bec-rg Resource group: tw-main-prd2-bec-rg Production cluster 2 — own Terraform workspace AKS — prd2 private cluster SKU tier Standard (SLA-backed API) system + apps pools, 3–5 nodes Standard_D4s_v5 · autoscaling Istio add-on · Key Vault CSI driver ingress-nginx on internal LB PRIVATE Observability per cluster Log Analytics — 90 day retention Azure Monitor workspace (Prometheus) Azure Managed Grafana Node RG: tw-aksnodes-prd2-bec-rg How production differs from non-production Two differences, both deliberate. Its own spoke VNet, so no non-production change can reach production and each side owns a full set of private DNS zones — which removes the shared-zone ownership workaround entirely. And one configuration set instead of two, because both clusters must behave identically. Standalone vector file: 05-prod-architecture.svg
Diagram 6 Delivery sequence The current pvt and sbx environments are removed before the new build starts. They are not redundant infrastructure that can be left running in parallel: they occupy the address ranges, the private endpoint subnet and the private DNS zone links that the new environments need. Steps 2 and 3 carry no risk to anything running; step 1 is the only destructive one, and it applies to non-production only.
Delivery Sequence The existing pvt/sbx pair is removed first: it holds the addresses, the private endpoints and the DNS zones the new build needs GREENFIELD REBUILD 1 Decommission Destroy the current pvt and sbx stacks — clusters, PaaS, subnets, private endpoints, DNS zones. Export anything still needed first: Key Vault secrets, container images. destructive 2 Refactor Move DNS zones and PaaS into the platform stack; drop the public/ private switch from every module. No Azure change in this step — code and review only. no downtime risk 3 Platform Apply the platform stack: DNS zones, shared subnet, registry, Key Vaults, App Configs, fleet. Deployed once. Everything the clusters depend on now exists. no downtime risk 4 dev cluster Apply workspace dev: subnets, private AKS, ingress, observability. Join it to the fleet. First proof that the private-only model works end to end. additive 5 test cluster Apply workspace test — identical code path, different variables. Join it to the fleet. Confirms the model is reusable, not hand-tuned per cluster. additive Why not migrate in place The two builds cannot coexist: they share one spoke VNet, the same address ranges and the same privatelink.* zones, which Azure allows to be linked to a VNet only once per namespace. Moving the existing resources instead would mean an Azure resource move plus Terraform state surgery on every object — more moving parts, and more ways to fail, than rebuilding a non-production environment. Standalone vector file: 06-cutover-sequence.svg
Azure service icons are the official Microsoft Azure architecture icon set, used for architecture
documentation. Diagrams are generated from docs/architecture/generate.py — edit the script
and re-run it rather than hand-editing the SVG files.