Global EV IoT Platform — Dual-Region AKS & DevSecOps Hardening

The Docker build bottleneck was the most immediately painful problem the development team faced. Each build produced a 4.2 GB container image that included the Go-based IoT authentication service, embedded TLS certificate bundles, protocol buffer definitions, and a set of legacy C libraries required for backward compatibility with older charging station firmware. Builds ran sequentially on a single GitHub Actions runner because the Dockerfile had never been restructured for parallelism — every layer depended on the previous layer, meaning no step could execute concurrently. With 15 engineers pushing an average of 8–10 times per day, the 45-minute build time created a persistent queue that delayed feature integration by hours. Developers had adopted the workaround of batching their commits and pushing once or twice per day, which reduced CI load but introduced its own problems: larger changesets, harder-to-bisect regressions, and merge conflicts from infrequent integration.

The dual-region deployment challenge went far beyond provisioning a second AKS cluster. The existing Terraform codebase was written as a monolithic configuration targeting a single Azure region. Resource names were hardcoded, region-specific parameters were embedded in module definitions rather than extracted as variables, and the state file was a single blob with no workspace separation. Replicating this infrastructure in a second region would have required either duplicating the entire Terraform codebase (creating a maintenance nightmare) or refactoring it into region-agnostic modules with parameterized inputs — the correct approach but one that required touching every resource definition in the existing configuration without breaking the production EU deployment.

Data sovereignty enforcement required more than geographic separation. Authentication tokens generated in Europe could not be valid in Asia-Pacific and vice versa — each region needed its own token signing keys, its own Key Vault instance, and its own certificate chain. Cross-region session migration (when a European EV owner charges their vehicle while traveling in Asia) required a secure token exchange protocol that transferred the minimum necessary data without replicating the full session state across regions. The existing authentication service had no concept of regional token scoping — every token was globally valid, which was a compliance violation waiting to happen.

The observability cost crisis stemmed from a common anti-pattern: logging everything at debug level in production. When the platform was small, the cost of debug-level logging was negligible. As the platform scaled to 50,000 charging stations and 8 million daily authentication events, the telemetry volume exploded. Analysis of the Datadog ingestion showed that 60% of the data was debug-level logs that served no operational purpose in production — verbose protocol traces, individual authentication step timestamps, and raw request/response payloads. The remaining 40% included genuinely useful operational telemetry, but it was buried in the noise. Alerts fired constantly because thresholds were set against absolute values rather than percentile-based metrics, creating alert fatigue that caused the operations team to ignore notifications — including, on one occasion, a genuine partial outage that went unnoticed for 47 minutes.

The security gaps identified in the audit were individually manageable but collectively represented a significant exposure surface. The shared service principal — a single Azure AD application registration used by all services across all environments — meant that a credential compromise in the development environment would grant access to production resources. Azure Functions processing IoT webhook events were publicly accessible with no private endpoint or network restriction, meaning they could be targeted directly from the internet. Key Vault contained secrets but had no rotation policy — several secrets had not been rotated in over 14 months. There was no managed identity strategy: services authenticated to Azure resources using client secrets stored in environment variables rather than using Azure Managed Identity, which would have eliminated stored credentials entirely.

The absence of any DevSecOps practices in the CI/CD pipeline meant that these security gaps were not being detected or prevented systematically. Container images were pulled from Docker Hub without vulnerability scanning. No SAST tool analyzed the codebase for common vulnerability patterns. No DAST tool tested deployed endpoints for runtime security issues. Pod security policies were not configured on the AKS clusters, meaning containers could run as root, mount host filesystems, and escalate privileges. The security audit had identified these gaps, but without automated detection integrated into the development workflow, new vulnerabilities were being introduced as fast as existing ones were being addressed.

The Docker build optimization delivered the most dramatic improvement in developer experience. We restructured the monolithic Dockerfile into a multi-stage build with BuildKit parallel execution. The first stage compiled the Go authentication service. The second stage, running in parallel, prepared the TLS certificate bundles and protocol buffer definitions. The third stage assembled the legacy C libraries. The final stage combined the outputs from all three stages into a minimal runtime image based on distroless. We configured Azure Container Registry (ACR) as the layer cache backend, meaning that unchanged layers were not rebuilt — only modified code triggered a rebuild of the affected stage. Additional optimizations included separating dependency resolution (go mod download) from code compilation, so that dependency layers were cached until go.mod changed. The cumulative effect reduced build times from 45 minutes to 4.5 minutes — a 90% reduction. We also pushed shared base images to ACR on a weekly schedule, further reducing the cold-start build time for the rare cases where the cache was invalidated.

The Terraform refactoring transformed the monolithic single-region configuration into a modular architecture with region-agnostic modules and region-specific variable files. Each Azure resource — AKS cluster, Key Vault, ACR, virtual network, network security groups, private endpoints — was defined in a reusable module that accepted region, naming prefix, and configuration parameters as inputs. The EU-West deployment was imported into the refactored modules using terraform import, preserving the existing state without recreation. The APAC deployment was provisioned from the same modules with different variable files specifying the target region, naming convention, and region-specific parameters such as Key Vault access policies and network CIDR ranges. Both regions shared a common Terraform backend in Azure Blob Storage with workspace-level state isolation, preventing cross-region state corruption.

The dual-region AKS architecture implemented private endpoints for every PaaS service: ACR, Key Vault, Azure Monitor, and Azure Service Bus. Each cluster ran with Azure CNI networking, network policies enforcing pod-to-pod communication restrictions, and pod security standards preventing privileged container execution. Key Vault per region stored region-specific token signing keys, TLS certificates, and service credentials — with automated rotation policies configured for 90-day cycles. The cross-region token exchange for traveling EV owners was implemented as a dedicated microservice that validated the originating region's token, extracted the minimum necessary claims, and issued a time-limited regional token that expired after the charging session completed. This design ensured that no persistent customer data crossed regional boundaries.

The observability redesign replaced Datadog with a self-managed stack built on Azure Monitor for metrics collection, Grafana for visualization, and a tiered log retention strategy. Hot retention (7 days) stored full-fidelity logs in Azure Monitor Logs for recent incident investigation. Warm retention (90 days) stored aggregated metrics and sampled logs in Azure Blob Storage for trend analysis. Cold retention (365 days) stored compressed archives for compliance audit requirements. Log level governance was enforced at the application level: production deployments automatically set log level to INFO, with DEBUG available only through a feature flag that expired after 4 hours. KEDA auto-scaling managed the log processing pipeline, scaling from 1 to 30 processor nodes based on ingestion volume — ensuring that log spikes during incident scenarios did not create processing backlogs while keeping steady-state costs minimal.

The DevSecOps pipeline integrated security scanning at every stage of the CI/CD workflow. Trivy scanned container images for known vulnerabilities before they were pushed to ACR, blocking any image with critical or high-severity CVEs from reaching the registry. SonarQube SAST analyzed every pull request for code quality issues and common vulnerability patterns, with quality gate enforcement preventing merges that introduced new issues. Azure Managed Identity replaced all service principal credentials — every workload authenticated to Azure resources using its pod-level managed identity, eliminating stored secrets entirely. KEDA autoscaling for event-driven workloads ensured that the Azure Functions processing IoT webhook events scaled based on Service Bus queue depth rather than running at fixed capacity. All Azure Functions were migrated to private endpoints, removing public internet exposure. The combined effect was a security posture that would have passed the most stringent enterprise security audit — a requirement the client anticipated as they pursued contracts with government-operated charging networks in Asia-Pacific.