Enterprise AI/ML Platform — MLOps Pipeline & GPU Cost Optimization

The 8-hour deployment cycle was the most visible symptom of a deeper problem: the absence of any operational automation for the ML lifecycle. Every step in the deployment process — from building a Docker image to verifying the deployed model's output quality — was performed manually, by hand, every single time. There was no pipeline, no runbook automation, and no tooling beyond SSH terminals and the Azure portal.

Model building consumed 2 hours per deployment. The Docker images were large (8–15 GB) because they included model weights, inference code, and all dependencies in a single monolithic layer. Builds were not cached — every deployment rebuilt the entire image from scratch. Multi-stage builds, layer caching, and parallel build strategies had not been explored. The build process ran on a single developer workstation because the team had never configured a build server with GPU support.

GPU node provisioning added another 1.5 hours. The DevOps engineer would log into the Azure portal, navigate to the AKS cluster, modify the node pool configuration, wait for nodes to provision, verify GPU driver installation, and confirm that the Kubernetes device plugin recognized the available GPUs. This process was manual because the team did not use infrastructure-as-code for GPU node pools — they had been created interactively through the portal and were managed through the portal exclusively.

Endpoint configuration consumed 1 hour and was the most error-prone step. Model endpoints were configured through Kubernetes manifests that were hand-edited for each deployment. The manifests referenced specific Docker image tags, GPU resource limits, replica counts, and environment variables that varied between models. A single typo — a wrong image tag, an incorrect GPU limit, a missing environment variable — would cause the deployment to fail silently, producing errors only when inference requests arrived at the new endpoint.

Smoke testing took 3.5 hours — the longest single step — because it was entirely manual. The data science team would submit test inference requests against the newly deployed model, compare outputs against a reference set, and make a subjective judgment about whether the model was performing acceptably. There were no automated quality checks, no statistical comparison against baseline metrics, and no defined acceptance criteria beyond "it looks right." This meant that model quality regressions could reach production if the person running smoke tests missed a subtle degradation.

The absence of model versioning compounded every other problem. When a deployment failed or a model showed degraded performance in production, the only "rollback" option was to find the Docker image tag of the last known good deployment — which lived in a Slack message or a wiki page, not in any systematic registry — rebuild the endpoint configuration from memory, and re-deploy. This took 2–4 hours and was itself a source of errors.

GPU compute costs were a separate but equally urgent concern. The client ran all workloads — training, evaluation, and inference — on the same pool of reserved N-series GPU instances. Training workloads (which are batch, interruptible, and tolerant of preemption) were consuming reserved instance capacity that could have been served by spot instances at 60–70% lower cost. There was no auto-scaling — the GPU node pool ran at a fixed size 24/7 regardless of actual demand, which varied significantly between business hours (high inference volume) and off-hours (minimal activity but ongoing training jobs).

The experiment tracking gap compounded the deployment problem and created a separate category of operational risk. A/B testing of model variants — essential for validating that a new model version outperformed its predecessor — was conducted by deploying both versions behind a load balancer and manually comparing inference logs after 48 hours. There was no systematic framework for defining test hypotheses, selecting evaluation metrics, controlling traffic splits, or determining statistical significance. The data science team relied on qualitative assessment: a senior data scientist would review a sample of outputs from each variant and make a judgment call. This approach had led to at least two documented regressions where an improved model variant that appeared better on the sample outputs actually performed worse on edge cases that the manual review missed. Model rollback in these situations required archaeological effort: locating the last known good Docker image tag — typically found by searching Slack message history or scrolling through Azure Container Registry tags — then rebuilding the endpoint manifest from memory and re-deploying through the 8-hour manual process. The entire cycle from regression detection to production recovery averaged 3 business days.

The solution was a fully automated MLOps pipeline that covered the entire lifecycle from code commit to production inference, with self-service deployment for data scientists and optimized GPU scheduling for cost efficiency.

The CI/CD pipeline, built on GitHub Actions, replaced the manual 8-hour deployment process with an automated flow triggered by PR merge. Docker builds were restructured using multi-stage builds with aggressive layer caching: base images with dependencies were pre-built and cached, model weights were stored in Azure Blob Storage and mounted at runtime rather than baked into the image, and inference code was built as a thin final layer. These optimizations reduced image build times from 2 hours to 12 minutes. We also enabled parallel builds for multi-model deployments — when the platform shipped updates to multiple models simultaneously, each model built concurrently rather than sequentially.

GPU node provisioning was automated through KEDA-driven auto-scaling. We configured separate node pools for training (using Azure Spot VMs with A100 GPUs) and inference (using reserved instances with H100 GPUs for latency-sensitive workloads). Training workloads submitted through the pipeline automatically targeted the spot node pool, with checkpoint-based fault tolerance ensuring that preempted training jobs resumed from their last checkpoint rather than restarting from scratch. Inference workloads scaled horizontally based on request queue depth — KEDA monitored the inference queue and added or removed GPU nodes to maintain response time SLAs. During off-hours, the inference pool scaled to a minimal footprint, reducing idle GPU costs by approximately 40%.

The RAG (Retrieval-Augmented Generation) pipeline was a significant technical deliverable. We architected a full retrieval-augmented generation system using Azure OpenAI for the generation component, PostgreSQL with pgvector for dense vector storage, and Weaviate as a dedicated vector database for semantic search across large document corpora. The pipeline ingested documents, chunked them using a sliding-window strategy, generated embeddings via Azure OpenAI's embedding models, stored them in both pgvector and Weaviate, and served retrieval queries through a unified API that combined semantic search results with keyword search for hybrid retrieval. This RAG architecture replaced the client's previous approach of fine-tuning models on domain-specific data — retrieval augmentation provided better accuracy with lower latency and dramatically lower training costs.

Model versioning and automated rollback were built on MLflow's model registry. Every model deployed through the pipeline was registered with full lineage metadata: training data version, hyperparameters, evaluation metrics, and the exact Git commit of the inference code. Health checks ran continuously against deployed models, comparing inference latency, error rates, and output quality metrics against baseline thresholds. If any metric degraded beyond the threshold, the pipeline automatically rolled back to the last known good model version — a process that took under 5 minutes compared to the previous 2–4 hour manual rollback.

Self-service deployment for data scientists was the final and highest-impact component. We created a model configuration repository where data scientists could submit model updates as pull requests: changing the model version, adjusting inference parameters, or deploying a new model variant for A/B testing. The PR triggered a validation pipeline that checked configuration schema, ran automated quality tests against a reference dataset, and produced a deployment preview showing exactly what would change. Upon PR approval (by a data science lead, not a DevOps engineer), the pipeline deployed automatically. This eliminated the 3-day average wait time for DevOps involvement and gave the data science team complete ownership of their deployment cadence.

The automated evaluation suite replaced the 3.5-hour manual smoke test with a comprehensive quality verification pipeline that ran as part of every deployment. The suite executed 500+ inference requests against a curated reference dataset spanning all supported model tasks: document summarisation, entity extraction, question answering, image generation, and custom domain-specific queries. Each response was evaluated against ground-truth annotations using task-specific metrics — ROUGE scores for summarisation, F1 for entity extraction, cosine similarity for embedding quality, and FID scores for image generation. Statistical significance testing compared the new model's aggregate metrics against the production baseline, with deployment proceeding automatically only if all metrics met or exceeded predefined thresholds. When a metric fell below threshold, the pipeline generated a detailed regression report identifying which task categories degraded and by how much, enabling the data science team to diagnose the issue without re-running the full evaluation manually. This evaluation pipeline was itself versioned and deployed through the same CI/CD system, ensuring that quality standards evolved alongside the models they validated.