KubeBench: Domain-Expert Code Writing AI & Comprehensive Benchmark for Kubernetes LLMs

What KubeBench measures

Instead of grading YAML by string similarity, KubeBench evaluates how models behave against a live Kubernetes API.

• Real production tasks. Each prompt reflects a concrete cluster scenario, such as configuring quotas for a team namespace or tightening RBAC for a service account.
• Three-dimensional scoring. Models are evaluated on whether manifests parse, deploy, and actually satisfy the operational intent.
• Difficulty tiers. Tasks span easy, intermediate, and advanced levels to mirror day-to-day infrastructure engineering work.
• LLM-as-judge reasoning. A judge model inspects cluster state and explains whether the produced YAML truly fulfills the task rather than just “working by accident.”

Why This Matters for Infrastructure Engineers

Prior work in this area has recently focused on creating fully autonomous, self-healing Kubernetes clusters. KubeBench is instead intentionally building for human infrastructure engineers and the AI agents that exist in their systems. For this reason, we've created several fine tuned LLMs based on open-source foundational base models that are designed and trained to assist Kubernetes engineers.

KubeBench measures progress toward creating better tools for human engineers who want to deploy intelligences that hold a specific domain expertise. Our Kubernetes models are instruction fine-tuned on a large repository of Kubernetes documentation and open source code bases, and are measured for their ability to create resources that are necessarily for any production cluster.

Tools for Infrastructure Development & KubeBench Models

Available KubeBench-Trained Models:

• Qwen K8s Assistant – 0.5B (qwen_0.5b_k8s-qlora_v4)
• Ultra-lightweight model optimized for manifest generation and quick scaffolding in resource-constrained environments
• Ideal for CI/CD pipelines, edge deployments, and rapid iteration on small clusters

• Qwen K8s Specialist – 1.5B (qwen_1.5b_k8s-qlora_v3)
• Balanced model with enhanced multi-step reasoning for production workflows and troubleshooting
• Strong performance on rollout planning, CI/CD configurations, and operational tasks

• Llama 3.1 – K8s Expert 8B (llama_v4)
• Production-grade model deeply fine-tuned for RBAC, controllers, operators, and enterprise deployment patterns
• Provides opinionated best-practice defaults with both code generation and architectural guidance

• Gemma 2 – K8s Deep Reasoner 2B (gemma_2b_k8s-qlora_v4)
• Specialized for complex reasoning tasks including migration planning, debugging sessions, and architecture reviews
• Excels at providing detailed explanations and "what & why" narrative alongside configurations

Kubernetes deployments are extremely diverse. A banking institution's cluster is vastly different from a genomics research platform. The human knowledge required to navigate HIPAA-compliant healthcare clusters differs fundamentally from other use cases. Our project's goal is then to augment Kubernetes engineering as it is happening today.

• Context-Aware RBAC Configuration: KubeBench models demonstrate strong performance on Role-Based Access Control resources,   critical resources that are required by every cluster regardless of industry or deployment type.
• Documentation-Grounded Responses: Fine-tuning on current Kubernetes documentation helps models generate configurations aligned with current best practices rather than outdated patterns from general training data.
• Confident Foundational Resource Expertise: KubeBench models have been specifically measured against their ability to create  the eight resource types (clusterrole, clusterrolebinding, configmap, namespace, role, rolebinding, secret, serviceaccount) that form the foundation of production deployments.
• Context-Aware Generation and Model Selection: Our methodology analyzes supervised fine tuning interventions on open source instruction-following and code-completion models from Meta, Alibaba, and Alphabet. We've analyzed models' ability to interpret scenario-based prompts and generate configurations tailored to specific team, security, and operational requirements.

Offline Model Development

Fine-Tuning Approach: QLoRA and Domain-Specific Training

Our model development strategy employs Quantized Low-Rank Adaptation (QLoRA), a parameter-efficient fine-tuning technique that combines quantized base models with Low-Rank Adaptation (LoRA) modules. QLoRA enables us to fine-tune larger language models on reasonable compute resources by significantly reducing memory requirements during training. While this approach introduces a slight performance trade-off compared to full fine-tuning, the efficiency gains make it practical to experiment with multiple model architectures and sizes—critical for identifying which base models respond best to Kubernetes domain adaptation.

Model Candidates

We selected four model families based on their demonstrated strengths in code generation and instruction-following capabilities:

Gemma Series: Open-weight models with strong instruction-following and code reasoning performance, providing a solid foundation for domain-specific adaptation.

Qwen Series: Optimized for code generation with strong multilingual task performance, particularly effective at understanding structured configuration syntax.

Llama Series: High-quality coding performance with robust instruction-following across diverse domains, offering reliable baseline capabilities for infrastructure code generation.

Mamba Series: Efficient structured generation architecture with strong performance on code and technical tasks, representing an alternative to traditional transformer-based approaches.

Training Data Strategy

Dataset Source: The Stack

Our fine-tuning dataset is derived from The Stack, a comprehensive corpus containing over 6TB of permissively-licensed source code files covering 358 programming languages. Created as part of the BigCode Project—an open scientific collaboration focused on the responsible development of Large Language Models for Code—The Stack serves as a foundational pre-training dataset for Code LLMs. These code-generating AI systems enable the synthesis of programs from natural language descriptions as well as code completion and transformation from existing snippets.

Kubernetes-Specific Filtering and Curation

From The Stack's extensive collection, we extracted and focused specifically on YAML configurations from Kubernetes-related repositories. We implemented rigorous data quality controls including filtering for correctness (removing buggy or outdated examples), validating against current Kubernetes API schemas, and ensuring structural consistency. The resulting dataset is cleaned, structured, and formatted specifically for fine-tuning workflows—prioritizing examples that demonstrate best practices for the eight foundational Kubernetes resources evaluated in KubeBench.

Dataset Source: Kubernetes Documentation

We also created and curated our own dataset from Kubernetes open source documentation from official Kubernetes websites that are covered by the Creative Commons CC-4.0 and Apache2.0 such as kubernetes.io or kubectl.

Overall, our datasets contained hundreds of thousands of valid Kubernetes samples which to use to implement a fine tuning workflow our selected models.

System Architecture

Platform Overview

KubeBench is built as a cloud-native platform that handles three primary functions: fine-tuning domain-specific models, evaluating them against the benchmark, and serving them for production use. The frontend is a React-based application hosted on DigitalOcean that allows users to explore benchmark results, compare model performance, and download HuggingFace-format model artifacts and adapters. The backend infrastructure runs entirely on Kubernetes with GPU-aware scheduling for compute-intensive evaluation workloads.

Core Infrastructure Components

The evaluation and inference workloads run as containerized APIs deployed to Kubernetes clusters. The Evaluation API handles the three-stage assessment process—applying generated YAML to live clusters, querying the Kubernetes control plane for operational validation, and coordinating LLM-as-judge reasoning assessments. The Inference API serves fine-tuned models for code generation, handling higher memory and CPU requirements for real-time manifest generation.

Both APIs are backed by Redis for caching and task coordination, reducing redundant API calls to the Kubernetes control plane and improving response times for repeated queries. Relational metadata and job state are stored in CloudSQL, while Dagshub and DVC manage dataset and model versioning to ensure reproducibility across training runs and evaluation cycles.

Kubernetes Resources & Deployment

The platform leverages Kubernetes' native resource model for secure and isolated evaluation:

Namespaces and RBAC: A dedicated evaluation namespace with a service account bound to cluster-admin permissions and impersonation capabilities allows the Evaluation API to create, inspect, and clean up test resources without affecting production workloads.

GPU Lifecycle Management: Automated CronJobs scale GPU-enabled evaluation nodes up during active hours and down during off-peak periods to optimize compute costs.

Dual Ingress Architecture: Separate ingress controllers for evaluation and inference traffic isolate the workloads so that evaluations and inference from users scale independently. We also isolated security policies and traffic management to logically separate compute-heavy batch evaluations from more latency-sensitive inference requests from the frontend.

Environment-Specific Deployment: Our team utilized Skaffold workflows support both local development on Minikube and production deployment to GKE. We used Kustomize to manage environment-specific configurations, secrets injection, and resource limits across development and cloud environments.

Supporting Services

The architecture separates concerns between compute, storage, and delivery. Model artifacts and training datasets are versioned through Dagshub and DVC to provide model lineage tracking and reproducibility. The frontend delivery infrastructure runs independently on DigitalOcean to decouple the frontend application from backend workloads. This separation allows the platform to scale evaluation capacity on Kubernetes clusters while maintaining consistent frontend performance and availability to showcase our ongoing model development and evolving capabilities of KubeBench models.

Project Accomplishments & Roadmap

Overview

KubeBench was designed to address a fundamental gap in infrastructure engineering: general-purpose LLMs often generate Kubernetes code that appears correct but fails in production. By fine-tuning domain-specific models and evaluating them through actual cluster deployment, we've built a platform that delivers measurable improvements in code quality, with the goal of supporting greater developer productivity and safe deployment of AI agents.

Model Development & Fine-Tuning Results

We applied QLoRA to train 16+ Kubernetes-specialized models across different size classes. We trained models of varying sizes - from ultra-lightweight 0.5B parameter models suitable for CI/CD pipelines to more capable 8B parameter models for complex reasoning tasks; and have made four of the best models available on KubeBench.dev. The results demonstrate clear performance improvements over baseline models:

• +18–25% execution validity: Generated YAML successfully validates and applies to Kubernetes clusters
• +21–27% YAML quality: Configurations align with best practices and operational requirements
• +18–23% overall benchmark score: Combined improvement across runtime validity, operational correctness and reasoning assessment

Notably, smaller models (0.5B–2B parameters) benefited most from domain-specific fine-tuning. This finding has significant implications for edge deployments and resource-constrained environments where developers still need fast and accurate assistance but are without internet or large compute. 

Benchmark Pipeline Development

We built a repeatable and scalable benchmark generator that currently covers the eight core Kubernetes resource types with a focus on CREATE operations. The evaluation framework automates scoring for any model against baseline performance through three-stage validation: schema compliance, successful cluster deployment and operational correctness assessment via LLM-as-judge. This pipeline enables rapid iteration on model development and provides researchers with a standardized methodology for comparing Kubernetes code generation capabilities.

Production Deployment

The platform is fully deployed on DigitalOcean infrastructure with FastAPI inference endpoints powering an interactive web interface. Users can engage with working chat functionality, explore model capabilities and download any of the four fine-tuned model files hosted for community use and research.

Key Learnings from Initial Deployment

Early user feedback and evaluation results revealed important insights about fine-tuning for infrastructure code generation. Fine-tuned models demonstrate substantially deeper Kubernetes knowledge and greater specificity than their base model counterparts, particularly in generating correct RBAC configurations and understanding resource relationships. However, fine-tuning alone may not produce the desired effect on already instruction-tuned models at larger sizes—achieving strong performance required substantial additional work in prompt engineering, dataset curation and evaluation methodology refinement.

Importantly, user testimonials confirm that perfection is neither a desired nor expected outcome. Infrastructure engineers value models that reduce cognitive overhead and accelerate initial scaffolding, even if manual review and adjustment remains required. This aligns with our design philosophy: augmenting human expertise rather than replacing it.

Future Directions

Benchmark Expansion: Our immediate priority is extending coverage beyond CREATE operations to include READ, UPDATE and DELETE tasks, as well as CLI generation scenarios. We aim to expand from the current 810 tasks to a 10,000-task research-grade benchmark covering the full spectrum of Kubernetes resource types and operational workflows.

Fine-Tuning Improvements: We will increase dataset curation through enhanced filtering, deduplication and validation against current Kubernetes API schemas. Experimentation with hybrid training approaches combining QLoRA with targeted instruction tuning may improve performance on complex multi-resource tasks. Adding automated loss-curve analysis and error-pattern detection will accelerate training iteration cycles.

Front-End & Agent Capabilities: Planned upgrades include context window management for session memory, unique chat experiences for each fine-tuned model, file and document upload features for context-aware generation, multi-file editing with live schema validation and an "explain mode" that shows why the model generated each field.

Architectural Evolution: Long-term development includes RAG integration with official Kubernetes documentation for real-time reference, domain-specific context templates tailored to common infrastructure workflows and agent orchestration capabilities for handling larger multi-step infrastructure tasks that span multiple resource types and deployment stages.

KubeBench demonstrates that domain-specific fine-tuning of coding LLMs can address the inaccuracy and inefficiency of general-purpose models when generating infrastructure code. The result is more reliable YAML output, faster developer workflows and measurable productivity gains for engineers managing production Kubernetes environments.