Running AI/ML workloads on Kubernetes has become genuinely standard practice for production deployments — but it requires real, specific considerations beyond a typical stateless web application deployment.
apiVersion: apps/v1
kind: Deployment
metadata:
name: llm-inference-server
spec:
replicas: 2
template:
spec:
containers:
- name: inference
image: my-registry/llm-server:latest
resources:
limits:
nvidia.com/gpu: 1 # request a dedicated GPU per pod - unusual for typical web apps
requests:
memory: "32Gi"
nodeSelector:
gpu-type: "a100" # schedule specifically onto GPU-equipped nodes
Genuine AI-Specific Considerations
- GPU scheduling — requires the NVIDIA device plugin and careful, deliberate node pool configuration; GPUs generally can’t be as flexibly shared or overcommitted across pods the way CPU typically can
- Model loading time — large models can genuinely take minutes to load into memory, meaningfully affecting rolling deployment strategy and how readiness probes need to be configured
- Autoscaling on genuinely meaningful metrics — scaling based on GPU utilization or actual queue depth is often far more meaningful than scaling on CPU usage alone for AI inference workloads specifically
- Model storage — large model weight files genuinely need efficient, fast-loading storage (often a shared network volume) rather than baking multi-gigabyte weights directly into a container image, which would make every single deployment painfully slow
Kubernetes’ orchestration strengths (auto-restart, rolling deployments, horizontal scaling) genuinely still apply well to AI workloads — the meaningful difference is mostly in resource requests (GPUs specifically, not just CPU/memory) and the real operational patterns around model loading and storage.