AI Infrastructure & MLOps Interview Questions

1. How do you schedule GPU workloads on Kubernetes?

The NVIDIA device plugin, usually installed by the GPU Operator, advertises GPUs as an extended resource, nvidia.com/gpu, and a pod asks for them in its limits. In the default setup each unit is a whole physical GPU, so a container cannot ask for half a card and the scheduler will not hand one card to two pods.

Put GPU nodes in their own pool and taint them so ordinary pods stay off. GPU workloads get the matching toleration plus node affinity for the GPU type they need. When a small model would waste a whole card, share it: MIG splits an A100 or H100 into up to seven instances with hardware memory and fault isolation, while time-slicing shares one GPU with no isolation at all, so keep it for dev and trusted batch jobs.

Since Kubernetes 1.34, Dynamic Resource Allocation is GA. A pod can describe the device it needs, such as GPU model or memory, instead of only a count, and the scheduler finds a match.

What they listen for: Taints plus affinity for GPU pools, and the isolation difference between MIG and time-slicing.

2. How do you autoscale an LLM inference service?

Not on CPU, and not on GPU utilization either. That number only says a kernel was running, and latency climbs when the request queue and the KV cache fill up, often while utilization looks fine. Scale on what the inference server reports: requests waiting, KV-cache usage or time to first token, through KEDA or a custom-metrics HPA.

Then plan for cold starts. Pulling tens of gigabytes of weights and loading them onto a GPU can take minutes, so keep a floor of warm replicas, cache weights on the node or a fast volume, and scale ahead of known peaks. Round-robin load balancing also works badly for LLM traffic; the Gateway API Inference Extension routes on model-server load instead.

What they listen for: A queue or KV-cache signal instead of CPU, and a plan for cold starts.

3. How would you cut GPU costs without hurting training or latency?

Measure first. The DCGM exporter gives per-GPU utilization and memory, and most fleets find cards sitting idle or oversized. Then right-size per workload: a small model serves fine on an L4-class card. Share cards for small models with MIG, batch requests on the inference side, and use a quantized model where the quality holds up.

For training, run on spot or preemptible capacity with frequent checkpoints, so an eviction costs minutes instead of the whole run. Scale dev GPU nodes and idle notebooks to zero outside working hours, and track cost per thousand requests or per training run rather than the monthly bill.

What they listen for: Measuring utilization before buying anything, and checkpointing to make spot safe for training.

4. How do you monitor an LLM service in production?

In layers. Requests: time to first token and time per output token tracked separately, throughput, errors and timeouts. Model server: queue depth, KV-cache usage and batch size. Hardware: GPU utilization, memory, temperature and XID errors from DCGM. Money: tokens and cost per request, broken down by tenant or feature.

Then quality, which normal monitoring misses. A model can answer fast and wrong, so every new version runs an offline eval, and a sample of production answers gets reviewed. For tracing, OpenTelemetry has GenAI semantic conventions for model calls, but they are still marked as in development, so pin the version you build on.

What they listen for: The two latency numbers kept apart, and evals for quality alongside the usual metrics.

5. How do you roll out a new model version safely?

Treat the model like any other release artifact: versioned in a registry, immutable, with its eval results and a reference to its training data attached. It has to pass an offline eval gate against a fixed test set and the current model’s scores before it sees any traffic.

Next comes shadow traffic, where real requests go to the new model and the outputs are compared but never returned to users. Then a small canary with automatic rollback on latency, errors and quality. Rollback means pointing serving back at the previous version, which is still in the registry and, if you can afford it, still warm.

What they listen for: An eval gate before traffic, shadow before canary, and rollback as a pointer change.

6. How do you secure model files in the supply chain?

A model file can be executable. Pickle-based formats, including older PyTorch .bin and .pt files, can run code when they load, so prefer safetensors, which stores only tensors, and scan anything else before loading it. Pull models from a private registry or mirror, not straight from a public hub at deploy time.

Sign models when you publish them and verify before serving; the OpenSSF model-signing project, which reached v1.0 in April 2025, does this with Sigstore. A signature proves who produced the file and that it hasn’t changed, not that it is safe, so scanning still applies. Keep lineage too, so you can say which data and code produced the model running today.

What they listen for: Knowing pickle runs code on load, and that signing and scanning answer different questions.