GPU and AI platforms
MIG or time-slicing and vLLM topology for model serving on Kubernetes
A per-endpoint guide to choosing MIG instances or time-sliced GPUs and a single-node or multi-node vLLM topology for model serving on Kubernetes, so node loss has a declared, tested outcome rather than an assumed one.

What to decide, endpoint by endpoint
For each inference endpoint doing model serving on Kubernetes, choose the GPU sharing mode and the vLLM parallelism topology together, because both set the failure domain. The NVIDIA MIG User Guide describes MIG as partitioning supported GPUs into isolated instances with dedicated compute and memory; the NVIDIA GPU Operator documentation states that time-slicing has no memory or fault isolation between replicas. The outcome on node loss is a contract your team declares and tests, not a property the tools hand you.
What the GPU Operator installs and what to prepare first
The NVIDIA GPU Operator installation page states that a default installation deploys the NVIDIA GPU driver, the NVIDIA Container Toolkit, the NVIDIA Device Plugin, DCGM Exporter and MIG Manager as pods on every GPU worker node. The current patch release of that documentation is v26.7.2, and the page offers the chart either as an OCI artifact or from the classic NVIDIA Helm repository.
The documented prerequisites are concrete. The kubectl and helm CLIs must be available on a client machine. Nodes must be configured with a container engine such as CRI-O or containerd. If you plan to use ClusterPolicy for driver configuration, all worker nodes or node groups that run GPU workloads must run the same operating-system version to use the NVIDIA GPU Driver container; if you pre-install the driver on the nodes, you can run different operating systems. Node Feature Discovery is a dependency on each node, and the Operator deploys it by default unless it is already running; the documentation offers a kubectl command using jq that checks for feature.node.kubernetes.io labels to determine whether NFD is already present. If your cluster uses Pod Security Admission, the documentation says to label the Operator namespace with pod-security.kubernetes.io/enforce set to privileged.
For an offline or isolated install, our inference is that the environment needs the chart, the container images and the Cosign public key staged inside it before install. The reasoning: the documented install commands reference network locations, namely the OCI artifact path under nvcr.io, the Helm repository at helm.ngc.nvidia.com, and a Cosign verification step that fetches a public key from api.ngc.nvidia.com. Confirm this inference against the full documentation and your internal registry setup before planning the isolated install.
The documented verification step is to confirm the Operator is healthy by checking that all GPU Operator pods are running, using kubectl get pods in the gpu-operator namespace and comparing against the Running statuses in the documentation's example output. Our proposal for verifying with the network disabled is to stage every image the Operator and its components need locally, then run that same pod check in staging with the network cut, and only rely on the result once it has been proven there. This is a proposed diligence step, not a documented procedure.
Ownership is a suggested division to adapt, not a fact from the sources: the platform team plausibly owns the Operator install, node operating-system consistency and the namespace label, and whoever builds the node images plausibly owns the container engine and any internal registry. Map these roles to your own organization before the install.
MIG versus time-slicing as an isolation choice
The NVIDIA MIG User Guide states that MIG partitions supported GPUs into multiple isolated instances, each with dedicated compute and memory resources, and that MIG enables efficient GPU utilization across multiple users or workloads with guaranteed performance. The GPU Operator time-slicing page states the trade directly: time-slicing lets workloads scheduled on oversubscribed GPUs interleave, but unlike MIG there is no memory or fault isolation between replicas, and the page frames this as better than not being able to share at all for some workloads.
The same page documents the sharing semantics. A typical resource request provides exclusive access; a request for a time-sliced GPU provides shared access. A request for more than one time-sliced GPU does not guarantee a proportional amount of GPU compute power. Each pod can run as many processes on the underlying GPU as it wants, and the GPU provides an equal share of time to all GPU processes across all pods. Time-slicing can be configured cluster-wide, per node, or both, and node labels change after configuration: the documentation gives the example nvidia.com/gpu.replicas = 8 and nvidia.com/gpu.product = A100-SXM4-40GB-SHARED on a DGX A100, where the -SHARED suffix lets node selectors distinguish shared from exclusive GPUs.
One bounded inference: because NVIDIA documents no memory or fault isolation between time-sliced replicas, a fault in one workload may affect its co-tenants on the same GPU. Treat co-tenant impact as a behavior to verify for your own workloads in staging, not as either guaranteed or impossible.
vLLM topology: one GPU, one node, or several
The vLLM parallelism documentation gives a decision sequence for a single-model replica, with hedges worth keeping. If the model fits on a single GPU, distributed inference is probably unnecessary; run inference on that GPU. If the model is too large for one GPU but fits on one node, use tensor parallelism, for example tensor_parallel_size=4 on a node with 4 GPUs. If the model is too large for a single node, combine tensor parallelism with pipeline parallelism: set tensor_parallel_size to the number of GPUs per node and pipeline_parallel_size to the number of nodes, for example tensor_parallel_size=8 and pipeline_parallel_size=2 on 2 nodes with 8 GPUs each. For serving, the flags are --tensor-parallel-size and --pipeline-parallel-size on the vllm serve command.
Two edge cases matter. If the model fits within a single node but the GPU count does not evenly divide the model size, the documentation says to enable pipeline parallelism with tensor_parallel_size set to 1 and pipeline_parallel_size set to the number of GPUs. And if the GPUs on the node lack NVLINK interconnect, such as L40S, the documentation recommends pipeline parallelism over tensor parallelism for higher throughput and lower communication overhead. For multi-node serving, the default distributed runtimes are Ray for multi-node and native Python multiprocessing for single-node, overridable with distributed_executor_backend or --distributed-executor-backend set to mp or ray.
For multi-node deployment, the documentation requires that every node provide an identical execution environment, including the model path and Python packages, and recommends container images as a convenient way to keep environments consistent and hide host heterogeneity.
What node loss means for each topology
This section is our inference from the vLLM topology guidance, not a documented failover procedure. If a replica uses tensor parallelism across the GPUs of one node, losing that node removes the entire tensor-parallel group, so the replica cannot continue on the remaining GPUs without a new placement. If a replica combines tensor parallelism per node with pipeline parallelism across nodes, a node loss removes one pipeline stage and with it the whole multi-node replica. In both cases the surviving capacity is other replicas, not the damaged one. Whether Kubernetes reschedules the work, whether in-flight requests retry, and whether a smaller model can absorb traffic are behaviors your environment determines; the staging test in the final section is how your team settles them.
The observability caveat for time-slicing
The GPU Operator documentation states that DCGM-Exporter does not support associating metrics to containers when GPU time-slicing is enabled with the NVIDIA Kubernetes Device Plugin, and that the Operator does not monitor changes to a time-slicing config map. An inference: a platform that increases sharing through time-slicing should not assume per-container GPU metrics exist, and should verify its observability path before relying on container-level attribution for on-call diagnosis.
A per-endpoint diligence table, proposed
We propose the platform team keep a service-owned manifest with one row per endpoint: model fit, GPU sharing mode, tensor and pipeline parallel sizes, replica count, declared failure outcome, and observability caveat. The declared outcome should be one of three: retry on another replica, fall back to a smaller model, or fail fast. This is a proposal; the table and the fallback behaviors are our construction, not a documented procedure. The vLLM documentation does give one sizing signal: the GPU KV cache size log line reports the total tokens that can be stored in the GPU KV cache at once, and the maximum concurrency line estimates how many requests can be served concurrently at a tokens-per-request value taken from the model configuration's maximum sequence length, ModelConfig.max_model_len. If those numbers are lower than your throughput requirements, the documentation says to add GPUs or nodes. As a suggested division, application owners own the declared outcome, the platform team owns the sharing mode, topology and replica count, and on-call reads the manifest during an incident; adapt this to your organization.
The pre-production node-loss test, proposed
We propose testing the declared path in staging before production: remove a GPU node from the serving pool and observe whether each endpoint retries elsewhere, falls back, or fails as declared, then record the evidence in the table. This is a proposed diligence step, not a documented test procedure.
The recommendation changes under conditions the sources do support. If endpoints need hardware isolation between co-tenants, prefer MIG over time-slicing: the GPU Operator page states time-slicing trades the memory and fault isolation MIG provides for broader sharing, and also notes time-slicing can share older GPUs that do not support MIG, and that MIG and time-slicing can be combined to share MIG instances. If the multi-node vLLM requirement of an identical execution environment on every node is hard to meet in a heterogeneous pool, our recommendation, not a source conclusion, is to weigh keeping replicas single-node; note that single-node replicas carry their own exposure, since losing the node loses the whole replica. The documentation itself only recommends container images to keep multi-node environments consistent.
Checklist
- Confirm every GPU worker node runs a supported container engine, CRI-O or containerd, before installing the GPU Operator.
- Verify all GPU worker nodes run the same operating-system version if using ClusterPolicy driver configuration, or pre-install the driver.
- Check whether Node Feature Discovery is already running before letting the Operator deploy it.
- If installing offline, stage the chart, container images and Cosign public key, and treat network-disabled verification as untested until proven in staging.
- Record, per endpoint, the sharing mode, tensor and pipeline parallel sizes, replica count and declared node-loss outcome.
- Confirm whether DCGM-Exporter per-container metrics are available if time-slicing is enabled.
- Remove a GPU node in staging and verify each endpoint behaves as declared before production traffic.
Questions
Does Kubernetes or vLLM automatically fail over inference when a GPU node is lost?
Treat the failover path as a contract your team declares and verifies in staging. The NVIDIA pages define the sharing modes and the vLLM page defines the parallelism topologies; the node-loss test proposed in this guide is how you settle what actually happens in your environment.
Can I combine MIG and time-slicing?
Yes. The NVIDIA GPU Operator documentation states you can combine MIG and time-slicing to provide shared access to MIG instances, and that time-slicing also provides shared access on older GPUs that do not support MIG.
Which sharing mode gives isolation?
Per the NVIDIA MIG User Guide, MIG instances have dedicated compute and memory with hardware isolation. Per the GPU Operator documentation, time-sliced replicas have no memory or fault isolation between replicas.
Does the GPU Operator documentation cover air-gapped installation?
The install commands shown in the documentation reference network locations: the OCI artifact under nvcr.io, the Helm repository at helm.ngc.nvidia.com, and a Cosign public key fetched from api.ngc.nvidia.com. Staging the chart, images and key for an offline install is our inference; check the full documentation before planning an isolated install.