Self-hosted and air-gapped deployment

What an offline update bundle must prove before you accept it: egress-free start, no vendor assistance, rollback

An offline update for an air-gapped LLM serving stack should be accepted only after a customer-runnable gate proves import-time configuration, telemetry opt-out, pinned GPU Operator artifacts and a rollback set. This guide lists the vendor-documented checks and what to preserve.

Atom XII ·

A packed auditorium during a technology keynote

What the update bundle must prove

Before you apply an offline update to an LLM serving stack, the vendor bundle should contain evidence you can run yourself, with the network disabled: that required configuration is set before the process starts, that no telemetry leaves the box, that the exact container images the new version needs are pinned in your local registry, and that the prior image set and configuration are preserved so you can roll back.

Three vendor-documented behaviors define the checks: huggingface_hub reads its environment variables at import time, vLLM collects usage data by default, and the NVIDIA GPU Operator depends on locally hosted images and OS-specific driver tags in air-gapped clusters.

Import-time configuration is a start-order problem

The Hugging Face Hub documentation states that all huggingface_hub environment variables are read at import time of the library, and that any modification made afterwards is not taken into account. The same page tells you to set environment variables before importing huggingface_hub. It also documents where local data lives: HF_HOME configures where huggingface_hub stores the token and cache, and defaults to ~/.cache/huggingface unless XDG_CACHE_HOME is set. HF_HUB_CACHE, HF_XET_CACHE and HF_ASSETS_CACHE default to subfolders of HF_HOME for repositories, Xet shard data and downstream assets respectively.

The inference for updates is that configuration written after the application process starts may not take effect for huggingface_hub. An acceptance test should therefore set and verify the environment before import or start, not after. Two documented timeouts matter for degraded-network behavior: HF_HUB_ETAG_TIMEOUT, default 10 seconds, after which huggingface_hub falls back to locally cached files when the metadata call times out, and HF_HUB_DOWNLOAD_TIMEOUT, default 10 seconds, after which a download raises a TimeoutError. HF_HUB_VERBOSITY accepts debug, info, warning, error or critical and defaults to warning; setting it to debug for the acceptance run is a proposed diligence step, not a documented requirement.

Telemetry opt-out must be verified, not assumed

vLLM documentation states that vLLM collects anonymous usage data by default, that the data does not contain sensitive information, and that a cleaned, aggregated subset will be publicly released. The collected fields include hardware details such as gpu_count, gpu_type, gpu_memory_per_device, model_architecture, vllm_version, dtype, tensor_parallel_size and quantization, as shown in the v0.4.0 example in the vLLM usage stats page.

The documented opt-outs are the environment variables VLLM_NO_USAGE_STATS or DO_NOT_TRACK, or a file at ~/.config/vllm/do_not_track. You can preview what would be sent by running tail against ~/.config/vllm/usage_stats.json. Because collection is on by default, the inference is that an offline deployment intending no outbound telemetry should explicitly set an opt-out and verify it, rather than assume a blocked network equals silence. The acceptance gate should check that one of the documented opt-outs is active in the running environment.

GPU Operator artifacts: registry contents and driver tags

NVIDIA's air-gapped installation documentation states that by default the GPU Operator requires internet access for two reasons: container images must be pulled during installation, and the driver container must download several OS packages before driver installation. To address this, the documentation says it may be necessary to create a local image registry and/or a local package repository. The documentation also states that using precompiled driver containers removes the need for the driver containers to download operating system packages and removes the need to create a local package repository.

The driver image has a documented caveat: the version field must be appended with the OS name running on the worker node. The documentation's example pulls the driver image for Ubuntu 22.04 with the tag 595.91.07-ubuntu22.04, and the operator image with the tag v26.7.3. Images are then tagged with the local registry prefix and pushed, and the repository field in values.yaml is updated with the local registry information. The general image syntax in the documentation is repository, image and version fields combined.

For the update bundle, the proposal is that the vendor pin the GPU Operator, driver and workload image references to your local registry and record the OS-specific driver tag, so you can verify the same artifacts are used in rollback. This is a proposed diligence step; the sources describe installation, not update bundles.

What a default install manages becomes your compatibility surface

The NVIDIA GPU Operator installation documentation states that a default installation deploys the NVIDIA GPU driver, NVIDIA Container Toolkit, NVIDIA Device Plugin, DCGM Exporter and MIG Manager as pods on every GPU worker node. The inference is that an offline update changing any of those components should treat them as part of the bundle compatibility surface, not as cluster prerequisites outside the update.

The same documentation states prerequisites that shape the gate. If ClusterPolicy is used for driver configuration, all worker nodes or node groups running GPU workloads must run the same operating system version to use the NVIDIA GPU Driver container; pre-installing the driver allows different operating systems. Node Feature Discovery is a dependency on each node, is deployed automatically by default, and must be disabled if NFD is already running; one documented check is inspecting node labels for keys starting with feature.node.kubernetes.io. Nodes need a container engine such as CRI-O or containerd, and if Pod Security Admission restricts pods, the gpu-operator namespace must be labeled with pod-security.kubernetes.io/enforce=privileged.

The documented post-install verification is checking that all GPU Operator pods are running with kubectl get pods -n gpu-operator. A proposed acceptance step is to capture that pod list and NFD state before the update and compare after, before you declare the update successful.

Multi-node consistency and start evidence

vLLM's parallelism documentation states that for multi-node deployment every node must provide an identical execution environment, including the model path and Python packages, and recommends container images as a way to keep environments consistent. It also documents start-up log lines worth capturing as evidence: the GPU KV cache size line and the Maximum concurrency line, which estimates how many requests can be served concurrently given the model's maximum sequence length from ModelConfig.max_model_len. The documentation says if these numbers are lower than your throughput requirements, add more GPUs or nodes.

For the acceptance gate, capturing those log lines from the new version with egress blocked is a proposed check that the stack started fully from local artifacts. Relevant serving flags named in the documentation include --tensor-parallel-size, --pipeline-parallel-size and --distributed-executor-backend, with mp for multiprocessing and ray for Ray.

The acceptance gate, with owners

The proposed gate, which is our recommendation rather than a documented vendor procedure, runs in this order. The vendor owns assembling the bundle: the pinned image list with OS-specific driver tags, the values.yaml or equivalent configuration, the environment variable set, the telemetry opt-out instruction, and the acceptance script itself. The customer operator owns running it: block egress, set and verify environment variables before process start, confirm the vLLM opt-out is active, start the new version, capture pod health, NFD state and the vLLM cache and concurrency log lines, and compare against the pre-update capture. The customer operator owns the rollback decision. The proposed operating rule: if any step requires the vendor on the call, the update fails the gate. A successful first start is not proof of an update channel; the documented dependencies on import-time configuration, default telemetry, local registry contents and OS-specific driver tags each create a separate check.

When the approach changes

The NVIDIA air-gapped documentation distinguishes environments by network flow: full air-gap without a proxy pulls images and packages from local sources only; full air-gap with an HTTP proxy routes driver-container package downloads through the proxy to a local package repository; proxy environments with full internet access differ again. Your gate should match the case you actually run. Precompiled driver containers change the preparation: no local package repository is needed. Pre-installing the NVIDIA driver on nodes removes the same-OS requirement for GPU worker nodes. If NFD already runs in your cluster, the Operator's NFD deployment must be disabled at install time, which changes what the pod-health comparison should expect.

Checklist

  • Verify environment variables are set before huggingface_hub import, not after process start
  • Confirm a documented vLLM opt-out is active: VLLM_NO_USAGE_STATS, DO_NOT_TRACK or ~/.config/vllm/do_not_track
  • Check every pinned image, including the OS-specific driver tag, exists in the local registry
  • Capture pre-update GPU Operator pod health and Node Feature Discovery state for comparison
  • Start the new version with egress blocked and capture the GPU KV cache size and Maximum concurrency log lines
  • Preserve the prior image set and prior values.yaml or equivalent configuration for rollback
  • Record whether precompiled driver containers or a pre-installed driver change the preparation steps
  • Treat any step that needs vendor assistance as a failed acceptance gate

Questions

Does a blocked network prove there is no telemetry?

No. vLLM collects usage data by default, so the gate should verify a documented opt-out such as VLLM_NO_USAGE_STATS, DO_NOT_TRACK or the ~/.config/vllm/do_not_track file is active, rather than infer silence from the firewall.

Why can configuration set after start fail silently?

The Hugging Face Hub documentation states huggingface_hub reads all its environment variables at import time and ignores later changes, so values must be set and verified before the process starts.

What has to be preserved for rollback?

The prior image set in your local registry and the prior values.yaml or equivalent configuration, including repository, image, version and the OS-specific driver tag such as the documented 595.91.07-ubuntu22.04 example.

Sources

  1. Environment variables - Hugging Face Hub documentation
  2. Usage Stats Collection - vLLM documentation
  3. Install NVIDIA GPU Operator in Air-Gapped Environments
  4. Installing the NVIDIA GPU Operator
  5. Parallelism and Scaling - vLLM documentation