Private AI operations

Before you approve a self-hosted LLM pilot: the pre-start defaults audit

Vendor defaults are not evidence of isolation. This page turns vLLM telemetry, Hugging Face cache variables and NVIDIA GPU Operator dependencies into a checkable pre-pilot audit, with owners, offline verification steps and the limits of what the documentation proves.

Atom XII ·

A software team at a whiteboard

The decision

No. Vendor defaults are not sufficient evidence that a pilot is isolated. vLLM collects usage data by default, Hugging Face Hub reads its environment variables only at import time, and NVIDIA's GPU Operator expects internet access unless a local registry and package source are prepared first. Require a documented defaults audit before first launch: telemetry off and verified, cache paths set before the process starts, and the GPU isolation mode written down.

The audit is narrow on purpose. It compares three default behaviors in a common serving stack and turns each into a check someone performs before the first launch. It does not repeat an install procedure, and it does not certify that the pilot cannot disturb a clinical, research or plant system. It answers a narrower question: did the team verify the defaults that matter, or did it assume them.

This gate is a proposed diligence step, not a vendor guarantee. The documents behind it describe how specific products behave by default: what vLLM collects, when Hugging Face Hub reads its variables, and what the GPU Operator needs in order to install. Whether those defaults are safe beside your clinical, research or plant systems is a question about your environment, and the audit answers it with verification rather than assumption. Treat every default as configuration to verify.

What vLLM sends by default

The vLLM documentation states that vLLM collects anonymous usage data by default, to help its engineering team understand which hardware and model configurations are widely used. The documentation says the collected data is transparent and does not contain sensitive information, and that a subset of the data, after cleaning and aggregation, will be publicly released for the community's benefit. The page links to a 2024 usage report as an example of that release.

The same page shows an example of a collected record, labeled as of v0.4.0. The example includes a uuid, the provider (GCP in the example), num_cpu, cpu_type, cpu_family_model_stepping, total_memory, architecture, platform, gpu_count, gpu_type (NVIDIA L4 in the example), gpu_memory_per_device, model_architecture (OPTForCausalLM in the example), vllm_version, context, log_time, source, dtype, tensor_parallel_size, block_size, gpu_memory_utilization, quantization, kv_cache_dtype, enable_lora, enable_prefix_caching, enforce_eager and disable_custom_all_reduce. The documentation directs readers to vllm/usage/usage_lib.py for the list of what the latest version collects, which is the list your reviewer should read rather than the example.

Opting out is documented and simple. The vLLM documentation states that usage stats collection can be disabled by setting the VLLM_NO_USAGE_STATS or DO_NOT_TRACK environment variable, or by creating a ~/.config/vllm/do_not_track file. It also documents how to preview what is collected: run tail against ~/.config/vllm/usage_stats.json. That file is the audit's evidence artifact. The serving engineer sets the opt-out; the security or data-governance reviewer confirms the collected-fields list against policy before accepting it.

Hugging Face cache paths and the import-time rule

The Hugging Face Hub documentation states that all environment variables are read at import time of huggingface_hub, and that any modification made afterwards will not be taken into account. It instructs users to set environment variables before importing the library. This is the detail that makes cache configuration a pre-start check rather than a runtime adjustment.

Two variables define where data lands. HF_HOME configures where huggingface_hub locally stores data, including the token and the cache, and defaults to ~/.cache/huggingface unless XDG_CACHE_HOME is set. HF_HUB_CACHE configures where repositories from the Hub are cached locally, covering models, datasets and spaces, and defaults to $HF_HOME/hub. HF_TOKEN_PATH, which defaults to a token file under HF_HOME, controls where the User Access Token is stored.

The documentation also covers two timeout behaviors relevant to restricted networks. HF_HUB_ETAG_TIMEOUT, defaulting to 10 seconds, defines how long to wait for a metadata response before fetching a file; if the request times out, huggingface_hub defaults to the locally cached files. HF_HUB_DOWNLOAD_TIMEOUT, also defaulting to 10 seconds, governs file downloads, and a timeout raises a TimeoutError.

Here is the inference the audit rests on, labeled as such: because the variables are read at import time, setting HF_HOME or HF_HUB_CACHE after the library has been imported will not redirect the cache for that process. If cache location is part of your isolation boundary, the variables must be set in the service unit or container specification before the process starts. The proposed check is to record the values of HF_HOME and HF_HUB_CACHE as the process sees them at startup, and to confirm the model files load from the intended local path.

What the GPU Operator needs before an offline install

NVIDIA's GPU Operator documentation for air-gapped environments states that, by default, the GPU Operator requires internet access for two reasons: container images need to be pulled during installation, and the driver container needs to download several OS packages prior to driver installation. The document adds that using precompiled driver containers removes the need for the driver containers to download operating system packages and removes the need to create a local package repository.

Without internet access, the same document states, the GPU Operator requires all images to be hosted in a local image registry accessible to all nodes in the cluster. The preparation sequence it describes starts with the values.yaml file used for GPU Operator configuration, fetched from the NVIDIA gpu-operator repository with a version such as v26.7.3 that you replace with the version you want. Images follow the syntax repository, image and version; the document's examples include pulling nvcr.io/nvidia/gpu-operator:v26.7.3 and, for the driver, nvcr.io/nvidia/driver:595.91.07-ubuntu22.04.

The driver image carries a caveat the audit should check directly: the version field must be appended by the OS name running on the worker node. The document's Ubuntu 22.04 example appends -ubuntu22.04 to the driver version. A mismatch here is a concrete, checkable failure. After pulling, images are tagged with the local registry path as a prefix, pushed to the local registry, and the repository field in values.yaml is updated with the local registry information.

The document also distinguishes environments by connectivity: an HTTP proxy with full internet access, an HTTP proxy with limited internet access, and full air-gapped setups with or without an HTTP proxy. It notes that Kubernetes nodes must be able to reach the local DNS servers, and that public name resolution for the image registry and package repositories is mandatory for the proxy use cases. For Red Hat OpenShift in air-gapped environments, it points to a separate disconnected-environment procedure. Your platform engineer should record which use case the cluster falls into, because that determines whether a local package repository is needed at all.

GPU isolation: what MIG gives you

NVIDIA's Multi-Instance GPU User Guide states that MIG partitions supported NVIDIA GPUs into multiple isolated instances, each with dedicated compute and memory resources, and that MIG enables efficient GPU utilization across multiple users or workloads with guaranteed performance. The guide covers MIG concepts, supported hardware, setup steps, and integration with tools like Docker and Kubernetes.

Two limits matter for the audit. First, MIG applies to supported NVIDIA GPUs; the excerpt cited here does not enumerate them, so the team must confirm the pilot's hardware against the guide's supported-hardware section before naming MIG as the isolation mode. Second, the guide describes what MIG does on the GPU: partitioning with dedicated compute and memory per instance. Whether that is sufficient beside your clinical or plant systems is a question about your environment and the owners of those systems; the audit asks for a documented isolation decision, not a guarantee.

The proposed check is a one-line record: the chosen isolation mode, the hardware it runs on, and the document that supports the choice. If MIG is chosen, cite the MIG User Guide and the specific GPU. If another mode is chosen, cite that mode's documentation. What the audit rejects is silence, where the isolation behavior of the pilot is simply whatever the default happened to be.

The audit gate: who owns each step

This allocation is proposed, not sourced. It divides the audit so that no single person both configures and approves. The platform or infrastructure engineer owns the registry and driver work: hosting all images in a local registry reachable by every node, confirming the driver image tag carries the worker node's OS name, and preparing either precompiled driver containers or a local package repository for restricted clusters. The serving engineer owns the process configuration: setting VLLM_NO_USAGE_STATS or DO_NOT_TRACK, or creating the do_not_track file, and setting HF_HOME and HF_HUB_CACHE before huggingface_hub is imported.

The security or data-governance reviewer owns the evidence review: reading the collected-fields list in vllm/usage/usage_lib.py against policy, confirming the cache paths the process actually used, and checking the offline verification results. The CIO, CTO or chief data or AI officer owns the approval decision itself, and signs the gate only on the completed evidence.

The gate has four evidence items. First, telemetry opt-out is set and the usage stats file is verified unchanged after a run. Second, the Hugging Face cache variables were set before import and the model loaded from the intended path. Third, the GPU isolation mode is documented with its supporting hardware. Fourth, for restricted or air-gapped clusters, there is evidence of a local image registry and either precompiled driver containers or a local package repository. A pilot that cannot produce these four items has not been audited; it has merely been launched.

Verifying with the network disabled

These verification steps are proposed, built on documented behaviors. Block egress from the pilot's nodes and run the stack once, end to end, before approving it. Before the run, note the contents of ~/.config/vllm/usage_stats.json if it exists. After the run, check it again; if the opt-out is working, the documented collection should not have appended a record. This is a check of the vLLM opt-out specifically, and it uses the preview command the vLLM documentation itself provides.

Confirm the model files load from the local cache path you set through HF_HOME and HF_HUB_CACHE, not from a fresh download. The Hugging Face documentation's timeout behavior supports the expectation that a process with no egress either reads from cache or fails loudly; a silent success after a supposed download is a signal to investigate. Confirm that the GPU Operator pods pulled only from the local registry, and that the driver container made no package downloads, which NVIDIA's documentation says is the expected behavior when precompiled driver containers are used.

Record the verification in the audit document: date, who ran it, what egress was blocked, and what the artifacts showed. The purpose is not to prove the cluster is air-gapped forever. It is to prove the defaults were actually changed, rather than assumed.

What changes the recommendation

The preparation burden depends on the connectivity use case NVIDIA documents. A cluster behind an HTTP proxy with full internet access does not need a local image registry or package repository in the same way, though public name resolution for the registry and package repositories is mandatory for those use cases, and NVIDIA directs proxy deployments to its proxy-environment steps. A fully air-gapped cluster without a proxy needs both a local image registry and a local package source, unless precompiled driver containers are used, which NVIDIA states removes the package repository requirement.

The telemetry decision also has a documented alternative. vLLM states the collected data is transparent and contains no sensitive information, and that a cleaned subset will be publicly released. An organization whose policy permits anonymous hardware and configuration telemetry may accept the default and record that acceptance instead of opting out. An organization whose policy forbids any egress from the pilot environment should opt out and verify. Both are defensible; only an unexamined default is not.

The isolation recommendation changes with hardware. If the pilot's GPUs appear in the MIG User Guide's supported hardware, MIG is a documented option with dedicated compute and memory per instance. If they do not, the team needs a different documented mode, and the audit record should say which. Finally, an OpenShift deployment in an air-gapped environment follows NVIDIA's separate disconnected-environment procedure rather than the general one, which changes who does the work and what evidence looks like.

Open questions the audit carries forward

The pages cited here cover telemetry, cache placement, install dependencies and GPU partitioning. Three questions a buyer will also want answered fall outside that scope: what the serving layer logs per request, how it behaves under overload, and whether the pilot can disturb adjacent clinical, research or plant systems. The audit carries these as open questions. Answering them requires the serving stack's own documentation and the owners of the adjacent systems, and the approval record should name who owns each question.

For governance framing, NIST describes its AI Risk Management Framework as intended for voluntary use. The NIST page excerpt cited here provides context on the framework's existence and voluntary nature rather than technical controls, so it cannot substitute for the engineering checks above.

The honest position for the approving officer is this: the defaults audit closes specific, documented gaps in telemetry, cache placement, install dependencies and GPU isolation. It does not close the logging, overload and interference questions. Approve the pilot with the audit complete and the open questions assigned, or defer it until they are answered. Do not approve it on defaults alone.

Default behaviors and the pre-start check for each
ComponentDocumented defaultPre-start checkSources
vLLMCollects anonymous usage data by defaultSet VLLM_NO_USAGE_STATS or DO_NOT_TRACK, or create ~/.config/vllm/do_not_track; verify usage_stats.json unchanged after a run[3]
Hugging Face HubEnvironment variables read at import time; HF_HOME defaults to ~/.cache/huggingface, HF_HUB_CACHE to $HF_HOME/hubSet HF_HOME and HF_HUB_CACHE before huggingface_hub is imported; confirm model loads from the intended path[2]
GPU Operator imagesContainer images are pulled during installationHost all images in a local image registry accessible to all nodes[1]
GPU Operator driverDriver container downloads OS packages before driver installationUse precompiled driver containers or a local package repository; confirm the driver tag ends with the worker node OS name[1]
GPU isolationMIG partitions supported NVIDIA GPUs into isolated instances with dedicated compute and memoryDocument the chosen isolation mode and confirm the hardware is supported per the MIG User Guide[4]

Checklist

  • VLLM_NO_USAGE_STATS or DO_NOT_TRACK is set, or ~/.config/vllm/do_not_track exists, before first launch.
  • HF_HOME and HF_HUB_CACHE are set in the service definition, before huggingface_hub is imported.
  • The driver image tag ends with the worker node OS name, for example 595.91.07-ubuntu22.04.
  • All images are hosted in a local registry reachable by every node, or the cluster's documented internet use case is recorded.
  • The GPU isolation mode is written down, with hardware support confirmed against the MIG User Guide.
  • After a run with egress blocked, ~/.config/vllm/usage_stats.json is unchanged and the model loaded from the local cache path.

Questions

How much time does the defaults audit add before a pilot can start?

The cited pages describe configuration steps, not timelines. The work items are pulling, tagging and pushing images to a local registry, preparing precompiled driver containers or a package repository, setting environment variables, and one verification run with egress blocked. Estimate from your own environment.

Does disabling vLLM usage stats change how vLLM runs?

The vLLM documentation documents VLLM_NO_USAGE_STATS, DO_NOT_TRACK and the do_not_track file as the ways to disable collection, and describes what is collected. Treat any effect on serving behavior as an open question and confirm it in your own pilot run rather than assuming either way.

Which GPUs can use MIG?

The MIG User Guide says it covers supported hardware, but the portion cited here does not enumerate the GPUs. Check the guide's supported-hardware section before naming MIG as the isolation mode.

Is the vLLM telemetry actually sensitive?

vLLM states the data is transparent and contains no sensitive information. The example record lists hardware and configuration fields such as GPU type, model architecture and memory settings. The full list for the latest version is in vllm/usage/usage_lib.py; review that list against your policy.

Can we run behind a proxy instead of fully air-gapped?

Yes. NVIDIA documents proxy use cases with full and limited internet access. Public name resolution for the image registry and package repositories is mandatory for those use cases, and NVIDIA directs proxy deployments to its proxy-environment steps.

What if we want to skip building a local package repository?

NVIDIA states that precompiled driver containers remove the need for driver containers to download operating system packages and remove the need to create a local package repository. That is the documented alternative.

Sources

  1. Install NVIDIA GPU Operator in Air-Gapped Environments
  2. Environment variables - Hugging Face Hub documentation
  3. Usage Stats Collection - vLLM documentation
  4. NVIDIA Multi-Instance GPU User Guide
  5. AI Risk Management Framework | NIST