inference-host

|

What it does

Inference host bring-up + tuning suite — Linux/GPU bare-metal host work that sits beneath any inference framework (vLLM, sglang, TensorRT-LLM). Two phases. Bring-up (nvidia-datacenter-bringup): Ubuntu 24.04 LTS from clean OS to gpu-operator cuda-validator passing — B300/B200/H100/A100/L40S/L4 driver + fabricmanager + NVLSM + DOCA-OFED install order, NVIDIA CUDA repo + DOCA repo + air-gap mirror, MOK + DKMS sign-on-build under Secure Boot, Dell PowerEdge XE9780/XE9785 baseboard firmware via iDRAC Redfish DellOemChassis.ExtendedReset, gpu-operator pre-installed-driver-mode integration. Tuning (gpu-host-tuning): read-only snapshot of CPU power state, C-states, NUMA topology, PCIe link state, GPU settings, kernel boot params, sysctl, ulimits, IRQ affinity, container runtime; optional pinned-host↔GPU memcpy bench (torch + numactl); per-lever cheat-sheets to flip settings (governor, EPP, cpuidle, persistence, ECC, hugepages, intel_iommu, NCCL env, tuned-adm profiles, BIOS guidance for Dell XE / SMC / HPE). Surfaces config gaps that bottleneck LMCache CPU-tier throughput, KV offload, NCCL bandwidth, prefix-cache rebuild.