Local LLM
I run open-weight models on my own hardware and publish what I learn: inference tooling, kernel and speculative-decoding experiments, reproducible benchmarks, and the cooling hardware that keeps long runs from throttling.
The homelab
Inference cluster
4× NVIDIA DGX Spark
Always-on GB10 cluster for tensor-parallel serving, multi-node experiments, and 24/7 local inference.
- Unified memory512 GB (4 × 128 GB)
- InterconnectConnectX-7 200 GbE
- AccessDocker, Tailscale
Workstation
4× RTX PRO 6000 Blackwell
Threadripper PRO host for FP4 serving, kernel work, and benchmarking from one to four GPUs.
- VRAM384 GB (4 × 96 GB)
- CPUAMD Threadripper PRO
- System RAM256 GB
Cooling
Custom fan shrouds
3D-printed shrouds with an RP2040 closed-loop fan controller, so long runs don't throttle.
- Peak dropUp to 18.6 °C
- ControlRP2040 + host app
- Fits6 GB10 chassis
Software: vLLM, SGLang, llama.cpp, NVFP4, Speculative decoding, Triton, CUDA graphs, Docker, Tailscale.
Open-source work
- SparkDeck Local-first control plane for GB10 clusters: deploy, monitor, and benchmark vLLM, SGLang, and llama.cpp across every node. 300+merged PRs
- Qwen3.8 Flash-Next FP4 · 1× RTX PRO 6000 Single-GPU inference work: pruned and dynamic MTP draft vocabulary, depth-4 speculation, and exact CUDA-graph sizes. +11.5%C1 coding tok/s
- DGX Spark · Qwen3.8-27B NVFP4 Fix and replication guide for tool calling on NVIDIA's vLLM 26.07 image (xgrammar mismatch), with MTP speculative decoding. ~79tok/s on one Spark
- DGX Spark Fan Shroud Open hardware: STL/STEP files, schematics, RP2040 firmware, and a host app for temperature-driven fan control on GB10 systems. −18 °CGPU peak
- Heretic DGX Distributed execution layer for p-e-w/heretic: runs directional-ablation optimization across two or four DGX Spark nodes. TP4multi-node
- GLM5.3 Flash FP4 · 2× RTX PRO 6000 Tensor-parallel FP4 serving experiments for GLM5.3 Flash across two Blackwell workstation GPUs. WIPin progress
- OpenCodeReview Manager Web control plane and MCP server for Alibaba's OpenCodeReview: queue, run, and triage reviews against local or hosted models. MCPagent-ready
How I work
- Measure against the original. Every optimization is benchmarked against the unmodified upstream build with interleaved launches, greedy decoding, and the raw results committed alongside the code.
- Keep outputs exact. Speed-ups come from the serving path (draft heads, kernels, graph capture, scheduling), not from changing what the target model produces.
- Make it reproducible. Each repo ships pinned images, launch flags, and scripts, so another GB10 or Blackwell owner can get the same numbers.