Back to skills
twinspark
by raullenchai
CodexโNot yet rated
Your rating
Brief
Production-grade recipe for DeepSeek-V4-Flash-0731 (284B MoE) on 2x NVIDIA DGX Spark: self-healing 2-node vLLM cluster, reboot-verified, tuned DSpark speculative decoding (~75 tok/s), full benchmarks, OpenAI Codex CLI integration
Description
# twinspark โ DeepSeek-V4-Flash-0731 (284B MoE) on 2ร NVIDIA DGX Spark
*Production-grade recipe: self-healing 2-node vLLM cluster, reboot-verified, tuned DSpark speculative decoding (~75 tok/s), full benchmarks, and OpenAI Codex CLI integration. Snapshot as of 2026-08-11 โ versions are pinned; expect drift.*
## TL;DR
- **Model**: `deepseek-ai/DeepSeek-V4-Flash-0731` โ 284B MoE (13B active), native FP4 experts + FP8 attention, 1M context, MIT license. 167 GB of safetensors.
- **Hardware**: 2ร DGX Spark (GB10, 121 GiB unified memory each, 4 TB NVMe), linked by a single 400G QSFP DAC cable โ shows up as **2ร 200GbE RoCE links**.
- **Engine**: vLLM 0.25.2 (community image `ghcr.io/anemll/dspark-vllm-gx10:0.1.1` with GB10/sm_121 kernels: FlashInfer-B12X MoE, native FP8 DS-MLA KV cache, DSpark speculative decoding) + Ray for 2-node TP=2.
- **Results**: **~75 tok/s single-stream decode** (DSpark speculative decoding = 2.74ร over the 27 tok/s bandwidth floor), ~1.7k tok/s prefill, 1M context enabled, KV capacity 2.68M tokens. Full benchmark tables below.
- **Daily driver**: OpenAI Codex CLI pointed at the local endpoint (`wire_api = "responses"`) โ agentic coding evaluated on 3 greenfield projects **plus one real feature landed in a 1,831-file codebase** (results below).
---
## Quick start
Assumes both Sparks are through DGX OS first-boot, the QSFP DAC is plugged (before power-on!), and you can SSH between them.
```bash
# 0. on BOTH nodes: RDMA prerequisites (memlock etc.) โ see ยง2 gotchas #1-2
# 1. on BOTH nodes: install the netplan file (swap IPs on node 2)
sudo cp netplan/99-qsfp.yaml /etc/netplan/ && sudo chmod 600 /etc/netplan/99-qsfp.yaml && sudo netplan apply
# 2. verify the fabric BEFORE touching any model (do not skip):
scripts/preflight.sh # links at 200000Mb/s, MTU 9000, RoCE ACTIVE
# then nccl-tests all_reduce_perf across nodes โ expect ~20 GB/s bus BW (ยง1)
# 3. get the weights on node 1, rsync to node 2 over the QSFP link
hf download deepseek-ai/DeepSeek-V4-Flash-0731 --local-dir ~/models/DeepSeek-V4-Flash-0731
# 4. build the serving image on BOTH nodes (adds Ray to the anemll image):
docker build -t ds0731-vllm:ray - <<'DF'
FROM ghcr.io/anemll/dspark-vllm-gx10:0.1.1
RUN pip install --no-cache-dir "ray[default]"
DF
# 5. edit scripts/twinspark.env (your IPs/usernames), then from node 1:
scripts/cluster-start.sh # ~8 min to serving
scripts/status.sh
curl -s localhost:8000/v1/models | jq -r '.data[0].id' # -> ds-0731
```
## 1. Hardware & Topology
```
โโโโโโโโโโโ LAN switch (management, 2.5GbE) โโโโโโโโโโโ
โ โ
spark1 (192.168.1.10) spark2 (192.168.1.11)
โ ConnectX-7 โโโ 400G QSFP DAC (0.5 m) โโโ ConnectX-7
โ 10.0.0.1 / 10.0.1.1 10.0.0.2 / 10.0.1.2
โโโโโโโ RoCE v2, MTU 9000, NCCL over 2 rails โโโโโโ
```
Each DGX Spark: GB10 (Grace 20-core ARM + Blackwell GPU, sm_121), 128 GB unified LPDDR5x (121 GiB usable), ~273 GB/s memory bandwidth, 4 TB NVMe, DGX OS 7.2.3 (Ubuntu 24.04 arm64), CUDA 13.0, driver 580.173.02.
### How two Sparks serve one model (TP=2, not load balancing)
A common misconception: "requests get load-balanced between the two Sparks." No โ **both machines compute every single token together**. The model is 167 GB of weights; one Spark has 121 GiB. It physically cannot fit on one node, so vLLM shards it with tensor parallelism (TP=2): half the attention heads and half the MoE experts live on each machine.
```
your clients (Codex agents, curl, ...)
โ all traffic โ node 1, port 8000 โ the ONLY endpoint
โผ
spark1: vLLM API server + scheduler
โ continuous batching: concurrent requests merge into one batch
โผ
every forward pass:
spark1 GPU โโโ each computes its half of every layer
โโโ NCCL allreduce per layer (over the 2ร 200G rails)
spark2 GPU โโโ
โผ
each request receives its own tokens
```
Consequences worth internalizing:
- **spark2 is invisible to clients.** There is nothing to route, balance, or failover between. One endpoint, period.
- **Both GPUs sit at ~95% utilization even with a single request** โ that's not waste, that's each node computing its half.
- **The QSFP link is on the critical path of every token.** Each transformer layer does an allreduce across the cable, which is why fabric quality (ยง1) matters more than any vLLM flag: if NCCL silently falls back to TCP, decode collapses.
- **Multiple agents are "balanced" in the batch dimension, not across nodes.** The scheduler on node 1 merges concurrent requests into one batch; each forward pass advances all of them by one token. Aggregate throughput goes *up* with concurrency (weights are read once per pass regardless of batch size โ see ยง5 addendum).
Data parallelism (one full model replica per node + a round-robin proxy in front) is the setup where "load balancing" would apply โ that only works for models that fit on a single node, which this one does not.
### Discovery #1: one cable, two NICs
The single QSFP cage is internally wired to **two independent ConnectX-7 controllers**. Plugging one 400G DAC lights up **two** 200G links (`enp1s0f0np0` + `enP2p1s0f0np0`, RDMA `rocep1s0f0` + `roceP2p1s0f0`). Each controller sits on a **PCIe Gen5 x4** link (~126 Gb/s usable), so per-rail RDMA tops out at ~109 Gb/s (`ib_write_bw`, 87% of the PCIe ceiling โ the NIC is not the bottleneck, the PCIe lane is).
Give **both** rails to NCCL:
```bash
export NCCL_IB_HCA=rocep1s0f0,roceP2p1s0f0
export NCCL_NET_GDR_LEVEL=5
export NCCL_SOCKET_IFNAME=enP7s7 # bootstrap over the management LAN
```
Measured cross-node `all_reduce_perf` (nccl-tests, NCCL 2.30.7): **21.7 GB/s bus bandwidth** peak at 1 GB messages, 17.8 GB/s average. If you see ~2 GB/s, NCCL silently fell back to TCP โ fix your RDMA before loading any model.
### Discovery #2: the CX-7 is power-gated
With no cable plugged, DGX OS powers the CX-7 down and removes it from the PCI bus entirely (dmesg shows `Detected insufficient power on the PCIe slot (27W)` then the devices vanish). Don't panic when `lspci` shows nothing โ **plug the cable before boot** (or `echo 1 > /sys/bus/pci/rescan`).
### Netplan (`/etc/netplan/99-qsfp.yaml`, mirrored with .1/.2 swapped)
```yaml
network:
version: 2
ethernets:
enp1s0f0np0: { addresses: [10.0.0.1/24], mtu: 9000, dhcp4: false, dhcp6: false, optional: true }
enP2p1s0f0np0: { addresses: [10.0.1.1/24], mtu: 9000, dhcp4: false, dhcp6: false, optional: true }
```
MTU 9000 verified end-to-end (`ping -M do -s 8972`), RoCE `active_mtu` goes 1024 โ 4096.
---
## 2. Gotchas that cost us real time (read this before you start)
| # | Symptom | Root cause | Fix |
|---|---------|-----------|-----|
| 1 | NCCL `unhandled system error`, `ncclIbRegMrDmaBufInternal -> 2` | default `memlock` ulimit is **8 MB**; RDMA can't register memory | `* soft/hard memlock unlimited` in `/etc/security/limits.d/99-rdma.conf` + `DefaultLimitMEMLOCK=infinity` for systemd + `--ulimit memlock=-1` on every container |
| 2 | limits.d fix "doesn't work" | **Tailscale SSH bypasses pam_limits** โ sessions through `tailscale up --ssh` keep the old 8 MB | test via OpenSSH (`ssh localhost`), not Tailscale SSH |
| 3 | Ray worker dies during model load, `killed due to memory pressure (OOM)` while 117 GiB is free | DGX Spark has **unified memory** โ vLLM's "GPU" allocation counts as host RAM; Ray's memory monitor kills the worker | `RAY_memory_monitor_refresh_ms=0` on **both** nodes |
| 4 | Model loads, `/v1/completions` is perfect, but **chat output is nondeterministic token salad with random Chinese** | DSpark speculative decoding: rejected draft tokens leave dirty KV context; plus vLLM auto-enables breakable CUDA graphs | `VLLM_DSPARK_GPU_REJECTED_CONTEXT_MASK=1` and `VLLM_USE_BREAKABLE_CUDAGRAPH=0` |
| 5 | `--speculative-config` JSON mangled (`Value {method:dspark} cannot be converted`) | quoting through nested `bash -lc "..."` layers | never inline the launch command โ ship a **script file** into the container and exec it |
| 6 | vLLM never starts, log file absent | `pkill -f 'vllm serve'` matches the wrapper shell's own command line and kills itself | `pkill -f 'bin/vllm'` |
| 7 | Official `vllm/vllm-openai:cu130-nightly` can't load the model | it's pinned to vLLM 0.19.x โ predates `DeepseekV4ForCausalLM` | community image `ghcr.io/anemll/dspark-vllm-gx10:0.1.1` (vLLM 0.25.2 + sm_121 kernels) |
| 8 | vLLM stable arm64 images crash on GB10 | bundled torch compiled through sm_120 only; GB10 is **sm_121** | same as #7 |
| 9 | Codex CLI: `wire_api = "chat"` rejected | removed in codex 0.147 | `wire_api = "responses"` โ vLLM 0.25 ships `/v1/responses` |
| 10 | Codex warns `Model metadata for ds-0731 not found` | unknown model โ fallback metadata | undocumented `model_catalog_json` key + catalog JSON (schema below) |
| 11 | After restarting the head container, it hangs forever "waiting for 2 GPUs" | Ray head restart = **new GCS session**; the worker's old raylet never re-registers, yet `ray status` from the worker still succeeds against the new GCS โ naive watchdogs never fire | worker watchdog must verify **its own IP is in `ray.nodes()` alive list**; param-change restarts bounce worker first, then head |
| 12 | Agent sessions degrade at ~100โ140k context: the model narrates intentions in a loop ("Let me grepโฆ Let me viewโฆ") or repeats identical tool calls, while the server is perfectly healthy | **Model-side, not your rig**: DeepSeek's own tech report shows retrieval stable only to **128K**; past it, self-reinforcing in-context repetition takes over. Reproduced on official DashScope serving ([qwen-code #4695](https://github.com/QwenLM/qwen-code/issues/4695) โ 43 identical tool calls while the model outputs "Let me stop this loop") | break the loop with `/compact`; prevent it with auto-compaction. The fence depends on KV precision (ยง5 Addendum 2): with native `fp8_ds_mla` KV โ **150k** (`model_auto_compact_token_limit = 150000`); with `nvfp4_ds_mla` โ 100โ120k. 1M context is real for *reading*, not for *accumulated agent history* |
| 13 | After ~29h of stable serving, the whole engine crash-loops: `RayWorkerProc rank=[1] died unexpectedly` on the head, nothing in `docker logs`, container exits **code 0** | **Kernel OOM on the worker node.** Ray sets its own workers to `oom_score_adj=1000` โ first in line for the kernel OOM killer. If the worker node does double duty (agents, git checkouts, anything spiky), one memory spike kills the model worker, and TP=2 means the whole engine dies with it. Verify with `dmesg | grep -i oom` on the worker node | run an OOM guard on **both hosts** (systemd loop resetting `oom_score_adj=-500` for `ray::`/`raylet`/`EngineCore`/`vllm serve` โ must run on the host: containers lack `CAP_SYS_RESOURCE` to lower it). Kernel then sacrifices the cheap-to-restart processes instead of the 8-minute-reload model |
| 14 | Long-context requests **randomly** collapse from ~100k โ code-salad, XML repetition loops, or โthis is a prompt injection testโ refusals โ while identical requests sometimes pass, and short requests are always perfect | V4โs sparse attention picks top-512 compressed KV blocks per query via a small FP4 โlightning indexerโ; `nvfp4_ds_mla` KV quantization adds noise to exactly the representations that indexer scores. Wrong blocks โ the answer is literally invisible, or format-heavy blocks lock generation into loops. Stochastic and content-dependent by nature | use the **native** `--kv-cache-dtype fp8_ds_mla`: same speed (64.1 vs 63.9 tok/s), โ2% KV capacity, **4ร fewer real long-context failures** (measured โ ยง5 Addendum 2) |
---
## 3. Software stack
| Layer | Version |
|---|---|
| DGX OS | 7.2.3 (Ubuntu 24.04.4, kernel 6.17.0-1029-nvidia, aarch64) |
| Driver / CUDA | 580.173.02 / 13.0 |
| Container | `ghcr.io/anemll/dspark-vllm-gx10:0.1.1` + `pip install ray[default]` (ray 2.57.0), rebuilt as `ds0731-vllm:ray` |
| vLLM | 0.25.2.dev0+g752a3a504 (DeepseekV4ForCausalLM, DSpark, fp8_ds_mla KV (native โ see gotcha #14), flashinfer_b12x MoE) |
| NCCL | 2.30.7+cuda13.3 |
| Model | `deepseek-ai/DeepSeek-V4-Flash-0731` (48 shards, 167 GB, native FP4/FP8 โ **this is the highest precision that exists**; don't waste time on GGUF re-quants) |
| Codex CLI | 0.147.0 (aarch64-musl static binary) |
Weights are transferred to node 2 over the QSFP link (`rsync` @ ~440 MB/s, ssh-crypto-bound, 380 s) โ don't download twice.
---
## 4. Architecture: self-healing containers
Both machines run one container each with `--restart unless-stopped`. The container entrypoints implement the full recovery logic, so **a reboot of either or both machines auto-recovers to a serving state** with zero manual steps (~8 min to ready):
- **spark1 / `ds0731-head`**: `scripts/boot-head.sh` โ start Ray head โ wait until cluster has 2 GPUs โ `exec launch-vllm.sh` (vLLM as PID 1; if it crashes, the container restarts and the whole sequence re-runs).
- **spark2 / `ds0731-worker`**: `scripts/boot-worker.sh` โ join Ray head (retry loop) โ watchdog: if head disappears, exit โ docker restarts the container โ rejoin fresh.
- All launch parameters live in **one file on the host** (`launch/launch-vllm.sh`, bind-mounted). Change params โ `restart-vllm.sh` (= `docker restart ds0731-head`; worker follows automatically).
### Final launch configuration (`launch/launch-vllm.sh`)
```bash
vllm serve /models/DeepSeek-V4-Flash-0731 \
--served-model-name ds-0731 \
--host 0.0.0.0 --port 8000 \
--tensor-parallel-size 2 \
--distributed-executor-backend ray \
--trust-remote-code \
--tokenizer-mode deepseek_v4 \
--tool-call-parser deepseek_v4 \
--reasoning-parser deepseek_v4 \
--enable-auto-tool-choice \
--kv-cache-dtype fp8_ds_mla \
--moe-backend flashinfer_b12x \
--block-size 256 \
--max-model-len 1048576 \
--gpu-memory-utilization 0.85 \
--max-num-seqs 6 \
--max-num-batched-tokens 8192 \
--max-cudagraph-capture-size 24 \
--generation-config vllm \
--async-scheduling \
--enable-chunked-prefill \
--enable-prefix-caching \
--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'
```
### Container environment (both nodes)
```
NCCL_IB_HCA=rocep1s0f0,roceP2p1s0f0 NCCL_NET_GDR_LEVEL=5
NCCL_SOCKET_IFNAME=enP7s7 GLOO_SOCKET_IFNAME=enP7s7
RAY_memory_monitor_refresh_ms=0 RAY_DISABLE_MEMORY_MONITOR=1
VLLM_USE_FLASHINFER_SAMPLER=1 VLLM_USE_B12X_MOE=1
VLLM_USE_B12X_WO_PROJECTION=1 VLLM_DSPARK_GPU_REJECTED_CONTEXT_MASK=1
VLLM_USE_BREAKABLE_CUDAGRAPH=0 DEFAULT_THINKING=low
HF_HUB_OFFLINE=1
docker: --network host --ipc=host --ulimit memlock=-1 --ulimit stack=67108864
--cap-add=IPC_LOCK --device=/dev/infiniband --gpus all
```
Memory budget after load: model 2ร 79.2 GiB, KV cache 20.5 + 18.4 GiB โ **2,682,310 tokens of KV capacity** (fp8_ds_mla โ the native format. We initially ran nvfp4_ds_mla: it buys only ~2% more capacity and quadruples long-context failures, see gotcha #14 / ยง5 Addendum 2).
---
## 5. Speculative decoding tuning
DSpark = DeepSeek's built-in MTP draft module (the extra ~20B params in the 304B checkpoint). Acceptance rate measured ~75% over 5 draft positions (pos0 93%, declining to ~60% at pos4).
| config | single-stream tok/s (mean of 3) | 4-way concurrent agg | 7k-prompt TTFT | prefill tok/s |
|---|---|---|---|---|
| spec decode OFF | 27.3 (27.3/27.3/27.3 โ the pure bandwidth floor) | 71.3 | 4.08 s | 1745 |
| num_speculative_tokens=3 | 64.0 | **100.2** | 4.12 s | 1728 |
| num_speculative_tokens=5 (recipe default) | 70.7 | 64.1 | 4.7 s | 1517 |
| **num_speculative_tokens=7 โ winner** | **74.8** (74.4โ75.2) | 82.6 | 4.26 s | 1674 |
Takeaways:
- **DSpark = 2.74ร single-stream speedup** (27.3 โ 74.8 tok/s). The no-spec floor of 27.3 tok/s is exactly what the memory-bandwidth math predicts for 13B active @ ~4.4 bit over 2ร 273 GB/s with per-layer all-reduce over the QSFP link.
- Acceptance stays high enough (~75% over 5 positions) that deeper drafts keep paying: 7 beats 5 beats 3 for a single stream.
- **Speculative decoding inverts at concurrency**: at 4 concurrent streams, spec3 wins (100 tok/s agg) and spec5/7 lose to plain decoding's batching. Tune for your workload โ we serve 1โ2 Codex streams, so 7 it is.
### Addendum: multi-agent production tuning (measured after a day of real use)
Running 2+ Codex agents against the endpoint revealed two things the clean-prompt numbers hide:
1. **Agentic traffic is prefill-dominated.** Every tool round re-submits a large context; we observed ~8k tok/s of prompt processing sustained while decode sat at ~51 tok/s aggregate. Prefix caching absorbs most of the cost, but prefill still competes with decode under chunked prefill.
2. **DSpark acceptance drops from ~75% to ~40% on agent contexts** โ draft models predict clean prose/code well, but not tool output (file listings, test logs). Deep drafts waste compute exactly when you're busiest.
So we ship two launch profiles (`launch/`):
| profile | config | single-stream | 4-way agg | prefill | 7k TTFT |
|---|---|---|---|---|---|
| `launch-vllm.sh` (default) | spec7, seqs 6 | **74.8 tok/s** | 82.6 | 1674 | 4.26 s |
| `launch-vllm-multiagent.sh` | spec3, seqs 8 | 66.3 | **105.7** | **1953** | **3.65 s** |
Swap = copy over `launch-vllm.sh` + run `scripts/restart-vllm.sh` (~8 min).
**Winner: `num_speculative_tokens=7`** โ baked into `launch/launch-vllm.sh`.
---
### Addendum 2: the long-context minefield, measured (and why KV precision is the fuse)
We probed needle-retrieval reliability at growing context sizes: non-guessable codes planted at multiple depths in a synthetic log, 3-needle exact-match queries, temperature 0, 5 trials per size. โReal failureโ = wrong/missing/refused answer; โjunkโ = correct answer with trailing garbage.
| context | nvfp4_ds_mla KV | fp8_ds_mla KV (native) |
|---|---|---|
| 74k | 0/5 | 0/5 |
| 92k | 0/5 | โ |
| 110k | 1/5 | 0/5 (2 junk) |
| 129k | 2/5 | 0/5 (3 junk) |
| 148k | 1/5 | 0/5 (1 junk) |
| 166k | 2/5 | 2/5 |
| 184k | 3/5 | 0/5 (1 junk) |
| **real failures total** | **8/25** | **2/25** |
What this says:
- **There is no hard โ128K cliffโ โ itโs a stochastic minefield.** The same request can pass or collapse on different runs. That matches the architecture: the sparse-attention โlightning indexerโ (FP4, top-512 block selection) is a small network whose scoring gets noisy as the haystack grows; near-tie scores flip block selection run to run.
- **4-bit KV quantization is an amplifier, not a bystander.** Switching to the native `fp8_ds_mla` cut real failures 4ร at identical speed and โ2% capacity. The failure pocket at ~166k persists on both โ that one looks model-side.
- **Real-agent burn-in (fp8 KV):** a Claude Code session stayed fully coherent through **163k of accumulated tool-call history**, then survived its own compaction cycle (~184k โ 141k) and kept working โ zero loops. On nvfp4 KV we saw agent loops from ~100k in daily use.
Practical fences: with `fp8_ds_mla` โ auto-compact at ~150k (codex `model_auto_compact_token_limit = 150000`; Claude Code `CLAUDE_CODE_MAX_CONTEXT_TOKENS=160000` compacts โ148k). With `nvfp4_ds_mla` โ stay at 100โ120k. Either way, fence *below* the ~166k pocket.
Claude Code works against this stack directly, by the way โ vLLM 0.25 serves an Anthropic-compatible `/v1/messages`, so `ANTHROPIC_BASE_URL=http://<node1>:8000` + `ANTHROPIC_MODEL=ds-0731` is all it takes.
## 6. Full benchmark (`vllm bench serve`, random dataset)
| scenario | conc | output tok/s | total tok/s | TTFT p50 / p99 | TPOT p50 / p99 |
|---|---|---|---|---|---|
| 1k in / 1k out | 1 | 33.8 | 67.5 | 683 / 706 ms | 29.6 / 38.4 ms |
| 1k / 1k | 2 | 51.6 | 103.3 | 466 / 829 ms | 31.4 / 55.6 ms |
| 1k / 1k | 4 | 56.9 | 113.8 | 610 / 937 ms | 69.4 / 100.2 ms |
| 1k / 1k | 6 | 72.0 | 144.0 | 716 / 1200 ms | 74.6 / 109.9 ms |
| 8k / 1k | 1 | 20.0 | 180.2 | 5.1 / 5.5 s | 50.3 / 56.4 ms |
| 8k / 1k | 4 | 39.5 | 355.7 | 2.3 / 5.9 s | 98.8 / 121.1 ms |
| **64k / 512** | 1 | 6.4 | 827.8 | **42 / 84 s** | 50.2 / 51.2 ms |
**Important caveat โ random-token inputs sandbag speculative decoding.** `vllm bench serve --dataset-name random` feeds random token soup, so the DSpark draft model can't predict continuations and acceptance collapses: 33.8 tok/s single-stream here vs **74.8 tok/s on real coding prompts** (our `measure.py`, temp 0.2). Real agentic workloads run at the higher number; the table above is the pessimistic floor. TPOT p50 of 29.6 ms @ conc 1 also implies ~34 tok/s worst-case per stream.
Long context: 64k-token prompts prefill at ~1.55k tok/s (42 s TTFT) and decode speed holds at 50 ms/token โ the sparse-MLA indexer doing its job. 1M context is configured and KV capacity (2.74 M tokens) covers it, but budget ~11 min of prefill for a full 1M prompt.
**Thermals under sustained load**: worst case across all runs โ spark1 79 ยฐC / 73 W, spark2 78 ยฐC / 69 W (64k-prefill group). Fans audible but far from throttling. Idle: ~43 ยฐC / 12 W each.
---
## 7. Codex CLI as the daily-driver client
### Config that works (codex 0.147)
`~/.codex/config.toml`:
```toml
model = "ds-0731"
model_provider = "spark"
model_catalog_json = "/home/USER/.codex/ds0731-model-catalog.json"
[model_providers.spark]
name = "DeepSeek V4 Flash 0731 (local vLLM)"
base_url = "http://localhost:8000/v1" # or the Tailscale IP from other machines
wire_api = "responses" # "chat" was removed in codex 0.147
env_key = "VLLM_API_KEY" # export VLLM_API_KEY=dummy (vLLM has no auth)
```
The catalog JSON (undocumented but shipped feature) registers context window (1,048,576), tool support, and base instructions for the unknown model โ kills the "fallback metadata" warning. Also set `model_auto_compact_token_limit = 150000` (with the native fp8_ds_mla KV cache; use 100โ120k if you run nvfp4 KV) โ see gotchas #12/#14: agent sessions loop in long contexts, and auto-compaction is the fence. Ubuntu 24.04 also needs `kernel.apparmor_restrict_unprivileged_userns=0` for codex's bubblewrap sandbox.
### Agentic evaluation: 3 medium projects, unattended
We gave `codex exec` (workspace-write sandbox, no human in the loop) three medium-difficulty specs and independently verified everything afterward โ running the test suites ourselves, hitting the live services with real traffic, and writing adversarial tests the model never saw.
| project | spec highlights | wall time | tokens | independent verification | supervision rounds |
|---|---|---|---|---|---|
| **Expense tracker CLI** (Python/sqlite3/argparse) | 5 subcommands, budgets with exit-code semantics, CSV export, pytest | 111 s (+51 s fix round) | 35k | 5/5 tests green; CLI acceptance & CSV verified by hand | **1** โ printed the over-budget warning but exited 0 where the spec required exit code 2; fixed correctly from a plain-English bug report in 51 s |
| **URL shortener API** (FastAPI/sqlite3) | base62 codes, custom aliases + 409, 307 redirects, click stats, admin-token delete, per-IP sliding-window rate limit, pytest | 564 s | 62k | 6/6 tests green; live uvicorn session: shorten/redirect/stats/409/422/404/401 + rate-limit 429 all correct | **0** |
| **Rate limiter library** (pure stdlib, thread-safe) | TokenBucket + SlidingWindowLog + decorator, concurrency no-over-grant test | 104 s | 15k | 10/10 tests ร3 runs; 6 adversarial checks (burst cap, 8-thread over-grant, blocking-timeout semantics, key isolation, expiry, decorator raise) all pass | **0** |
**Verdict**: 2 of 3 projects were correct zero-shot end-to-end; the third needed exactly one bug-report iteration. ~13 minutes of total agent wall time for three working, tested projects. Subjectively: it reads specs carefully, iterates until tests pass, writes clean idiomatic Python, and its self-reports were accurate (everything it claimed passed did pass โ it just missed one exit-code requirement). At 75 tok/s the interactive feel is comparable to a fast cloud API. This is a genuinely usable local daily-driver for agentic coding.
### The hard test: a real feature in a 1,831-file codebase
Small greenfield projects don't separate frontier models from strong mid-tier ones โ big codebases do. So we pointed the same unattended `codex exec` at a fresh clone of [Rapid-MLX](https://github.com/raullenchai/Rapid-MLX) (1,831 files, a **17,392-test** suite, green baseline) and asked for a real roadmap feature (issues #1335 + #319): **multi-model serving behind one endpoint with lazy load, single-flight, idle unload, and LRU eviction** โ an architecture-level concurrency feature touching CLI, config, server assembly, and four route families.
**Round 1 (unattended, 27 min, 278k tokens, 152 tool calls):** it shipped a genuinely well-designed implementation โ a self-contained 492-line pool module with an injectable clock for testability, per-model `asyncio.Future` single-flight, and even the subtle `asyncio.wait_for(asyncio.shield(future))` idiom so one waiter's timeout can't cancel a shared load (a detail human reviewers regularly get wrong). 27 new tests, all green; `DESIGN.md` included; honest disclosure of what it couldn't verify. **But**: it renamed a module-level seam (`routes.chat.get_engine`) that 200 existing tests monkeypatch โ tests it *couldn't run* because its sandbox has no Metal โ and shipped the regression: **200 failed / 16,707 passed**.
**Round 2 (we played CI, 16 min, 207k tokens):** given only the failing-test list and one sample traceback โ no root-cause hint, still no ability to run the affected tests itself โ it diagnosed the seam breakage from source, restored the interface while keeping the feature intact, and left the codebase at **16,900+ tests fully green** with its 27 new tests still passing.
**Long-horizon verdict:** one supervision round to land a real concurrency feature in a large codebase. The failure mode was interesting and very human: not hallucination, not spaghetti โ it broke a contract it had no way to see, and disclosed the blind spot in its own report. Our honest placement after both evals: **frontier-competitive on well-specified, self-contained work; roughly one supervision round behind top frontier models on large-codebase integration work** โ which, for 13 B active parameters running on two desk-side boxes, is remarkable.
---
## 8. Ops runbook
```bash
scripts/status.sh # one-glance health: containers, vLLM, GPUs, QSFP
scripts/restart-vllm.sh # apply new params from launch-vllm.sh (~8 min)
scripts/cluster-start.sh # (re)create both containers from scratch
scripts/cluster-stop.sh # stop everything
tail -f ~/twinspark/logs/vllm.log
curl -s localhost:8000/metrics | grep spec_decode # DSpark acceptance telemetry
```
Reboot behavior: power-cycle either or both machines โ docker restarts containers โ Ray re-forms โ vLLM re-serves. **Verified by actually rebooting both nodes simultaneously**: SSH back in 40 s, Ray cluster re-formed ~2.5 min after boot, vLLM serving again ~9 min after power-on โ zero manual steps.
---
## Repo layout
```
scripts/ cluster lifecycle: twinspark.env (edit me) ยท cluster-start/stop ยท boot-head/worker ยท restart-vllm ยท status ยท preflight
launch/ launch-vllm.sh โ the single source of truth for all vLLM parameters
bench/ measure.py (real-prompt perf) ยท bench.sh (vllm bench serve suite) ยท results/*.json (raw data)
codex/ Codex CLI config + model-catalog examples
netplan/ QSFP link template
```
*Setup, debugging, tuning and this write-up were done by Claude (Anthropic) driving the cluster end-to-end. Hardware: 2ร DGX Spark. Time from first boot to serving: about one evening, including hitting every gotcha above. Pinned to the versions in ยง3 โ PRs updating for newer stacks welcome, but this repo is a snapshot, not a maintained product.*
Installation
Install the skill from https://github.com/raullenchai/twinsparkThis skill is not published on the ClawHub registry. Paste the prompt into your coding agent โ Claude Code, Cursor, Codex โ and it will install the skill straight from the source repository.
Open the repository