Two GPUs, Four Crashes, and 384 Benchmark Runs

It started with a one-line question: how viable is this model for fully local dev workflows on my current machine?

The model was Muse-Glimmer-30B, a 30-billion-parameter model with strong published SWE-bench numbers. The machine was my home server, the same box whose inference queue I wrote about earlier: an Intel i7-8700K from 2017, 32GB of DDR4, and two NVIDIA GPUs the industry has written off, a Tesla P40 with 24GB of VRAM and a GTX 1070 with 8GB. Both are Pascal, compute capability 6.1. No tensor cores. FP16 runs at 1/64 rate, so the default numeric format of modern inference is useless. CUDA 13 dropped the architecture entirely. The one redeeming instruction is dp4a, an INT8 dot product. It’s the P40’s entire compute advantage, and it’s possible to build your inference stack in a way that silently turns it off.

The honest answer to the question would have taken one afternoon: yes, a 15GB Q4 quant fits in 24GB and generates about 14 tokens per second. But the question behind the question was bigger. Should this model be the one? What would it take to make this machine genuinely useful for local AI work instead of a curiosity? Answering that took benchmarks. Building the benchmarks took infrastructure. And running the infrastructure took the machine down. Repeatedly.

What this is (in one breath)

A silent-success check is a check whose failure mode is indistinguishable from its success mode—a build that succeeds without its fast path, a test that passes because it tested nothing, a write that “works” by writing the wrong thing. This post is a field guide to hunting them, disguised as the story of turning a leftover server into a local-LLM box that earns its keep: four models behind one authenticated endpoint, surviving reboots unattended. Getting there was three problems braided together—making the machine stable, making Pascal fast, and building benchmarks honest enough to decide which models deserved the VRAM. Silent-success checks turned up in all three. If you self-host models on aging hardware, or you’re building any kind of eval harness, most of what follows transfers.

The machine fights back

The first benchmark runs died along with everything else—SSH, the session driving the work, the machine itself. Hard lockup, no response on any interface. Then it happened again: SSH healthy at 16:09, last journal entry at 16:11, unresponsive by 16:16. No kernel oops, no thermal event, no out-of-memory kill. The journal simply stopped, because journald syncs periodically and a hard lockup eats the last five minutes of whatever you were about to learn.

The only recovery was the power button, which is the worst possible answer three times over: it skips filesystem sync, it destroys the evidence, and it teaches you nothing. After the second time I walked over and held that button, I stopped benchmarking entirely. Before anything else, this needed to be a machine I could diagnose.

I put in three pieces of boring infrastructure. The hardware watchdog ended the power-button era by itself: if systemd stops petting /dev/watchdog for two minutes, the chipset resets the box on its own. The other two earned their keep later:

Netconsole to a Raspberry Pi. Kernel messages stream over UDP to a Pi on the same LAN, which appends them to a file. When the server dies too hard to write its own disk, the Pi still has its last words.

A crash-resumable work queue. The original sin was driving the benchmarks from a session running on the machine being benchmarked: when it died, it took every in-flight job with it. The replacement is a dumb, durable pattern—a systemd service reading shell commands from a JSONL queue, executing them one at a time, recording per-item state on disk. Attempt counters increment before execution, so a job that locks the machine up still burns an attempt and gets skipped after three, rather than crash-looping the box forever.

The culprit was the RAM the whole time

With netconsole in place, crash four finally produced a body to autopsy. The Pi’s log captured what the server’s own journal never could:

llama-bench: Corrupted page table at address 618c34358170
Oops: Bad pagetable: 000d [#1] SMP PTI

Error code 0x0d decodes to a reserved-bit violation: a page-table entry had bits set that the CPU requires to be zero. No driver writes reserved bits into a PTE. That’s not a software bug. That’s a memory cell flipping.

Six minutes earlier, the clincher: an unrelated Node.js process that never touches the GPU had died with trap invalid opcode in libnode.so. Node executing garbage instructions exonerated the NVIDIA driver, the inference stack, the PSU, and every other theory I’d been circling. Two different processes, no shared code path, both reading corrupted memory.

The cause was almost embarrassing. The DDR4 was running its XMP profile at 3200 MT/s on a platform whose JEDEC-rated maximum is 2666—a 20% factory overclock, enabled with one BIOS toggle years ago, silently corrupting memory under sustained load. Non-ECC RAM, so nothing ever complained. The machine just occasionally died, and the deaths looked like anything and everything.

One BIOS trip later (XMP → Auto), the crashes stopped. Measured cost of the fix: about 0.1% of throughput. The machine had been trading its stability for a rounding error.

The lesson generalizes: sustained AI inference is the best memory stress test most consumer hardware will ever run. If a machine that “never crashes” starts crashing when you give it LLM work, check XMP before you blame the GPU stack.

Making Pascal fast

The compiler that ate two-thirds of the performance

llama.cpp’s Vulkan backend compiles its compute shaders with glslc at build time. Ubuntu 24.04 ships a glslc too old to compile the integer-dot-product shaders—the ones that use dp4a, the single instruction that makes Pascal worth anything. The build does not fail. It succeeds, silently, without them.

The difference, measured on the same model and card:

A 2.9× performance loss plus a crash, from a compiler version, with zero warnings. The fix was extracting a current glslc from the LunarG Vulkan SDK and pointing cmake at it. The rule that survived: read the device line before trusting a benchmark. int dot: 1, or the numbers are garbage.

The build succeeded either way. Only the numbers changed.

“CUDA is impossible” lasted eighteen hours

With CUDA 13 refusing to target compute_61, I wrote down, with some confidence, that Vulkan was the only backend this hardware would ever run.

The conclusion was wrong in an instructive way: it was true of the CUDA installed on the machine, not of CUDA. NVIDIA still publishes CUDA 12.6, and 12.6 targets sm_61 just fine. Installing the older toolkit alongside the existing driver—without touching the driver—and rebuilding llama.cpp produced a CUDA backend that beat Vulkan by roughly 2.7× on prefill while matching it on decode. On a box destined to serve coding agents, where every request front-loads a large prompt, prefill is exactly the number that matters.

“Impossible” claims about software support deserve a version qualifier. What’s impossible in the current release is often routine in the previous one.

Learning to measure quality (by measuring it wrong)

My first quality benchmark graded tool-use tasks by string-matching the final answer. One task’s grader checked that the answer contained "78771". Every model computed the right number. Several answered "78,771", because that’s how literate English formats a number that size. String match: fail. Six models, scored wrong, by a comma.

Small bug, big moral: grade state, not prose. The rewrite moved to world-state scoring—tasks run against a simulated world (a small filesystem, a ticket tracker, an outbox), and the grader asserts on what the world looks like afterward. Did the file get written? Was the ticket closed with the right severity? Prose never enters into it.

I baked in two more rules, both earned through false starts. Golden and null gates: before any model sees a task, replay the known-good solution (must score 1.0) and score doing nothing at all (must score ~0). The null gate caught three tasks that awarded free credit for inaction. A blind judge: a second scoring layer rates how the goal was reached (planning, recovery, efficiency) on transcripts anonymized with a per-task shuffle, never seeing the execution score.

The tool-use round produced one finding that shaped everything after it. Qwen3.6-35B-A3B is a mixture-of-experts model: 35B total parameters, roughly 3B active per token. Decode speed is governed by active parameters, so it decodes at 55 tokens per second where a dense 27B manages 13.5, a genuine 4×, while landing statistically indistinguishable quality scores. On Pascal, where every FLOP is expensive, that trade is enormous.

It also flagged Muse early: more than half its runs burned every allotted turn. At that point it read as “struggles with tool use.” The coding benchmark sharpened the diagnosis considerably.

Benchmark the benchmark

Then I sharpened the question: forget tool use, measure coding. Published numbers didn’t settle it; public benchmarks leak into training data, and Muse’s excellent SWE-bench scores were the very claim being audited. So: 32 tasks across four families (agentic repo fixes, one-shot function writing, traceback debugging, code comprehension), two difficulty tiers, a hard cap on tool-calling rounds per task, hidden tests the models never see, and a Docker sandbox for anything they execute.

The scoring machinery is where a benchmark earns trust, and every mechanism exists because of a specific way to cheat it. Scoring restores test files from the task definition, so rewriting the visible tests to assert True gains nothing. Repo scoring is multiplicative—fixed × retained—because an additive formula hands a do-nothing model free credit for the tests that already passed. And every task must clear a gate battery before any model sees it: golden (known-correct edits score exactly 1.0), null (doing nothing scores ~0), vandal (golden edits plus vandalized visible tests still score 1.0).

That battery felt thorough. It wasn’t thorough enough.

The benchmark shipped broken, in the most educational way possible

Six models, 32 tasks, two reps: 384 runs. The strongest model finished first, and its scoreboard contained a result too clean to be real: 0.00 on every single repo task. Both tiers. A model that aces everything else scoring zero on exactly one family is not a model problem.

The transcripts told the story in one line. The task’s goal said “the retry-budget helper’s suite is failing.” The model, sensibly, ran the tests first. The tool returned:

1 passed, 0 failed

Every repo task had shipped with a visible test suite that was already green. The bug each task described was real—but only the hidden tests could see it, and models can’t see hidden tests. So the models were told the suite was failing, observed with their own tools that it wasn’t, and burned their entire round budget searching for a defect that was, from their side of the wall, unobservable.

How did this get past the gate battery? Because golden, null, and vandal all validate scoring. Not one of them asks whether the task is solvable from what the model can see. The fix: every repo task gained a visible test that actually reproduces its defect—with input literals deliberately different from the hidden tests’, so the generalization pressure survives—and the battery gained a fourth gate, signal: the visible suite must fail on the pristine repo and pass after the golden edits. Re-run on three of the fixed tasks, the previously-zero model went 0.00 → 1.00 on each, finishing in five to seven rounds instead of capping.

The silent-success check in its purest form: a benchmark defect whose observable symptom—model scores zero—is identical to the symptom of a bad model. If the zeros had come from a small model first, “small models can’t do repo work” would have been an easy, wrong conclusion to keep.

It wasn’t the only harness bug, either. A path-handling inconsistency I introduced myself—one tool accepting absolute paths, the others rejecting them—turned out to hit the weakest models hardest: one burned 106 of its 183 file operations on it, inflating exactly the strong-versus-weak gap the benchmark existed to measure. The nastiest part: writing to /src/budget.py didn’t error. It silently created a second file at that literal path, leaving the buggy original untouched. I re-executed the affected runs and marked one model’s results confounded rather than disproven in the findings log.

The model that solves the task and cannot stop

Muse—the model whose reputation started this whole project—produced the benchmark’s strangest pattern: it hit the round cap on 84% of its runs while scoring perfectly on many of them. The transcripts showed the same shape every time: explore, read, edit, run tests, watch them pass—and then keep going. Another search. Another listing. Re-read a file it had already read. Cap.

Across its 64 runs, 53% ended with the task already solved and the model still calling tools. It never once emitted the plain text message that ends an agent loop.

This reframes its benchmark reputation rather than contradicting it. SWE-bench-style evaluation asks one question: is the final patch correct? By that measure Muse is genuinely good—its hard-tier repo score here was a perfect 1.00. But an agent loop asks a second question static benchmarks never do: does the model know when it’s done? Muse doesn’t. In production, that means unbounded token burn on completed work. I amended the judging rubric to keep the two failure modes separate—”capped without solving” is a reasoning failure, “solved and couldn’t stop” is a termination failure—because collapsing them into one score hides which models can actually code.

What the judge saw that execution scoring couldn’t

After the fleet finished—320 clean runs across five models, the smallest having been pulled mid-fleet once it was clear its runs were both hopeless and the most expensive in wall-clock terms—every transcript went through blind judging, with mandatory line-number citations for every score. Two tasks make the case for the second layer better than any argument.

The stale-records task. Goal: stale records are winning when a key repeats. All five models scored a perfect 1.00 on execution. The diffs split cleanly in two. Three models replaced the dedup guard with an unconditional last record wins—which passes the hidden tests only because the fresher record happens to appear last in the test input. Reverse the input order and the stale record wins again. The bug the task describes is still there. Two models actually compared the sequence field. Execution scoring is structurally incapable of telling these apart. The craft axis scored them 2 and 5.

The CSV task. The defect: a naive line.split(',') breaking on quoted fields. Three of five models hand-rolled character-by-character quote-tracking state machines, 19–26 lines each, correct on the tested cases, escaped-quote handling quietly absent. Two models wrote four lines using Python’s csv module, which handles all of it. All five: execution 1.00. This is the boring technology test administered to language models, and most of them failed it.

And one dog that didn’t bark: across all 160 judged transcripts, not a single model edited a test file to pass. The anti-gaming machinery was never exercised in anger. It was still worth building.

What the results actually support

Three results deserve to outlive the table.

Execution scoring saturated, and admitting it matters. Five of five models scored 1.00 on repo work. My instinct that small models would floor at zero was exactly backwards. At saturation, execution scoring answers “can this model do the job at all” (yes—even the 4.6GB one) and cannot answer “which is better.” The final report calls the top four a statistical tie in plain words, rather than dressing a 0.935-versus-0.915 gap up as a ranking.

Process and craft come apart, and a blended score hides it. The 4.6GB gemma posted the highest judge score overall (best planning, zero round-caps, perfect verification) and the lowest craft score in the fleet. It’s the model most likely to hand you a passing diff you’d bounce in code review. Muse is its mirror image: genuinely good code, and an efficiency score of 1.72 out of 5 because it cannot stop writing it. Splitting one number into five axes turned “which model is best?” into the answerable “best at what?”

Only one separation survived the statistics. Exactly one model sits outside everyone else’s confidence intervals: Muse, below the field. Everything else is a tie broken by secondary criteria: decode speed, craft, VRAM. One strong negative and a cluster of “indistinguishable; choose on other grounds.” Less satisfying than a leaderboard, and considerably more honest.

The final call diverged from the report’s own recommendation, which is worth recording: the report said delete Muse; I kept it. Its repo-patching is real, and a hard round limit neutralizes the termination problem. The benchmark’s job is to make the trade visible, not to make the decision. 102GB of deletions later, the model directory was down from 160GB to 58GB and four survivors.

The stack at the end

With models chosen, the serving stack is deliberately anticlimactic. Three layers, each doing one thing:

 Aider · Claude Code · Open WebUI
              ↓
 LiteLLM      — auth, OpenAI + Anthropic formats, LAN + VPN
              ↓
 llama-swap   — localhost only, never reachable off-box
              ↓
 llama-server — launched via a wrapper that asserts GPU identity or refuses
   P40:  Qwen3.6-35B-A3B | Qwen3.6-27B | Muse-Glimmer-30B  (auto-swapped)
   1070: gemma-4-E4B                                        (pinned, never evicted)

llama-swap loads whichever deep model a request names, evicting the previous one; the small model on the second GPU is pinned, so something always answers instantly. LiteLLM is the only thing reachable off-box, key-authenticated. And the endpoint names are the real model names, not deep and fast. An alias invites misreading which model produced an answer, which is precisely the ambiguity that forced the benchmark’s judging to be blind.

One measurement from this phase deserves its own paragraph. Quantizing the KV cache to 8-bit roughly doubles every context window, but requires flash attention, and flash attention on a tensor-core-less Pascal card is the kind of thing everyone assumes is slow. I fixed the adoption rule before measuring: adopt only if decode cost stays under 15%. Measured cost: 0.74%. Effectively free. Every context window doubled: the MoE to 128k, the 27B to 96k. Two models are now capped by their trained context length rather than by VRAM. The assumption was reasonable, widely held, and worth twenty minutes of llama-bench to demolish.

The doubling mattered immediately. Claude Code’s system prompt is ~41k tokens before you type a word—arithmetic that ruled the 8GB card out as its host and made the 128k MoE the only comfortable fit.

The whole stack has to pass one verification script—services up, llama-swap unreachable from the LAN as a tested property, keyless requests rejected, all four models answering through the front door, the pinned model’s VRAM byte-stable across a swap. The final gate was a reboot with no hands on the keyboard. Everything came back green, with the VRAM figures a couple of mebibytes off their pre-reboot values. That small drift is the proof: the models genuinely reloaded rather than the check reading stale state. After a month of checks that couldn’t tell failure from success, this one can tell a fresh success from a cached one.

The five lessons, if the rest gets cut

1. Silent-success checks are the enemy, and they are everywhere. The most useful reflex I built all month: verify the outcome, never the action.

2. Benchmarks break in ways that impersonate model failure. Gate every task on golden, null, and vandal replays—and above all on solvability from the model’s side of the wall.

3. Measure, don’t assume, in both directions. Flash attention was assumed slow, CUDA impossible, XMP harmless. Each was an afternoon of measurement against months of living with the assumption.

4. Outcome scores saturate; process scores discriminate. Once every model passes the tests, “passes the tests” stops being information.

5. Termination is a first-class capability. No static benchmark measures “knows when it’s done,” and agent loops live and die by it.

The machine that started this story crashed four times running its first benchmarks. It now serves four models behind one authenticated endpoint, survives reboots unattended, and—more to the point—I trust the numbers that chose those models, because every layer of the measurement had to prove itself before it was believed. Like the priority queue in front of Ollama, the end state is boring on purpose. The interesting part was earning the right to be bored.

Self-hosting LLMs on hardware the industry gave up on, or fighting the same benchmark-integrity battles? I’d love to compare notes. Reach out at architgupta941@gmail.com or find me on X.