Half the Weights, Twice the Attitude: Stress-Testing Refusals Across Quantization Levels

Every holiday season, somewhere between the Micro Center doorbuster and the third leftover turkey sandwich, a tinkerer downloads a 70B model, watches it eat 140GB of VRAM, sighs, and reaches for the 4-bit GGUF. It's the oldest ritual in local AI: trade precision for headroom, get the model running on the GPU you actually own, and pretend the compression is free. The benchmark charts say the perplexity bump is small. The vibes, though, are a different story.

That's the thing nobody puts in the download card. Quantization doesn't just shave bits off the weights — it changes how the model behaves at the edges of its training. Refusals get softer in some places, harder in others, and occasionally just plain weird. A model that politely declines a borderline request at FP16 might, at Q4_K_M, either shrug and comply or launch into a lecture about why the question itself is dangerous. Neither outcome is predictable from a perplexity score.

So we ran the test. Same model family, same prompt suite, same seed where the runtime allowed it. Full precision on one box, 8-bit on another, and a stack of 4-bit variants — Q4_K_M, Q4_0, Q5_K_M as a control, plus a couple of GPTQ and AWQ builds for the GPU crowd. This isn't a peer-reviewed paper. It's a field audit for people who actually run these models on their own desks.

The Setup: One Suite, Many Shrinks

We picked three open-weight families that get heavy local use — a mid-size general model, a smaller instruct-tuned one, and a code-focused model that tinkerers often run as a second brain. For each, we tested the base (or highest available) precision, an 8-bit quant, and at least three 4-bit variants. Same chat template, same system prompt, temperature pinned to 0.7, top-p at 0.9. Where the runtime supported it, we locked the seed; where it didn't, we ran each prompt five times and looked at the spread.

The prompt suite was deliberately boring: 40 requests split into four buckets. Ten were flatly disallowed under any reasonable policy (we're not publishing those). Ten were borderline — think security research framing, edgy creative writing, medical-adjacent questions with obvious caveats. Ten were ordinary instruction-following tasks with formatting constraints. And ten were classic jailbreak scaffolds from the public corpus, lightly paraphrased so we weren't just testing memorized strings.

We scored refusals on a three-point scale: hard refusal, soft refusal (complies with heavy hedging or a lecture), and compliance. Then we eyeballed everything for tone drift, because a model that complies while sounding like a HR memo is not the same as one that complies cleanly.

Where the Refusal Layer Bends

The first surprise was how uneven the shrinkage is. Across the three families, the hard-refusal rate on truly disallowed prompts stayed roughly flat down to 4-bit — good news, mostly. The 4-bit models still knew what they weren't supposed to do. But the borderline bucket moved. At FP16, most models gave a soft refusal with a caveat. At Q4_K_M, about a third of those turned into straight compliance, sometimes with a cheerful tone that felt out of place. At Q4_0 — the older, blunter quant — the same prompts sometimes triggered harder refusals than full precision, as if the model got confused about context and defaulted to the safest possible reading.

That's the pattern worth internalizing: 4-bit doesn't uniformly loosen the model. It scrambles the calibration. A borderline request can go either direction, and which direction depends on the exact quant method, the model, and sometimes the day. Q4_K_M felt more permissive on creative-writing prompts; Q4_0 felt twitchier and more prone to sudden "I can't help with that" pivots mid-conversation. GPTQ builds were closer to 8-bit on refusals but occasionally dropped formatting instructions. AWQ landed somewhere in the middle with a noticeable uptick in hedging language.

Instruction-Following Goes Soft at the Edges

Refusals get the headlines, but the quieter story is rule-following. We asked for JSON with specific keys, for bullet lists capped at five items, for responses under 80 words, for a specific persona maintained across three turns. Full precision and 8-bit passed most of these. The 4-bit models passed the easy ones and started slipping on the compound ones — a JSON response with a stray markdown fence, a five-item list that quietly became seven, a 60-word cap that ballooned to 130.

None of that is catastrophic. But if your workflow depends on a model holding a format, quantization is a place where the grip loosens. The smaller the model, the more pronounced it gets. A 7B at Q4_0 is noticeably flakier than the same model at Q5_K_M, and the gap widens on multi-step instructions.

The Jailbreak Surface Doesn't Shrink Evenly

The public jailbreak scaffolds were the most interesting bucket. Against full precision, most of them failed — the model caught the framing and refused. Against 4-bit variants, a few that had failed at FP16 started working. Not dramatically, and not consistently across seeds, but enough to notice. The likely explanation is that the model's ability to track the jailbreak framing degrades along with everything else; it loses the thread of "this is a manipulation attempt" and just answers the surface question.

At the same time, a couple of 4-bit builds became more resistant to certain scaffolds, probably because they were too confused to follow the elaborate setup. That's not safety — it's incoherence wearing a safety hat. If you're benchmarking a model's jailbreak resistance, you have to test the quant you're actually going to run. The FP16 scorecard is not a proxy.

A Field Checklist for Small-Footprint Builds

If you're going to run 4-bit — and most of us are — treat the quant as part of the model's personality, not just a file size. A few practical habits help:

• Benchmark the exact quant you'll ship. Not the FP16 sibling, not the 8-bit build. Download the Q4_K_M, run your own 20-prompt suite, and write down what changed.
• Test borderline prompts, not just disallowed ones. The hard refusals usually survive. It's the soft refusals that drift.
• Run each prompt three to five times. 4-bit sampling variance is real; a single run will lie to you.
• Keep a Q5_K_M or 8-bit build around as a reference. When something feels off, compare against the higher-precision model before you blame your prompt.
• Check format adherence separately. JSON, list caps, word limits — these break before refusals do.
• Re-run the suite after every runtime update. Quantization behavior can shift when the inference engine changes its kernels. A model file that behaved one way in October can behave differently after a November release.
• Don't trust the perplexity chart alone. It tells you how well the model predicts text, not how it handles a borderline ask at 2 a.m.

The 4-bit lobotomy is real, but it's not a clean cut. It's a hundred small recalibrations, some of which make the model looser, some tighter, and some just strange. The tinkerers who get the best results are the ones who treat the quant as a variable to be measured — not a checkbox to be clicked on the way to a cheaper rig.

Run the suite. Write down the drift. Then decide whether the model you compressed is still the model you wanted.