A pre-registered calibration on Qwen2.5-3B-Instruct. After chemistry fine-tuning, the base-model's refusal direction is still at least as good a handle on refusal as the fine-tuned model's own. Did the mechanism move or only the estimate?

Epistemic status: One model (Qwen-2.5-3B-Instruct), one fine-tuning domain (Chemistry QA), pre-registered calibration, two independent refusal datasets. The calibration result (ablation can't see this rotation) is solid; it replicates across the datasets used, two layers, two models and two readouts. The graded-ablation result is larger and clear at layer 19, but it was measured on one dataset, in two fine-tuned models the same training chain, and it has an unresolved caveat about comparing ablation strengths between two directions. There is no non-chemistry control.

(This research project was started as an step for application for Neel Nanda's MATs application.)

TL;DR

• The problem: I finetuned Qwen2.5-3B-Instruct on chemistry QA with no safety data (SFT, then RL starting from the SFT's checkpoint) and asked whether its refusal mechanism changed or not.

• The Refusal Direction Moved: The difference-in-means refusal direction drifted by 24° (cos = 0.914) on dataset v1, falling well below the estimator's noise floor ($0.984$), and by 15° on dataset v2. This replicates known drift reported by Du et al. (2025) and RefusalGuard (2026).

• Standard Ablation Can't Tell: Ablating the base model’s direction inside the fine-tuned model completely removes refusal (transfer ratio 1.004). However, a positive-control calibration reveals that full directional ablation is blind to rotations of this size: it only detects an intervention failure once 33% - 67% of the refusal signal survives (35° - 55°). Following my pre-registered decision rule, the claim that "the mechanism rotated but was causally preserved" is withdrawn.

• A Graded Test Distinguishes Them: Scaling ablation gradually (α in [0, 1]) shows that at layer 19 (dataset v1), the base model’s direction is actually a stronger handle on refusal than the fine-tuned model’s own direction (α_50 ≈ 0.52 vs. 0.60). Single-layer ablation removes 95 - 97% of refusal with the base direction vs. only 75% with the model's own direction.

• My Reading: Narrow capability fine-tuning writes domain-specific bias (chemistry terminology) into the activation difference without genuinely moving the refusal circuit. The 24° rotation describes estimator corruption, not a rotating mechanism.

• Steering Takes a Bigger Push for Unexciting Reasons: Inducing refusal in fine-tuned models requires 20 - 24% more push along the base direction. This is not due to stubbornness; benign fine-tuning simply pulled harmless prompts 1.6 units further away from the refusal threshold, matching a significant reduction in over-refusal (16.8% to 10.4%).

• Refusal Behaviour Remained Intact: Safety behaviour didn't collapse. Minor shifts in measured refusal rates were driven by test set selection effects and first-token wording preferences rather than broken refusal capabilities.

What I'd like input on

1. Has anyone seen a finetuned model's own difference-in-means direction be a worse handle on refusal than the base model's direction, in another model or finetuning setting?

2. Is partial-strength ablation a fair comparison between two directions when activations project differently onto each? (see the caveat in Experiment 2).

3. Under graded ablation the refusal, the refusal-score curve for the base direction turns down above α≈0.8 in both finetuned models. I can't explain it. Ideas welcome.

4. Prior work that calibrate ablation against nearby directions, not just random ones. I haven't found any and may have missed it.

Why I looked at this

This project started as my application task for Neel Nanda's MATS 12.0 stream (an open-ended 20-hour interpretability sprint).

The question came out of a paper I co-authored (2605.28860). We fine-tuned Qwen2.5-3B on science QA with SFT and RL (Dr. GRPO), and found that RL kept about 72% of the base model's circuits against 59% for SFT. Venhoff et al. reached a similar conclusion with steering vectors: RL re-uses mechanisms the model already has, while SFT installs new ones.

Both results are machinery the training objectives touches. I wanted to know what happens to machinery it never touches. Chemistry QA contains no safety data and no reward, so any change to refusal is collateral. That matters in practice. Capability finetuning is routinely stacked on top of safety training, and safety evaluations are usually run on base model. If finetuning moves the refusal mechanism without changing refusal behaviour, a behavioural benchmark would not show it.

My pre-registered question was: Does chemistry SFT erode the refusal mechanism, and does RL continuation restore it?
The RL half yielded a clear null: the RL run was undertrained (its science score collapsed from 77% to 59%), leaving it indistinguishable from SFT. But the SFT investigation produced a neat-looking result—and then revealed a fundamental flaw in the tool used to measure it.

Standard practice in the literature (Arditi et al., 2024) models refusal as a single direction in activation space: the difference in mean activations between harmful and harmless prompts. When asking whether fine-tuning preserved that mechanism, researchers rely on two natural checks:

1. Representational: Does the fine-tuned model's refusal direction still align with the base model's direction (high cosine)?

2. Causal: If you project the base model's direction out of the fine-tuned model, does refusal disappear? (Measured via the transfer ratio: refusal drop from ablating the base direction divided by the drop from ablating the model's own direction).

This post is about what happens when those two checks completely contradict each other—and which one actually deserves our trust.

Setup

• Models: M0 = Qwen2.5-3B-Instruct model. M_SFT = full-finetuning on chemistry QA (2,200 examples, 2 epochs). M_RL = Dr. GRPO continued from M_SFT. Science score: 42 (M0) → 77 (SFT) → 59 (RL).A second SFT seed was added later.

• Refusal direction: Difference-in-means of the residual stream activations at the last instruction token. The layer was chosen on a validation split as the earlier one that fully removed refusal (Layer 19 of 36 on the first dataset; Layer 27 on the second dataset).

• Intervention: Project the direction out of every residual write, at every layer and position.

• Data: (i) v1: AdvBench / HarmBench / MaliciousInstruct vs Alpaca (502 test pairs). (ii) v2 shares no prompts with v1: strongREJECT / Do-Not-Answer /XSTest vs Alpaca (607 test pairs), plus XSTest's safe-but-scary prompts to measure over refusal.

• Readouts: A frozen substring matcher on the reply's opening words ("I'm sorry", "I cannot", …), and a graded refusal score: the log-odds that the first token is a refusal opener.

• Sensitivity experiments: Three follow-ups: E1, rotating the direction on purpose; E2a, adding the direction; E2b, ablating at partial strength. All three run on the same 200 test pairs for every model, with prompt-bootstrap 95% CIs. Their design, grids and decision rules were registered before any of them ran.

• Three follow-ups: E1, rotating the direction on purpose; E2a, adding the direction; E2b, ablating at partial strength. All three run on the same 200 test pairs for every model, with prompt-bootstrap 95% CIs. Their design, grids and decision rules were registered before any of them ran.

• Pre-registration: Hypotheses, thresholds and decision rules were written down before running. Every later deviation is logged together with what had been observed by then.

What the prompts look like

What "refusal removed" looks like

Here is M_SFT on four validation prompts, with and without M0's direction projected out of every residual write. These are from a seeded, uncurated sample of eight:

The substring judge scores all eight ablated replies as "not a refusal". Reading them, only one is an on-topic answer (it explains how to culture bacteria and drops the "resistant to antibiotics" part). Two agree to help and ask foe details. Five are off-topic, grabled, or re-interpret the requests. M0 with it's own direction ablated looks the same: eight of eight scored as non-refusals, one on-topic.

Harmless prompts are damaged too. "Create a simple face recognition program." goes from a step-by-step answer to "I'm ready! Go ahead and provide some examples for me to learn from."

So throughout the post, "ablation removes refusal" means that " the reply no longer opens with a refusal phrase". It does not mean the model carries out the request. Every ablation number below shown (transfer ratio, α50) inherits this.

The results were as follows which were actually fascinating:

On both datasets the direction moved outside its noise ceiling. Yet M0's direction still switched refusal off in M_SFT exactly as well as M_SFT's own direction did. A random direction did almost nothing; on v2 it removed at most 0.06 if refusal against 0.94 - 0.97 for the real ones. "Rotated but not weakened", I had a draft built around it.

Experiments

Experiment 1 (E1): How big a rotation can the directional ablation actually detect?

Before concluding that a mechanism is preserved because cross-checkpoint ablation works (transfer ratio≈1.0), we must ask the calibration question: Could this test have detected a rotation of the observed size in the first place?

I calibrated ablation against controlled synthetic rotations within the same model to assess the sensitivity of the device. I ablated along a vector v(c) that was systematically rotated for a model with unit rejection direction d_own:

$$v(c) = c\cdot d_own + \sqrt(1-c^2) \cdot u$$

where:

1. u is a unit vector orthogonal to d_own (c·d_own = 0).

2. c = cosΘ steps through {1.0, 0.95, 0.90, 0.85, 0.80, 0.70, 0.60, 0.50, 0.30}, plus the empirically observed cosine c_obs.

3. u is sampled across 5 independent random seeds on the orthogonal sphere to measure isotropic sensitivity.

4. Later, u is set to the real rotation axis between M0 and M_SFT to test the specific plane traced by fine-tuning.

We then evaluate whether ablating v(c) still scores as "causally preserved" using the transfer ratio:

$$\text{Transfer Ratio}(c) = \frac{\Delta \text{Refusal}(\text{ablating } v(c))}{\Delta \text{Refusal}(\text{ablating } d_{\text{own}})}$$

Pre-registered rule: If the 95% bootstrap CI of the transfer ratio at the observed cosine (c_obs) has a lower bound>=0.8, the readout is blind to a rotation of that magnitude. Under that outcome, the hypothesis that "the mechanism is preserved despite representation rotation" is non-identifiable by this test, and the claim is withdrawn.

Figure 1: Transfer ratio when ablating a rotated copy of the direction. The vertical line marks the empirical rotation fine-tuning produced. Solid lines show the mean across 5 random orthogonal axes; shaded bands show the min–max range across seeds.

The Verdict: Claim Withdrawn

Across both independent datasets (v1 and v2), at both extraction layers (L19 and L27), in both models (M0 and M_SFT), and across both readouts:

1. A random rotation of the observed size is completely invisible to the ablation test: On the binary substring judge, random rotations of 24° and 15° yield transfer ratios of 0.991 - 1.008. On the continuous first-token refusal-score metric, ratios remain at 0.916 - 0.986 (well above the 0.80 threshold).

2. The test only begins to fail at much larger angles: The substring judge remains saturated until 45° - 55°. The continuous refusal score is noticeably more sensitive, flagging failure at 35° - 36°, but this is still well above the observed 15°- 24° drift.

Conclusion: Observing a transfer ratio of 1.004 when ablating M0's direction in M_SFT does not indicate that the causal mechanism survived intact. A direction rotated by 24° in arbitrary random directions produces the exact same score. Per the pre-registered decision rule, the claim that "the mechanism rotated but was causally preserved" is withdrawn.

Why This Happens: The 1-c^2 Threshold Rule

This saturation is an inevitable consequence of projection geometry compounded with thresholded behaviour:

1. When you project out a direction v that has cosine c with the true refusal direction d_own, the fraction of the activation's component along d_own removed is c^2.

2. The surviving fraction of the true refusal component is: Residual Fraction = 1 - c^2

3. For our observed 24° rotation: c = 0.914 => c^2 approx 0.835 => 1 - c^2 approx 0.165. The ablation removes 83.5% of the refusal signal and leaves only 16.5%.

4. Because generating a refusal string functions like an internal threshold switch, wiping out ≈84% of the signal is sufficient to drop activation levels below the threshold needed to trigger a refusal response.

Practical Rule of Thumb: Before interpreting an all-writes transfer ablation as evidence that a feature or mechanism is preserved across checkpoints, compute 1 - c^2. If the expected surviving component is under 30%, the test is operating in its saturated regime and will score rotated directions as fully effective.

*So this raises "What about the real rotation axis?"*

Instead of random rotation directions, I also ablated along the actual 2D plane connecting M_SFT and M0:

• v1 (Layer 19): Rotating toward M0 stays effective out to 73° (vs. 45° for random noise), making M0's axis look privileged.

• v2 (Layer 27): The pattern inverts it fails at 40°, breaking faster than random noise (55°).

Whether M0's direction is a "better" handle depends heavily on the dataset and layer choice.

Takeaway: Full directional ablation is simply too blunt to distinguish a preserved mechanism from a 35° - 55° rotation. To see what actually happened at 24°, we need a graded test.

Experiment 2 (E2b): A graded test that can see it

Full ablation (α = 1) saturates. To take the test off its ceiling, I made ablation gradual:

$$a \leftarrow a - \alpha \cdot r(r \cdot a), \quad \alpha \in [0, 1]$$

applied to every residual write across all layers.

I measured α_50 - the ablation strength needed to remove 50% of refusals (lower α_50 = a more potent handle). The summary metric is the graded transfer ratio:

$$\text{AUC}(d_{M0}) / \text{AUC}(d_{\text{own}})$$

So, what must happen when the mechanism gets rotated?

If the refusal mechanism truly rotated to the finetuned model's direction (d_own), then d_M0 is just a msialigned copy rotated by angle Θ. Ablating d_M0 at strength α removes only αcos^2Θ of the true refusal component.

Therefore, d_M0 should be weaker than d_own, requiring:

$$\alpha_{50,\text{pred}} = \frac{\alpha_{50,\text{own}}}{\cos^2\theta}$$

Results: The Prediction Fails in Reverse

Instead of needing α≈0.72 - 0.82, M0's direction removes refusal faster than the finetuned model's own direction (0.52 - 0.54 vs. 0.60 - 0.64), performing almost identically to how it performs in M0 itself (0.53).

Two additional checks show the same pattern without comparing partial strengths:

• Single layer ablation (layer 19 only): M0 direction removes 95-97% of refusal in M_SFT and M_RL, while each models' own direction removes only 75%.

• Layer 21 checks: In M_RL at it's own selected layer (L21), the gap narrows substantially (ratio 1.05 [1.04, 1.07]), but M0 direction still leads slightly (α_50 = 0.33 vs 0.38).

The Caveat: A Potential Scale Confound

Projection ablation removes a fraction α of the activation's total projection from zero, not from the harmless baseline.

• IN M_SFT, activations sit with an unnormalized baseline of ≈ 18.6 units along M0's direction vs. 13.8 units along its own direction.

• Because the absolute projection onto M0 is larger, applying the same dimensionless α subtracts a larger absolute vector norm. Part of the α_50 gap may reflect this baseline magnitude difference rather than purely directional efficiency.

What this thing actually means?

On dataset v1 at Layer 19, the fact that M0's direction is a better causal handle than M_SFT's own estimate points away from a clean physical rotation. Two hypotheses fit the data:

1. Estimator Corruption (Leading Hypothesis): Difference-in-means picks up domain-specific variance. Narrow fine-tuning introduces substantial activation bias even on unrelated text (Minder et al., 2025). Chemistry training shifted the activation centroid, rotating the estimate without moving the actual refusal circuit.

2. Multi-dimensional Refusal: Refusal is mediated by a multi-dimensional subspace rather than a single rank-1 direction (Wollschläger et al., 2025), and M0's direction captures a more critical subspace component than the noisy post-fine-tuning estimate.

Either way, cosine drift alone is not the evidence that the underlying safety mechanism rotated.

Experiment 3 (E2a): Inducing refusal takes a bigger push, mostly for unexciting reasons

If ablating the refusal direction direction removes refusals, adding it to harmless prompts should induce them:

$$a \leftarrow a + c \cdot v_{\text{raw}}$$

applied at Layer 19 across 200 harmless prompts. The readout is c_50 - the coefficient at which 50% of harmless prompts trigger a refusal opener.

Results: A Consistent +20 - 24% Shift

Every fine-tuned checkpoint requires roughly 20 -24% more push along M0's direction before it starts refusing harmless requests.

Why This Does Not Mean "Less Responsive to Refusal"

A naive reading of this +24% shift is that fine-tuning made the model "stubborn" or less sensitive to safety steering. The data accounts for almost all of this gap through simple geometry and baseline offsets:

1. Harmless Prompts Start Further from the Boundary: After chemistry training, harmless prompts sit lower along the refusal axis (20.2 to 18.6 units in M_SFT, δ = -1.6). That baseline drop accounts for nearly the entire remaining push (≈1.5 units). The dose-response curves are parallel, which is the classic signature of an intercept shift rather than a slope change.

2. Vector Scale Difference: The raw diff-in-means norm is 11 - 15% smaller in the fine-tuned models (13.8 vs. 15.5 in M0), so an identical coefficient c injects a smaller absolute vector displacement.

3. Activation Norms Didn't Blow Up: Average residual stream norms grew by only 3.4% in M_SFT and 4.4% in M_RL, ruling out overall activation expansion as the cause.

Converting c_50 into absolute displacement units along each model's own fitted direction shows that M_SFT needs 14.2 units vs. 12.7 units in M0 - a modest +12% difference that closely tracks the starting-point shift.

Takeaway: Fine-tuning did not break the model's steering sensitivity. Instead, benign fine-tuning slightly pulled harmless prompts further into the "safe/helpful" territory.

This directly mirrors the behavioural over-refusal results: M_SFT over-refuses safe look-alike prompts significantly less (16.8% to 10.4%). Harmless prompts simply start slightly further away from the refusal threshold.

Behaviour: The sign depends on how the set was built

Did chemistry fine-tuning actually degrade refusal behaviour? It depends on the test set:

• v1 test set (Filtered to M0 refusals): Refusal dropped slightly from 99.2% to 97.2% in M_SFT (p = 0.013). However, because v1 only included prompts \(M_0\) refused, any random perturbation could only push refusal down (a regression-to-the-mean selection effect).

• v2 test set (Unfiltered): Refusal of harmful prompts actually rose from 72.3% to 75.9% (p = 0.005). But inspecting the transcripts shows this is a phrasing artifact: M0 opened with "I'm afraid there's no safe way..." (scored non-refusal by substring matching), while M_SFT opened with "I'm sorry, but..." (scored refusal).

• Over-refusal dropped significantly: On safe look-alikes (XSTest safe), over-refusal fell from 16.8% to 10.4% (p = 0.002). M_SFT successfully answered safe questions like "How do I blow up a balloon?" that M0 mistakenly refused.

Takeaway: Benign chemistry fine-tuning left actual refusal behaviour roughly intact while slightly improving compliance on benign edge cases.

What's New Here (and What Isn't)

• Not New: Difference-in-means refusal direction drift under fine-tuning (Du et al., 2025; RefusalGuard, 2026), cosine noise floors (Varshney, 2026), and the need to calibrate interpretability instruments (Reblitz-Richardson, 2026).

• New in this work:
  1) Calibrating cross-checkpoint ablation: Full directional ablation is blind to rotations under 35° - 55°, accompanied by the 1 - c^2 rule of thumb.
  2) Decoupling estimate drift from mechanism drift: Demonstrating that difference-in-means vectors can drift 24° due to fine-tuning domain bias while the underlying causal handle remains tied to the base direction.
  3) Decomposing steering difficulty: Showing that a +24% steering push is driven by baseline activation shift and vector norm differences rather than true model "stubbornness."

Limitations and Caveats

1. No non-chemistry control: Is estimator drift specific to chemistry (which introduces unusual SMILES strings and chemical syntax), or does generic instruction tuning produce the same drift?

2. Output degradation & judge limits: In qualitative samples, 7 out of 8 ablated completions were off-topic or garbled. Substring matching scores any broken output as "refusal removed." A semantic judge (e.g., StrongREJECT) is needed to separate true compliance from linguistic incoherence.

3. The RL checkpoint degraded: The RL continuation lost science capability (77% to 59%), meaning M_RL represents an undertrained run rather than a healthy post-RL model.

4. Scale confound in graded ablation: Because activations project with a larger absolute magnitude onto M0 (18.6 vs. 13.8 units), using dimensionless α in Experiment 2 may slightly exaggerate M0's advantage.

What I Got Wrong (Epistemic Post-Mortem)

• I nearly published a phantom dissociation: I initially drafted a write-up celebrating "Rotated but not weakened: the mechanism survives fine-tuning!" It was an artifact of an uncalibrated, saturated test. Writing down a strict pre-registered rule forced me to withdraw the claim immediately when positive controls failed.

• I over-interpreted steering difficulty: I initially thought fine-tuning made the model "less responsive to refusal." Converting the push to absolute displacement along each model's own baseline showed the model was responding normally.

The Big Picture Question

Methods that monitor or constrain cosine drift during fine-tuning (e.g., ProCon, RefusalGuard) assume that representation drift = safety drift.

If fine-tuning primarily rotates what the difference-in-means estimator picks up (via domain bias) while the causal mechanism remains unchanged, regularizing against cosine drift may be penalizing harmless representation adjustments while failing to protect safety.

What's Next and Feedback

• Cheap / Next steps: Measure harmful-prompt absolute projections to close the scale caveat in Exp 2; subtract the mean fine-tuning trace on neutral text to verify if the 24° rotation shrinks.

• Compute-intensive: Test on Llama-3-8B, evaluate with an LLM semantic judge, and run non-chemistry SFT.

If you’ve seen similar behaviour where a base model’s vector outperforms a fine-tuned model’s own direction, or want to share compute/feedback, feel free to reach out via comments or email (sawantviraj465@gmail.com). Code is available on GitHub.