Will Mayner

AI interpretability·consciousness·neuroscience

Fisher geometry doesn’t improve mechanism clustering in parameter decomposition

I’ve been having a lot of fun using Silico in beta! I gave it a hypothesis and it planned experiments, ran them, and reported the results.

I was learning about parameter decomposition (VPD; Bushnaq et al. (2026), developed at Goodfire), which aims to uncover interpretable mechanisms by learning a decomposition of weight matrices into rank 1 components, then grouping those components based on coactivation.

I wondered if I could use ideas from information geometry to improve the clustering: could approximations of the Fisher information yield better clusters than coactivation?

This seemed plausible to me because Fisher information measures which components affect the model’s output similarly, whereas coactivation is in part measuring co-occurrence of input in the data distribution.

With pretty minimal prompting, Silico was able to plan, orchestrate experiments to test several candidate clustering affinities, interpret results, and write a report. This took around 2 days total.

Result: negative! None of the measures I tried beat coactivation, either on toy models or on the 67M model from the VPD paper.

Figure 1. Mean true-K ARI against the parameter-derived mechanism labels, averaged over 20 residual-MLP checkpoints, for each clustering affinity.
Figure 1. Mean true-K ARI against the parameter-derived mechanism labels, averaged over 20 residual-MLP checkpoints, for each clustering affinity.

In hindsight, this makes sense.

First, consider mechanisms whose components are doing different things in the service of a shared computational goal: these will coactivate, yet influence the output in different, complementary ways.

So coactivation is the right measure here because it asks “are these needed at the same time?” vs. Fisher asking “do these do similar things?” (and Fisher also only measures single-component perturbations anyway, ignoring joint effects).

Second, VPD’s causal importance function is trained to predict which components can be jointly ablated, thus encoding mechanism structure. Coactivation is computed from that function’s outputs—so coactivation is reading out the grouping the decomposition learned!

That’s probably the deepest explanation; in some sense, Fisher information gives an independent notion of ‘mechanism’, and is computed only from the model’s weights and gradients, whereas coactivation is related by definition to the decomposition itself.

Figure 2. Why two components of one mechanism are grouped by coactivation but split by Fisher cosine.
Figure 2. Why two components of one mechanism are grouped by coactivation but split by Fisher cosine.

Interestingly, on a separate benchmark where the causal importance function didn’t carry any usable grouping signal, Fisher did outperform coactivation!1

Full report from Silico: https://share.goodfire.ai/a/shr_c305008cdfa35f7a7e314adb304303de/

Thoughts on Silico

Overall, I found that Silico removes a lot of the friction with this kind of research: handling the context that accumulates throughout an investigation (the most useful aspect, IMO), planning & orchestrating subagents, details of experimental design, dealing with GPUs and collecting results. And it integrates with GitHub and W&B.

Using it feels a bit like you’re leading a small research team, just with a faster iteration cycle. I had been using Claude Code with some of my own scaffolding, and Silico is definitely an upgrade.

I found that Silico can be overly rigorous at preliminary stages of research; it spends a lot of tokens on pre-registration, null controls, bootstrap CIs, etc. The Goodfire team told me they’re aware of this and are working on it :)

Acknowledgements

Thanks to @RaphaelSarfati and the rest of the team at Goodfire for providing beta access, and to Apollo Kaneko for some late-night hotfixes!

Footnotes

  1. The setup for this result was a two-block residual MLP with eight planted mechanisms, each several rank-one atoms under a random hidden-basis rotation so the true grouping is not neuron-aligned. Decompositions were trained at three corners of a trigger-correlation × output-angle grid, three target seeds and three decomposition seeds each, for 27 in total, at 64 learned components and 40 matched atoms per checkpoint. Fisher reached a true-K ARI of 0.3605 and full K-FAC 0.3140, while binary coactivation got −0.0007 and continuous coactivation got −0.0018, with binary coactivation’s pair-discrimination AUC at 0.4736. It’s unclear exactly why coactivation fails here; Silico’s report points to the importance-minimality coefficient of 1 × 10⁻⁵ used to train these decompositions as the possible cause. ↩︎

References

  1. Bushnaq, L., et al. (2026). Interpreting Language Model Parameters (VPD). Goodfire.
  2. Bushnaq, L., Braun, D., & Sharkey, L. (2025). Stochastic Parameter Decomposition. arXiv:2506.20790. (Library.)
  3. Braun, D., Bushnaq, L., Heimersheim, S., Mendel, J., & Sharkey, L. (2025). Interpretability in Parameter Space: Minimizing Mechanistic Description Length with Attribution-based Parameter Decomposition. arXiv:2501.14926.
interpretabilityAI safetyAI for research