Dictionary Learning as Unsupervised Steering-Vector Generation
Ontologizers vs. sparse autoencoders on SONAR sentence embeddings
Read the paper PDF · 77 pages · 1 MB
- 1 SAE, ReLU + L1 (10−5)17,723 bits + 1,438 coeffs 0.048
- 2 Ontologizer, hard code1,900 bits 0.154
- 3 SAE, ReLU + L1 (3×10−5)6,390 bits + 519 coeffs 0.220
- 4 Top-k SAE, k=1602,154 bits + 160 coeffs 0.258
- 5 1-layer hard variant1,972 bits + 160 coeffs 0.274
Reconstruction error (FVUw) on SONAR sentence embeddings, one frontier of converged codes. Lower is better; read it against each code’s bits and coefficients. Section 5.2.
- 1 Ontologizer, soft code4,960 mixture weights, 1 epoch 99.85%
- 2 Top-k SAE, k=128~1,580 bits + 128 coeffs 98.6%
- 3 Ontologizer, private heads960 bits 97.9%
- 4 Ontologizer, direct atoms800 bits 97.6%
- 5 Ontologizer, private heads800 bits 97.5%
- 6 Top-k SAE, k=32~394 bits + 32 coeffs 96.2%
Share of GPT-2 small’s loss recovered when the reconstruction of the block-8 residual is spliced back in. Higher is better; bars start at 95%. Ontologizers are one seed each. Table 8.
Sparse autoencoders are the usual tool for pulling features out of a model’s representations, and they are judged one feature at a time. We test a different goal. The Ontologizer is a stack of competing classifier heads trained only to reconstruct an embedding space. If the heads find the space’s natural contrasts, then switching one head’s choice is a steering move, found without labels.
We compare it with top-k SAEs on SONAR sentence embeddings under the same whitened objective, then take the hard code to GPT-2 small. These are working notes, and the results are preliminary.
Key findings
- A discrete code beats a sparse linear one. At 1,900 index bits and no continuous coefficients, the hard code reaches FVUw 0.154, 40% below a top-k SAE that also sends 160 coefficients.
- Soft heads are contrast sets, but not a code. Directions within a head repel, as a classifier’s should, but the plain-SAE frontier beats the soft model’s code on reconstruction at every measured capacity.
- Steering works without supervision. In the soft model, forcing an entry moves decodes along the supervised difference-of-means direction 0.43× as far as supervised steering itself, and 5× further than probe directions.
- Depth costs collateral. At fixed capacity the residual cascade is worth 1.79× over a flat layer, but a single-head intervention disturbs 66% of the other heads, against 33%.
- Only the first layer reproduces. Two seeds reach identical error, yet no head past the first layer appears in both, which limits how far heads can be named.
- The strongest head encodes topic and genre, not language, as a language-agnostic embedding should be organized.
- Error does not order downstream loss. On GPT-2, an 800-bit code recovers 97.6% of the loss, more than a k=32 SAE at nearly the same FVUw (96.2%).
The paper
The full working notes cover the evaluation frame, the structural ladder, steering fidelity, seed reproducibility and every configuration: read the PDF (77 pages).
Get in touch
If you work on dictionary learning or steering and want to compare notes, reach out to contact@flock.holdings or use the contact form.