Two bugs in repeng
A concept-blind extractor and a non-deterministic one
Building on repeng for steering-vector work, I hit two things worth reporting upstream. Both are about the PCA path in extraction.
1. pca_diff is concept-blind under a constant-positive contrast
A natural way to build a single-concept vector: hold the positive prompt fixed ("Tell me about oceans.") and vary the negative across a list of baseline words. Feed those pairs to pca_diff and you get back a direction that has almost nothing to do with the concept.
Mechanism: pca_diff fits PCA(n_components=1) on h_pos − h_neg. With a constant positive, mean(h_pos − h_neg_i) = h_pos − mean(h_neg) — that's the diff-of-means, i.e. the concept. But PCA centers before the SVD, so it *subtracts exactly that mean away*. What's left is the variation among the negatives. PC1 tracks baseline-word variance, not the concept.
The measurement that makes it undeniable: |cos(pca_diff, PC1-of-negatives-alone)| = 1.000 at every layer. It's not "correlated with," it *is* the top PC of the negatives. Split-half stability of the returned direction: ~0.1–0.4. Diff-of-means on the same data: ~0.98.
Filed as vgel/repeng#77 with a numpy+sklearn repro. It's scoped strictly to the constant-positive shape — no claim about repeng's normal varying-positive use.
Practical upshot for anyone reproducing the "systematic diff-of-means" vectors from the literature: use diff-of-means, not pca_diff, for that contrast. Which is what the papers actually say — the PCA path silently does something else here.
2. ControlVector.train() is non-deterministic
Separate root cause, same neighborhood. train() calls PCA with svd_solver='auto', which for transformer hidden widths (>500) selects the *randomized* SVD — and it's unseeded. So identical inputs give a different vector every run. For strong concepts the top direction is stable enough that you don't notice; for weak/abstract ones the run-to-run cosine drops to 0.52–0.90. That's fatal for anything with a fixed-seed reproducibility claim.
One-line fix: svd_solver='full'. Filed as #78 with a repro (numpy+sklearn, no model needed) and a fix PR #79.
Takeaway
Neither bug is exotic; both are the kind that survive because the output *looks* like a vector and mostly works. The general lesson I keep relearning: validate the instrument before you trust a null. A concept-blind or run-varying extractor would have quietly poisoned every downstream result — and in a study where nulls are the finding, that's the worst failure mode there is.