A Three-Phase Factual Recall Circuit in Gemma-2B and Gemma-12B-IT

This article investigates the internal mechanisms of factual recall in Gemma-2B and Gemma-12B-IT models, using activation patching to identify a three-phase circuit involving storage, routing, and readout across transformer layers. The findings highlight that the residual stream plays a dominant role in this process, offering insights into how transformer models retrieve factual information.