Beta. Content is under active construction and has not been peer-reviewed. Report errors on
GitHub
.
Disclaimer
Theorem
Path
Curriculum
Paths
Demos
Diagnostic
Search
Quiz Hub
/
Induction Heads
Induction Heads
2 questions
Difficulty 5-9
View topic
Intermediate
0 / 2
1 intermediate
1 advanced
Adapts to your performance
1 / 2
intermediate (5/10)
conceptual
Induction heads are a two-layer circuit in transformers that implement a specific in-context learning algorithm. What pattern do they detect and complete?
Hide and think first
A.
They implement attention-over-attention inside a single transformer layer using one head's output as another's input
B.
They memorize the most frequent bigrams seen during training and recall them at inference time
C.
They detect the pattern
[
A
]
[
B
]
…
[
A
]
and predict
[
B
]
by composing two attention layers
D.
They detect long-range syntactic patterns like subject-verb agreement using a single attention head
Submit Answer