Back to Module 1.9: Self-AttentionIn Progress
AI Lesson & Submodule
Multi-head Attention
Split query, key, and value vectors across multiple parallel heads.
Why This Matters
Multi-head attention lets models attend to different categories of information (e.g. grammar and names) simultaneously.
What You Will Learn
- •Split vector dimensions
- •Run parallel heads attention
- •Concatenate heads outputs
Concepts Covered
Parallel attention splitsDimensionality splitsHead outputs concatenation
Mapped Foundation Project: Mini Attention Notebook
Self-contained interactive workbook computing step-by-step scaled dot-product attention scores from raw matrices input.
Architecture Preview
Dynamic math workbook running forward projections, scaling matrix outputs, and plotting soft heatmaps.
Character Tokens ArrayQKV Projections WeightsQK Dot Product Score
Tech Stack Planned
ReactTypeScriptMath.js
Technical Interview Value
- ?Detail how inputs tensors are split and recombined across Multi-Head Attention blocks