Back to Module 1.9: Self-AttentionIn Progress
AI Lesson & Submodule
Scaled Dot Product Attention
Derive the mathematical formula of scaled dot-product attention.
Why This Matters
Scaling scores prevents Softmax outputs from reaching flat regions where gradients vanish.
What You Will Learn
- •Compute QK^T matrix
- •Scale by key dimensions
- •Apply softmax functions
Concepts Covered
QK^T dot product mathSquare root key dimension scalingSoftmax normalizations curves
Mapped Foundation Project: Mini Attention Notebook
Self-contained interactive workbook computing step-by-step scaled dot-product attention scores from raw matrices input.
Architecture Preview
Dynamic math workbook running forward projections, scaling matrix outputs, and plotting soft heatmaps.
Character Tokens ArrayQKV Projections WeightsQK Dot Product Score
Tech Stack Planned
ReactTypeScriptMath.js
Technical Interview Value
- ?Derive why self-attention values must be scaled by 1/sqrt(dk) to prevent gradient vanishing