Back to Module 1.9: Self-AttentionIn Progress
AI Lesson & Submodule
Causal Masking
Implement causal mask filters to enforce autoregressive generation constraints.
Why This Matters
Decoder-only models must not see future words during training to learn autoregressive prediction rules.
What You Will Learn
- •Write causal masks
- •Prevent future token access
- •Verify decoding shapes
Concepts Covered
Lower triangular matricesNegative infinity logit masksAutoregressive constraints
Mapped Foundation Project: Mini Attention Notebook
Self-contained interactive workbook computing step-by-step scaled dot-product attention scores from raw matrices input.
Architecture Preview
Dynamic math workbook running forward projections, scaling matrix outputs, and plotting soft heatmaps.
Character Tokens ArrayQKV Projections WeightsQK Dot Product Score
Tech Stack Planned
ReactTypeScriptMath.js
Technical Interview Value
- ?How does applying a causal attention mask prevent next-token leakage during parallel batch training?