AI Inference & Serving
Multi-Head Attention
An attention arrangement with multiple learned query, key and value projections. Each head calculates an attention result; those results are combined for the next computation. Multiple heads are not multiple independent agents or necessarily human-readable specialties. In conventional multi-head attention, each query head has its own key and value head.
Reviewed
Sources
Free complete lesson
This definition and the complete practical lesson are free. The interactive reader loads the lesson examples without a membership.