Sequence models that replace recurrence with attention.
Built on . Introduced in [@vaswani2017].
Self-attention
Every token attends to every other token in one step:
Attention(Q,K,V) = softmax(QK⊤ / √dk) V
See the paper, page 3:
LocalOffline readyMarkdown60 words · saved
Keep typing to filter: /ta finds Table. ↑↓ to move, Enter to insert, Esc to close, Tab to jump between placeholders.
There are about a hundred blocks: text, lists, Callouts, Math, research and study templates, media, tasks and dates. Pin favourites with ★. Every item is also in Mod+K.