This is a living document. Everything in this document is made in good faith of being accurate, but like I just said; we don't yet know everything about what's going on.
Update: I've disabled comments as of 2025-01-26 to avoid everyone having notifications for something a year on if someone wants to suggest a correction. Folks are free to email to suggest corrections still, of course.
| import gc | |
| from typing import Tuple | |
| import torch | |
| import torch.nn.functional as F | |
| import triton | |
| import triton.language as tl | |
| import triton.testing | |
| from kernels import get_kernel |
| from __future__ import annotations | |
| """ | |
| Fused Attention | |
| =============== | |
| This is a Triton implementation of the Flash Attention v2 algorithm from Tri Dao (https://tridao.me/publications/flash2/flash2.pdf) | |
| Credits: OpenAI kernel team | |
| Extra Credits: |
