Posts tagged Sparse Attention
Optimize Sparse Attention in FFA (Coming Soon)
- 25 January 2026
The upcoming blog post will be released in the near future. Stay tuned!
Dynamic Attention Solver
- 21 January 2026
Context Parallelism (CP) shards sequence activations across distributed devices to overcome memory constraints in long-context training. In MagiAttention, the static attn solver has largely solved CP scheduling for standard long-context workloads where masks are static or deterministic prior to the iteration (e.g. causal, causal document, sliding window). By leveraging initial token dispatch before the iteration starts, the static solver ensures compute balance across ranks while restricting data movement to \(\mathrm{KV}\)-communication only, keeping Query/Output (\(\mathrm{QO}\)) tokens fixed on their host devices.