Posts tagged Distributed Attention

How to Ensure Kernels Actually Overlapped

• 15 February 2026
Yunpeng Huang
China
English
MagiAttention
Computation-Communication Overlap Distributed Attention Context Parallelism

While the CPU scheduler controls kernel launch order to favor overlap, the GPU Hyper-Q driver [Bradley, 2013] ultimately determines actual execution order non‑deterministically, influenced by transient GPU resource occupancy as well.

Read more ...

---

Distributed-Native FFA

• 14 February 2026
Yunpeng Huang
China
English
MagiAttention
Flex-Flash-Attention Distributed Attention Context Parallelism

The upcoming blog post will be released in the near future. Stay tuned!

Read more ...

---

Attention Engine for Inference

• 08 February 2026
Zewei Tao
China
English
MagiAttention
AF Disaggregation Distributed Attention Context Parallelism

The upcoming blog post will be released in the near future. Stay tuned!

Read more ...

---

Support Native Group Collective Based on DeepEP

• 24 January 2026
Yunpeng Huang , Zewei Tao
China
English
MagiAttention
Group Collective DeepEP Distributed Attention Context Parallelism

The upcoming blog post will be released in the near future. Stay tuned!

Read more ...

---

Dynamic Attention Solver

• 21 January 2026
Jin Li , Zewei Tao , Qiangang Wang , Yunpeng Huang
China
English
MagiAttention
Dynamic Load Balance Sparse Attention Hybrid Attention Distributed Attention Context Parallelism

The upcoming blog post will be released in the near future. Stay tuned!

Read more ...

---

Long-Context Attention Benchmark

• 19 October 2025
Tao Bu , Qiangang Wang , Bowen Zeng , Hanwen Sun , Yunpeng Huang , Zewei Tao
China
English
MagiAttention
Benchmark Blackwell Flex-Flash-Attention Distributed Attention Context Parallelism

From Kernel Efficiency to Distributed Scalability

Read more ...

---

MagiAttention

• 21 April 2025
Zewei Tao , Yunpeng Huang , Qiangang Wang , Hanwen Sun , Jin Li , Tao Bu , Bowen Zeng
China
English
MagiAttention
Attention Slice Representation Computation Load-Balance Zero-Redundant Communication Multi-Stage Overlap Flex-Flash-Attention Group Collective Flash-Attention Distributed Attention Context Parallelism

A Distributed Attention Towards Linear Scalability for Ultra-Long Context, Heterogeneous Mask Training

Read more ...

---