Posts by Jin Li

Support Muon QK-Clip

• 04 February 2026
Jin Li , Yunpeng Huang
China
English
MagiAttention
Muon QK-Clip Flex-Flash-Attention Flash-Attention

The Muon optimizer [Jordan et al., 2024], which leverages matrix orthogonalization, has shown faster convergence than traditional optimizers such as Adam [Kingma and Ba, 2017, Loshchilov and Hutter, 2019] on smaller language models and was subsequently demonstrated to scale to large models by Kimi [Liu et al., 2025].

Read more ...

---

Optimize Sparse Attention in FFA (Coming Soon)

• 25 January 2026
Zewei Tao , Hanwen Sun , Bowen Zeng , Jin Li , Yunpeng Huang
China
English
MagiAttention
Sparse Attention NSA DSA Flex-Flash-Attention Flash-Attention

The upcoming blog post will be released in the near future. Stay tuned!

Read more ...

---

Dynamic Attention Solver (Coming Soon)

• 21 January 2026
Jin Li , Zewei Tao , Qiangang Wang , Yunpeng Huang
China
English
MagiAttention
Dynamic Load Balance Sparse Attention Hybrid Attention Distributed Attention Context Parallelism

The upcoming blog post will be released in the near future. Stay tuned!

Read more ...

---

MagiAttention

• 21 April 2025
Zewei Tao , Yunpeng Huang , Qiangang Wang , Hanwen Sun , Jin Li , Tao Bu , Bowen Zeng
China
English
MagiAttention
Attention Slice Representation Computation Load-Balance Zero-Redundant Communication Multi-Stage Overlap Flex-Flash-Attention Group Collective Flash-Attention Distributed Attention Context Parallelism

A Distributed Attention Towards Linear Scalability for Ultra-Long Context, Heterogeneous Mask Training

Read more ...

---