谷歌于 11 月 19 日将 Streaming DiLoCo 分布式训练方法整合进 MaxText 框架。这项技术让分布在不同城市、不同数据中心的计算节点能够高效协同训练大语言模型。Google integrated the Streaming DiLoCo distributed training method into its MaxText framework on Nov 19, enabling compute nodes spread across cities and data centers to efficiently co-train large language models.
在官方测试中,在 10 Gbit/s 带宽条件下,跨 slice 的通信流量相比传统方案减少逾 60 倍(特定配置下最高约 65 倍),单步训练速度提升 9.57 倍,而模型 loss 与数据并行方案基本持平。这意味着地理上分散的算力集群可以像在同一机房一样协同工作,大幅降低大规模训练对高带宽专线的依赖。In official tests at 10 Gbit/s bandwidth, cross-slice communication traffic dropped over 60x (up to ~65x in some settings) and per-step speed rose 9.57x, with loss on par with data parallelism. That lets geographically scattered clusters cooperate like a single room, sharply cutting reliance on high-bandwidth dedicated lines.