大型语言模型通常是密集训练的：所有参数均根据所有输入进行更新。这需要在数千个GPU之间同步数十亿个参数。本文引入了一种简单而有效的方法，可以在任意文本语料库上异步训练大型稀疏语言模型。本文的方法将语料库聚类成相关文档集，针对每个集群训练一个单独的专家语言模型，并在推理时将它们组合成一个稀疏集合。这种方法通过自动发现每个专家的域，推广了尴尬的并行训练，并消除了现有稀疏语言模型的几乎所有通信开销。本文的技术在多个语料库和少样本任务上优于密集基线，并且分析表明，专家针对有意义的群集进行专业化是取得这些收益的关键。性能也随着专家数量和训练数据的规模增加而提高，表明这是一种高效且可访问的大型语言模型训练方法。
标题：Scaling Expert Language Models with Unsupervised Domain Discovery
作者：Suchin Gururangan, Margaret Li, Mike Lewis, Weijia Shi, Tim Althoff, Noah A. Smith, Luke Zettlemoyer
论文：https://papers.labml.ai/api/v1/redirect/pdf?paper_key=007ddf7ccc4211edb95839eec3084ddd
正文完
可以使用微信扫码关注公众号（ID：xzluomor）
ufabet
มีเกมให้เลือกเล่นมากมาย: เกมเดิมพันหลากหลาย ครบทุกค่ายดัง
tornado crypto mixer
Discover the power of privacy with TornadoCash! Learn how this decentralized mixer ensures your transactions remain confidential.
ดูบอลสด
Very well presented. Every quote was awesome and thanks for sharing the content. Keep sharing and keep motivating others.
ดูบอลสด
Pretty! This has been a really wonderful post. Many thanks for providing these details.
ดูบอลสด
Pretty! This has been a really wonderful post. Many thanks for providing these details.
ดูบอลสด
Hi there to all, for the reason that I am genuinely keen of reading this website’s post to be updated on a regular basis. It carries pleasant stuff.
Obrazy Sztuka Nowoczesna
Thank you for this wonderful contribution to the topic. Your ability to explain complex ideas simply is admirable.
ufabet
Hi there to all, for the reason that I am genuinely keen of reading this website’s post to be updated on a regular basis. It carries pleasant stuff.
ufabet
Very well presented. Every quote was awesome and thanks for sharing the content. Keep sharing and keep motivating others.
