{"id":59240,"date":"2022-03-25T15:12:53","date_gmt":"2022-03-25T13:12:53","guid":{"rendered":"https:\/\/forklog.com\/en\/?p=59240"},"modified":"2025-09-04T20:48:35","modified_gmt":"2025-09-04T17:48:35","slug":"researchers-to-build-a-large-open-source-language-model","status":"publish","type":"post","link":"https:\/\/u1f987.com\/en\/researchers-to-build-a-large-open-source-language-model\/","title":{"rendered":"Researchers to build a large open-source language model"},"content":{"rendered":"<p>An international team of BigScience developers has begun training an open-source AI language model with 176 billion parameters.<\/p>\n<blockquote class=\"twitter-tweet\">\n<p lang=\"en\" dir=\"ltr\">BigScience main training just started\ud83d\udca5 A large language model created as a tool for research\ud83d\udd2c<\/p>\n<p>Model: 176 billion parameters<br \/>\ud83d\udcd6<a href=\"https:\/\/t.co\/7gz2Gibybx\">https:\/\/t.co\/7gz2Gibybx<\/a><\/p>\n<p>Data: 46 languages<br \/>\ud83d\udcd6<a href=\"https:\/\/t.co\/EOgshEDrnw\">https:\/\/t.co\/EOgshEDrnw<\/a><\/p>\n<p>Cluster: 416 GPU \u2014 low carbon energy<br \/>\ud83d\udcd6<a href=\"https:\/\/t.co\/VA1u4OpnVr\">https:\/\/t.co\/VA1u4OpnVr<\/a><\/p>\n<p>Follow it live\ud83d\udc47<\/p>\n<p>\u2014 BigScience Research Workshop (@BigscienceW) <a href=\"https:\/\/twitter.com\/BigscienceW\/status\/1503775017653391369?ref_src=twsrc%5Etfw\">March 15, 2022<\/a><\/p><\/blockquote>\n<p> <script async=\"\" src=\"https:\/\/platform.twitter.com\/widgets.js\" charset=\"utf-8\"><\/script><\/p>\n<p>The model is trained on data in 46 languages. The training runs on the Jean Zay supercomputer of the French Institute for Development and Resources in High-Performance Computing. It is built on Nvidia V100 and A100 GPUs. The system&#8217;s peak performance exceeds 28 petaflops.<\/p>\n<p>Hugging Face&#8217;s head of research, Dau Kiela, said the training would take three to four months.<\/p>\n<p>According to the developers, the project is intended for research purposes. Proprietary language models from companies like OpenAI, Google or Microsoft exhibit similarly problematic behavior, spawning toxic language, bias, and misinformation, engineers say. The open-source algorithm will help researchers understand these issues and fix them, they add.<\/p>\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p>\u201cIf we care about democratizing research progress and want to ensure that the world can use this technology \u2014 we must find a solution for this. This is exactly what big science should be about,\u201d Kiela said.<\/p>\n<\/blockquote>\n<p>The open BigScience project brings together about a thousand developers from around the world who create and maintain large datasets for training language models.<\/p>\n<p>In January, OpenAI announced the <a href=\"https:\/\/u1f987.com\/en\/news\/openai-has-created-a-less-toxic-version-of-gpt-3\">creation of a less toxic version of GPT-3<\/a>.<\/p>\n<p>In December 2021, DeepMind introduced the <a href=\"https:\/\/u1f987.com\/en\/news\/deepmind-unveils-a-280-billion-parameter-language-model\">language model with 280 billion parameters<\/a>.<\/p>\n<p>In October, Microsoft and Nvidia developed an algorithm three times larger than GPT-3.<\/p>\n<p>Subscribe to ForkLog AI news on Telegram: <a href=\"https:\/\/t.me\/forklogAI\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">ForkLog AI<\/a> \u2014 all the news from the world of AI!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>An international team of BigScience developers has begun training an open-source AI language model with 176 billion parameters.<\/p>\n","protected":false},"author":1,"featured_media":26216,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"select":"1","news_style_id":"1","cryptorium_level":"","_short_excerpt_text":"","creation_source":"","_metatest_mainpost_news_update":false,"footnotes":""},"categories":[3],"tags":[438],"class_list":["post-59240","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-news-and-analysis","tag-artificial-intelligence"],"aioseo_notices":[],"amp_enabled":true,"views":"9","promo_type":"1","layout_type":"1","short_excerpt":"","is_update":"","_links":{"self":[{"href":"https:\/\/u1f987.com\/en\/wp-json\/wp\/v2\/posts\/59240","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/u1f987.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/u1f987.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/u1f987.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/u1f987.com\/en\/wp-json\/wp\/v2\/comments?post=59240"}],"version-history":[{"count":1,"href":"https:\/\/u1f987.com\/en\/wp-json\/wp\/v2\/posts\/59240\/revisions"}],"predecessor-version":[{"id":59241,"href":"https:\/\/u1f987.com\/en\/wp-json\/wp\/v2\/posts\/59240\/revisions\/59241"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/u1f987.com\/en\/wp-json\/wp\/v2\/media\/26216"}],"wp:attachment":[{"href":"https:\/\/u1f987.com\/en\/wp-json\/wp\/v2\/media?parent=59240"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/u1f987.com\/en\/wp-json\/wp\/v2\/categories?post=59240"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/u1f987.com\/en\/wp-json\/wp\/v2\/tags?post=59240"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}