Gemini Team

Gemini: A Family of Highly Capable Multimodal Models

This number is an estimate based on limited evidence. In particular, we combine information about the performance of Gemini Ultra on various benchmarks compared to other models, and guesstimates about the hardware setup used for training to arrive at our estimate. Our reasoning and calculations are detailed in this Colab notebook. https://colab.research.google.com/drive/1sfG91UfiYpEYnj_xB5YRy07T5dv-9O_c

"Gemini models are trained on a dataset that is both multimodal and multilingual. Our pretraining dataset uses data from web documents, books, and code, and includes image, audio, and video data... We find that data quality is critical to a highlyperforming model, and believe that many interesting questions remain around finding the optimal dataset distribution for pretraining."

This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding. The Gemini family consists of Ultra, Pro, and Nano sizes, suitable for applications ranging from complex reasoning tasks to on-device memory-constrained use-cases. Evaluation on a broad range of benchmarks shows that our most-capable Gemini Ultra model advances the state of the art in 30 of 32 of these benchmarks — notably being the first mode

API access: https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models

Aidan Clark, Alex Paino, Jacob Menick, Liam Fedus, Luke Metz, Clemens Winter, Lia Guy, Sam Schoenholz, Daniel Levy, Nitish Keskar, Alex Carney, Alex Paino, Ian Sohl, Qiming Yuan, Reimar Leike, Arka Dhar, Brydon Eastman, Mia Glaese, Ben Sokolowsky, Andrew Kondrich, Felipe Petroski Such, Henrique Ponde de Oliveira Pinto, Jiayi Weng, Randall Lin, Youlong Cheng, Nick Ryder, Lauren Itow, Barret Zoph, John Schulman, Mianna Chen, Adam Lerer, Adam P. Goucher, Adam Perelman, Akila Welihinda, Alec Radford

Hello GPT-4o

https://openai.com/index/hello-gpt-4o/ https://openai.com/index/gpt-4o-system-card/

Not known. Inference costs in the API are 2x cheaper than GPT-4 Turbo

Training compute estimated from benchmark scores.

"With GPT-4o, we trained a single new model end-to-end across text, vision, and audio."

We’re announcing GPT-4o, our new flagship model that can reason across audio, vision, and text in real time. GPT-4o (“o” for “omni”) is a step towards much more natural human-computer interaction—it accepts as input any combination of text, audio, image, and video and generates any combination of text, audio, and image outputs. It can respond to audio inputs in as little as 232 milliseconds, with an average of 320 milliseconds, which is similar to human response time(opens in a new window) in a

Definitely a new model, not a GPT-4 finetune

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu,

The Llama 3 Herd of Models

405B

Stated in paper. Also, 6 * 405B * 15.6T training tokens = 3.8e25

15.6T tokens

Modern artificial intelligence (AI) systems are powered by foundation models. This paper presents a new set of foundation models, called Llama 3. It is a herd of language models that natively support multilinguality, coding, reasoning, and tool usage. Our largest model is a dense Transformer with 405B parameters and a context window of up to 128K tokens. This paper presents an extensive empirical evaluation of Llama 3. We find that Llama 3 delivers comparable quality to leading language models s

Llama 3.1 model license: https://huggingface.co/meta-llama/Meta-Llama-3.1-8B/blob/main/LICENSE must seek separate license if over 700m monthly users, acceptable use restrictions training code here: https://github.com/meta-llama/llama-recipes/blob/main/src/llama_recipes/utils/train_utils.py#L70

Claude 3.5 Sonnet

Training compute estimated from benchmark scores. Blog post by Dario Amodei includes some info on 3.5 Sonnet compute: https://darioamodei.com/on-deepseek-and-export-controls "Claude 3.5 Sonnet is a mid-sized model that cost a few $10M's to train (I won't give an exact number). Also, 3.5 Sonnet was not trained in any way that involved a larger or more expensive model (contrary to some rumors)."

Training data cutoff Apr 2024

Zhipu AI

GLM-4-Plus

Estimated using benchmark imputation

At the KDD International Conference on Data Mining and Knowledge Discovery, the Zhipu GLM team unveiled the new generation of base large model—GLM-4-Plus. As the latest version of Zhipu’s fully self-developed GLM large model, GLM-4-Plus signifies Zhipu AI’s continuous dedication in the field of general artificial intelligence, advancing the independent and autonomous innovation of large model technology.

Claude 3.7 Sonnet

"Claude 3.7 Sonnet is trained on a proprietary mix of publicly available information on the Internet, as well as non-public data from third parties, data provided by data labeling services and paid contractors, and data we generate internally. While trained on publicly available information on the internet through November 2024, Claude 3.7 Sonnet’s knowledge cut-off date is the end of October 2024. This means the model’s knowledge base is most extensive and reliable on information and events up

Today, we’re announcing Claude 3.7 Sonnet1, our most intelligent model to date and the first hybrid reasoning model on the market. Claude 3.7 Sonnet can produce near-instant responses or extended, step-by-step thinking that is made visible to the user. API users also have fine-grained control over how long the model can think for. Claude 3.7 Sonnet shows particularly strong improvements in coding and front-end web development.

Grok-2 Beta Release

Estimate based on xAI statements comparing Grok-2 compute to GPT-4 and Grok-3. Full estimate here: https://docs.google.com/document/d/1C_dABuZrAqYE_ui4_GZ4bRLtq3TBjIGoBSktaPElhEU/edit?usp=sharing

Grok-2 is our frontier language model with state-of-the-art reasoning capabilities. This release includes two members of the Grok family: Grok-2 and Grok-2 mini. Both models are now being released to Grok users on the 𝕏 platform.

Doubao General Model Pro (Doubao-pro)

[Speculative] Doubao's large language model has scaled up from 35 billion parameters to 800 billion, with 500 billion and 800 billion parameter models currently under training. https://xueqiu.com/9637001584/309910396?md5__1038=7qmx2DyDuie4cDBqDTQEWqDtMvO4iTphD

6ND = 6 * 500*10^9 * 8350*10^9 = 2.505e+25

Doubao's data sources primarily rely on proprietary business data, accounting for 50-60%; externally sourced data comprises 15-20%; and synthetic data has been used since June of this year, although Doubao is cautious in feeding synthetic data due to its uncertain quality.

[Speculative] Doubao's pre-training data volume is approximately 500TB, with only about 10% of this data actually used for training. The current version employs a non-Mixture-of-Experts (MoE) architecture. In the future, MoE architecture may be introduced to increase parameter count and performance, while also integrating multimodal data solutions. So this model is dense, and the training data is probably all text tokens, not multimodal. 50TB * 167M tokens/GB ~= 8.35 trillion tokens

A professional-grade, self-developed LLM supporting up to 128k tokens, enabling fine-tuning across the entire series.

New models and developer products announced at DevDay

Not known. Maybe smaller/sparser than GPT-4.

Estimated using benchmark imputation

Today, we shared dozens of new additions and improvements, and reduced pricing across many parts of our platform. These include: New GPT-4 Turbo model that is more capable, cheaper and supports a 128K context window

Albert Jiang, Alexandre Sablayrolles, Alexis Tacnet, Alok Kothari, Antoine Roux, Arthur Mensch, Audrey Herblin-Stoop, Augustin Garreau, Austin Birky, Bam4d, Baptiste Bout, Baudouin de Monicault, Blanche Savary, Carole Rambaud, Caroline Feldman, Devendra Singh Chaplot, Diego de las Casas, Diogo Costa, Eleonore Arcelin, Emma Bou Hanna, Etienne Metzger, Gaspard Blanchet, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Harizo Rajaona, Henri Roussez, Hichem Sattouf, Ian Mack, Jean-Malo Delignon, Je

Top-tier reasoning for high-complexity tasks, for your most sophisticated needs.

Details are sparse, but we can hazard a guess based on evidence about the training cluster they may have used, the scale up in compute they likely would have used relative to Mistral Large 1, and from the model's MMLU score. Extended reasoning given here: https://docs.google.com/document/d/1I2ZWBLFMpRZYcdMMUfKAGZFJrOJpduNDS9ZeVFIHnd8/edit?usp=sharing

Today, we are announcing Mistral Large 2, the new generation of our flagship model. Compared to its predecessor, Mistral Large 2 is significantly more capable in code generation, mathematics, and reasoning. It also provides a much stronger multilingual support, and advanced function calling capabilities.

"We are releasing Mistral Large 2 under the Mistral Research License, that allows usage and modification for research and non-commercial usages. For commercial usage of Mistral Large 2 requiring self-deployment, a Mistral Commercial License must be acquired by contacting us."

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie C

GPT-4 Technical Report

90% CI: 8.2E+24 to 4.4E+25 NOTE: this is a rough estimate based on public information, much less information than most other systems in the database. Calculation and confidence intervals here: https://colab.research.google.com/drive/1O99z9b1I5O66bT78r9ScslE_nOj5irN9?usp=sharing

Speculative. Reported secondhand by online sources such as Semianalysis, but not verified by OpenAI. If total number of tokens seen was 13T, text was repeated for 2 epochs, and text was the majority of tokens, then dataset size roughly is 13T*0.75/2 = 4.9T words. Note this examines only the text dataset, since GPT-4 was first and foremost a language model. However, the vision component had its own vision dataset, which we believe accounted for a much smaller part of the compute budget.

We report the development of GPT-4, a large-scale, multimodal model which can accept image and text inputs and produce text outputs. While less capable than humans in many real-world scenarios, GPT-4 exhibits human-level performance on various professional and academic benchmarks, including passing a simulated bar exam with a score around the top 10% of test takers. GPT-4 is a Transformer-based model pre-trained to predict the next token in a document. The post-training alignment process results

Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H. Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, Sirshak Das, Ayush Dattagupta, Olivier Delalleau, Leon Derczynski, Yi Dong, Daniel Egert, Ellie Evans, Aleksander Ficek, Denys Fridman, Shaona Ghosh, Boris Ginsburg, Igor Gitman, Tomasz Grzegorzek, Robert Hero, Jining Huang, Vibhu Jawa, Joseph Jennings, Aastha Jhunjhunwala, John Kamalu, Sadaf Khan, Oleksii Kuchaiev, Patrick LeGresley, Hui Li, Jiwe

NVIDIA Releases Open Synthetic Data Generation Pipeline for Training Large Language Models

340B

9 trillion tokens for training 6 * 340B * 9T = 1.8E25 alternatively, can do a hardware estimate with a few extra steps: According to the technical report, Nemotron-4 340B was trained using up to 6144 H100 GPUs. Helpfully, they also report the model FLOP utilization (MFU), which was 41-42% (Table 2). This is the ratio of the actual output of their GPUs, in FLOP used for training, relative to their theoretical max of 989 teraFLOP/s per GPU. Unfortunately, the report omits the last ingredient, w

The technical report for the 340B model cites the report for the 15B version (https://arxiv.org/pdf/2402.16819 ) from that paper: "We train Nemotron-4 15B on a pre-training dataset consisting of 8 trillion tokens. At a high-level, the data blend is split into three different types of data: English natural language data (70%), multilingual natural language data (15%), and source-code data (15%). The English corpus consists of curated documents from a variety of sources and domains including web

9T training tokens. They first train on an 8T token dataset and then an additional 1T tokens, it's slightly unclear if that's more data or a partial second epoch 6.75T words using 1 token = 0.75 words

We release the Nemotron-4 340B model family, including Nemotron-4-340B-Base, Nemotron-4- 340B-Instruct, and Nemotron-4-340B-Reward. Our models are open access under the NVIDIA Open Model License Agreement, a permissive model license that allows distribution, modification, and use of the models and its outputs. These models perform competitively to open access models on a wide range of evaluation benchmarks, and were sized to fit on a single DGX H100 with 8 GPUs when deployed in FP8 precision. We

Permissive commercial license: https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf

The Claude 3 Model Family: Opus, Sonnet, Haiku

Training compute estimated from benchmark scores.

Claude 3 models are trained on a proprietary mix of publicly available information on the Internet as of August 2023, as well as non-public data from third parties, data provided by data labeling services and paid contractors, and data we generate internally. We employ several data cleaning and filtering methods, including deduplication and classification. The Claude 3 suite of models have not been trained on any user prompt or output data submitted to us by users or customers, including free us

We introduce Claude 3, a new family of large multimodal models – Claude 3 Opus, our most capable offering, Claude 3 Sonnet, which provides a combination of skills and speed, and Claude 3 Haiku, our fastest and least expensive model. All new models have vision capabilities that enable them to process and analyze image data. The Claude 3 family demonstrates strong performance across benchmark evaluations and sets a new standard on measures of reasoning, math, and coding. Claude 3 Opus achieves sta

Gemini Team

Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

MoE architecture

Training compute imputed from benchmark scores.

API access: https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models

Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang, Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaohan Zhang, Xiaotao Gu, Xin Lv, Xinghan Liu, Xinyi Liu, Xinyue Yang, Xixuan Song, Xunkai Zh

ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools

https://arxiv.org/abs/2406.12793 https://zhipuai.cn/en/devday

ChatGLM was 130B parameters, and the paper implies GLM-4 was scaled larger than previous models.

- 0116 has slightly worse performance than 0520 - “the GLM-4 models are pre-trained on ten trillions of tokens” - I did not find any information about parameters or compute. Over here they speculatively estimate GLM-4 to be 200B parameters (which seems plausible to me), though no source provided. - “GLM-4 gets close to the state-of-the-art models (GPT-4-Turbo, Gemini 1.5 Pro, and Claude 3 Opus)” none of these models has parameters disclosed or compute estimation. 6*10000000000000*200000000000

"To date, the GLM-4 models are pre-trained on ten trillions of tokens mostly in Chinese and English, along with a small set of corpus from 24 languages, and aligned primarily for Chinese and English usage"

We introduce ChatGLM, an evolving family of large language models that we have been developing over time. This report primarily focuses on the GLM-4 language series, which includes GLM-4, GLM-4-Air, and GLM-4-9B. They represent our most capable models that are trained with all the insights and lessons gained from the preceding three generations of ChatGLM. To date, the GLM-4 models are pre-trained on ten trillions of tokens mostly in Chinese and English, along with a small set of corpus from 24

GLM-4 (0116) has been made available through the GLM-4 API at https://bigmodel.cn

Mistral Large, our new flagship model

https://www.wsj.com/tech/ai/the-9-month-old-ai-startup-challenging-silicon-valleys-giants-ee2e4c48 Mistral spent <20 million euro (meaning approximately 20 million?) to train Mistral Large: https://x.com/EMostaque/status/1762152740938031484?s=20 "assuming this is on H100s with @Scaleway who are €1.9/hour => 10m H100 hours (c 30m A100 hrs), 3 months at 4k H100s :timer_clock:" -Emad Mostaque Assuming bf16 or fp16, H100 SXM performance is 989 TFLOPS At 1.9 euro per H100-hour and 30% utilization,

Saudi Aramco

Saudi Aramco unveils industry’s first generative AI model

"It has 250 billion parameters that are adjustable during training to generate outputs or make predictions."

6*250B*7T=1.05e+25

"The AI was trained using seven trillion data points, collecting more than 90 years of company history."

Inflection-2: The Next Step Up

"Inflection-2 was trained on 5,000 NVIDIA H100 GPUs in fp8 mixed precision for ~10²⁵ FLOPs" (the second 1 is there because of airtable being wonky, it's not a real sig fig)

Today we are proud to announce that we have completed training of Inflection-2, the best model in the world for its compute class and the second most capable LLM in the world today. Our mission at Inflection is to create a personal AI for everyone. Just a few months ago, we announced Inflection-1 — a best-in-class language model that currently powers Pi. Our new model, Inflection-2, is substantially more capable than Inflection-1, demonstrating much improved factual knowledge, better stylistic c

via Pi, no API

Inflection-2.5: meet the world's best personal AI

"Inflection-1 used approximately 4% the training FLOPs of GPT-4 and, on average, performed at approximately 72% GPT-4 level on a diverse range of IQ-oriented tasks. Inflection-2.5, now powering Pi, achieves more than 94% the average performance of GPT-4 despite using only 40% the training FLOPs." This is a weird one - we estimated GPT-4 at 2.1e25 FLOP (which could be off somewhat, or Inflection could believe a different number). 40% of that is ~8e24. But Inflection 2, the previous model, was tr

At Inflection, our mission is to create a personal AI for everyone. Last May, we released Pi—a personal AI, designed to be empathetic, helpful, and safe. In November we announced a new major foundation model, Inflection-2, the second best LLM in the world at the time. Now we are adding IQ to Pi’s exceptional EQ. We are launching Inflection-2.5, our upgraded in-house model that is competitive with all the world's leading LLMs like GPT-4 and Gemini. It couples raw capability with our signature p

Introducing Grok-1.5, our latest model capable of long context understanding and advanced reasoning. Grok-1.5 will be available to our early testers and existing Grok users on the 𝕏 platform in the coming days.

Lower bound is taken from Grok-1 estimation Upper bound is taken from Grok-2 estimation geometric mean: sqrt(2.90000000001*29.6)*10^24 = 9.26e+24

Musk noted that Grok-1.5 will power xAI’s ChatGPT-challenging chatbot on the X platform, while Grok-2, the successor of the new model, is still in the training phase

Aitor Ormazabal Che Zheng Cyprien de Masson d’Autume Dani Yogatama Deyu Fu Donovan Ong Eric Chen Eugenie Lamprecht Hai Pham Isaac Ong Kaloyan Aleksiev Lei Li Matthew Henderson Max Bain Mikel Artetxe Nishant Relan Piotr Padlewski Qi Liu Ren Chen Samuel Phua Yazheng Yang Yi Tay Yuqi Wang Zhongkai Zhu Zhihui Xie

Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Models

No direct information about Reka Core model ("Reka Core has not finished training and is still improving.") The smaller dense model Reka Flash has 21B parameters and was trained on 5 trillion language tokens. There is information about compute: "Our setup comprises of clusters from a mixture of vendors with our peak compute being approximately 2.5K H100s and 2.5K A100s." If we assume 2 months of training with 2.5k H100s and 2.5k A100s at utilization 0.5 we get 8.4e24 FLOP (2500*9.9e14+2500*3.

The training data comprises a mixture of publicly available and proprietary/licensed datasets with a dataset knowledge cutoff of November 2023. The dataset ingested by our model comprises of text, images, videos, and audio clips. Reka Flash and Reka Edge were trained on approximately 5 trillion and 4.5 trillion extensively deduplicated and filtered language tokens, respectively. While the classification of corpora is not strictly defined to one class or category, approximately 25% of our pretrai

AI Singapore

SEA-LION V3

Llama3.1 70B base model: 7.929e+24 Additional pretraining compute: Stage 1: 200*60*60*989500000000000*64*0.3=1.36788×10^22 Stage 2: 495*60*60*989500000000000*128*0.3=6.77103×10^22 Total: 7.929×10^24 + 1.36788×10^22 + 6.77103×10^22 = 8.0103891 × 10^24

Trained on a mix of datasets including StackV2 and Dolma (see https://huggingface.co/aisingapore/llama3.1-70b-cpt-sea-lionv3-base#data)

"pre-trained on 200B tokens"

Our SEA-LION v3 Llama3.1 8B and 70B base models have been continued pre-trained on top of the Llama3.1 8B and 70B models respectively. Both have a context length of 128K, making them the SEA-LION models with the longest context length to date.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu,

The Llama 3 Herd of Models

70B

Huggingface page says 3.1-70B used 7.0M H100 hours and trained over 15T tokens. https://huggingface.co/meta-llama/Llama-3.1-70B The paper also says that 3.1-405B got MFU of between 38-43%; presumably 70B was around the same or a bit higher. I'll assume utilization of 40% 6ND: 6 * 15T * 70B = 6.3e24 FLOPs Hardware: 7M * 9.9e14 * 3600 * 0.4 = 9.98e24 FLOPs Geometric mean: sqrt(6.3e24 * 9.98e24) = 7.929e24 Note that Llama 3-70B also said it used 15T tokens, but only 6.4M H100 hours. This sugges

Modern artificial intelligence (AI) systems are powered by foundation models. This paper presents a new set of foundation models, called Llama 3. It is a herd of language models that natively support multilinguality, coding, reasoning, and tool usage. Our largest model is a dense Transformer with 405B parameters and a context window of up to 128K tokens. This paper presents an extensive empirical evaluation of Llama 3. We find that Llama 3 delivers comparable quality to leading language models s

Llama 3.1 license: https://huggingface.co/meta-llama/Meta-Llama-3.1-8B/blob/main/LICENSE must seek separate license if over 700m monthly users, acceptable use restrictions code here: https://github.com/meta-llama/llama3/tree/main

Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy Zhang, Makesh Narsimhan Sreedhar, Oleksii Kuchaiev

200000 monthly HF download as of Nov 2024

Taken from Llama 3.1 70B as the finetuning compute is multiple orders of magnitude lower

Overall, we have 7,118 preference pairs with 6,766 pairs in the training set and 352 pairs in the validation set.

High-quality preference datasets are essential for training reward models that can effectively guide large language models (LLMs) in generating high-quality responses aligned with human preferences. As LLMs become stronger and better aligned, permissively licensed preference datasets, such as Open Assistant, HH-RLHF, and HelpSteer need to be updated to remain effective for reward modeling. Methods that distil preference data from proprietary LLMs such as GPT-4 have restrictions on commercial usa

Llama 3.1 70B: 7.929e+24 FT (see Appendix F): 32+64=96 hours on a single H100 Compute: 96*60*60*989500000000000*0.3=102591360000000000000=1e20

Aaditya Singh; Aaron Grattafiori; Abhimanyu Dubey; Abhinav Jauhri; Abhinav Pandey; Abhishek Kadian; Adam Kelsey; Adi Gangidi; Ahmad Al-Dahle; Amit Sangani; Ahuva Goldstand; Aiesha Letman; Ajay Menon; Akhil Mathur; Alan Schelten; Alex Vaughan; Amy Yang; Andrei Lupu; Andres Alvarado; Andrew Gallagher; Andrew Gu; Andrew Ho; Andrew Poulton; Andrew Ryan; Angela Fan; Ankit Ramchandani; Anthony Hartshorn; Archi Mitra; Archie Sravankumar; Artem Korenev; Arun Rao; Ashley Gabriel; Ashwin Bharambe; Assaf E

Introducing Meta Llama 3: The most capable openly available LLM to date

Arithmetic calculation: 6 * 15T tokens * 70B parameters = 6.3e24 GPU calculation: https://huggingface.co/meta-llama/Meta-Llama-3-70B indicates training took 6.4M GPU-hours We also know their larger scale training runs for 405B were getting between 0.38-0.41 MFU. Presumably the 70B model gets at least 0.43 utilization (405B has to be split across two nodes, while 70B should fit on one). 990 TFLOPS per GPU * 6.4 million GPU hours * 3600s * 0.43 = 9.808e24 Geometric mean: sqrt(6.3e24 * 9.808e24)

"Llama 3 is pretrained on over 15T tokens that were all collected from publicly available sources. Our training dataset is seven times larger than that used for Llama 2, and it includes four times more code. To prepare for upcoming multilingual use cases, over 5% of the Llama 3 pretraining dataset consists of high-quality non-English data that covers over 30 languages."

https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md License A custom commercial license is available at: https://llama.meta.com/llama3/license

Qwen Team

Qwen2.5: A Party of Foundation Models!

Number of Parameters: 72.7B Number of Paramaters (Non-Embedding): 70.0B

6ND = 6*72700000000 parameters *18000000000000 tokens = 7.8516e+24

"In terms of Qwen2.5, the language models, all models are pretrained on our latest large-scale dataset, encompassing up to 18 trillion tokens."

Qwen2.5 is the latest series of Qwen large language models. For Qwen2.5, we release a number of base language models and instruction-tuned language models ranging from 0.5 to 72 billion parameters. Qwen2.5 brings the following improvements upon Qwen2: Significantly more knowledge and has greatly improved capabilities in coding and mathematics, thanks to our specialized expert models in these domains. Significant improvements in instruction following, generating long texts (over 8K tokens), unde

requires permission to use in applications with 100K+ users https://huggingface.co/Qwen/Qwen2.5-72B-Instruct

Qwen Team

Qwen2.5: A Party of Foundation Models!

72.7B

Training dataset size was 18 trillion 6ND = 6 * 72.7 billion parameters * 18 trillion tokens = 7.8e24

"In terms of Qwen2.5, the language models, all models are pretrained on our latest large-scale dataset, encompassing up to 18 trillion tokens."

"In terms of Qwen2.5, the language models, all models are pretrained on our latest large-scale dataset, encompassing up to 18 trillion tokens"

In the past three months since Qwen2’s release, numerous developers have built new models on the Qwen2 language models, providing us with valuable feedback. During this period, we have focused on creating smarter and more knowledgeable language models. Today, we are excited to introduce the latest addition to the Qwen family: Qwen2.5. We are announcing what might be the largest opensource release in history! Let’s get the party started!

license: allows commercial. weights only https://huggingface.co/Qwen/Qwen2.5-72B/blob/main/LICENSE

Pre-training leads Aidan Clark, Alex Paino, Jacob Menick Post-training leads Liam Fedus, Luke Metz Architecture leads Clemens Winter, Lia Guy Optimization leads Sam Schoenholz, Daniel Levy Long-context lead Nitish Keskar Pre-training Data leads Alex Carney, Alex Paino, Ian Sohl, Qiming Yuan Tokenizer lead Reimar Leike Human data leads Arka Dhar, Brydon Eastman, Mia Glaese Eval lead Ben Sokolowsky Data flywheel lead Andrew Kondrich Inference lead Felipe Petroski Such Inference Producti

GPT-4o mini: advancing cost-efficient intelligence

Training compute estimated from benchmark scores. 90% CI [3.23e+24, 2.05e+25]

OpenAI is committed to making intelligence as broadly accessible as possible. Today, we're announcing GPT-4o mini, our most cost-efficient small model. We expect GPT-4o mini will significantly expand the range of applications built with AI by making intelligence much more affordable. GPT-4o mini scores 82% on MMLU and currently outperforms GPT-41 on chat preferences in LMSYS leaderboard(opens in a new window). It is priced at 15 cents per million input tokens and 60 cents per million output toke

Andrew M. Dai, David R. So, Dmitry Lepikhin, Jonathan H. Clark, Maxim Krikun, Melvin Johnson, Nan Du, Rohan Anil, Siamak Shakeri, Xavier Garcia, Yanping Huang, Yi Tay, Yong Cheng, Yonghui Wu, Yuanzhong Xu, Yujing Zhang, Zachary Nado, Bryan Richter, Alex Polozov, Andrew Nystrom, Fangxiaoyu Feng, Hanzhao Lin, Jacob Austin, Jacob Devlin, Kefan Xiao, Orhan Firat, Parker Riley, Steven Zheng, Yuhuai Wu, Zhongtao Liu, Jiahui Yu, Guy Gur-Ari, Weikang Zhou, Sneha Kudugunta, Sunipa Dev, Frederick Liu, Gus

PaLM 2 Technical Report

Model Architecture: "PaLM-2 is a new state-of-the-art language model. We have small, medium, and large variants that use stacked layers based on the Transformer architecture, with varying parameters depending on model size. Further details of model size and architecture are withheld from external publication." However, the parameter count was leaked to CNBC: https://www.cnbc.com/2023/05/16/googles-palm-2-uses-nearly-five-times-more-text-data-than-predecessor.html

Compute Requirements "Not reported." Paper suggests heuristic of C=6ND. Based on 340B parameters and 3.6T tokens, training compute would be around 7.3*10^24 FLOP.

"The PaLM 2 pre-training corpus is composed of a diverse set of sources: web documents, books, code, mathematics, and conversational data. The pre-training corpus is significantly larger than the corpus used to train PaLM (Chowdhery et al., 2022). PaLM 2 is trained on a dataset that includes a higher percentage of non-English data than previous large language models, which is beneficial for multilingual tasks" (page 9)

"The pre-training corpus is significantly larger than the corpus used to train PaLM" so greater than 6e+11. According to the leaked documents viewed by CNBC, the corpus was 3.6 trillion tokens or around 2.7*10^12 words. https://www.cnbc.com/2023/05/16/googles-palm-2-uses-nearly-five-times-more-text-data-than-predecessor.html

We introduce PaLM 2, a new state-of-the-art language model that has better multilingual and reasoning capabilities and is more compute-efficient than its predecessor PaLM (Chowdhery et al., 2022). PaLM 2 is a Transformer-based model trained using a mixture of objectives similar to UL2 (Tay et al., 2023). Through extensive evaluations on English and multilingual language, and reasoning tasks, we demonstrate that PaLM 2 has significantly improved quality on downstream tasks across different model

Zihan Wang and Xinzhang Liu and Shixuan Liu and Yitong Yao and Yuyao Huang and Zhongjiang He and Xuelong Li and Yongxiang Li and Zhonghao Che and Zhaoxi Zhang and Yan Wang and Xin Wang and Luwen Pu and Huihan Xu and Ruiyu Fang and Yu Zhao and Jie Zhang and Xiaomeng Huang and Zhilong Lu and Jiaxin Peng and Wenjun Zheng and Shiquan Wang and Bingkai Yang and Xuewei he and Zhuoru Jiang and Qiyi Xie and Yanhan Zhang and Zhongqiu Li and Lingling Shi and Weiwei Fu and Yin Zhang and Zilu Huang and Sishi

TeleChat Technical Report

6ND: 6 * 115B * 10T = 6.9e24

The open source TeleChat2-115B model is trained using 10 trillion tokens of high-quality Chinese and English corpus

Meta Llama 3.3 multilingual large language model (LLM)

70B

6ND = 6*70*10^9*15*10^12= 6.3e+24 7000000*3600*989500000000000*0.3= 7.48062e+24 sqrt(7.48062e+24*6.3e+24) = 6.8649768e+24

"A new mix of publicly available online data."

"Overview: Llama 3.3 was pretrained on ~15 trillion tokens of data from publicly available sources. The fine-tuning data includes publicly available instruction datasets, as well as over 25M synthetically generated examples. Data Freshness: The pretraining data has a cutoff of December 2023."

The Meta Llama 3.3 multilingual large language model (LLM) is a pretrained and instruction tuned generative model in 70B (text in/text out). The Llama 3.3 instruction tuned text only model is optimized for multilingual dialogue use cases and outperforms many of the available open source and closed chat models on common industry benchmarks. Model developer: Meta Model Architecture: Llama 3.3 is an auto-regressive language model that uses an optimized transformer architecture. The tuned versions

License A custom commercial license, the Llama 3.3 Community License Agreement, is available at: https://github.com/meta-llama/llama-models/blob/main/models/llama3_3/LICENSE "Llama 3.3 is intended for commercial and research use in multiple languages."

NVIDIA: Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, Daniel Dworakowski, Jiaojiao Fan, Michele Fenzi, Francesco Ferroni, Sanja Fidler, Dieter Fox, Songwei Ge, Yunhao Ge, Jinwei Gu, Siddharth Gururani, Ethan He, Jiahui Huang, Jacob Huffman, Pooya Jannaty, Jingyi Jin, Seung Wook Kim, Gergely Klár, Grace Lam, Shiyi Lan, Laura Leal-Taixe, Anqi Li, Zhaoshuo Li, Chen-Hsuan Lin, Tsung-Yi Lin, Huan Ling, Min

Cosmos World Foundation Model Platform for Physical AI

14B

989500000000000 * 0.4 * 10000 * 3600 * 3 *30 *24 = 3.0777408e+25 (total training compute) assuming this model is 1/5 of it: 3.0777408e+25 / 5 = 6.1554816e+24 (Likely confidence)

"Suite of first-generation video models trained on 9,000 trillion tokens, including 20 million hours of robotics and driving data - generating high-quality videos from multimodal inputs like images, text, or video."

Physical AI needs to be trained digitally first. It needs a digital twin of itself, the policy model, and a digital twin of the world, the world model. In this paper, we present the Cosmos World Foundation Model Platform to help developers build customized world models for their Physical AI setups. We position a world foundation model as a general-purpose world model that can be fine-tuned into customized world models for downstream applications. Our platform covers a video curation pipeline, pr

NVIDIA Open Model License Agreement Under the NVIDIA Open Model License, NVIDIA confirms: Models are commercially usable. You are free to create and distribute Derivative Models. NVIDIA does not claim ownership to any outputs generated using the Models or Derivative Models. Important Note: If you bypass, disable, reduce the efficacy of, or circumvent any technical limitation, safety guardrail or associated safety guardrail hyperparameter, encryption, security, digital rights management, or aut

Introducing Amazon Nova foundation models: Frontier intelligence and industry leading price performance

"probably just below 1e25 stemming from the Llama 70B serving speed. If Llama 70B is trained proportionally to 405B, then it's at ~ 6.6e24. Nova Pro is served at 100tk/s, while Llama 70B is served at 70tk/s on average, and 100tk/s by together.ai at FP8. So Nova Pro would be >1e25 if they roughly 2x the amount of training compared to Llama 70B which [seems unlikely]"

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

671B total 37B activated https://github.com/deepseek-ai/DeepSeek-R1/tree/main

4.56e+24 FLOP (estimated base model Deepseek V3 training compute) + 6.1e23 FLOP = 5.17e+24 FLOP

RL + SFT When reasoning-oriented RL converges, we utilize the resulting checkpoint to collect SFT (Supervised Fine-Tuning) data for the subsequent round. Unlike the initial cold-start data, which primarily focuses on reasoning, this stage incorporates data from other domains to enhance the model’s capabilities in writing, role-playing, and other general-purpose tasks.

We introduce our first-generation reasoning models, DeepSeek-R1-Zero and DeepSeek-R1. DeepSeek-R1-Zero, a model trained via large-scale reinforcement learning (RL) without supervised fine-tuning (SFT) as a preliminary step, demonstrates remarkable reasoning capabilities. Through RL, DeepSeek-R1-Zero naturally emerges with numerous powerful and intriguing reasoning behaviors. However, it encounters challenges such as poor readability, and language mixing. To address these issues and further enhan

6.1e23 FLOP from these estimations: https://epoch.ai/gradient-updates/what-went-into-training-deepseek-r1

MIT licensed

200B dense model https://importai.substack.com/p/import-ai-365-wmd-benchmark-amazon

trained using NVIDIA NeMo: https://blogs.nvidia.com/blog/nemo-amazon-titan/ 13,760 NVIDIA A100 chips (using 1,720 P4d nodes). It took 48 days to train. from https://importai.substack.com/p/import-ai-365-wmd-benchmark-amazon counting operations: 6*200000000000*4000000000000=4.8e+24 gpu usage: 312000000000000(FLOP/s)*0.3*13760*1152*3600=5.3413281792e+24

4T tokens of data, based on comments from amazon engineer James Hamilton at a 2024 talk: https://perspectives.mvdirona.com/2024/01/cidr-2024/ Also cited here: https://lifearchitect.ai/titan/

DeepSeek-V3 Technical Report

Mixture-of-Experts (MoE) language model with 671B total parameters with 37B activated for each token.

6*37B*14.8T = 3.2856e+24 Alternatively, they say: "DeepSeek-V3 requires only 2.788M H800 GPU hours for its full training." and we know they trained in FP8. H800s get 1.513e15 FLOP/s in FP8: 2.788M * 3600 * 1.513e15 * 0.3 = 4.56e24 Utilization may be somewhat lower for FP8. Upper bound estimate: 50% utilization would mean 7.59e24

"We pre-train DeepSeek-V3 on 14.8 trillion diverse and high-quality tokens, followed by Supervised Fine-Tuning and Reinforcement Learning stages to fully harness its capabilities"

We present DeepSeek-V3, a strong Mixture-of-Experts (MoE) language model with 671B total parameters with 37B activated for each token. To achieve efficient inference and cost-effective training, DeepSeek-V3 adopts Multi-head Latent Attention (MLA) and DeepSeekMoE architectures, which were thoroughly validated in DeepSeek-V2. Furthermore, DeepSeek-V3 pioneers an auxiliary-loss-free strategy for load balancing and sets a multi-token prediction training objective for stronger performance. We pre-tr

MIT and deepseek license https://github.com/deepseek-ai/DeepSeek-V3?tab=readme-ov-file

Andy Narayanan, Aonan Zhang, Bowen Zhang, Chen Chen, Chong Wang, Chung-Cheng Chiu, David Qiu, Deepak Gopinath, Dian Ang Yap, Dong Yin, Feng Nan, Floris Weers, Guoli Yin, Haoshuo Huang, Jianyu Wang, Jiarui Lu, John Peebles, Ke Ye, Mark Lee, Nan Du, Qibin Chen, Quentin Keunebroek, Ruoming Pang, Sam Wiseman, Syd Evans, Tao Lei, Tom Gunter, Vivek Rathod, Xiang Kong, Xianzhi Du, Yanghao Li, Yongqiang Wang, Yuan Gao, Zaid Ahmed, Zhaoyang Xu, Zhiyun Lu, Zirui Wang, Al Rashid, Albin Madappally Jose, Ale

Apple Intelligence Foundation Language Models

"The AFM base models are dense decoder-only models that build on the Transformer architecture" "We train AFM-server from scratch for 6.3T tokens on 8192 TPUv4 chips, using a sequence length of 4096 and a batch-size of 4096 sequences." "For both models we perform continued pre-training at a sequence length of 8192, with another 1T tokens from a mixture that upweights math and code, and down-weights the bulk web-crawl." "The sustained model-flop-utilization (MFU) for this training run was appro

6.3T tokens of web text, code, and math, plus another 1T in the second stage and 100B in the third. See section 3.1 for details.

Not explicitly mentioned, but I assume the 7.4T tokens do not involve multiple epochs.

Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong, Yulu Jia, Sun He, Hongmin Chen, Zhihao Bai, Qi Hou, Shipeng Yan, Ding Zhou, Yiyao Sheng, Zhuo Jiang, Haohan Xu, Haoran Wei, Zhang Zhang, Pengfei Nie, Leqi Zou, Sida Zhao, Liang Xiang, Zherui Liu, Zhe Li, Xiaoying Jia, Jianxi Ye, Xin Jin, Xin Liu

MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs

Production run is stated to have "hundreds of billions of parameters". Since the authors also do a number of experiments with a 530B model, I speculate they've used 530B for the production model.

Speculative. The model is stated to have trained for "several weeks". Assuming 530B parameters and "several" = 3, compute can be estimated from the 175B model's stated PFLOP/sec: 2166.3 aggregate PFlops/sec * 3 weeks * 7 days/week * 24 hours/day * 3600 seconds/hour = 3.9e+24. As an upper bound, say 8e+24.

Speculative. Authors note production system was trained on "multi-trillions of tokens". This could refer to training for multiple epochs on the same 300B tokens used to train the 175B and 530B models outlined in more detail in the paper. Alternatively, it could refer to a larger dataset of perhaps 3-9 trillion tokens.

We present the design, implementation and engineering experience in building and deploying MegaScale, a production system for training large language models (LLMs) at the scale of more than 10,000 GPUs. Training LLMs at this scale brings unprecedented challenges to training efficiency and stability. We take a full-stack approach that co-designs the algorithmic and system components across model block and optimizer design, computation and communication overlapping, operator optimization, data pip

Code for MegaScale (also called veScale) training system are released under Apache Licence: https://github.com/volcengine/vescale The model itself is unreleased.

SenseTime Launches “SenseNova” Foundation Model Sets and AI Computing Systems, Advancing AGI Development

https://www.thepaper.cn/newsDetail_forward_22639611 Translation: "SenseTime launched the "SenseNova" large model system, which includes natural language generation, image generation services, pre-labeling for perception models, and model development. The "SenseChat" application platform, powered by a 180-billion parameter Chinese language model, supports ultra-long text comprehension and offers capabilities such as question answering, understanding, and generation in Chinese." Link says "hundre

“Over the course of five years, SenseTime has built SenseCore, a leading AI infrastructure with 27,000 GPUs, capable of delivering a total computational power of 5,000 petaflops” Assuming they used this entire cluster with 30 days of training (rough average of frontier model training times since 2016), 30% utilization rate: 5000e15 * 0.3 * 30 * 24 * 60 * 60 = 3.89e24 FLOP. Assuming the model is dense and trained Chinchilla-optimal: 20 tokens/parameter * (180e9 parameters)**2 * 6 = 3.89e24 FLOP

SenseTime hosted a Tech Day event, sharing their strategic plan for advancing AGI (Artificial General Intelligence) development through the combination of “foundation models + large-scale computing” systems. Under this strategy, SenseTime unveiled the “SenseNova” foundation model set, introducing a variety of foundation models and capabilities in natural language processing, content generation, automated data annotation, and custom model training. At the event, SenseTime not only showcased their

https://www.anthropic.com/index/claude-2, https://www-files.anthropic.com/production/images/Model-Card-Claude-2.pdf

From model card: "Claude models are trained on a proprietary mix of publicly available information from the Internet, datasets that we license from third party businesses, and data that our users affirmatively share or that crowd workers provide. Some of the human feedback data used to finetune Claude was made public [12] alongside our RLHF [2] and red-teaming [4] research. Claude 2’s training data cuts off in early 2023, and roughly 10 percent of the data included was non-English."

Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, Guilherme Penedo

The Falcon Series of Open Language Models

https://falconllm.tii.ae/falcon-180b.html; https://arxiv.org/abs/2311.16867

"Falcon 180B is a super-powerful language model with 180 billion parameters"

43,500 petaflop-days per Table 1 of the paper 43500 * 1e15 * 24 * 3600 = 3.76e24 C = 6ND = 6 FLOP/token/parameter * 3.5 trillion tokens * 180 billion parameters = 3.78*10^24 FLOP

"The Falcon series is made of three causal decoder-only models trained on up to 4,096 A100. We assembled a pretraining dataset of 3,500 billion tokens, predominantly sourced from our work on RefinedWeb (Penedo et al., 2023)–a massive filtered and deduplicated web dataset" Training dataset composition is described in Table 3. Falcon was trained for 1 epoch.

3.5 trillion tokens * (~3 words per 4 tokens) ~= 2.625 trillion words

Falcon 180B is a super-powerful language model with 180 billion parameters, trained on 3.5 trillion tokens. It's currently at the top of the Hugging Face Leaderboard for pre-trained Open Large Language Models and is available for both research and commercial use. This model performs exceptionally well in various tasks like reasoning, coding, proficiency, and knowledge tests, even beating competitors like Meta's LLaMA 2. Among closed source models, it ranks just behind OpenAI's GPT 4, and perfo

"Falcon 180b can be commercially used but under very restrictive conditions, excluding any "hosting use"." https://huggingface.co/blog/falcon-180b

Qwen Team

QwQ-32B: Embracing the Power of Reinforcement Learning

Architecture: transformers with RoPE, SwiGLU, RMSNorm, and Attention QKV bias Number of Parameters: 32.5B Number of Paramaters (Non-Embedding): 31.0B Number of Layers: 64 Number of Attention Heads (GQA): 40 for Q and 8 for KV

Assuming the same dataset size as for Qwen2.5 training (18T tokens): 6ND = 6 * 32500000000 parameters * 18 * 10^12 tokens = 3.51 × 10^24 'Speculative' confidence

Speculativley might be similar to Qwen2.5 models (18T tokens)

QwQ is the reasoning model of the Qwen series. Compared with conventional instruction-tuned models, QwQ, which is capable of thinking and reasoning, can achieve significantly enhanced performance in downstream tasks, especially hard problems. QwQ-32B is the medium-sized reasoning model, which is capable of achieving competitive performance against state-of-the-art reasoning models, e.g., DeepSeek-R1, o1-mini.

https://huggingface.co/Qwen/QwQ-32B Apache 2

Qwen Team

Qwen2.5: A Party of Foundation Models!

32.5B

6 * 32.5B parameters * 18 trillion tokens = 3.51 × 10^24

"In terms of Qwen2.5, the language models, all models are pretrained on our latest large-scale dataset, encompassing up to 18 trillion tokens"

In the past three months since Qwen2’s release, numerous developers have built new models on the Qwen2 language models, providing us with valuable feedback. During this period, we have focused on creating smarter and more knowledgeable language models. Today, we are excited to introduce the latest addition to the Qwen family: Qwen2.5. We are announcing what might be the largest opensource release in history! Let’s get the party started! The Qwen2.5-7B model surpasses its predecessors and counte

Apache 2.0 https://huggingface.co/Qwen/Qwen2.5-32B

Xingwu Sun, Yanfeng Chen, Yiqing Huang, Ruobing Xie, Jiaqi Zhu, Kai Zhang, Shuaipeng Li, Zhen Yang, Jonny Han, Xiaobo Shu, Jiahao Bu, Zhongzhi Chen, Xuemeng Huang, Fengzong Lian, Saiyong Yang, Jianfeng Yan, Yuyuan Zeng, Xiaoqin Ren, Chao Yu, Lulu Wu, Yue Mao, Jun Xia, Tao Yang, Suncong Zheng, Kan Wu, Dian Jiao, Jinbao Xue, Xipeng Zhang, Decheng Wu, Kai Liu, Dengpeng Wu, Guanghui Xu, Shaohua Chen, Shuang Chen, Xiao Feng, Yigeng Hong, Junqiang Zheng, Chengcheng Xu, Zongwei Li, Xiong Kuang, Jianglu

Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent

"a total of 389 billion parameters and 52 billion activation parameters"

52B activated parameters 6ND = 6*52*10^9*7*10^12 = 2.184 × 10^24 They also suggest more precise formula to calculate MoE compute budget: 9.59ND + 2.3 × 10^8D = 9.59*52*10^9*7*10^12 + 2.3 × 10^8 × 7*10^12 = 3.49237×10^24 which seems closer to projected compute on Figure 3

"# Trained Tokens 7T" Table 1

In this paper, we introduce Hunyuan-Large, which is currently the largest open-source Transformer-based mixture of experts model, with a total of 389 billion parameters and 52 billion activation parameters, capable of handling up to 256K tokens. We conduct a thorough evaluation of Hunyuan-Large's superior performance across various benchmarks including language understanding and generation, logical reasoning, mathematical problem-solving, coding, long-context, and aggregated tasks, where it outp

the license doesn't regulate usage in the EU also requires additional licensing in case of massive commercial use

Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuolin Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, Wei Ping

NVLM: Open Frontier-Class Multimodal LLMs

72B

3.02e24 FLOP (Qwen2-72B compute) + 19818086000000000000000 = 3.0398181e+24

Captioning COCO [72], CC3M [127], SBU [114], LAION-115M (sanitized) [123; 66] VQA (natural image) VQAv2 [38], Visual Genome [59] Chart DVQA [51] Document Docmatix [90] OCR / Scene-Text OCR-VQA [98], COCO-Text [144], TextOCR [132], ReCTs [170], RRC-ArT [22], RRC-LSVT [134] RCTW [128], synthdog-en [57], pdfa-eng-wds [117] Math CLEVR-Math [73]

Pre-training Global batch size 2,048 Sequence length in the LLM decoder 512 Downsampling of visual tokens 1024->256 # of visual token per tile 256 # of tiles 1 # of training steps 20K 2048 * (512 + 256 * 1) * 20000 = 31,457,280,000 SFT: Global batch size 256 Sequence length in the LLM decoder 1,024 # of visual token per tile 256 # of tiles 6+1 # of training steps 20K 256 * (1,024 + 256*7) * 20000 = 14417920000 31,457,280,000 +14417920000 = 45875200000

We introduce NVLM 1.0, a family of frontier-class multimodal large language models (LLMs) that achieve state-of-the-art results on vision-language tasks, rivaling the leading proprietary models (e.g., GPT-4o) and open-access models (e.g., Llama 3-V 405B and InternVL 2). Remarkably, NVLM 1.0 shows improved text-only performance over its LLM backbone after multimodal training. In terms of model design, we perform a comprehensive comparison between decoder-only multimodal LLMs (e.g., LLaVA) and cro

6*72B*45875200000 = 1.9818086e+22

Qwen Team

Hello Qwen2

https://qwenlm.github.io/blog/qwen2/ https://arxiv.org/abs/2407.10671

72.71B parameters in total, of which 70.21B are non-embedding parameters

72 billion params, 7 trillion tokens 6 * 72 billion * 7 trillion ~= 3.02e24

"All models were pre-trained on a high-quality, large-scale dataset comprising over 7 trillion tokens, covering a wide range of domains and languages. Compared to previous editions of Qwen, Qwen2 includes a broader spectrum of linguistic data, enhancing the quantity and quality of code and mathematics content. "

"All models were pre-trained on a high-quality, large-scale dataset comprising over 7 trillion tokens, covering a wide range of domains and languages."

After months of efforts, we are pleased to announce the evolution from Qwen1.5 to Qwen2. This time, we bring to you: - Pretrained and instruction-tuned models of 5 sizes, including Qwen2-0.5B, Qwen2-1.5B, Qwen2-7B, Qwen2-57B-A14B, and Qwen2-72B; - Having been trained on data in 27 additional languages besides English and Chinese; - State-of-the-art performance in a large number of benchmark evaluations; - Significantly improved performance in coding and mathematics; - Extended context length sup

Apache 2.0

Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuolin Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, Wei Ping

NVLM: Open Frontier-Class Multimodal LLMs

72B

Uses Qwen2-72B as a backbone, which trained with 3.02e24 FLOP, as well as InternViT-6B. It's unclear how many FLOP were spent training but probably negligible; e.g. PaLI trained ViT-e with ~4B parameters using 1.07e23 FLOP. Fine-tuning FLOPs: 57,016,320,000 image/text tokens over all stages 6 * 72B * 57,016,320,000 = 2.463e22

Captioning COCO [72], CC3M [127], SBU [114], LAION-115M (sanitized) [123; 66] VQA (natural image) VQAv2 [38], Visual Genome [59] Chart DVQA [51] Document Docmatix [90] OCR / Scene-Text OCR-VQA [98], COCO-Text [144], TextOCR [132], ReCTs [170], RRC-ArT [22], RRC-LSVT [134] RCTW [128], synthdog-en [57], pdfa-eng-wds [117] Math CLEVR-Math [73]

Pre-training Global batch size 2,048 Sequence length in the LLM decoder 512 Downsampling of visual tokens 1024->256 # of visual token per tile 256 # of tiles 1 # of training steps 20K 2048 * (512 + 256 * 1) * 20000 = 31,457,280,000 SFT: Global batch size 128 Sequence length in the LLM decoder 3,200 # of visual token per tile 256 # of tiles 6+1 # of training steps 40K 128 * (3200 + 256*7) * 40000 = 25,559,040,000 31,457,280,000 + 25,559,040,000 = 57,016,320,000

We introduce NVLM 1.0, a family of frontier-class multimodal large language models (LLMs) that achieve state-of-the-art results on vision-language tasks, rivaling the leading proprietary models (e.g., GPT-4o) and open-access models (e.g., Llama 3-V 405B and InternVL 2). Remarkably, NVLM 1.0 shows improved text-only performance over its LLM backbone after multimodal training. In terms of model design, we perform a comprehensive comparison between decoder-only multimodal LLMs (e.g., LLaVA) and cro

Fine-tuning FLOPs: 57,016,320,000 image/text tokens over all stages 6 * 72B * 57,016,320,000 = 2.463e22

https://huggingface.co/nvidia/NVLM-D-72B Creative Commons Attribution: Non-Commercial 4.0 International *training code "coming soon"

Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuolin Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, Wei Ping

NVLM: Open Frontier-Class Multimodal LLMs

72B

Additional compute in this paper is negligible relative to the compute used to train the language model backbone (Qwen2-72B at 3.02e24 FLOP)