Thanks to theidioms.com

Google Brain Trains An AI Model Almost 6x Larger Than OpenAI’s GPT-3 Model

Switch Transformers: Google Brain trains an AI model almost 6x larger than OpenAI's GPT-3 model
Blog

Google Brain Trains An AI Model Almost 6x Larger Than OpenAI’s GPT-3 Model

This week, Google Brain researchers published a paper called ‘Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity’, which claims that they have trained a language model containing more than a trillion parameters. OpenAI’s GPT-3 only has about 175 billion parameters in comparison.

Google Brain trains an AI model almost 6x larger than OpenAI's GPT-3 model

The model uses a sparsely activated technique called Switch Transformers using 32 TPU cores. The model was trained using the Colossal Clean Crawled Corpus which is an 800+ GB dataset of text scraped from Wikipedia, Reddit, and other sources. Also, this model was able to achieve a 4x-speedup over the T5-XXL model (previously largest model by Google).

You can read the paper in detail from here: Read paper.

Leave your thought here

Your email address will not be published. Required fields are marked *