Today we walk you through the fascinating world of upcoming text-generating rivals
Analysis Text-generating language models are difficult to control. These systems have no sense of morality: they can spew hate speech and misinformation. Despite this, numerous companies believe this kind of software is good enough to sell. Are these organizations, and the wider world, ready for it? OpenAI launched its powerful GPT-3 to the masses in 2020; it also has an exclusive licensing deal with Microsoft. The upshot of this is that developers no longer have to be machine-learning gurus to create products that feature natural language processing. All the hard work of building, training, and running a massive neural network has been done for them, and is neatly packaged behind the GPT-3 API. Last year, two startups released their own proprietary text-generation APIs. AI21 Labs, based in Israel, launched its 178-billion-parameter Jurassic-1 in August 2021, and Cohere, headquartered in Canada, released a range of models nicknamed small, medium, and large, three months later. Now, Cohere has an extremely large -sized system, which is right now only available to beta testers. Cohere hasn’t disclosed how many parameters its models contain. For comparison, OpenAI’s GPT-3 has 175 billion parameters. Aidan Gomez, co-founder and CEO of Cohere, said he toyed with the idea of launching a generative language model startup before GPT-3 was announced. He was part of the team at Google Brain, which came up with the transformer-based architecture at the heart of these systems. Gomez argued there are benefits to having a few centralized, powerful text-generation systems as opposed to a sprawl of individual deployments. “We really shouldn’t have a world where every single company is training their own GPT-3, it would be massively environmentally costly, compute costly, and we should be trying to share resources as much as possible,” Gomez told The Register. “I saw the opportunity for an independent player to come out and to basically centralize the cost of pre-training these massive models and then open up access and amortize those costs across a huge number of users. By reducing the cost you make it accessible to more people.” Starting a language model company that can compete with the likes of OpenAI is a tall order because the barrier to entry is so high. New ventures must come armed with deep pockets to pay for the huge amount of computational resources required to train and run these models, and hire experts in cutting-edge research and machine-learning engineering. Cohere raised $40m in its series-A funding round, and just announced $125m in series-B funding this month, while AI21 Labs has collected $54.5m over four rounds of funding. OpenAI secured $250m in its latest round, technically its series A. Each startup has partnered with a different company to provide cloud computing. Cohere has entered a multi-year contract with Google. OpenAI and AI21 Labs are supported by Microsoft and AWS, respectively. “Training these large models is always expensive,” Yoav Shoham, co-CEO of AI21 Labs and a retired Stanford computer-science professor, told The Register. “If you’re not smart enough, you can easily run into tens of millions of dollars if you’re not careful. You need to make sure that you know unit economics so that you don’t lose money on every customer and only make it up in volume.” AI21 Labs and Cohere are also choosy about the customers they onboard.
