About

Alexa

Alexa at the Science Hub is a team of MIT and Amazon researchers working together on advances in conversational AI and natural language processing. Relevant research areas include commonsense reasoning, natural language understanding, and AI democratization. By working together to advance research in these areas, the Alexa team creates intuitive voice experiences that enable people to naturally communicate with computers.

Efficient AI: Efficient training of Large Language Models

Large language models have been shown to exhibit surprisingly general-purpose language capabilities. Critical to their performance is the sheer scale at which they are trained, with continuous improvements coming from increasingly larger models. However, training such models from scratch is extremely resource-intensive. This proposal aims to investigate methods for more efficiently training large language models by leveraging smaller language models that have already been pretrained. In particular, we propose to exploit the linear algebraic structure of smaller, pretrained networks to better initialize and train the parameters of larger networks.

Yoon Kim, MIT

honeycomb shape photo of Yoon
Alexa