Dive into Cohere For AI’s community selection of March 2023's NLP research, featuring cutting-edge language models, unparalleled text generation, and revolutionary summarization techniques! Stay ahead, and stay informed! 🌐🧠

TL;DR:

Explore the C4AI community's top NLP research picks for March 2023. This post features an array of topics, encompassing the latest advancements in large language models and more.

As NLP enthusiasts, we know that this technology constantly pushes the boundaries of what's possible in language models and their real-world applications. That's why staying up-to-date with the latest breakthroughs and advancements is crucial. In this post, our team at Cohere has done the heavy lifting by scouring the web and consulting with our research community to bring you the most current and relevant developments in natural language processing.

From embodied multimodal language models like PaLM-E that tackle the challenge of grounding language models in real-world robotic applications to the impressive Vid2Seq architecture that leverages narrated videos for dense video captioning, researchers are working to continuously expand the horizons of natural language processing. Meanwhile, MathPrompter addresses the limitations of large language models in arithmetic reasoning, while the study of in-context-learning explores the fascinating interplay between semantic priors and input-label-mappings.

At Cohere, our goal is to make NLP technology more accessible to developers and organizations. We believe that the democratization of NLP is key to unlocking its full potential. That's why we are always looking for new community members to join us on this exciting journey. If you're passionate about NLP and want to be part of the research community that is driving the future of this technology, we would love to have you at Cohere For AI. Don't hesitate to apply and be a part of this exciting journey.

Top Papers of March 2023 Highlighted by Our Research Discord Community

These papers were highlighted by C4AI research discord community members. Big thank you to MajorMelancholy#1836, hails#6601, bun#9632, bharat#0287, Antonio J.#4677, bess#8732, Ujan#3046, and the rest of the Cohere For AI NLP research community for participating.

PaLM-E: An Embodied Multimodal Language Model

Authors: Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, Pete Florence

In this paper, the authors tackle the challenge of grounding in large language models for real-world applications, such as robotics problems, by introducing embodied language models. These novel models incorporate continuous sensor modalities from the real world, such as visual and state estimation inputs, directly into language models, creating a connection between words and percepts. By training these encodings end-to-end alongside a pretrained large language model, the authors demonstrate the effectiveness of their approach across various embodied tasks, including sequential robotic manipulation planning, visual question answering, and captioning.

Meet PaLM-E, a single large embodied multimodal model that can handle a wide range of embodied reasoning tasks across multiple observation modalities and embodiments. Not only does PaLM-E showcase the power of diverse joint training across language, vision, and visual-language domains, but it also exhibits positive transfer, improving its performance as it learns from different tasks. The largest model, PaLM-E-562B, boasts 562 billion parameters and sets the bar high with state-of-the-art performance on OK-VQA, all while maintaining its generalist language capabilities as it scales.

MathPrompter: Mathematical Reasoning using Large Language Models

Authors: Shima Imani, Liang Du, Harsh Shrivastava

In this paper, the authors address the challenge of limited performance in large language models (LLMs) when solving arithmetic reasoning tasks. LLMs often provide incorrect answers to math problems, which typically have a single correct answer, unlike natural language understanding. Also, these models fail to indicate their level of confidence in their responses, leading to a trust deficit and impeding their adoption. To tackle this problem, the authors propose MathPrompter, a technique that not only enhances the performance of LLMs on arithmetic problems but also increases the trust in their predictions.

MathPrompter utilizes the zero-shot chain-of-thought prompting technique to generate multiple algebraic expressions or Python functions, solving the same math problem in various ways and ultimately boosting the confidence level in the output results. This approach differs from other prompt-based chain-of-thought methods that lack checks on the validity of intermediate steps. The authors demonstrate the effectiveness of their technique by achieving a significant improvement in performance on the MultiArith dataset (78.7% to 92.5%), evaluated using a 175 billion-parameter GPT-based LLM.

Larger Language Models Do In-Context Learning Differently

Authors: Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, Tengyu Ma

In this paper, the authors delve into the fascinating world of in-context learning (ICL) in language models, studying the effects of semantic priors versus input-label mappings. They explore two setups: ICL with flipped labels and ICL with semantically-unrelated labels, using various model families such as GPT-3, InstructGPT, Codex, PaLM, and Flan-PaLM. Their experiments on ICL with flipped labels reveal that overriding semantic priors is an emergent ability tied to model scale. While smaller language models rely mostly on semantic priors from pretraining and ignore flipped labels presented in-context, larger models can override these priors when faced with contradicting in-context exemplars.

The authors then investigate semantically-unrelated label ICL (SUL-ICL), where labels bear no semantic relation to their inputs, forcing language models to learn input-label mappings from in-context exemplars. They discover that the ability to perform SUL-ICL is also primarily dependent on scale, with larger models even capable of linear classification in an SUL-ICL setting. Lastly, they evaluate instruction-tuned models and find that instruction tuning enhances both the utilization of semantic priors and the capacity to learn input-label mappings, though the former sees a more significant improvement.

High-Throughput Generative Inference of Large Language Models with a Single GPU

Authors: Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y. Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E. Gonzalez, Percy Liang, Christopher Ré, Ion Stoica, Ce Zhang

The race to efficiently run large language models (LLMs) on limited resources, such as a single commodity GPU, is fueled by the growing demand for latency-insensitive tasks with batched processing. In this paper, the authors introduce FlexGen, a high-throughput generation engine designed to operate LLMs with limited GPU memory. FlexGen's unique ability to be flexibly configured under various hardware resource constraints allows it to aggregate memory and computation from the GPU, CPU, and disk. By using a linear programming optimizer, FlexGen searches for efficient patterns to store and access tensors, while further compressing weights and attention cache to 4 bits with negligible accuracy loss.

These techniques expand FlexGen's range of batch size options and significantly enhance maximum throughput. When running OPT-175B on a single 16GB GPU, FlexGen outperforms state-of-the-art offloading systems, achieving a generation throughput of 1 token/s for the first time with an effective batch size of 144. On the HELM benchmark, FlexGen successfully benchmarks a 30B model with a 16GB GPU on 7 representative sub-scenarios in just 21 hours. The exciting advances demonstrated by FlexGen bring us one step closer to harnessing the power of LLMs on more accessible hardware, making it a game-changer for researchers and developers alike.

Language Is Not All You Need: Aligning Perception with Language Models

Authors: Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei

In this paper, the authors present Kosmos-1, a groundbreaking Multimodal Large Language Model (MLLM) that converges language, multimodal perception, action, and world modeling, taking a significant stride towards artificial general intelligence. Kosmos-1 is capable of perceiving general modalities, learning in context through few-shot learning, and following instructions using zero-shot learning. The model is trained from scratch on web-scale multimodal corpora, incorporating arbitrarily interleaved text and images, image-caption pairs, and text data.

Experimental results reveal that Kosmos-1 demonstrates remarkable performance in language understanding, generation, OCR-free NLP, perception-language tasks like multimodal dialogue, image captioning, visual question answering, and vision tasks such as image recognition with descriptions. The authors also show that MLLMs can benefit from cross-modal transfer, enabling the transfer of knowledge between language and multimodal domains. Additionally, the researchers introduce a dataset based on the Raven IQ test, which assesses the nonverbal reasoning capabilities of MLLMs, providing valuable insight into the model's ability to reason beyond language.

Simfluence: Modeling the Influence of Individual Training Examples by Simulating Training Runs

Authors: Kelvin Guu, Albert Webson, Ellie Pavlick, Lucas Dixon, Ian Tenney, Tolga Bolukbasi

In this paper, the authors introduce Simfluence, a groundbreaking paradigm for Training Data Attribution (TDA) that moves beyond the simplifying assumption of additivity in influence scores. Instead of assigning a single influence score per example, Simfluence focuses on developing a training run simulator. This innovative approach allows users to explore counterfactual questions about their model's learning under different training curricula and observe when and where the learning occurs during the training process.

The authors present Simfluence-Linear, a simulator capable of capturing non-additive interactions and predicting the trajectory of individual example losses with impressive accuracy. They also demonstrate that existing TDA methods, such as TracIn and influence functions, can be considered special cases of Simfluence-Linear. This insight allows for direct comparison of methods in terms of simulation accuracy and encompasses several previous TDA evaluation approaches. Through experiments on large language model (LLM) fine-tuning, the authors show that their method significantly outperforms existing TDA methods in predicting loss trajectories, doubling Spearman's correlation and reducing mean-squared error by 75% across various tasks, models, and training methods.

The Quantization Model of Neural Scaling

Authors: Eric J. Michaud, Ziming Liu, Uzay Girit, Max Tegmark

In this paper, the authors introduce the Quantization Model of neural scaling laws, a novel approach that not only elucidates the power law dropoff of loss observed with model and data size but also accounts for the sudden emergence of new capabilities as models scale up. The foundation of this model lies in the Quantization Hypothesis, which posits that learned network capabilities are compartmentalized into discrete chunks, or quanta. The authors demonstrate that when quanta are learned in descending order of usage frequency, a power law in usage frequencies can effectively explain the observed power law scaling of loss.

To validate their groundbreaking theory, the researchers first test it on toy datasets before delving into the complexities of large language models. By examining language model internals, they auto-discover a diverse array of model capabilities (quanta) and find preliminary evidence that the distribution over corresponding subproblems in natural text prediction aligns with the power law predicted by the neural scaling exponent. This fascinating discovery lends support to the authors' innovative Quantization Model, which has the potential to reshape our understanding of neural scaling laws and the future development of large-scale models.

Scaling Expert Language Models with Unsupervised Domain Discovery

Authors: Suchin Gururangan, Margaret Li, Mike Lewis, Weijia Shi, Tim Althoff, Noah A. Smith, Luke Zettlemoyer

In this paper, the authors present a groundbreaking method for asynchronously training large, sparse language models on arbitrary text corpora. This innovative approach involves clustering a corpus into sets of related documents, training a separate expert language model on each cluster, and combining them in a sparse ensemble for inference. Not only does this technique generalize embarrassingly parallel training by autonomously identifying the domains for each expert, but it also virtually eliminates the communication overhead associated with existing sparse language models.

The authors' method consistently outperforms dense baselines on multiple corpora and few-shot tasks, with their analysis revealing that specializing experts to meaningful clusters is crucial for achieving these gains. Furthermore, performance continues to improve as the number of experts and the size of training data increase, suggesting that this novel approach offers an efficient and accessible way to train large language models. This paradigm-shifting technique promises to be a game-changer in the field of NLP, opening the door to even more powerful and efficient models in the future.

The Nordic Pile: A 1.2TB Nordic Dataset for Language Modeling

Authors: Joey Öhman, Severine Verlinden, Ariel Ekgren, Amaru Cuba Gyllensten, Tim Isbister, Evangelia Gogoulou, Fredrik Carlsson, Magnus Sahlgren

In this paper, the authors tackle the challenge of building Large Language Models (LLMs) for smaller languages, specifically focusing on the Nordic languages where text corpora availability is limited. To facilitate the development of LLMs in Danish, Icelandic, Norwegian, and Swedish, the researchers curate an impressive high-quality dataset consisting of 1.2TB of text, along with some high-quality English data. The paper meticulously details the team's considerations and processes for collecting, cleaning, and filtering the dataset.

This groundbreaking work sets the stage for a new era of NLP advancements in the underrepresented Nordic languages. By creating a comprehensive and high-quality dataset, the authors have opened the door to the development of powerful LLMs for these languages, which will significantly contribute to closing the gap in language technology across various linguistic communities. This resource is poised to become a valuable asset for NLP researchers and practitioners working with North Germanic languages, driving innovation and language model development in these lesser-studied linguistic domains.

Vid2seq: A Pretrained Visual Language Model for Describing Multi-Event Videos

Authors: Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, Cordelia Schmid

In this paper, the authors introduce Vid2Seq, a cutting-edge multi-modal single-stage dense event captioning model that's pretrained on narrated videos – a resource that's abundant and readily available. The Vid2Seq architecture enhances a language model with special time tokens, enabling it to simultaneously predict event boundaries and textual descriptions within the same output sequence. Since a unified model like this demands extensive training data not found in current annotated datasets, the authors demonstrate the possibility of harnessing unlabeled narrated videos for dense video captioning by cleverly redefining sentence boundaries of transcribed speech as pseudo-event boundaries and using transcribed speech sentences as pseudo-event captions.

The resulting Vid2Seq model, pretrained on the YT-Temporal-1B dataset, surpasses state-of-the-art in a variety of dense video captioning benchmarks, including YouCook2, ViTT, and ActivityNet Captions. Moreover, Vid2Seq exhibits remarkable generalization capabilities in video paragraph captioning, video clip captioning, and few-shot settings. With the authors making their code publicly accessible, Vid2Seq is poised to revolutionize the field of video captioning, paving the way for more advanced and efficient models.

Final Thoughts

Are you ready to revolutionize the way you work with large volumes of text? Look no further than incorporating large language models into your workflow. This list of cutting-edge research on NLP serves as your guide to unlocking the full potential of this powerful technology. But don't just take our word for it—experiment and tweak to find the perfect model for your specific needs. And the journey doesn't have to be a solitary one—join our Discord community to share your discoveries and collaborate with like-minded individuals. Ready to dive in? Try out our NLP API on the Cohere playground and start building the future of natural language processing today.