Allen Institute for AI (Ai2) has introduced OLMo 2, a new series of open-source language models designed to make advanced AI more accessible and equitable. The release underscores Ai2’s commitment to fostering innovation and collaboration within the AI community. With OLMo 2, researchers and developers can leverage powerful tools to advance various applications in natural language processing.
OLMo 2 builds upon earlier iterations by increasing the scale and sophistication of the models. The advancements reflect ongoing efforts in the open-source AI community to close the performance gap with proprietary systems. Users benefit from the improved capabilities and expanded resources provided by this new release.
Release Specifications
The OLMo 2 models are offered in 7 billion and 13 billion parameter configurations. Trained on a vast dataset comprising up to five trillion tokens, these models demonstrate competitive performance benchmarks.
“Since the release of the first OLMo in February 2024, we’ve seen rapid growth in the open language model ecosystem, and a narrowing of the performance gap between open and proprietary models,”
stated Ai2.
Technical Enhancements
Ai2’s development team introduced several innovations to improve OLMo 2’s performance. These include enhanced training stability, staged training approaches, and advanced post-training methods based on their Tülu 3 framework. Key technical upgrades involve replacing nonparametric layer norm with RMSNorm and integrating rotary positional embedding.
Community and Evaluation
Emphasizing transparency, Ai2 has made comprehensive resources available, including model weights, data, code, and instruction-tuned models. This effort facilitates thorough examination and reproducibility by the AI community. Additionally, the introduction of the OLMES evaluation system, featuring 20 benchmarks, allows for the assessment of knowledge recall, commonsense reasoning, and mathematical reasoning capabilities.
OLMo 2 sets a new standard in open-source AI development by providing robust, transparent, and high-performing language models. This release not only enhances the resources available to the AI community but also fosters greater collaboration and innovation. With its competitive performance and open-access approach, OLMo 2 is poised to significantly impact natural language processing applications and drive future advancements in the field.
