Orca: Ultra-Fast and Ultra-Intelligent SOTA Language Model 🚀🌌

Orca is a state-of-the-art language model that pushes the boundaries of natural language understanding and generation. Designed for high performance and efficiency, Orca is built upon advanced techniques that make it a strong contender against the likes of OpenAI's GPT-4 and PALM.

Usage

Get started:

1. Clone the repository and install the required packages.

git clone https://github.com/kyegomez/Orca
cd Orca
pip3 install -r requirements.txt
cd Orca
python3 training_distributed.py

Training

First:

Accelerate Config

Enable Deepspeed 3:

Accelerate launch train_distributed_accelerate.py