Understanding LLMs

A hands-on collection of Jupyter notebooks exploring key concepts, architectures, and optimizations in Large Language Models (LLMs).

Notebooks

1. LLM from Scratch - Part 1

   • Build a basic LLM using PyTorch.
   • Covers data preprocessing, model architecture, training loop, and inference.

2. LLM from Scratch - Part 2

   • Enhances the previous notebook with modern techniques:
     • RMSNorm
     • Gated (SwiGLU) FFN
     • Rotary Positional Embeddings (RoPE)
   • Includes gradient accumulation and mixed precision training.

3. Rotary Positional Embeddings (RoPE)

   • Deep dive into RoPE in PyTorch.
   • Compares different RoPE implementations.