👀 Overview 📊 Slides @ samforeman.me/talks/openskai25/training/slides 📄 HTML version: samforeman.me/talks/openskai25/training 📑 Outline Scaling: Overview Data Parallel Training Communication Why Distributed Training? Beyond Data Parallelism Additional Parallelism Strategies Large Language Models Hands On 🚀 Scaling:...

🎯 AuroraGPT: Goals AuroraGPT: General purpose scientific LLM Broadly trained on a general corpora plus scientific {papers, texts, data} Explore pathways towards a “Scientific Assistant” model Build with international partners (RIKEN, BSC, others) Multilingual English, 日本語, French, German, Spanish Multimodal: images,...

# %matplotlib inline import matplotlib_inline matplotlib_inline.backend_inline.set_matplotlib_formats('svg') import os os.environ['COLORTERM'] = 'truecolor' import lovely_tensors as lt lt.monkey_patch() lt.set_config(color=False) # automatically detect and reload local changes to modules %load_ext autoreload...

🏖️ Shell Environment Helper function to get timestamp: tstamp() { date +"%Y-%m-%d-%H%M%S" } Load frameworks module: module load frameworks Deactivate conda environment: conda deactivate Create new conda environment: ENV_PATH="/flare/datascience/foremans/miniconda/2025-06-15" conda create --prefix "${ENV_PATH}" --y...

What When 🎨 Mixing Between Distributions While Training 2025-10-06 📊 pbs-tui: TUI for PBS Job Scheduler Monitoring 2025-09-17 🍹 BlendCorpus + TorchTitan @ ALCF 2025-09-12 🏗️ Building PyTorch 2.8 from Source on Aurora 2025-06-14 🧜‍♀️ Mermaid 2025-06-02 📰 Nice Headings 2025-06-01 🚧 Frameworks Issue with numpy > 2...

flowchart LR subgraph D["`Data`"] direction TB x("`x₀`") x1("`x₁`") x2("`x₂`") end direction LR subgraph G0["`GPU0`"] direction LR subgraph N0["`NN`"] end L0["`L0`"] end subgraph G1["`GPU1`"] direction LR subgraph N1["`NN`"] end L1["`L1`"] end subgraph G2["`GPU2`"] direction LR subgraph N2["`NN`"] end L2["`L2`"] end...

What When 🏗️ Building PyTorch 2.8 from Source on Aurora 2025-06-14 🧜‍♀️ Mermaid 2025-06-02 📰 Nice Headings 2025-06-01 No matching items CitationBibTeX citation:@online{foreman2025, author = {Foreman, Sam}, title = {06}, date = {2025-06-01}, url = {https://samforeman.me/posts/2025/06/}, langid = {en} } For...

I like headings. They help organize content and make it easier to read. Inspired by my neovim config, I wanted to recreate a similar style for headings in my website. Light Theme Dark Theme Figure 1: My neovim headings in light and dark themes. I think they turned out pretty well! Heading 1 Heading 2 Heading 3...

ALCF Incite Hackathon 2025 2025 ALCF INCITE GPU Hackathon (20-May 22, 2025) LLMs on Aurora1: 🍋 Hands-On: ezpz 🌌 Overview: AuroraGPT 🎯 AuroraGPT: Goals AuroraGPT: General purpose scientific LLM Broadly trained on a general corpora plus scientific {papers, texts, data} Explore pathways towards a “Scientific Assistant”...

📍 Currently Figure 1: Current state of LLM Pretraining. [Source] 💬 LLMs on Aurora 🍋 ezpz 🤗 transformers 🏎️ Megatron-DeepSpeed 🍋 ezpz Write once, run anywhere 🐣 Getting Started Submit interactive job: qsub -I -l select=2 -l walltime=01:00:00 \ -l filesystems=home:flare \ -A gpu_hack \ -q gpu_hack_prio Source1 the...

Something I just learned The TensorFlow that is included in the new frameworks module (aurora_nre_models_frameworks-2025.0.0) was built with numpy==1.26.4 (< 2). Unfortunately, if you then (for whatever reason) then tries to install / upgrade a package1 that has numpy in its dependencies, e.g.: python3 -m pip...

PyTorch 2.6 on Aurora ; source <(curl -L https://bit.ly/ezpz-utils) && ezpz_setup_env ; export CXX=$(which g++) ; export CC=$(which gcc) ; export REL_WITH_DEB_INFO=1 ; export USE_CUDA=0 ; export USE_ROCM=0 ; export USE_MKLDNN=1 ; export USE_MKL=1 ; export USE_ROCM=0 ; export USE_CUDNN=0 ; export USE_FBGEMM=0 ;...