In ancient times1, back in ~ 2022–2023, virtually all (production) PyTorch code was designed to run on NVIDIA GPUs. In April 2023, AMD announced day-zero support for PyTorch 2.0 within the ROCm 6.0 ecosystem, leveraging new features like TorchDynamo for performance gantt title AMD and Intel PyTorch Enablement...

I’d like to try and post more this year. Ideally these would be less-polished, more-frequent updates on what I’m thinking about / working on. Ongoing Projects AuroraGPT: Large Language Models for Scientific Applications on leadership-class supercomputers1. Additional details can be found in some of my recent talks:...

🧑🏻‍💻 About Me 🏡 samforeman.me UIUC (2015): Engineering Physics + Applied Mathematics University of Iowa (2015–2019): PhD. Physics1 ANL (2019–2022): Postdoctoral Researcher ANL (2022–Present): Assistant Computational Scientist Member of the AI/ML Group at ALCF Current Research: AuroraGPT: Foundation Models for...

📉 Simple Experiment to Compare Validation Loss Cool Down Comparison Note📑 W&B Report See W&B Report: Cooling Down Checkpoints for more details. ☃️ Cooling Down 256 Nodes of Aurora: Cooled down over last 10%: W&B Run: volcanic-blaze-4312 Explicit command: LR_DECAY_STYLE=constant \ OPT=ipex.fusedlamb \...

🧑🏻‍💻 About Me 🏡 samforeman.me UIUC (2015): Engineering Physics + Applied Mathematics University of Iowa (2015–2019): PhD. Physics1 ANL (2019–2022): Postdoctoral Researcher ANL (2022–Present): Assistant Computational Scientist Member of the AI/ML Group at ALCF Current Research: AuroraGPT: Foundation Models for...

🌐 Distributed Training 🚀 Scaling: Overview ✅ Goal: Minimize: Cost (i.e. amount of time spent training) Maximize: Performance Note📑 Note See 🤗 Performance and Scalability for more details In this talk, we will explore the intricacies of training foundation models on supercomputers. We will discuss the architecture of...

🌎 AERIS Figure 1: arXiv:2509.13523 ACM Gordon Bell Prize for Climate Modeling Finalist @ SC’25 We demonstrate a significant advancement in AI weather and climate modeling with AERIS by efficient scaling of window-based transformer models. We have performed global medium-range forecasts with performance competitive...

Motivation When training on multiple data sources or domains, it is often desirable to smoothly interpolate between two distributions rather than switching abruptly. This ensures stable optimization and avoids sudden shifts in gradient statistics. We can achieve this with an annealing schedule that gradually shifts...

👀 Scaling: Overview ✅ Goal: Minimize: Cost (i.e. amount of time spent training) Maximize: Performance Note📑 Note See 🤗 Performance and Scalability for more details 🐢 Training on a Single Device See also: Scientific AI at Scale: Distributed Training 🤗 Methods and tools for efficient training on a single GPU flowchart...

pbs-tui Figure 1: A terminal dashboard for monitoring PBS Pro schedulers 👀 Overview A terminal user interface built with Textual for monitoring PBS Pro schedulers at the Argonne Leadership Computing Facility. The dashboard surfaces job, queue, and node activity in a single view and refreshes itself automatically so...

📌 Source Repositories Things are changing quickly, so to avoid confusion, here are the exact branches used for this demo: Using: auroraGPT-ANL/torchtitan @ saforem2/blendcorpus saforem2/blendcorpus @ saforem2/reorg-imports1 🏃‍♂️ Running Clone repo: git clone https://github.com/auroraGPT-ANL/torchtitan cd torchtitan...

🎯 AuroraGPT: Goals AuroraGPT: General purpose scientific LLM Broadly trained on a general corpora plus scientific {papers, texts, data} Explore pathways towards a “Scientific Assistant” model Build with international partners (RIKEN, BSC, others) Multilingual English, 日本語, French, German, Spanish Multimodal: images,...