The ENCCS & NSC Megatron Bootcamp provide exciting opportunities for participants to train GPT models with NVIDIA Megatron-LM under the guidance of expert mentors from National Labs, Universities and Industry leaders in a collaborative environment.
This 3-day Bootcamp is designed to give NLP researchers a step-wised overview on the fundamentals of NVIDIA Megatron-LM ( NVIDIA open-source framework for training very large language models). The event will have many hands-on exercises complimented by tutorials, code snippets and presentations to help you kick-start training your GPT Megatron models for own language.
This 3-day Bootcamp will be hosted online in the Central European Summer Time (CEST) on October 25th to 27th. All communication will be done through Zoom, Slack and email.
Experience with training distributed deep learning models with Python and Pytorch is preferred, no GPU programming knowledge is required.
This event has limited capacity, so please make sure that prerequisites are met before applying. Participants will receive acceptance email a week before the event.
GPU Compute Resource
Teams attending the event will be given access to "Berzelius" for the duration of the hackathon. Berzelius is the premier AI/ML cluster at NSC. To learn more about Berzelius, please visit the links below for more information.
Day 1: October 25, 2021 (9:00 AM to 12:00 PM CEST)
- 09:00 AM: Welcome
- 09:00 AM - 09:15 AM: Connecting to a cluster
- 09:15 AM - 09:30 AM: Bootcamp Overview
- 09:30 AM - 09:50 AM: Quick start – docker | singularity | slurm
- 09:50 AM - 10:15 AM: Multi-nodes Megatron training
- 10:15 AM - 10:30 AM: Break
- 10:30 AM - 10:45 AM: Team-Up time
- 10:45 AM - 11:30 AM: Challenge overview
- 11:00 AM - 12:00 PM: Discussion & What’s up next day
Day 2: October 26, 2021 (9:00 AM to 1:30 PM CEST)
- 09:00 AM - 09:15 AM: Environment prep
- 09:15 AM - 09:45 AM: Introduction to Megatron
- 09:45 AM - 10:15 AM: Tutorial part 1 – Pre-requisite
- 10:15 AM - 10:45 AM: Tutorial part 2 – Megatron’s core MPU
- 10:45 AM - 11:00 AM: Break
- 11:00 AM - 11:40 AM: Tutorial part 3 – data preprocessing
- 11:40 AM - 12:00 PM: Intro to profiling
- 12:00 PM - 12:10 PM: Ask the Experts
- 12:10 PM - 12:40 PM: Tutorial part 4 – GPT config vs GPUs performance
- 12:40 PM - 01:10 PM: Challenge overview
- 01:10 PM - 01:30 PM: Discussion & What’s up next day
Day 3: October 27, 2021 (9:00 AM to 2:00 PM CEST)
- 09:00 AM - 09:15 AM: Recap and Overview of Day 3
- 09:15 AM - 09:30 AM: About acquiring your own
- 09:30 AM - 10:10 AM: Tutorial part 1 – data cleaning & filter
- 10:10 AM - 10:30 AM: Mini challenge – approaching ground-truth
- 10:30 AM - 11:00 AM: Tutorial part 2 – train your own GPT tokenizer
- 11:00 AM - 11:30 AM: Tutorial part 3 – data preprocessing
- 11:30 AM - 12:00 PM: Mini challenge – customize preprocessing script
- 12:00 PM - 12:30 PM: Break
- 12:30 PM - 12:50 PM: Tutorial part 4 – recap on Megatron model parallelism
- 12:50 PM - 01:20 PM: Challenge – Go BIG or go home
- 01:20 PM - 01:40 PM: Ask the Experts with NVIDIA data scientists on what's next for NLP
- 01:40 PM - 02:00 PM: Discussion & Final remarks
Important Event Dates
ENCCS & NSC Megatron Bootcamp Application Deadline
ENCCS & NSC Megatron Bootcamp Day 1
ENCCS & NSC Megatron Bootcamp Day 2
ENCCS & NSC Megatron Bootcamp Day 3