Multi-GPU Training with PyTorch: Data and Model Parallelism

The first part of this workshop will show participants how to optimize single-GPU training. We will then illustrate the necessary code changes for data-parallel and model-parallel training using multiple GPUs. Model-parallel training is essential for producing new Large Language Models (LLMs).

This workshop aims to prepare researchers to use the new H100 GPU nodes as part of Princeton Language and Intelligence.

Workshop format: Presentation, demo and hands-on

Target audience: This workshop is geared toward researchers looking to train neural networks in PyTorch using multiple GPUs.

Knowledge prerequisites: Participants should have some familiarity with training neural networks with PyTorch.

Hardware/software prerequisites: For this workshop, users must have an account on the Adroit cluster, and they should confirm that they can SSH into Adroit several hours beforehand. Request an account on Adroit: https://bit.ly/3wicSaH (VPN required if off-campus). Details on all of the above can be found in this guide (https://bit.ly/3QER9Sv).

Research Computing | View More Events

Contact the organizers