Domain Adaptive Pretraining for Model Customization with NVIDIA

This workshop is an overview of how pre-trained LLMs can be customized and applied to specific domains using domain adaptation techniques: custom tokenization and domain adapted continued pre-training with curated domain specific data, and supervised fine tuning with domain specific instructions. NVIDIA will walk through how these techniques were applied to ChipNeMo - an LLM customized for industrial chip design which was then used for building a code generation assistant, a bug summarization and analysis assistant, and an engineering chatbot assistant. Results show that these domain adaptation techniques enable significant LLM performance improvements over general-purpose base models in domain-related downstream tasks without degradation in generic capabilities.

References:

• https://research.nvidia.com/index.php/publication/2023-10_chipnemo-domain-adapted-llms-chip-design
• https://arxiv.org/abs/2311.00176

Format: Presentation with open Q&A

See the full PICSciE/RC spring training program or subscribe to the PICSciE/RC mailing list.

PICSciE/Research Computing | View More Events
Co-hosted with: GradFUTURES

Contact the organizers