Embark on a journey to orchestrate thousands of GPUs to scale LLM training to the largest compute clusters today. Starting with the memory and compute anatomy of model training we then explore 5 dimensions of parallelism to distribute training efficiently. From there we dive deeper into how GPUs are designed and how specialised kernels help increase training efficiency further.
This book is a great starting point if you want to get into training ever larger models efficiently at scale!, Distributed Training, Artificial Intelligence, Large Language Models, Generative AI, High Performance Computing, Machine Learning, Nouamane Tazi| | 246PAGES