Training and Inference of LLMs with PyTorch Fully Sharded Data Parallel and Better Transformer
In this blog we show how to perform efficient and optimized distributed training and inference of large language models using PyTorch’s Fully Sharded Data Parallel and Better Transformer implementations, on the Spark platform. In this implementation, we combine Microsoft Fabric for data preparation and model inference, and Azure Databricks for model training, having all our data under Microsoft Fabric’s OneLake. The code for this blog is available at this GitHub repository, as a series of PySpark notebooks for Microsoft Fabric and Azure Databricks.Accelerating Azure Databricks Runtime for Machine Learning
Data Scientists love working in the Azure Databricks environment when developing their machine learning and artificial intelligence models. By simply installing Intel-optimized versions of scikit-learn and TensorFlow as described in this blog post, they can see potentially large performance gains that will save them time and money.
