Training Large Vision Models (LVMs): Benchmarking AMD vs. NVIDIA GPUs

Landing AI’s workloads for training Large Vision Models (LVMs) are computationally intensive. Looking for efficient model training options, we benchmarked AMD Instinct MI250 vs. NVIDIA’s A100, using comparable server configurations, with 1-4 GPUs: (i) MI250 128 GB, (ii) A100 SXM 80GB. We found their performance comparable, with AMD offering a slightly better price-performance tradeoff. Furthermore, our LVM training code, which we had developed in PyTorch, required no code modifications to run on either AMD or NVIDIA hardware, using, respectively, AMD’s open-source ROCm and NVIDIA’s CUDA frameworks to execute on the GPU.

Benchmark results for two practical LVM workloads, in our preferred BF16 representation, showing total training time (less is better):

Although not our preferred representation for efficient training, we also found that AMD’s MI250 was about 2x faster than A100 when training using FP32 precision.

The benchmarking for the AMD Instinct MI250 was done on the AMD Accelerator Cloud (AAC), where Landing AI has tested multiple ML vision workloads, and we plan to include more benchmark results in a future report.

Technical details: 

Large Vision Models

Landing AI trains domain-specific Large Vision Models (LVMs) for customers with hundreds of thousands or sometimes millions of unlabeled images in specific domains. LVMs are the image modality equivalent of Large Language Models (LLMs). For images, we have found the best option is to train models in a specific domain, like manufacturing, rather than use the entire diversity of images on the internet. The resulting LVMs are highly efficient in solving vision tasks in their domain.

We benchmarked training of two practical LVMs, in the histopathology and semiconductor domains. We used the Vision Transformer architecture (ViT), with the following setup:

Given the importance of data security for Landing AI’s customers, we offer training on a variety of cloud service providers and on-premises as well. Thus, it has been important to offer customers the flexibility of compute environment choices. In this blog, we benchmark LVMs on A100 and MI250 compute environments. 

Hardware configuration

For MI250, we used a server configuration with 4 MI250 GPUs. For A100 configuration, we picked the comparable A100 SXM 80GB, in a server configuration with 8 GPUs (but we benchmarked with up to 4 GPUs

Benchmark results 

For both LVMs, we ran with 1,2,4 GPU configurations, using the largest batch size (BS) that fit in GPU memory for both MI250 and A100. We used three precisions: (i) FP32, (ii) FP16, (iii) BF16. For FP16/BF16, we used mixed precision (MP), accumulating results in FP32. Our preferred representation for training is BF16, where we get a significant speedup in the training time, without loss in training performance. We report times for complete training runs, with 300 epochs. Some of the longer runs reported below were cut after a few epochs and results extrapolated to complete runs.

Histopathology LVM:

Semiconductor LVM:

For the smaller histopathology LVM, MI250 is 1.5x faster, training the model in 8 hours compared to 12 hours for A100 (BF16, 4 GPU configuration). For the larger semiconductor LVM, MI250 and A100 training times are comparable, taking about 3 days to train.

When running with FP32 precision, MI250 is 2-2.5x faster than A100. We use FP32 precision during the initial exploration of new LVM training methods and architectures

Conclusion

We believe domain specific LVMs will become indispensable in a wide range of real-world scenarios, revolutionizing industries and enhancing the efficiency and capabilities of various systems. 

If your organization has a large (100K images or more) set of images that you want to extract significant value from your data using domain-specific Large Vision Models, submit a request to Start Your LVM Journey.