wirawan0/RCS-knowledge-base
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
Files
1c0f2aa57771f060ae414c0693031759f85fc85b
RCS-knowledge-base/deep-learning/20231115.FP-precision-notes.md
Wirawan Purwanto 1c0f2aa577 * Added initial notes concerning the role of floating point precision 
in deep learning applications.
2023-12-13 13:30:26 -05:00

1.0 KiB
Raw Blame History

Notes on Floating Point Precisions in Deep Learning Computations

ECCV 2020 Tutorial on Accelerating Computer Vision with Mixed Precision

https://nvlabs.github.io/eccv2020-mixed-precision-tutorial/

Topics of the tutorial:

  • Training Neural Networks with Tensor Cores
  • PyTorch Performance Tuning Guide
  • Mixed Precision Training for Conditional GANs
  • Mixed Precision Training for FAZE: Few-shot Adaptive Gaze Estimation
  • Mixed Precision Training for Video Synthesis
  • Mixed Precision Training for Convolutional Tensor-Train LSTM
  • Mixed Precision Training for 3D Medical Image Analysis

Has PDF of the slides and the videos.

Q&A:

What's the difference between FP32 and TF32 modes? FP32 cores perform scalar instructions. TF32 is a Tensor Core mode, which performs matrix instructions - they are 8-16x faster and more energy efficient. Both take FP32 as inputs. TF32 mode also rounds those inputs to TF32.