Rethinking "Batch" in BatchNorm
source link: https://ppwwyyxx.com/publication/batch-norm/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Rethinking "Batch" in BatchNorm
Yuxin Wu, Justin Johnson
May 2021
Abstract
BatchNorm is a critical building block in modern convolutional neural networks. Its unique property of operating on “batches” instead of individual samples introduces significantly different behaviors from most other operations in deep learning. As a result, it leads to many hidden caveats that can negatively impact model’s performance in subtle ways. This paper thoroughly reviews such problems in visual recognition tasks, and shows that a key to address them is to rethink different choices in the concept of “batch” in BatchNorm. By presenting these caveats and their mitigations, we hope this review can help researchers use BatchNorm more effectively.
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK