[2306.11987] Training Transformers with 4-bit Integers
source link: https://arxiv.org/abs/2306.11987
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Computer Science > Machine Learning
[Submitted on 21 Jun 2023 (v1), last revised 22 Jun 2023 (this version, v2)]
Training Transformers with 4-bit Integers
Quantizing the activation, weight, and gradient to 4-bit is promising to accelerate neural network training. However, existing 4-bit training methods require custom numerical formats which are not supported by contemporary hardware. In this work, we propose a training method for transformers with all matrix multiplications implemented with the INT4 arithmetic. Training with an ultra-low INT4 precision is challenging. To achieve this, we carefully analyze the specific structures of activation and gradients in transformers to propose dedicated quantizers for them. For forward propagation, we identify the challenge of outliers and propose a Hadamard quantizer to suppress the outliers. For backpropagation, we leverage the structural sparsity of gradients by proposing bit splitting and leverage score sampling techniques to quantize gradients accurately. Our algorithm achieves competitive accuracy on a wide range of tasks including natural language understanding, machine translation, and image classification. Unlike previous 4-bit training methods, our algorithm can be implemented on the current generation of GPUs. Our prototypical linear operator implementation is up to 2.2 times faster than the FP16 counterparts and speeds up the training by up to 35.1%.
| Comments: | 9 pages, 8 figures |
| Subjects: | Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE) |
| Cite as: | arXiv:2306.11987 [cs.LG] |
| (or arXiv:2306.11987v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2306.11987 |
Submission history
From: Haocheng Xi [view email][v1] Wed, 21 Jun 2023 02:45:01 UTC (3,822 KB)
[v2] Thu, 22 Jun 2023 20:09:02 UTC (3,822 KB)
Recommend
-
13
BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingJacob Devlin,
-
7
[Submitted on 11 Oct 2018 (v1), last revised 24 May 2019 (this version, v2)] BERT: Pre-training of Deep Bidirectional Transformers for Language Understandin...
-
4
perf: Avoid an Option in the `Map*` futures #2306
-
132
Self-Supervised Vision Transformers with DINO PyTorch implementation and pretrained models for DINO. For details, see Emerging Properties in Self-Supervised Vision Transformers. [
-
9
[Submitted on 31 May 2023] Bytes Are All You Need: Transformers Operating Directly On File Bytes
-
4
Corinna Kramer June 22, 2023 4 minute read...
-
7
Pensieve: 2306 2023-06-25 21:30 仍然是玩游戏中抽空读书, 这个月好歹读了三本. 第一本是一本漫画, Revenge of the Librarians, 基本算...
-
4
微软推出 2306 版 Windows 11虚拟机，更新“Moment 3” 作者：远洋 2023-06-26 06:50:25 WDE 2306 基于 Windows 11 企业版 SKU，版本号是 22621.1848，这是目前稳定通道中最新的版本，22621.1848 也被称为“Moment 3...
-
5
[Submitted on 16 Jun 2023 (v1), last revised 21 Jun 2023 (this version, v3)] The False Dawn: Reevaluating Google's Reinforcement...
-
2
Microsoft Outlook Microsoft 365 MSO (Version 2306 Build 16.0.16529.20100) 32-bit - Remote C...
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK