![](/style/images/good.png)
![](/style/images/bad.png)
Github GitHub - netease-youdao/EMLL: Edge Machine Learning Library
source link: https://github.com/netease-youdao/EMLL
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Edge ML Library - High-performance Compute Library for On-device Machine Learning Inference
Edge ML Library (EMLL) offers optimized basic routines like general matrix multiplications (GEMM) and quantizations, to speed up machine learning (ML) inference on ARM-based devices. EMLL supports fp32, fp16 and int8 data types. EMLL accelerates on-device NMT, ASR and OCR engines of Youdao, Inc.
Features
Performance-Oriented Design
The matrix-multiplication routines are heavily-optimized for matrix shapes common in on-device ML tasks, including "skinny" ones. The matrix-multiplication kernels are tuned for specific CPUs with a large portion of inline assembly codes.
Here are benchmarks of SGEMM on 2 machines[1]:
armv8a cortex-A35 4-thread armv8a cortex-A53 4-thread
[1].The fomular of GEMM: C[MxN] = A[MxK] B[KxN]; For each test case, the better performance in all-row-major and all-column-major situations is selected.
Facile Interface
The data and parameters are passed straightforward without wrappings. Matrices and arrays are passed with base address + dimensions. GEMM parameters seldom used in on-device inference like LDA-LDC are excluded from the interface. There is no dependency on any third-party compute libraries.
Extensibility
EMLL abstracts the core structures of CPU-based high-performance matrix multiplication algorithms and also bias/quant functions to general macros (see files under include/common), which can be applied to a variety of processors. When developing for a new architecture, a lot of coding works can be saved with these macros.
EMLL APIs
EMLL provides a series of C functions. See Usage_EN.md for details.
Type Name Parameters Matrix Multiplication data_type + "gemm" matrix_orders, addresses of matrices, M, N, K, beta, number of threads Fully-connect Layer (fp32) "fc" addresses of src/weight/bias/output, dimensions M/K/N, orders of source matrices, (number of threads) Quantization "quantize_" + "symmetric"/"asymmetric" + input_type + output_type input array, output array, (zero point), scale, size of array, input range Requantization "requantize_" + "symmetric/asymmetric" + "_XtoY" input array, output array, (zero point), output scale, size of array, input range Bias "bias" + data_type the matrix to be biased, scalar bias to all elements, vector bias along major direction, vector bias along minor direction, dimensions of the matrixSupported Architectures and Data Types
Target CPU Matrix Multiplication Bias Quantization Requantization ARMv7a 32-bit fp32 -> fp32, (u)int8 -> (u)int32 fp32, int32 fp32 -> (u)int8/(u)int16 int32 -> (u)int8/(u)int16, int16 -> (u)int8 ARMv8a 64-bit fp32 -> fp32, (u)int8 -> (u)int32, fp16 -> fp16 fp32, fp16, int32 fp32 -> (u)int8/(u)int16 int32 -> (u)int8/(u)int16, int16 -> (u)int8Supported OS: Linux & Android
Supported Compilers: GCC & Clang
Future Plan
EMLL may support on-device GPUs and NPUs in the future, with the expansion of available functions, according to business requirements.
License
Apache 2.0
Reference
Eigen: [https://eigen.tuxfamily.org]
OpenBLAS: [https://github.com/xianyi/OpenBLAS]
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK