mlc-llm/android at main · mlc-ai/mlc-llm · GitHub
source link: https://github.com/mlc-ai/mlc-llm/tree/main/android
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Introduction to MLC-LLM for Android
We are excited to share that we have enabled the Android support for MLC-LLM. Checkout the instruction page for instructions to download and install our Android app. Checkout the announcing blog post for the technical details throughout our process of making MLC-LLM possible for Android.
App Build Instructions
-
Install TVM Unity. We have some local changes to TVM Unity, so please try out the mlc/relax repo for now. We will migrate change back to TVM Unity soon.
git clone https://github.com/mlc-ai/relax.git --recursive cd relax mkdir build cp cmake/config.cmake build
in build/config.cmake, set
USE_OPENCL
andUSE_LLVM
as ONmake -j export TVM_HOME=$(pwd) export PYTHONPATH=$PYTHONPATH:$TVM_HOME/python
-
Get Model Weight.
Currently we support LLaMA and Vicuna.
- Get the original LLaMA weights in the HuggingFace format by following the instructions here.
- Use instructions here to get vicuna weights.
- Create a soft link to the model path under dist/models.
mkdir -p dist/models ln -s your_model_path dist/models/model_name # For example: # ln -s path/to/vicuna-v1-7b dist/models/vicuna-v1-7b
-
Build model to library.
git clone https://github.com/mlc-ai/mlc-llm.git --recursive cd mlc-llm python3 build.py --model vicuna-v1-7b --dtype float16 --target android --quantization-mode int4 --quantization-sym --quantization-storage-nbit 32 --max-seq-len 768
-
Build libraries for Android app.
cd android ./prepare_libs.sh
-
Download Android Studio, and install Android APK and NDK either inside Android Studio (recommended) or separately. Connect your Android device to your machine. Use Android Studio to open folder
android/MLCChat
as the project. In the menu bar, clickBuild - Make Project
. Once the build is finished, clickRun - Run 'app'
, and you will see the app launched on your phone.
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK