• Home
  • Line#
  • Scopes#
  • Navigate#
  • Raw
  • Download
1# Building and Running Llama 3 8B Instruct with Qualcomm AI Engine Direct Backend
2
3This tutorial demonstrates how to export Llama 3 8B Instruct for Qualcomm AI Engine Direct Backend and running the model on a Qualcomm device.
4
5## Prerequisites
6
7- Set up your ExecuTorch repo and environment if you haven’t done so by following [the Setting up ExecuTorch](../getting-started-setup.md) to set up the repo and dev environment.
8- Read [the Building and Running ExecuTorch with Qualcomm AI Engine Direct Backend page](../build-run-qualcomm-ai-engine-direct-backend.md) to understand how to export and run a model with Qualcomm AI Engine Direct Backend on Qualcomm device.
9- Follow [the README for executorch llama](https://github.com/pytorch/executorch/tree/main/examples/models/llama) to know how to run a llama model on mobile via ExecuTorch.
10- A Qualcomm device with 16GB RAM
11  - We are continuing to optimize our memory usage to ensure compatibility with lower memory devices.
12- The version of [Qualcomm AI Engine Direct SDK](https://developer.qualcomm.com/software/qualcomm-ai-engine-direct-sdk) is 2.26.0 or above.
13
14## Instructions
15
16### Step1: Prepare the checkpoint of the model and optimized matrix from [Spin Quant](https://github.com/facebookresearch/SpinQuant)
17
181. For Llama 3 tokenizer and checkpoint, please refer to https://github.com/meta-llama/llama-models/blob/main/README.md for further instructions on how to download `tokenizer.model`, `consolidated.00.pth` and `params.json`.
192. To get the optimized matrix, please refer to [SpinQuant on GitHub](https://github.com/facebookresearch/SpinQuant). You can download the optimized rotation matrices in the Quantized Models section. Please choose **LLaMA-3-8B/8B_W4A16KV16_lr_1.5_seed_0**.
20
21### Step2: Export to ExecuTorch with Qualcomm AI Engine Direct Backend
22Deploying large language models like Llama 3 on-device presents the following challenges:
23
241. The model size is too large to fit in device memory for inference.
252. High model loading and inference time.
263. Difficulty in quantization.
27
28To address these challenges, we have implemented the following solutions:
291. Using `--pt2e_quantize qnn_16a4w` to quantize activations and weights, thereby reducing the on-disk model size and alleviating memory pressure during inference.
302. Using `--num_sharding 8` to shard the model into sub-parts.
313. Performing graph transformations to convert or decompose operations into more accelerator-friendly operations.
324. Using `--optimized_rotation_path <path_to_optimized_matrix>` to apply R1 and R2 of [Spin Quant](https://github.com/facebookresearch/SpinQuant) to improve accuracy.
335. Using `--calibration_data "<|start_header_id|>system<|end_header_id|..."` to ensure that during the quantization of Llama 3 8B instruct, the calibration includes special tokens in the prompt template. For more details on the prompt template, refer to [the model card of meta llama3 instruct](https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-3/).
34
35To export Llama 3 8B instruct with the Qualcomm AI Engine Direct Backend, ensure the following:
36
371. The host machine has more than 100GB of memory (RAM + swap space).
382. The entire process takes a few hours.
39
40```bash
41# Please note that calibration_data must include the prompt template for special tokens.
42python -m examples.models.llama.export_llama -t <path_to_tokenizer.model>
43llama3/Meta-Llama-3-8B-Instruct/tokenizer.model -p <path_to_params.json> -c <path_to_checkpoint_for_Meta-Llama-3-8B-Instruct>  --use_kv_cache  --qnn --pt2e_quantize qnn_16a4w --disable_dynamic_shape --num_sharding 8 --calibration_tasks wikitext --calibration_limit 1 --calibration_seq_length 128 --optimized_rotation_path <path_to_optimized_matrix> --calibration_data "<|start_header_id|>system<|end_header_id|>\n\nYou are a funny chatbot.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nCould you tell me about Facebook?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
44```
45
46### Step3: Invoke the Runtime on an Android smartphone with Qualcomm SoCs
471. Build executorch with Qualcomm AI Engine Direct Backend for android
48    ```bash
49    cmake \
50        -DCMAKE_TOOLCHAIN_FILE="${ANDROID_NDK_ROOT}/build/cmake/android.toolchain.cmake" \
51        -DANDROID_ABI=arm64-v8a \
52        -DCMAKE_INSTALL_PREFIX=cmake-android-out \
53        -DCMAKE_BUILD_TYPE=Release \
54        -DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
55        -DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
56        -DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
57        -DEXECUTORCH_BUILD_QNN=ON \
58        -DQNN_SDK_ROOT=${QNN_SDK_ROOT} \
59        -DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
60        -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
61        -DEXECUTORCH_BUILD_KERNELS_CUSTOM=ON \
62        -Bcmake-android-out .
63
64    cmake --build cmake-android-out -j16 --target install --config Release
65    ```
662. Build llama runner for android
67```bash
68    cmake \
69        -DCMAKE_TOOLCHAIN_FILE="${ANDROID_NDK_ROOT}"/build/cmake/android.toolchain.cmake  \
70        -DANDROID_ABI=arm64-v8a \
71        -DCMAKE_INSTALL_PREFIX=cmake-android-out \
72        -DCMAKE_BUILD_TYPE=Release -DPYTHON_EXECUTABLE=python \
73        -DEXECUTORCH_BUILD_QNN=ON \
74        -DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
75        -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
76        -DEXECUTORCH_BUILD_KERNELS_CUSTOM=ON \
77        -Bcmake-android-out/examples/models/llama examples/models/llama
78
79    cmake --build cmake-android-out/examples/models/llama -j16 --config Release
80```
813. Run on Android via adb shell
82*Pre-requisite*: Make sure you enable USB debugging via developer options on your phone
83
84**3.1 Connect your android phone**
85
86**3.2 We need to push required QNN libraries to the device.**
87```bash
88# make sure you have write-permission on below path.
89DEVICE_DIR=/data/local/tmp/llama
90adb shell mkdir -p ${DEVICE_DIR}
91adb push ${QNN_SDK_ROOT}/lib/aarch64-android/libQnnHtp.so ${DEVICE_DIR}
92adb push ${QNN_SDK_ROOT}/lib/aarch64-android/libQnnSystem.so ${DEVICE_DIR}
93adb push ${QNN_SDK_ROOT}/lib/aarch64-android/libQnnHtpV69Stub.so ${DEVICE_DIR}
94adb push ${QNN_SDK_ROOT}/lib/aarch64-android/libQnnHtpV73Stub.so ${DEVICE_DIR}
95adb push ${QNN_SDK_ROOT}/lib/aarch64-android/libQnnHtpV75Stub.so ${DEVICE_DIR}
96adb push ${QNN_SDK_ROOT}/lib/hexagon-v69/unsigned/libQnnHtpV69Skel.so ${DEVICE_DIR}
97adb push ${QNN_SDK_ROOT}/lib/hexagon-v73/unsigned/libQnnHtpV73Skel.so ${DEVICE_DIR}
98adb push ${QNN_SDK_ROOT}/lib/hexagon-v75/unsigned/libQnnHtpV75Skel.so ${DEVICE_DIR}
99```
100
101**3.3 Upload model, tokenizer and llama runner binary to phone**
102```bash
103adb push <model.pte> ${DEVICE_DIR}
104adb push <tokenizer.model> ${DEVICE_DIR}
105adb push cmake-android-out/lib/libqnn_executorch_backend.so ${DEVICE_DIR}
106adb push cmake-out-android/examples/models/llama/llama_main ${DEVICE_DIR}
107```
108
109**3.4 Run model**
110```bash
111adb shell "cd ${DEVICE_DIR} && ./llama_main --model_path <model.pte> --tokenizer_path <tokenizer.model> --prompt \"<|start_header_id|>system<|end_header_id|>\n\nYou are a funny chatbot.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nCould you tell me about Facebook?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n\" --seq_len 128"
112```
113You should see the message:
114```
115<|start_header_id|>system<|end_header_id|>\n\nYou are a funny chatbot.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nCould you tell me about Facebook?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nHello! I'd be delighted to chat with you about Facebook. Facebook is a social media platform that was created in 2004 by Mark Zuckerberg and his colleagues while he was a student at Harvard University. It was initially called "Facemaker" but later changed to Facebook, which is a combination of the words "face" and "book". The platform was initially intended for people to share their thoughts and share information with their friends, but it quickly grew to become one of the
116```
117
118## What is coming?
119- Improve the performance for Llama 3 Instruct
120- Reduce the memory pressure during inference to support 12GB Qualcomm devices
121- Support more LLMs
122
123## FAQ
124
125If you encounter any issues while reproducing the tutorial, please file a github
126issue on ExecuTorch repo and tag use `#qcom_aisw` tag
127