1# How to Cross-Compile ArmNN on x86_64 for arm64 2 3* [Introduction](#introduction) 4* [Cross-compiling ToolChain](#installCCT) 5* [Build and install Google's Protobuf library](#buildProtobuf) 6* [Build Caffe for x86_64](#buildCaffe) 7* [Build Boost library for arm64](#installBaarch) 8* [Build Compute Library](#buildCL) 9* [Build ArmNN](#buildANN) 10* [Run Unit Tests](#unittests) 11* [Troubleshooting and Errors](#troubleshooting) 12 13 14#### <a name="introduction">Introduction</a> 15These are the step by step instructions on Cross-Compiling ArmNN under an x86_64 system to target an Arm64 system. This build flow has been tested with Ubuntu 16.04. 16The instructions show how to build the ArmNN core library and the Boost, Protobuf, Caffe, Tensorflow, Tflite, Flatbuffer and Compute Libraries necessary for compilation. 17 18#### <a name="installCCT">Cross-compiling ToolChain</a> 19* Install the standard cross-compilation libraries for arm64: 20 ``` 21 sudo apt install crossbuild-essential-arm64 22 ``` 23 24#### <a name="buildProtobuf">Build and install Google's Protobuf library</a> 25 26## We support protobuf version 3.12.0 27* Get protobuf from here: https://github.com/protocolbuffers/protobuf : 28 ```bash 29 git clone -b v3.12.0 https://github.com/google/protobuf.git protobuf 30 cd protobuf 31 git submodule update --init --recursive 32 ./autogen 33 ``` 34* Build a native (x86_64) version of the protobuf libraries and compiler (protoc): 35 (Requires cUrl, autoconf, llibtool, and other build dependencies if not previously installed: sudo apt install curl autoconf libtool build-essential g++) 36 ``` 37 mkdir x86_64_build 38 cd x86_64_build 39 ../configure --prefix=$HOME/armnn-devenv/google/x86_64_pb_install 40 make install -j16 41 cd .. 42 ``` 43* Build the arm64 version of the protobuf libraries: 44 ``` 45 mkdir arm64_build 46 cd arm64_build 47 export CC=aarch64-linux-gnu-gcc \ 48 export CXX=aarch64-linux-gnu-g++ \ 49 ../configure --host=aarch64-linux \ 50 --prefix=$HOME/armnn-devenv/google/arm64_pb_install \ 51 --with-protoc=$HOME/armnn-devenv/google/x86_64_pb_install/bin/protoc 52 make install -j16 53 cd .. 54 ``` 55 56#### <a name="buildCaffe">Build Caffe for x86_64</a> 57* Ubuntu 16.04 installation. These steps are taken from the full Caffe installation documentation at: http://caffe.berkeleyvision.org/install_apt.html 58* Install dependencies: 59 ```bash 60 sudo apt-get install libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev 61 sudo apt-get install --no-install-recommends libboost-all-dev 62 sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev 63 sudo apt-get install libopenblas-dev 64 sudo apt-get install libatlas-base-dev 65 ``` 66* Download Caffe from: https://github.com/BVLC/caffe. We have tested using tag 1.0 67 ```bash 68 git clone https://github.com/BVLC/caffe.git 69 cd caffe 70 git checkout eeebdab16155d34ff8f5f42137da7df4d1c7eab0 71 cp Makefile.config.example Makefile.config 72 ``` 73* Adjust Makefile.config as necessary for your environment, for example: 74 ``` 75 #CPU only version: 76 CPU_ONLY := 1 77 78 #Add hdf5 and protobuf include and library directories (Replace $HOME with explicit /home/username dir): 79 INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial/ $HOME/armnn-devenv/google/x86_64_pb_install/include/ 80 LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu/hdf5/serial/ $HOME/armnn-devenv/google/x86_64_pb_install/lib/ 81 ``` 82* Setup environment: 83 ```bash 84 export PATH=$HOME/armnn-devenv/google/x86_64_pb_install/bin/:$PATH 85 export LD_LIBRARY_PATH=$HOME/armnn-devenv/google/x86_64_pb_install/lib/:$LD_LIBRARY_PATH 86 ``` 87* Compilation with Make: 88 ```bash 89 make all 90 make test 91 make runtest 92 ``` 93 These should all run without errors 94* caffe.pb.h and caffe.pb.cc will be needed when building ArmNN's Caffe Parser 95 96#### <a name="installBaarch">Build Boost library for arm64</a> 97* Build Boost library for arm64 98 Download Boost version 1.64 from http://www.boost.org/doc/libs/1_64_0/more/getting_started/unix-variants.html 99 Using any version of Boost greater than 1.64 will fail to build ArmNN, due to different dependency issues. 100 ```bash 101 tar -zxvf boost_1_64_0.tar.gz 102 cd boost_1_64_0 103 echo "using gcc : arm : aarch64-linux-gnu-g++ ;" > user_config.jam 104 ./bootstrap.sh --prefix=$HOME/armnn-devenv/boost_arm64_install 105 ./b2 install toolset=gcc-arm link=static cxxflags=-fPIC --with-test --with-log --with-program_options -j32 --user-config=user_config.jam 106 ``` 107 108#### <a name="buildCL">Build Compute Library</a> 109* Building the Arm Compute Library: 110 ```bash 111 git clone https://github.com/ARM-software/ComputeLibrary.git 112 cd ComputeLibrary/ 113 git checkout <branch_name> 114 git pull 115 scons arch=arm64-v8a neon=1 opencl=1 embed_kernels=1 extra_cxx_flags="-fPIC" -j8 internal_only=0 116 ``` 117 118 For example, if you want to checkout release branch of 20.02: 119 ```bash 120 git checkout branches/arm_compute_20_02 121 git pull 122 ``` 123 124#### <a name="buildtf">Build Tensorflow</a> 125* Building Tensorflow version 1.15: 126 ```bash 127 git clone https://github.com/tensorflow/tensorflow.git 128 cd tensorflow/ 129 git checkout 590d6eef7e91a6a7392c8ffffb7b58f2e0c8bc6b 130 ../armnn/scripts/generate_tensorflow_protobuf.sh ../tensorflow-protobuf ../google/x86_64_pb_install 131 ``` 132 133#### <a name="buildflatbuffer">Build Flatbuffer</a> 134* Building Flatbuffer version 1.12.0 135 ```bash 136 wget -O flatbuffers-1.12.0.tar.gz https://github.com/google/flatbuffers/archive/v1.12.0.tar.gz 137 tar xf flatbuffers-1.12.0.tar.gz 138 cd flatbuffers-1.12.0 139 rm -f CMakeCache.txt 140 mkdir build 141 cd build 142 cmake .. -DFLATBUFFERS_BUILD_FLATC=1 \ 143 -DCMAKE_INSTALL_PREFIX:PATH=$<DIRECTORY_PATH> \ 144 -DFLATBUFFERS_BUILD_TESTS=0 145 make all install 146 ``` 147 148* Build arm64 version of flatbuffer 149 ```bash 150 mkdir build-arm64 151 cd build-arm64 152 # Add -fPIC to allow us to use the libraries in shared objects. 153 CXXFLAGS="-fPIC" cmake .. -DCMAKE_C_COMPILER=/usr/bin/aarch64-linux-gnu-gcc \ 154 -DCMAKE_CXX_COMPILER=/usr/bin/aarch64-linux-gnu-g++ \ 155 -DFLATBUFFERS_BUILD_FLATC=1 \ 156 -DCMAKE_INSTALL_PREFIX:PATH=$<DIRECTORY_PATH> \ 157 -DFLATBUFFERS_BUILD_TESTS=0 158 make all install 159 ``` 160 161#### <a name="buildingONNX">Build Onnx</a> 162* Building Onnx 163 ```bash 164 git clone https://github.com/onnx/onnx.git 165 cd onnx 166 git fetch https://github.com/onnx/onnx.git 553df22c67bee5f0fe6599cff60f1afc6748c635 && git checkout FETCH_HEAD 167 export LD_LIBRARY_PATH=$<DIRECTORY_PATH>/protobuf-host/lib:$LD_LIBRARY_PATH 168 ../google/x86_64_pb_install/bin/protoc onnx/onnx.proto --proto_path=. --proto_path=../google/x86_64_pb_install/include --cpp_out ../onnx 169 ``` 170 171#### <a name="buildingtflite">Build TfLite</a> 172* Building TfLite 173 ```bash 174 mkdir tflite 175 cd tflite 176 cp ../tensorflow/tensorflow/lite/schema/schema.fbs . 177 ../flatbuffers-1.12.0/build/flatc -c --gen-object-api --reflect-types --reflect-names schema.fbs 178 ``` 179 180#### <a name="buildANN">Build ArmNN</a> 181* Compile ArmNN for arm64: 182 ```bash 183 git clone https://github.com/ARM-software/armnn.git 184 cd armnn 185 git checkout <branch_name> 186 git pull 187 mkdir build 188 cd build 189 ``` 190 191 For example, if you want to checkout release branch of 20.02: 192 ```bash 193 git checkout branches/armnn_20_02 194 git pull 195 ``` 196 197* Use CMake to configure your build environment, update the following script and run it from the armnn/build directory to set up the armNN build: 198 ```bash 199 #!/bin/bash 200 export CXX=aarch64-linux-gnu-g++ \ 201 export CC=aarch64-linux-gnu-gcc \ 202 cmake .. \ 203 -DARMCOMPUTE_ROOT=$HOME/armnn-devenv/ComputeLibrary \ 204 -DARMCOMPUTE_BUILD_DIR=$HOME/armnn-devenv/ComputeLibrary/build/ \ 205 -DBOOST_ROOT=$HOME/armnn-devenv/boost_arm64_install/ \ 206 -DARMCOMPUTENEON=1 -DARMCOMPUTECL=1 -DARMNNREF=1 \ 207 -DCAFFE_GENERATED_SOURCES=$HOME/armnn-devenv/caffe/build/src \ 208 -DBUILD_CAFFE_PARSER=1 \ 209 -DONNX_GENERATED_SOURCES=$HOME/onnx \ 210 -DBUILD_ONNX_PARSER=1 \ 211 -DTF_GENERATED_SOURCES=$HOME/tensorflow-protobuf \ 212 -DBUILD_TF_PARSER=1 \ 213 -DBUILD_TF_LITE_PARSER=1 \ 214 -DTF_LITE_GENERATED_PATH=$HOME/tflite \ 215 -DFLATBUFFERS_ROOT=$HOME/flatbuffers-arm64 \ 216 -DFLATC_DIR=$HOME/flatbuffers-1.12.0/build \ 217 -DPROTOBUF_ROOT=$HOME/google/x86_64_pb_install \ 218 -DPROTOBUF_ROOT=$HOME/armnn-devenv/google/x86_64_pb_install/ \ 219 -DPROTOBUF_LIBRARY_DEBUG=$HOME/armnn-devenv/google/arm64_pb_install/lib/libprotobuf.so.23.0.0 \ 220 -DPROTOBUF_LIBRARY_RELEASE=$HOME/armnn-devenv/google/arm64_pb_install/lib/libprotobuf.so.23.0.0 221 ``` 222 223* If you want to include standalone sample dynamic backend tests, add the argument to enable the tests and the dynamic backend path to the CMake command: 224 ```bash 225 -DSAMPLE_DYNAMIC_BACKEND=1 \ 226 -DDYNAMIC_BACKEND_PATHS=$SAMPLE_DYNAMIC_BACKEND_PATH 227 ``` 228* Run the build 229 ```bash 230 make -j32 231 ``` 232 233#### <a name="buildStandaloneBackend">Build Standalone Sample Dynamic Backend</a> 234* The sample dynamic backend is located in armnn/src/dynamic/sample 235 ```bash 236 mkdir build 237 cd build 238 ``` 239 240* Use CMake to configure your build environment, update the following script and run it from the armnn/src/dynamic/sample/build directory to set up the armNN build: 241 ```bash 242 #!/bin/bash 243 export CXX=aarch64-linux-gnu-g++ \ 244 export CC=aarch64-linux-gnu-gcc \ 245 cmake .. \ 246 -DCMAKE_CXX_FLAGS=--std=c++14 \ 247 -DBOOST_ROOT=$HOME/armnn-devenv/boost_arm64_install/ \ 248 -DBoost_SYSTEM_LIBRARY=$HOME/armnn-devenv/boost_arm64_install/lib/libboost_system.a \ 249 -DARMNN_PATH=$HOME/armnn-devenv/armnn/build/libarmnn.so 250 ``` 251 252* Run the build 253 ```bash 254 make 255 ``` 256 257#### <a name="unittests">Run Unit Tests</a> 258* Copy the build folder to an arm64 linux machine 259* Copy the libprotobuf.so.15.0.1 library file to the build folder 260* If you enable the standalone sample dynamic tests, also copy libArm_SampleDynamic_backend.so library file to the folder specified as $SAMPLE_DYNAMIC_BACKEND_PATH when you build ArmNN 261* cd to the build folder on your arm64 machine and set your LD_LIBRARY_PATH to its current location: 262 ``` 263 cd build/ 264 export LD_LIBRARY_PATH=`pwd` 265 ``` 266* Create a symbolic link to libprotobuf.so.23.0.0: 267 ``` 268 ln -s libprotobuf.so.23.0.0 ./libprotobuf.so.23 269 ``` 270* Run the UnitTests: 271 ``` 272 ./UnitTests 273 Running 567 test cases... 274 275 *** No errors detected 276 ``` 277#### <a name="troubleshooting">Troubleshooting and Errors:</a> 278#### Error adding symbols: File in wrong format 279* When building armNN: 280 ``` 281 /usr/local/lib/libboost_log.a: error adding symbols: File in wrong format 282 collect2: error: ld returned 1 exit status 283 CMakeFiles/armnn.dir/build.make:4028: recipe for target 'libarmnn.so' failed 284 make[2]: *** [libarmnn.so] Error 1 285 CMakeFiles/Makefile2:105: recipe for target 'CMakeFiles/armnn.dir/all' failed 286 make[1]: *** [CMakeFiles/armnn.dir/all] Error 2 287 Makefile:127: recipe for target 'all' failed 288 make: *** [all] Error 2 289 ``` 290* Boost libraries are not compiled for the correct architecture, try recompiling for arm64 291## 292#### Virtual memory exhausted 293* When compiling the boost libraries: 294 ```bash 295 virtual memory exhausted: Cannot allocate memory 296 ``` 297* Not enough memory available to compile. Increase the amount of RAM or swap space available. 298 299## 300#### Unrecognized command line option '-m64' 301* When compiling the boost libraries: 302 ```bash 303 aarch64-linux-gnu-g++: error: unrecognized command line option ‘-m64’ 304 ``` 305* Clean the boost library directory before trying to build with a different architecture: 306 ```bash 307 sudo ./b2 clean 308 ``` 309* It should show the following for arm64: 310 ```bash 311 - 32-bit : no 312 - 64-bit : yes 313 - arm : yes 314 ``` 315 316## 317#### Missing libz.so.1 318* When compiling armNN: 319 ```bash 320 /usr/lib/gcc-cross/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/bin/ld: warning: libz.so.1, needed by /home/<username>/armNN/usr/lib64/libprotobuf.so.23.0.0, not found (try using -rpath or -rpath-link) 321 ``` 322 323* Missing arm64 libraries for libz.so.1, these can be added by adding a second architecture to dpkg and explicitly installing them: 324 ```bash 325 sudo dpkg --add-architecture arm64 326 sudo apt-get install zlib1g:arm64 327 sudo apt-get update 328 sudo ldconfig 329 ``` 330* If apt-get update returns 404 errors for arm64 repos refer to section 5 below. 331* Alternatively the missing arm64 version of libz.so.1 can be downloaded and installed from a .deb package here: 332 https://launchpad.net/ubuntu/wily/arm64/zlib1g/1:1.2.8.dfsg-2ubuntu4 333 ```bash 334 sudo dpkg -i zlib1g_1.2.8.dfsg-2ubuntu4_arm64.deb 335 ``` 336## 337#### Unable to install arm64 packages after adding arm64 architecture 338* Using sudo apt-get update should add all of the required repos for arm64 but if it does not or you are getting 404 errors the following instructions can be used to add the repos manually: 339* From stackoverflow: 340https://askubuntu.com/questions/430705/how-to-use-apt-get-to-download-multi-arch-library/430718 341* Open /etc/apt/sources.list with your preferred text editor. 342 343* Mark all the current (default) repos as \[arch=<current_os_arch>], e.g. 344 ```bash 345 deb [arch=amd64] http://archive.ubuntu.com/ubuntu/ xenial main restricted 346 ``` 347* Then add the following: 348 ```bash 349 deb [arch=arm64] http://ports.ubuntu.com/ xenial main restricted 350 deb [arch=arm64] http://ports.ubuntu.com/ xenial-updates main restricted 351 deb [arch=arm64] http://ports.ubuntu.com/ xenial universe 352 deb [arch=arm64] http://ports.ubuntu.com/ xenial-updates universe 353 deb [arch=arm64] http://ports.ubuntu.com/ xenial multiverse 354 deb [arch=arm64] http://ports.ubuntu.com/ xenial-updates multiverse 355 deb [arch=arm64] http://ports.ubuntu.com/ xenial-backports main restricted universe multiverse 356 ``` 357* Update and install again: 358 ```bash 359 sudo apt-get install zlib1g:arm64 360 sudo apt-get update 361 sudo ldconfig 362 ``` 363## 364#### Undefined references to google::protobuf:: functions 365* When compiling armNN there are multiple errors of the following type: 366 ``` 367 libarmnnCaffeParser.so: undefined reference to `google::protobuf:* 368 ``` 369* Missing or out of date protobuf compilation libraries. 370 Use the command 'protoc --version' to check which version of protobuf is available (version 3.12.0 is required). 371 Follow the instructions above to install protobuf 3.12.0 372 Note this will require you to recompile Caffe for x86_64 373 374## 375#### Errors on strict-aliasing rules when compiling the Compute Library 376* When compiling the Compute Library there are multiple errors on strict-aliasing rules: 377 ``` 378 cc1plus: error: unrecognized command line option ‘-Wno-implicit-fallthrough’ [-Werror] 379 ``` 380* Add Werror=0 to the scons command: 381 ``` 382 scons arch=arm64-v8a neon=1 opencl=1 embed_kernels=1 extra_cxx_flags="-fPIC" -j8 internal_only=0 Werror=0 383 ``` 384