1# Pose estimation 2 3<img src="../images/pose.png" class="attempt-right" /> 4 5Pose estimation is the task of using an ML model to estimate the pose of a 6person from an image or a video by estimating the spatial locations of key body 7joints (keypoints). 8 9## Get started 10 11If you are new to TensorFlow Lite and are working with Android or iOS, explore 12the following example applications that can help you get started. 13 14<a class="button button-primary" href="https://github.com/tensorflow/examples/tree/master/lite/examples/pose_estimation/android"> 15Android example</a> 16<a class="button button-primary" href="https://github.com/tensorflow/examples/tree/master/lite/examples/pose_estimation/ios"> 17iOS example</a> 18 19If you are familiar with the 20[TensorFlow Lite APIs](https://www.tensorflow.org/api_docs/python/tf/lite), 21download the starter MoveNet pose estimation model and supporting files. 22 23<a class="button button-primary" href="https://tfhub.dev/s?q=movenet"> Download 24starter model</a> 25 26If you want to try pose estimation on a web browser, check out the 27<a href="https://storage.googleapis.com/tfjs-models/demos/pose-detection/index.html?model=movenet"> 28TensorFlow JS Demo</a>. 29 30## Model description 31 32### How it works 33 34Pose estimation refers to computer vision techniques that detect human figures 35in images and videos, so that one could determine, for example, where someone’s 36elbow shows up in an image. It is important to be aware of the fact that pose 37estimation merely estimates where key body joints are and does not recognize who 38is in an image or video. 39 40The pose estimation models takes a processed camera image as the input and 41outputs information about keypoints. The keypoints detected are indexed by a 42part ID, with a confidence score between 0.0 and 1.0. The confidence score 43indicates the probability that a keypoint exists in that position. 44 45We provides reference implementation of two TensorFlow Lite pose estimation 46models: 47 48* MoveNet: the state-of-the-art pose estimation model available in two 49 flavors: Lighting and Thunder. See a comparison between these two in the 50 section below. 51* PoseNet: the previous generation pose estimation model released in 2017. 52 53The various body joints detected by the pose estimation model are tabulated 54below: 55 56<table style="width: 30%;"> 57 <thead> 58 <tr> 59 <th>Id</th> 60 <th>Part</th> 61 </tr> 62 </thead> 63 <tbody> 64 <tr> 65 <td>0</td> 66 <td>nose</td> 67 </tr> 68 <tr> 69 <td>1</td> 70 <td>leftEye</td> 71 </tr> 72 <tr> 73 <td>2</td> 74 <td>rightEye</td> 75 </tr> 76 <tr> 77 <td>3</td> 78 <td>leftEar</td> 79 </tr> 80 <tr> 81 <td>4</td> 82 <td>rightEar</td> 83 </tr> 84 <tr> 85 <td>5</td> 86 <td>leftShoulder</td> 87 </tr> 88 <tr> 89 <td>6</td> 90 <td>rightShoulder</td> 91 </tr> 92 <tr> 93 <td>7</td> 94 <td>leftElbow</td> 95 </tr> 96 <tr> 97 <td>8</td> 98 <td>rightElbow</td> 99 </tr> 100 <tr> 101 <td>9</td> 102 <td>leftWrist</td> 103 </tr> 104 <tr> 105 <td>10</td> 106 <td>rightWrist</td> 107 </tr> 108 <tr> 109 <td>11</td> 110 <td>leftHip</td> 111 </tr> 112 <tr> 113 <td>12</td> 114 <td>rightHip</td> 115 </tr> 116 <tr> 117 <td>13</td> 118 <td>leftKnee</td> 119 </tr> 120 <tr> 121 <td>14</td> 122 <td>rightKnee</td> 123 </tr> 124 <tr> 125 <td>15</td> 126 <td>leftAnkle</td> 127 </tr> 128 <tr> 129 <td>16</td> 130 <td>rightAnkle</td> 131 </tr> 132 </tbody> 133</table> 134 135An example output is shown below: 136 137<img alt="Animation showing pose estimation" src="https://storage.googleapis.com/download.tensorflow.org/example_images/movenet_demo.gif"/> 138 139## Performance benchmarks 140 141MoveNet is available in two flavors: 142 143* MoveNet.Lightning is smaller, faster but less accurate than the Thunder 144 version. It can run in realtime on modern smartphones. 145* MoveNet.Thunder is the more accurate version but also larger and slower than 146 Lightning. It is useful for the use cases that require higher accuracy. 147 148MoveNet outperforms PoseNet on a variety of datasets, especially in images with 149fitness action images. Therefore, we recommend using MoveNet over PoseNet. 150 151Performance benchmark numbers are generated with the tool 152[described here](../../performance/measurement). Accuracy (mAP) numbers are 153measured on a subset of the [COCO dataset](https://cocodataset.org/#home) in 154which we filter and crop each image to contain only one person . 155 156<table> 157<thead> 158 <tr> 159 <th rowspan="2">Model</th> 160 <th rowspan="2">Size (MB)</th> 161 <th rowspan="2">mAP</th> 162 <th colspan="3">Latency (ms)</th> 163 </tr> 164 <tr> 165 <td>Pixel 5 - CPU 4 threads</td> 166 <td>Pixel 5 - GPU</td> 167 <td>Raspberry Pi 4 - CPU 4 threads</td> 168 </tr> 169</thead> 170<tbody> 171 <tr> 172 <td> 173 <a href="https://tfhub.dev/google/lite-model/movenet/singlepose/thunder/tflite/float16/4">MoveNet.Thunder (FP16 quantized)</a> 174 </td> 175 <td>12.6MB</td> 176 <td>72.0</td> 177 <td>155ms</td> 178 <td>45ms</td> 179 <td>594ms</td> 180 </tr> 181 <tr> 182 <td> 183 <a href="https://tfhub.dev/google/lite-model/movenet/singlepose/thunder/tflite/int8/4">MoveNet.Thunder (INT8 quantized)</a> 184 </td> 185 <td>7.1MB</td> 186 <td>68.9</td> 187 <td>100ms</td> 188 <td>52ms</td> 189 <td>251ms</td> 190 </tr> 191 <tr> 192 <td> 193 <a href="https://tfhub.dev/google/lite-model/movenet/singlepose/lightning/tflite/float16/4">MoveNet.Lightning (FP16 quantized)</a> 194 </td> 195 <td>4.8MB</td> 196 <td>63.0</td> 197 <td>60ms</td> 198 <td>25ms</td> 199 <td>186ms</td> 200 </tr> 201 <tr> 202 <td> 203 <a href="https://tfhub.dev/google/lite-model/movenet/singlepose/lightning/tflite/int8/4">MoveNet.Lightning (INT8 quantized)</a> 204 </td> 205 <td>2.9MB</td> 206 <td>57.4</td> 207 <td>52ms</td> 208 <td>28ms</td> 209 <td>95ms</td> 210 </tr> 211 <tr> 212 <td> 213 <a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/posenet_mobilenet_v1_100_257x257_multi_kpt_stripped.tflite">PoseNet(MobileNetV1 backbone, FP32)</a> 214 </td> 215 <td>13.3MB</td> 216 <td>45.6</td> 217 <td>80ms</td> 218 <td>40ms</td> 219 <td>338ms</td> 220 </tr> 221</tbody> 222</table> 223 224## Further reading and resources 225 226* Check out this 227 [blog post](https://blog.tensorflow.org/2021/08/pose-estimation-and-classification-on-edge-devices-with-MoveNet-and-TensorFlow-Lite.html) 228 to learn more about pose estimation using MoveNet and TensorFlow Lite. 229* Check out this 230 [blog post](https://blog.tensorflow.org/2021/05/next-generation-pose-detection-with-movenet-and-tensorflowjs.html) 231 to learn more about pose estimation on the web. 232* Check out this [tutorial](https://www.tensorflow.org/hub/tutorials/movenet) 233 to learn about running MoveNet on Python using a model from TensorFlow Hub. 234* Coral/EdgeTPU can make pose estimation run much faster on edge devices. See 235 [EdgeTPU-optimized models](https://coral.ai/models/pose-estimation/) for 236 more details. 237* Read the PoseNet paper [here](https://arxiv.org/abs/1803.08225) 238 239Also, check out these use cases of pose estimation. 240 241<ul> 242 <li><a href="https://vimeo.com/128375543">‘PomPom Mirror’</a></li> 243 <li><a href="https://youtu.be/I5__9hq-yas">Amazing Art Installation Turns You Into A Bird | Chris Milk "The Treachery of Sanctuary"</a></li> 244 <li><a href="https://vimeo.com/34824490">Puppet Parade - Interactive Kinect Puppets</a></li> 245 <li><a href="https://vimeo.com/2892576">Messa di Voce (Performance), Excerpts</a></li> 246 <li><a href="https://www.instagram.com/p/BbkKLiegrTR/">Augmented reality</a></li> 247 <li><a href="https://www.instagram.com/p/Bg1EgOihgyh/">Interactive animation</a></li> 248 <li><a href="https://www.runnersneed.com/expert-advice/gear-guides/gait-analysis.html">Gait analysis</a></li> 249</ul> 250
