• Home
  • Line#
  • Scopes#
  • Navigate#
  • Raw
  • Download
1# Integrate image embedders
2
3Image embedders allow embedding images into a high-dimensional feature vector
4representing the semantic meaning of an image, which can then be compared with
5the feature vector of other images to evaluate their semantic similarity.
6
7As opposed to
8[image search](https://www.tensorflow.org/lite/inference_with_metadata/task_library/image_searcher),
9the image embedder allows computing the similarity between images on-the-fly
10instead of searching through a predefined index built from a corpus of images.
11
12Use the Task Library `ImageEmbedder` API to deploy your custom image embedder
13into your mobile apps.
14
15## Key features of the ImageEmbedder API
16
17*   Input image processing, including rotation, resizing, and color space
18    conversion.
19
20*   Region of interest of the input image.
21
22*   Built-in utility function to compute the
23    [cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity) between
24    feature vectors.
25
26## Supported image embedder models
27
28The following models are guaranteed to be compatible with the `ImageEmbedder`
29API.
30
31*   Feature vector models from the
32    [Google Image Modules collection on TensorFlow Hub](https://tfhub.dev/google/collections/image/1).
33
34*   Custom models that meet the
35    [model compatibility requirements](#model-compatibility-requirements).
36
37## Run inference in C++
38
39```c++
40// Initialization
41ImageEmbedderOptions options:
42options.mutable_model_file_with_metadata()->set_file_name(model_path);
43options.set_l2_normalize(true);
44std::unique_ptr<ImageEmbedder> image_embedder = ImageEmbedder::CreateFromOptions(options).value();
45
46// Create input frame_buffer1 and frame_buffer_2 from your inputs `image_data1`, `image_data2`, `image_dimension1` and `image_dimension2`.
47// See more information here: tensorflow_lite_support/cc/task/vision/utils/frame_buffer_common_utils.h
48std::unique_ptr<FrameBuffer> frame_buffer1 = CreateFromRgbRawBuffer(
49      image_data1, image_dimension1);
50std::unique_ptr<FrameBuffer> frame_buffer1 = CreateFromRgbRawBuffer(
51      image_data2, image_dimension2);
52
53// Run inference on two images.
54const EmbeddingResult result_1 = image_embedder->Embed(*frame_buffer_1);
55const EmbeddingResult result_2 = image_embedder->Embed(*frame_buffer_2);
56
57// Compute cosine similarity.
58double similarity = ImageEmbedder::CosineSimilarity(
59    result_1.embeddings[0].feature_vector(),
60    result_2.embeddings[0].feature_vector());
61```
62
63See the
64[source code](https://github.com/tensorflow/tflite-support/blob/master/tensorflow_lite_support/cc/task/vision/image_embedder.h)
65for more options to configure `ImageEmbedder`.
66
67## Run inference in Python
68
69### Step 1: Install TensorFlow Lite Support Pypi package.
70
71You can install the TensorFlow Lite Support Pypi package using the following
72command:
73
74```sh
75pip install tflite-support
76```
77
78### Step 2: Using the model
79
80```python
81from tflite_support.task import vision
82
83# Initialization.
84image_embedder = vision.ImageEmbedder.create_from_file(model_path)
85
86# Run inference on two images.
87image_1 = vision.TensorImage.create_from_file('/path/to/image1.jpg')
88result_1 = image_embedder.embed(image_1)
89image_2 = vision.TensorImage.create_from_file('/path/to/image2.jpg')
90result_2 = image_embedder.embed(image_2)
91
92# Compute cosine similarity.
93feature_vector_1 = result_1.embeddings[0].feature_vector
94feature_vector_2 = result_2.embeddings[0].feature_vector
95similarity = image_embedder.cosine_similarity(
96    result_1.embeddings[0].feature_vector, result_2.embeddings[0].feature_vector)
97```
98
99See the
100[source code](https://github.com/tensorflow/tflite-support/blob/master/tensorflow_lite_support/python/task/vision/image_embedder.py)
101for more options to configure `ImageEmbedder`.
102
103## Example results
104
105Cosine similarity between normalized feature vectors return a score between -1
106and 1. Higher is better, i.e. a cosine similarity of 1 means the two vectors are
107identical.
108
109```
110Cosine similarity: 0.954312
111```
112
113Try out the simple
114[CLI demo tool for ImageEmbedder](https://github.com/tensorflow/tflite-support/tree/master/tensorflow_lite_support/examples/task/vision/desktop#imageembedder)
115with your own model and test data.
116
117## Model compatibility requirements
118
119The `ImageEmbedder` API expects a TFLite model with optional, but strongly
120recommended
121[TFLite Model Metadata](https://www.tensorflow.org/lite/models/convert/metadata).
122
123The compatible image embedder models should meet the following requirements:
124
125*   An input image tensor (kTfLiteUInt8/kTfLiteFloat32)
126
127    -   image input of size `[batch x height x width x channels]`.
128    -   batch inference is not supported (`batch` is required to be 1).
129    -   only RGB inputs are supported (`channels` is required to be 3).
130    -   if type is kTfLiteFloat32, NormalizationOptions are required to be
131        attached to the metadata for input normalization.
132
133*   At least one output tensor (kTfLiteUInt8/kTfLiteFloat32)
134
135    -   with `N` components corresponding to the `N` dimensions of the returned
136        feature vector for this output layer.
137    -   Either 2 or 4 dimensions, i.e. `[1 x N]` or `[1 x 1 x 1 x N]`.
138