xref: /frameworks/base/docs/html/training/articles/assistant.jd

page.title=Optimizing Content for the Assistant
page.metaDescription=Support contextually relevant actions through the Assist API.
page.tags=assist, accessibility, now, now on tap
meta.tags="assistant", "marshmallow", "now"
page.image=images/cards/card-assist_16-9_2x.png

page.article=true
@jd:body

<div id="tb-wrapper">
<div id="tb">
    <h2>In this document</h2>
    <ol>
      <li><a href="#assist_api">Using the Assistant</a>
      <ol>
        <li><a href="#source_app">Source app</a></li>
        <li><a href="#destination_app">Destination app</a></li>
      </ol>
      </li>
      <li><a href="#implementing_your_own_assistant">Implementing Your
      Own Assistant</a></li>
    </ol>
  </div>
</div>

<p>
  Android 6.0 Marshmallow introduces a new way for users to engage with apps
  through the assistant. The assistant is a top-level window that users can view to obtain
  contextually relevant actions for the current activity. These actions might include deep links
  to other apps on the device.</p>

<p>
  Users activate the assistant with a long press on the Home button or by saying a
  <a href="{@docRoot}reference/android/service/voice/AlwaysOnHotwordDetector.html">keyphrase</a>.
  In response, the system opens a top-level window that displays contextually
  relevant actions.
</p>

<p>
  Google App implements the assistant overlay window through a feature called
  Now on Tap, which works with the Android platform-level functionality. The system allows
  the user to select an assistant app, which obtains contextual information from your app
  using Android’s Assist API.
</p>
<p>
  This guide explains how Android apps use Android's Assist API to improve the assistant
  user experience.
<p/>
</p>


<h2 id="assist_api">Using the Assistant</h2>

<p>
  Figure 1 illustrates a typical user interaction with the assistant. When the user long-presses
  the Home button, the Assist API callbacks are invoked
  in the <em>source</em> app (step 1). The assistant renders the overlay window (steps 2 and 3),
  and then the user selects the action to perform. The assistant executes the selected action,
  such as firing an intent with a deep link to the (<em>destination</em>) restaurant app (step 4).
</p>

<div>
  <img src="{@docRoot}images/training/assistant/image01.png">
  <p class="img-caption" style="text-align:center;">
    Figure 1. Assistant interaction example with the Now on Tap feature of
    the Google App
  </p>
</div>

<p>
  Users can configure the assistant by selecting <strong>Settings > Apps > Default Apps >
  Assist &amp; voice input</strong>. Users can change system options such as accessing
  the screen contents as text and accessing a screenshot, as shown in Figure 2.
</p>

<div id="assist-input-settings" style="float:right;margin:1em;max-width:300px">
  <img src="{@docRoot}images/training/assistant/image02.png">
  <p class="img-caption" style="text-align:center;">
    Figure 2. Assist &amp; voice input settings
  </p>
</div>

<h3 id="source_app">Source app</h3>

<p>
  To ensure that your app works with the assistant as a source of information for the user,
  you need only follow <a href=
  "{@docRoot}guide/topics/ui/accessibility/apps.html">accessibility best
  practices</a>. This section describes how to provide additional information
  to help improve the assistant user experience as well as scenarios
  that need special handling, such as custom Views.
</p>
<h4 id="share_additional_information_with_the_assistant">Share additional information
 with the assistant</h4>

<p>
  In addition to the text and the screenshot, your app can share
  other information with the assistant. For example, your music
  app can choose to pass current album information so that the assistant can
  suggest smarter actions tailored to the current activity.
</p>

<p>
  To provide additional information to the assistant, your app provides
  <em>global application context</em> by registering an app listener and
  supplies activity-specific information with activity callbacks as shown in
  Figure 3:
</p>

<div>
  <img src="{@docRoot}images/training/assistant/image03.png">
  <p class="img-caption" style="text-align:center;">
    Figure 3. Assist API lifecycle sequence diagram
  </p>
</div>

<p>
  To provide global application context, the app creates an implementation of
  {@link android.app.Application.OnProvideAssistDataListener} and registers it
  using {@link
  android.app.Application#registerOnProvideAssistDataListener(android.app.Application.OnProvideAssistDataListener) registerOnProvideAssistDataListener()}.
  To provide activity-specific contextual information, the activity
  overrides {@link android.app.Activity#onProvideAssistData(android.os.Bundle) onProvideAssistData()}
  and {@link
  android.app.Activity#onProvideAssistContent(android.app.assist.AssistContent) onProvideAssistContent()}.
  The two activity methods are called <em>after</em> the optional global
  callback is invoked. Because the callbacks execute on the main thread, they should
  complete <a href="{@docRoot}training/articles/perf-anr.html">promptly</a>.
  The callbacks are invoked only when the activity is <a href=
  "{@docRoot}reference/android/app/Activity.html#ActivityLifecycle">running</a>.
</p>

<h5 id="providing_context">Providing context</h5>

<p>
  When the user activates the assistant,
  {@link android.app.Activity#onProvideAssistData(android.os.Bundle) onProvideAssistData()} is called to build a full
  {@link
  android.content.Intent#ACTION_ASSIST} Intent with all of the context of the
  current application represented as an instance of the {@link
  android.app.assist.AssistStructure}. You can override this method to place
  anything you like into the bundle to appear in the
  {@link android.content.Intent#EXTRA_ASSIST_CONTEXT} part of the assist intent.
</p>

<h5 id="describing_content">Describing content</h5>

<p>
  Your app can implement {@link
  android.app.Activity#onProvideAssistContent(android.app.assist.AssistContent) onProvideAssistContent()}
  to improve the assistant user experience by providing content-related references
  related to the current activity. You can describe the app content using the
  common vocabulary defined by <a href="https://schema.org" class="external-link">Schema.org</a>
  through a JSON-LD object. In the example below, a music app provides
  structured data to describe the music album that the user is currently
  viewing:
</p>

<pre class="prettyprint">
&commat;Override
public void onProvideAssistContent(AssistContent <strong>assistContent</strong>) {
  super.onProvideAssistContent(<strong>assistContent</strong>);

  String structuredJson = <strong>new </strong>JSONObject()
       .put(<strong>"@type"</strong>, <strong>"MusicRecording"</strong>)
       .put(<strong>"@id"</strong>, <strong>"https://example.com/music/recording"</strong>)
       .put(<strong>"name"</strong>, <strong>"Album Title"</strong>)
       .toString();

  <strong>assistContent</strong>.setStructuredData(structuredJson);
}
</pre>

<p>
 You can also improve the user experience with custom implementations of
 {@link
 android.app.Activity#onProvideAssistContent(android.app.assist.AssistContent) onProvideAssistContent()},
 which can provide the following benefits:
</p>
<ul>
  <li><a href="{@docRoot}reference/android/app/assist/AssistContent.html#setIntent(android.content.Intent)">
  Adjusts the provided content
  intent</a> to
  better reflect the top-level context of the activity.</li>
  <li><a href="{@docRoot}reference/android/app/assist/AssistContent.html#setWebUri(android.net.Uri)">
  Supplies the URI</a>
  of the displayed content.</li>
  <li>Fills in {@link
  android.app.assist.AssistContent#setClipData(android.content.ClipData) setClipData()} with additional
  content of interest that the user is currently viewing.</li>
</ul>
<p class="note">
  <strong>Note: </strong>Apps that use a custom text selection implementation likely need
  to implement {@link
  android.app.Activity#onProvideAssistContent(android.app.assist.AssistContent) onProvideAssistContent()}
  and call {@link android.app.assist.AssistContent#setClipData(android.content.ClipData) setClipData()}.
</p>

<h4 id="default_implementation">Default implementation</h4>

<p>
  If neither the {@link
  android.app.Activity#onProvideAssistData(android.os.Bundle) onProvideAssistData()} nor the {@link
  android.app.Activity#onProvideAssistContent(android.app.assist.AssistContent) onProvideAssistContent()}
  callback is implemented, the system still proceeds and passes the
  automatically collected information to the assistant unless the current
  window is flagged as <a href="#excluding_views">secure</a>.
  As shown in Figure 3, the system uses the default implementations of {@link
  android.view.View#onProvideStructure(android.view.ViewStructure) onProvideStructure()} and {@link
  android.view.View#onProvideVirtualStructure(android.view.ViewStructure) onProvideVirtualStructure()} to
  collect text and view hierarchy information. If your view implements custom
  text drawing, override {@link
  android.view.View#onProvideStructure(android.view.ViewStructure) onProvideStructure()} to provide
  the assistant with the text shown to the user by calling {@link
  android.view.ViewStructure#setText(java.lang.CharSequence) setText(CharSequence)}.
</p>

<p>
  <em>In most cases, implementing accessibility support enables the
  assistant to obtain the information it needs.</em> To implement accessibility support,
  observe the best practices described in <a href=
  "{@docRoot}guide/topics/ui/accessibility/apps.html">Making Applications
  Accessible</a>, including the following:</p>

<ul>
  <li>Provide {@link android.R.attr#contentDescription
  android:contentDescription} attributes.</li>
  <li>Populate {@link
  android.view.accessibility.AccessibilityNodeInfo} for custom views.</li>
  <li>Make
  sure that custom {@link android.view.ViewGroup ViewGroup} objects correctly
  <a href="{@docRoot}reference/android/view/ViewGroup.html#getChildAt(int)">expose</a>
  their children.</li>
</ul>

<h4 id="excluding_views">Excluding views from the assistant</h4>

<p>
  To handle sensitive information, your app can exclude the current view from the assistant
  by setting the {@link android.view.WindowManager.LayoutParams#FLAG_SECURE
  FLAG_SECURE} layout parameter of the {@link android.view.WindowManager}. You must set {@link
  android.view.WindowManager.LayoutParams#FLAG_SECURE
  FLAG_SECURE} explicitly for
  every window created by the activity, including dialogs. Your app can also use
  {@link android.view.SurfaceView#setSecure(boolean) setSecure()} to exclude
  a surface from the assistant. There is no
  global (app-level) mechanism to exclude all views from the assistant. Note
  that {@link android.view.WindowManager.LayoutParams#FLAG_SECURE
  FLAG_SECURE} does not cause the Assist API callbacks to stop
  firing. The activity that uses {@link android.view.WindowManager.LayoutParams#FLAG_SECURE
  FLAG_SECURE} can still explicitly
  provide information to the assistant using the callbacks described earlier
  this guide.
</p>

<p class="note"><strong>Note: </strong>For enterprise accounts (Android for Work),
 the administrator can disable
 the collection of assistant data for the work profile by using the {@link
 android.app.admin.DevicePolicyManager#setScreenCaptureDisabled(android.content.ComponentName, boolean)
 setScreenCaptureDisabled()} method of the {@link android.app.admin.DevicePolicyManager} API.</p>

<h4 id="voice_interactions">Voice interactions</h4>

<p>
  Assist API callbacks are also invoked upon
  <a href="{@docRoot}reference/android/service/voice/AlwaysOnHotwordDetector.html">keyphrase
  detection</a>. For more information, see the
  <a href="https://developers.google.com/voice-actions/" class="external-link">Voice
  Actions</a> documentation.
</p>

<h4 id="z-order_considerations">Z-order considerations</h4>

<p>
  The assistant uses a lightweight overlay window displayed on top of the
  current activity. Because the user can activate the assistant at any time,
  don't create permanent <a
  href="{@docRoot}reference/android/Manifest.permission.html#SYSTEM_ALERT_WINDOW">
  system alert</a> windows that interfere with the overlay window, as shown in
  Figure 4.
</p>

<div style="">
  <img src="{@docRoot}images/training/assistant/image04.png">
  <p class="img-caption" style="text-align:center;">
    Figure 4. Assist layer Z-order
  </p>
</div>

<p>
  If your app uses <a
  href="{@docRoot}reference/android/Manifest.permission.html#SYSTEM_ALERT_WINDOW">
  system alert</a> windows, remove them promptly because leaving them on the
  screen degrades the user experience.
</p>

<h3 id="destination_app">Destination app</h3>

<p>
  The assistant typically takes advantage of deep linking to find destination apps. To make your
  app a potential destination app, consider adding <a href=
  "{@docRoot}training/app-indexing/deep-linking.html">deep linking</a> support. The matching
  between the current user context and deep links or other potential actions displayed in the
  overlay window (shown in step 3 in Figure 1) is specific to the assistant’s implementation.
  For
  example, the Google App uses deep linking and <a href=
  "https://developers.google.com/app-indexing/" class="external-link">Firebase App Indexing</a> in order to
  drive traffic to destination apps.
</p>

<h2 id="implementing_your_own_assistant">Implementing Your Own Assistant </h2>

<p>
  You may wish to implement your own assistant. As shown in <a href="#assist-input-settings">Figure
  2</a>, the user can select the active assistant app. The
  assistant app must provide an implementation of {@link
  android.service.voice.VoiceInteractionSessionService} and {@link
  android.service.voice.VoiceInteractionSession} as shown in <a href=
  "https://android.googlesource.com/platform/frameworks/base/+/marshmallow-release/tests/VoiceInteraction/" class="external-link">
  this <code>VoiceInteraction</code> example</a>. It also requires the {@link
  android.Manifest.permission#BIND_VOICE_INTERACTION} permission. The assistant can then
  receive the text and view hierarchy represented as an instance of the {@link
  android.app.assist.AssistStructure} in {@link
  android.service.voice.VoiceInteractionSession#onHandleAssist(android.os.Bundle,
  android.app.assist.AssistStructure,android.app.assist.AssistContent) onHandleAssist()}.
  It receives the screenshot through {@link
  android.service.voice.VoiceInteractionSession#onHandleScreenshot(android.graphics.Bitmap)
  onHandleScreenshot()}.
</p>