1<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" 2"http://www.w3.org/TR/html4/loose.dtd"> 3<html> 4 5<head> 6<title>OpenSL ES for Android</title> 7</head> 8 9<body> 10 11<h1>OpenSL ES for Android</h1> 12 13This article describes the Android native audio APIs based on the 14Khronos Group OpenSL ES™ 1.0.1 standard, as of Android API level 9 (Android 15version 2.3). 16<p> 17OpenSL ES provides a C language interface that is also callable from C++, and 18exposes features similar to the audio portions of these Android APIs 19callable from Java programming language code: 20<ul> 21<li><a href="http://developer.android.com/reference/android/media/MediaPlayer.html"> 22android.media.MediaPlayer</a> 23<li><a href="http://developer.android.com/reference/android/media/MediaRecorder.html"> 24android.media.MediaRecorder</a> 25</ul> 26 27As with all of the Android Native Development Kit (NDK), the primary 28purpose of OpenSL ES for Android is to facilitate the implementation 29of shared libraries to be called from Java programming language code via Java Native 30Interface (JNI). NDK is not intended for writing pure C/C++ 31applications. That said, OpenSL ES is a full-featured API, and we 32expect that you should be able to accomplish most of your audio 33needs using only this API, without up-calls to Java. 34 35<p> 36Note: though based on OpenSL ES, Android native audio at API level 9 37is <i>not</i> a conforming implementation of any OpenSL ES 1.0.1 38profile (game, music, or phone). This is because Android does not 39implement all of the features required by any one of the profiles. 40Any known cases where Android behaves differently than the specification 41are described in section "Android extensions" below. 42 43<h2>Getting started</h2> 44 45<h3>Example code</h3> 46 47<h4>Recommended</h4> 48 49Supported and tested example code, usable as a model 50for your own code, is located in NDK folder 51<code>platforms/android-9/samples/native-audio/</code>. 52 53<h4>Not recommended</h4> 54 55The OpenSL ES 1.0.1 specification contains example code in the 56appendices (see section "References" below for the link to this 57specification). However, the examples in Appendix B: Sample Code 58and Appendix C: Use Case Sample Code use features 59not supported by Android API level 9. Some examples also contain 60typographical errors, or use APIs that are likely to change. 61Proceed with caution in referring to these; 62though the code may be helpful in understanding the full OpenSL ES 63standard, it should not be used as is with Android. 64 65<h3>Adding OpenSL ES to your application source code</h3> 66 67OpenSL ES is a C API, but is callable from both C and C++ code. 68<p> 69At a minimum, add the following line to your code: 70<pre> 71#include <SLES/OpenSLES.h> 72</pre> 73 74If you use Android extensions, also include these headers: 75<pre> 76#include <SLES/OpenSLES_Android.h> 77#include <SLES/OpenSLES_AndroidConfiguration.h> 78</pre> 79 80<h3>Makefile</h3> 81 82Modify your Android.mk as follows: 83<pre> 84LOCAL_LDLIBS += libOpenSLES 85</pre> 86 87<h3>Audio content</h3> 88 89There are many ways to package audio content for your 90application, including: 91 92<dl> 93 94<dt>Resources</dt> 95<dd> 96By placing your audio files into the <code>res/raw/</code> folder, 97they can be accessed easily by the associated APIs for 98<a href="http://developer.android.com/reference/android/content/res/Resources.html"> 99Resources</a>. However there is no direct native access to resources, 100so you will need to write Java programming language code to copy them out before use. 101</dd> 102 103<dt>Assets</dt> 104<dd> 105By placing your audio files into the <code>assets/</code> folder, 106they will be directly accessible by the Android native asset manager 107APIs. See the header files <code>android/asset_manager.h</code> 108and <code>android/asset_manager_jni.h</code> for more information 109on these APIs, which are new for API level 9. The example code 110located in NDK folder 111<code>platforms/android-9/samples/native-audio/</code> uses these 112native asset manager APIs in conjunction with the Android file 113descriptor data locator. 114</dd> 115 116<dt>Network</dt> 117<dd> 118You can use the URI data locator to play audio content directly from the 119network. However, be sure to read section "Security and permissions" below. 120</dd> 121 122<dt>Local filesystem</dt> 123<dd> 124The URI data locator supports the <code>file:</code> scheme for local files, 125provided the files are accessible by the application. 126Note that the Android security framework restricts file access via 127the Linux user ID and group ID mechanism. 128</dd> 129 130<dt>Recorded</dt> 131<dd>Your application can record audio data from the microphone input, 132store this content, and then play it back later. 133The example code uses this method for the "Playback" clip. 134</dd> 135 136<dt>Compiled and linked inline</dt> 137<dd> 138You can link your audio content directly into the shared library, 139and then play it using an audio player with buffer queue data locator. This is most 140suitable for short PCM format clips. The example code uses this 141technique for the "Hello" and "Android" clips. The PCM data was 142converted to hex strings using a <code>bin2c</code> tool (not supplied). 143</dd> 144 145<dt>Synthesis</dt> 146<dd> 147Your application can synthesize PCM data on the fly and then play it 148using an audio player with buffer queue data locator. This is a 149relatively advanced technique, and the details of audio synthesis 150are beyond the scope of this article. 151</dd> 152 153</dl> 154 155Finding or creating useful audio content for your application is 156beyond the scope of this article, but see the "References" section 157below for some suggested web search terms. 158<p> 159Note that it is your responsibility to ensure that you are legally 160permitted to play or record content, and that there may be privacy 161considerations for recording content. 162 163<h3>Debugging</h3> 164 165For robustness, we recommend that you examine the <code>SLresult</code> 166value which is returned by most APIs. Use of <code>assert</code> 167vs. more advanced error handling logic is a matter of coding style 168and the particular API; see the Wikipedia article on 169<a href="http://en.wikipedia.org/wiki/Assertion_(computing)">assert</a> 170for more information. In the supplied example, we have used <code>assert</code> 171for "impossible" conditions which would indicate a coding error, and 172explicit error handling for others which are more likely to occur 173in production. 174<p> 175Many API errors result in a log entry, in addition to the non-zero 176result code. These log entries provide additional detail which can 177be especially useful for the more complex APIs such as 178<code>Engine::CreateAudioPlayer</code>. 179<p> 180Use <a href="http://developer.android.com/guide/developing/tools/adb.html"> 181adb logcat</a>, the 182<a href="http://developer.android.com/guide/developing/eclipse-adt.html"> 183Eclipse ADT plugin</a> LogCat pane, or 184<a href="http://developer.android.com/guide/developing/tools/ddms.html#logcat"> 185ddms logcat</a> to see the log. 186 187<h2>Supported features from OpenSL ES 1.0.1</h2> 188 189This section summarizes available features in this API level. In some 190cases, there are limitations which are described in the next 191sub-section. 192 193<h3>Global entry points</h3> 194 195Supported global entry points: 196<ul> 197<li><code>slCreateEngine</code> 198<li><code>slQueryNumSupportedEngineInterfaces</code> 199<li><code>slQuerySupportedEngineInterfaces</code> 200</ul> 201 202<h3>Objects and interfaces</h3> 203 204The following figure indicates objects and interfaces supported by 205Android's OpenSL ES implementation. A green cell means the feature 206is supported. 207 208<p> 209<img src="chart1.png" alt="Supported objects and interfaces"> 210 211<h3>Limitations</h3> 212 213This section details limitations with respect to the supported 214objects and interfaces from the previous section. 215 216<h4>Buffer queue data locator</h4> 217 218An audio player or recorder with buffer queue data locator supports 219PCM data format only. 220 221<h4>Device data locator</h4> 222 223The only supported use of an I/O device data locator is when it is 224specified as the data source for <code>Engine::CreateAudioRecorder</code>. 225It should be initialized using these values, as shown in the example: 226<pre> 227SLDataLocator_IODevice loc_dev = 228 {SL_DATALOCATOR_IODEVICE, SL_IODEVICE_AUDIOINPUT, 229 SL_DEFAULTDEVICEID_AUDIOINPUT, NULL}; 230</pre> 231 232<h4>Dynamic interface management</h4> 233 234<code>RemoveInterface</code> and <code>ResumeInterface</code> are not supported. 235 236<h4>Effect combinations</h4> 237 238It is meaningless to have both environmental reverb and preset 239reverb on the same output mix. 240<p> 241The platform may ignore effect requests if it estimates that the 242CPU load would be too high. 243 244<h4>Effect send</h4> 245 246<code>SetSendLevel</code> supports a single send level per audio player. 247 248<h4>Environmental reverb</h4> 249 250Environmental reverb does not support the <code>reflectionsDelay</code>, 251<code>reflectionsLevel</code>, or <code>reverbDelay</code> fields of 252<code>struct SLEnvironmentalReverbSettings</code>. 253 254<h4>MIME data format</h4> 255 256The MIME data format can be used with URI data locator only, and only 257for player (not recorder). 258<p> 259The Android implementation of OpenSL ES requires that <code>mimeType</code> 260be initialized to either <code>NULL</code> or a valid UTF-8 string, 261and that <code>containerType</code> be initialized to a valid value. 262In the absence of other considerations, such as portability to other 263implementations, or content format which cannot be identified by header, 264we recommend that you 265set the <code>mimeType</code> to <code>NULL</code> and <code>containerType</code> 266to <code>SL_CONTAINERTYPE_UNSPECIFIED</code>. 267<p> 268Supported formats include WAV PCM, WAV alaw, WAV ulaw, MP3, Ogg 269Vorbis, AAC LC, HE-AACv1 (aacPlus), HE-AACv2 (enhanced aacPlus), 270and AMR [provided these are supported by the overall platform, 271and AAC formats must be located within an MP4 container]. 272MIDI is not supported. 273WMA is not part of the open source release, and compatibility 274with Android OpenSL ES has not been verified. 275<p> 276The Android implementation of OpenSL ES does not support direct 277playback of DRM or encrypted content; if you want to play this, you 278will need to convert to cleartext in your application before playing, 279and enforce any DRM restrictions in your application. 280 281<h4>Object</h4> 282 283<code>Resume</code>, <code>RegisterCallback</code>, 284<code>AbortAsyncOperation</code>, <code>SetPriority</code>, 285<code>GetPriority</code>, and <code>SetLossOfControlInterfaces</code> 286are not supported. 287 288<h4>PCM data format</h4> 289 290The PCM data format can be used with buffer queues only. Supported PCM 291playback configurations are 8-bit unsigned or 16-bit signed, mono 292or stereo, little endian byte ordering, and these sample rates: 2938000, 11025, 12000, 16000, 22050, 24000, 32000, 44100, or 48000 Hz. 294For recording, the supported configurations are device-dependent, 295however generally 16000 Hz mono 16-bit signed is usually available. 296<p> 297Note that the field <code>samplesPerSec</code> is actually in 298units of milliHz, despite the misleading name. To avoid accidentally 299using the wrong value, you should initialize this field using one 300of the symbolic constants defined for this purpose (such as 301<code>SL_SAMPLINGRATE_44_1</code> etc.) 302 303<h4>Playback rate</h4> 304 305A single playback rate range from 500 per mille to 2000 per mille 306inclusive is supported, with property 307<code>SL_RATEPROP_NOPITCHCORAUDIO</code>. 308 309<h4>Record</h4> 310 311The <code>SL_RECORDEVENT_HEADATLIMIT</code> and 312<code>SL_RECORDEVENT_HEADMOVING</code> events are not supported. 313 314<h4>Seek</h4> 315 316<code>SetLoop</code> enables whole file looping. The <code>startPos</code> 317and <code>endPos</code> parameters are ignored. 318 319<h4>URI data locator</h4> 320 321The URI data locator can be used with MIME data format only, and 322only for an audio player (not audio recorder). Supported schemes 323are <code>http:</code> and <code>file:</code>. 324A missing scheme defaults to the <code>file:</code> scheme. Other 325schemes such as <code>https:</code>, <code>ftp:</code>, and 326<code>content:</code> are not supported. 327<code>rtsp:</code> is not verified. 328 329<h3>Data structures</h3> 330 331Android API level 9 supports these OpenSL ES 1.0.1 data structures: 332<ul> 333<li>SLDataFormat_MIME 334<li>SLDataFormat_PCM 335<li>SLDataLocator_BufferQueue 336<li>SLDataLocator_IODevice 337<li>SLDataLocator_OutputMix 338<li>SLDataLocator_URI 339<li>SLDataSink 340<li>SLDataSource 341<li>SLEngineOption 342<li>SLEnvironmentalReverbSettings 343<li>SLInterfaceID 344</ul> 345 346<h3>Platform configuration</h3> 347 348OpenSL ES for Android is designed for multi-threaded applications, 349and is thread-safe. 350<p> 351OpenSL ES for Android supports a single engine per application, and 352up to 32 objects. Available device memory and CPU may further 353restrict the usable number of objects. 354<p> 355<code>slCreateEngine</code> recognizes, but ignores, these engine options: 356<ul> 357<li><code>SL_ENGINEOPTION_THREADSAFE</code> 358<li><code>SL_ENGINEOPTION_LOSSOFCONTROL</code> 359</ul> 360 361<h2>Planning for future versions of OpenSL ES</h2> 362 363The Android native audio APIs at level 9 are based on Khronos 364Group OpenSL ES 1.0.1 (see section "References" below). 365As of the time of this writing, the OpenSL ES working group 366is preparing a revised version of the standard. The revised version 367will likely include new features, clarifications, correction of 368typographical errors, and some incompatibilities. Most of the expected 369incompatibilities are relatively minor, or are in areas of OpenSL ES 370not supported by Android API level 9. However, even a small change 371can be significant for an application developer, so it important 372to prepare for this. 373<p> 374The Android team is committed to preserving future API binary 375compatibility for developers to the extent feasible. It is our 376intention to continue to support future binary compatibility of the 3771.0.1-based API, even as we add support for later versions of the 378standard. An application developed with this version should 379work on future versions of the Android platform, provided that 380you follow the guidelines listed in section "Planning for 381binary compatibility" below. 382<p> 383Note that future source compatibility will <i>not</i> be a goal. That is, 384if you upgrade to a newer version of the NDK, you may need to modify 385your application source code to conform to the new API. We expect 386that most such changes will be minor; see details below. 387 388<h3>Planning for binary compatibility</h3> 389 390We recommend that your application follow these guidelines, 391to improve future binary compatibility: 392<ul> 393<li> 394Use only the documented subset of Android-supported features from 395OpenSL ES 1.0.1. 396<li> 397Do not depend on a particular result code for an unsuccessful 398operation; be prepared to deal with a different result code. 399<li> 400Application callback handlers generally run in a restricted context, 401and should be written to perform their work quickly and then return 402as soon as possible. Do not do complex operations within a callback 403handler. For example, within a buffer queue completion callback, 404you can enqueue another buffer, but do not create an audio player. 405<li> 406Callback handlers should be prepared to be called more or less 407frequently, to receive additional event types, and should ignore 408event types that they do not recognize. Callbacks that are configured 409with an event mask of enabled event types should be prepared to be 410called with multiple event type bits set simultaneously. 411Use "&" to test for each event bit rather than a switch case. 412<li> 413Use prefetch status and callbacks as a general indication of progress, but do 414not depend on specific hard-coded fill levels or callback sequence. 415The meaning of the prefetch status fill level, and the behavior for 416errors that are detected during prefetch, may change. 417<li> 418See section "Buffer queue behavior" below. 419</ul> 420 421<h3>Planning for source compatibility</h3> 422 423As mentioned, source code incompatibilities are expected in the next 424version of OpenSL ES from Khronos Group. Likely areas of change include: 425 426<ul> 427<li>The buffer queue interface is expected to have significant changes, 428especially in the areas of <code>BufferQueue::Enqueue</code>, the parameter 429list for <code>slBufferQueueCallback</code>, 430and the name of field <code>SLBufferQueueState.playIndex</code>. 431We recommend that your application code use Android simple buffer 432queues instead, because we do not plan to change that API. 433In the example code supplied with the NDK, we have used 434Android simple buffer queues for playback for this reason. 435(We also use Android simple buffer queue for recording, but 436that is because standard OpenSL ES 1.0.1 does not support record to 437a buffer queue data sink.) 438<li>Addition of <code>const</code> to input parameters passed by reference, 439and to <code>SLchar *</code> struct fields used as input values. 440This should not require any changes to your code. 441<li>Substitution of unsigned types for some parameters that are 442currently signed. You may need to change a parameter type from 443<code>SLint32</code> to <code>SLuint32</code> or similar, or add a cast. 444<li><code>Equalizer::GetPresetName</code> will copy the string to 445application memory instead of returning a pointer to implementation 446memory. This will be a significant change, so we recommend that you 447either avoid calling this method, or isolate your use of it. 448<li>Additional fields in struct types. For output parameters, these 449new fields can be ignored, but for input parameters the new fields 450will need to be initialized. Fortunately, these are expected to all 451be in areas not supported by API level 9. 452<li>Interface 453<a href="http://en.wikipedia.org/wiki/Globally_unique_identifier"> 454GUIDs</a> will change. Refer to interfaces by symbolic name rather than GUID 455to avoid a dependency. 456<li><code>SLchar</code> will change from <code>unsigned char</code> 457to <code>char</code>. This primarily affects the URI data locator 458and MIME data format. 459<li><code>SLDataFormat_MIME.mimeType</code> will be renamed to <code>pMimeType</code>, 460and <code>SLDataLocator_URI.URI</code> will be renamed to <code>pURI</code>. 461We recommend that you initialize the <code>SLDataFormat_MIME</code> 462and <code>SLDataLocator_URI</code> 463data structures using a brace-enclosed comma-separated list of values, 464rather than by field name, to isolate your code from this change. 465In the example code we have used this technique. 466<li><code>SL_DATAFORMAT_PCM</code> does not permit the application 467to specify the representation of the data as signed integer, unsigned 468integer, or floating-point. The Android implementation assumes that 4698-bit data is unsigned integer and 16-bit is signed integer. In 470addition, the field <code>samplesPerSec</code> is a misnomer, as 471the actual units are milliHz. These issues are expected to be 472addressed in the next OpenSL ES version, which will introduce a new 473extended PCM data format that permits the application to explicitly 474specify the representation, and corrects the field name. As this 475will be a new data format, and the current PCM data format will 476still be available (though deprecated), it should not require any 477immediate changes to your code. 478</ul> 479 480Any actual source code incompatibilities will be explained in detail 481when the time comes. 482 483<h2>Android extensions</h2> 484 485The API for Android extensions is defined in <code>SLES/OpenSLES_Android.h</code>. 486Consult that file for details on these extensions. Unless otherwise 487noted, all interfaces are "explicit". 488<p> 489Note that use these extensions will limit your application's 490portability to other OpenSL ES implementations. If this is a concern, 491we advise that you avoid using them, or isolate your use of these 492with <code>#ifdef</code> etc. 493<p> 494The following figure shows which Android-specific interfaces and 495data locators are available for each object type. 496 497<p> 498<img src="chart2.png" alt="Android extensions"> 499 500<h3>Android configuration interface</h3> 501 502The Android configuration interface provides a means to set 503platform-specific parameters for objects. Unlike other OpenSL ES 5041.0.1 interfaces, the Android configuration interface is available 505prior to object realization. This permits the object to be configured 506and then realized. Header file <code>SLES/OpenSLES_AndroidConfiguration.h</code> 507documents the available configuration keys and values: 508<ul> 509<li>stream type for audio players (default <code>SL_ANDROID_STREAM_MEDIA</code>) 510<li>record profile for audio recorders (default <code>SL_ANDROID_RECORDING_PRESET_GENERIC</code>) 511</ul> 512Here is an example code fragment that sets the Android audio stream type on an audio player: 513<pre> 514// CreateAudioPlayer and specify SL_IID_ANDROIDCONFIGURATION 515// in the required interface ID array. Do not realize player yet. 516// ... 517SLAndroidConfigurationItf playerConfig; 518result = (*playerObject)->GetInterface(playerObject, 519 SL_IID_ANDROIDCONFIGURATION, &playerConfig); 520assert(SL_RESULT_SUCCESS == result); 521SLint32 streamType = SL_ANDROID_STREAM_ALARM; 522result = (*playerConfig)->SetConfiguration(playerConfig, 523 SL_ANDROID_KEY_STREAM_TYPE, &streamType, sizeof(SLint32)); 524assert(SL_RESULT_SUCCESS == result); 525// ... 526// Now realize the player here. 527</pre> 528Similar code can be used to configure the preset for an audio recorder. 529 530<h3>Android effects interfaces</h3> 531 532The Android effect, effect send, and effect capabilities interfaces provide 533a generic mechanism for an application to query and use device-specific 534audio effects. A device manufacturer should document any available 535device-specific audio effects. 536<p> 537Portable applications should use the OpenSL ES 1.0.1 APIs 538for audio effects instead of the Android effect extensions. 539 540<h3>Android file descriptor data locator</h3> 541 542The Android file descriptor data locator permits the source for an 543audio player to be specified as an open file descriptor with read 544access. The data format must be MIME. 545<p> 546This is especially useful in conjunction with the native asset manager. 547 548<h3>Android simple buffer queue data locator and interface</h3> 549 550The Android simple buffer queue data locator and interface are 551identical to the OpenSL ES 1.0.1 buffer queue locator and interface, 552except that Android simple buffer queues may be used with both audio 553players and audio recorders, and are limited to PCM data format. 554[OpenSL ES 1.0.1 buffer queues are for audio players only, and are not 555restricted to PCM data format.] 556<p> 557For recording, the application should enqueue empty buffers. Upon 558notification of completion via a registered callback, the filled 559buffer is available for the application to read. 560<p> 561For playback there is no difference. But for future source code 562compatibility, we suggest that applications use Android simple 563buffer queues instead of OpenSL ES 1.0.1 buffer queues. 564 565<h3>Dynamic interfaces at object creation</h3> 566 567For convenience, the Android implementation of OpenSL ES 1.0.1 568permits dynamic interfaces to be specified at object creation time, 569as an alternative to adding these interfaces after object creation 570with <code>DynamicInterfaceManagement::AddInterface</code>. 571 572<h3>Buffer queue behavior</h3> 573 574The OpenSL ES 1.0.1 specification requires that "On transition to 575the <code>SL_PLAYSTATE_STOPPED</code> state the play cursor is 576returned to the beginning of the currently playing buffer." The 577Android implementation does not necessarily conform to this 578requirement. For Android, it is unspecified whether a transition 579to <code>SL_PLAYSTATE_STOPPED</code> operates as described, or 580leaves the play cursor unchanged. 581<p> 582We recommend that you do not rely on either behavior; after a 583transition to <code>SL_PLAYSTATE_STOPPED</code>, you should explicitly 584call <code>BufferQueue::Clear</code>. This will place the buffer 585queue into a known state. 586<p> 587A corollary is that it is unspecified whether buffer queue callbacks 588are called upon transition to <code>SL_PLAYSTATE_STOPPED</code> or by 589<code>BufferQueue::Clear</code>. 590We recommend that you do not rely on either behavior; be prepared 591to receive a callback in these cases, but also do not depend on 592receiving one. 593<p> 594It is expected that a future version of OpenSL ES will clarify these 595issues. However, upgrading to that version would result in source 596code incompatibilities (see section "Planning for source compatibility" 597above). 598 599<h3>Reporting of extensions</h3> 600 601<code>Engine::QueryNumSupportedExtensions</code>, 602<code>Engine::QuerySupportedExtension</code>, 603<code>Engine::IsExtensionSupported</code> report these extensions: 604<ul> 605<li><code>ANDROID_SDK_LEVEL_9</code> 606(platforms that support a higher API level will report an extension of 607<code>ANDROID_SDK_LEVEL_#</code> where # is that API level) 608</ul> 609 610<h2>Programming notes</h2> 611 612These notes supplement the OpenSL ES 1.0.1 specification, 613available in the "References" section below. 614 615<h3>Objects and interface initialization</h3> 616 617Two aspects of the OpenSL ES programming model that may be unfamiliar 618to new developers are the distinction between objects and interfaces, 619and the initialization sequence. 620<p> 621Briefly, an OpenSL ES object is similar to the object concept 622in programming languages such as Java and C++, except an OpenSL ES 623object is <i>only</i> visible via its associated interfaces. This 624includes the initial interface for all objects, called 625<code>SLObjectItf</code>. There is no handle for an object itself, 626only a handle to the <code>SLObjectItf</code> interface of the object. 627<p> 628An OpenSL ES object is first "created", which returns an 629<code>SLObjectItf</code>, then "realized". This is similar to the 630common programming pattern of first constructing an object (which 631should never fail other than for lack of memory or invalid parameters), 632and then completing initialization (which may fail due to lack of 633resources). The realize step gives the implementation a 634logical place to allocate additional resources if needed. 635<p> 636As part of the API to create an object, an application specifies 637an array of desired interfaces that it plans to acquire later. Note 638that this array does <i>not</i> automatically acquire the interfaces; 639it merely indicates a future intention to acquire them. Interfaces 640are distinguished as "implicit" or "explicit". An explicit interface 641<i>must</i> be listed in the array if it will be acquired later. 642An implicit interface need not be listed in the object create array, 643but there is no harm in listing it there. OpenSL ES has one more 644kind of interface called "dynamic", which does not need to be 645specified in the object create array, and can be added later after 646the object is created. The Android implementation provides a 647convenience feature to avoid this complexity; see section "Dynamic 648interfaces at object creation" above. 649<p> 650After the object is created and realized, the application should 651acquire interfaces for each feature it needs, using 652<code>GetInterface</code> on the initial <code>SLObjectItf</code>. 653<p> 654Finally, the object is available for use via its interfaces, though 655note that some objects require further setup. In particular, an 656audio player with URI data source needs a bit more preparation in 657order to detect connection errors. See the next section 658"Audio player prefetch" for details. 659<p> 660After your application is done with the object, you should explicitly 661destroy it; see section "Destroy" below. 662 663<h3>Audio player prefetch</h3> 664 665For an audio player with URI data source, <code>Object::Realize</code> allocates resources 666but does not connect to the data source (i.e. "prepare") or begin 667pre-fetching data. These occur once the player state is set to 668either <code>SL_PLAYSTATE_PAUSED</code> or <code>SL_PLAYSTATE_PLAYING</code>. 669<p> 670Note that some information may still be unknown until relatively 671late in this sequence. In particular, initially 672<code>Player::GetDuration</code> will return <code>SL_TIME_UNKNOWN</code> 673and <code>MuteSolo::GetChannelCount</code> will return zero. 674These APIs will return the proper values once they are known. 675<p> 676Other properties that are initially unknown include the sample rate 677and actual media content type based on examining the content's header 678(as opposed to the application-specified MIME type and container type). 679These too, are determined later during prepare / prefetch, but there are 680no APIs to retrieve them. 681<p> 682The prefetch status interface is useful for detecting when all 683information is available. Or, your application can poll periodically. 684Note that some information may <i>never</i> be known, for example, 685the duration of a streaming MP3. 686<p> 687The prefetch status interface is also useful for detecting errors. 688Register a callback and enable at least the 689<code>SL_PREFETCHEVENT_FILLLEVELCHANGE</code> and 690<code>SL_PREFETCHEVENT_STATUSCHANGE</code> events. If both of these 691events are delivered simultaneously, and 692<code>PrefetchStatus::GetFillLevel</code> reports a zero level, and 693<code>PrefetchStatus::GetPrefetchStatus</code> reports 694<code>SL_PREFETCHSTATUS_UNDERFLOW</code>, then this indicates a 695non-recoverable error in the data source. 696This includes the inability to connect to the data source because 697the local filename does not exist or the network URI is invalid. 698<p> 699The next version of OpenSL ES is expected to add more explicit 700support for handling errors in the data source. However, for future 701binary compatibility, we intend to continue to support the current 702method for reporting a non-recoverable error. 703<p> 704In summary, a recommended code sequence is: 705<ul> 706<li>Engine::CreateAudioPlayer 707<li>Object:Realize 708<li>Object::GetInterface for SL_IID_PREFETCHSTATUS 709<li>PrefetchStatus::SetCallbackEventsMask 710<li>PrefetchStatus::SetFillUpdatePeriod 711<li>PrefetchStatus::RegisterCallback 712<li>Object::GetInterface for SL_IID_PLAY 713<li>Play::SetPlayState to SL_PLAYSTATE_PAUSED or SL_PLAYSTATE_PLAYING 714<li>preparation and prefetching occur here; during this time your 715callback will be called with periodic status updates 716</ul> 717 718<h3>Destroy</h3> 719 720Be sure to destroy all objects on exit from your application. Objects 721should be destroyed in reverse order of their creation, as it is 722not safe to destroy an object that has any dependent objects. 723For example, destroy in this order: audio players and recorders, 724output mix, then finally the engine. 725<p> 726OpenSL ES does not support automatic garbage collection or 727<a href="http://en.wikipedia.org/wiki/Reference_counting">reference counting</a> 728of interfaces. After you call <code>Object::Destroy</code>, all extant 729interfaces derived from the associated object become <i>undefined</i>. 730<p> 731The Android OpenSL ES implementation does not detect the incorrect 732use of such interfaces. 733Continuing to use such interfaces after the object is destroyed will 734cause your application to crash or behave in unpredictable ways. 735<p> 736We recommend that you explicitly set both the primary object interface 737and all associated interfaces to NULL as part of your object 738destruction sequence, to prevent the accidental misuse of a stale 739interface handle. 740 741<h3>Stereo panning</h3> 742 743When <code>Volume::EnableStereoPosition</code> is used to enable 744stereo panning of a mono source, there is a 3 dB reduction in total 745<a href="http://en.wikipedia.org/wiki/Sound_power_level"> 746sound power level</a>. This is needed to permit the total sound 747power level to remain constant as the source is panned from one 748channel to the other. Therefore, don't enable stereo positioning 749if you don't need it. See the Wikipedia article on 750<a href="http://en.wikipedia.org/wiki/Panning_(audio)">audio panning</a> 751for more information. 752 753<h3>Callbacks and threads</h3> 754 755Callback handlers are generally called <i>synchronously</i> with 756respect to the event, that is, at the moment and location where the 757event is detected by the implementation. But this point is 758<i>asynchronous</i> with respect to the application. Thus you should 759use a mutex or other synchronization mechanism to control access 760to any variables shared between the application and the callback 761handler. In the example code, such as for buffer queues, we have 762omitted this synchronization in the interest of simplicity. However, 763proper mutual exclusion would be critical for any production code. 764<p> 765Callback handlers are called from internal 766non-application thread(s) which are not attached to the Dalvik virtual machine and thus 767are ineligible to use JNI. Because these internal threads are 768critical to the integrity of the OpenSL ES implementation, a callback 769handler should also not block or perform excessive work. Therefore, 770if your callback handler needs to use JNI or do anything significant 771(e.g. beyond an <code>Enqueue</code> or something else simple such as the "Get" 772family), the handler should instead post an event for another thread 773to process. 774<p> 775Note that the converse is safe: a Dalvik application thread which has 776entered JNI is allowed to directly call OpenSL ES APIs, including 777those which block. However, blocking calls are not recommended from 778the main thread, as they may result in the dreaded "Application Not 779Responding" (ANR). 780 781<h3>Performance</h3> 782 783As OpenSL ES is a native C API, non-Dalvik application threads which 784call OpenSL ES have no Dalvik-related overhead such as garbage 785collection pauses. However, there is no additional performance 786benefit to the use of OpenSL ES other than this. In particular, use 787of OpenSL ES does not result in lower audio latency, higher scheduling 788priority, etc. than what the platform generally provides. 789On the other hand, as the Android platform and specific device 790implementations continue to evolve, an OpenSL ES application can 791expect to benefit from any future system performance improvements. 792 793<h3>Security and permissions</h3> 794 795As far as who can do what, security in Android is done at the 796process level. Java programming language code can't do anything more than native code, nor 797can native code do anything more than Java code. The only differences 798between them are what APIs are available that provide functionality 799that the platform promises to support in the future and across 800different devices. 801<p> 802Applications using OpenSL ES must request whatever permissions they 803would need for similar non-native APIs. For example, if your application 804records audio, then it needs the <code>android.permission.RECORD_AUDIO</code> 805permission. Applications that use audio effects need 806<code>android.permission.MODIFY_AUDIO_SETTINGS</code>. Applications that play 807network URI resources need <code>android.permission.NETWORK</code>. 808<p> 809Media content parsers and software codecs run within the context 810of the Android application that calls OpenSL ES (hardware codecs 811are abstracted, but are device-dependent). Malformed content 812designed to exploit parser and codec vulnerabilities is a known attack 813vector. We recommend that you play media only from trustworthy 814sources, or that you partition your application such that code that 815handles media from untrustworthy sources runs in a relatively 816sandboxed environment. For example you could process media from 817untrustworthy sources in a separate process. Though both processes 818would still run under the same UID, this separation does make an 819attack more difficult. 820 821<h2>Platform issues</h2> 822 823This section describes known issues in the initial platform 824release which supports these APIs. 825 826<h3>Dynamic interface management</h3> 827 828<code>DynamicInterfaceManagement::AddInterface</code> does not work. 829Instead, specify the interface in the array passed to Create, as 830shown in the example code for environmental reverb. 831 832<h2>References and resources</h2> 833 834Google Android: 835<ul> 836<li><a href="http://developer.android.com/resources/index.html"> 837Android developer resources</a> 838<li><a href="http://groups.google.com/group/android-developers"> 839Android developers discussion group</a> 840<li><a href="http://developer.android.com/sdk/ndk/index.html">Android NDK</a> 841<li><a href="http://groups.google.com/group/android-ndk"> 842Android NDK discussion group</a> (for developers of native code, including OpenSL ES) 843<li><a href="http://code.google.com/p/android/issues/"> 844Android open source bug database</a> 845</ul> 846 847Khronos Group: 848<ul> 849<li><a href="http://www.khronos.org/opensles/"> 850Khronos Group OpenSL ES Overview</a> 851<li><a href="http://www.khronos.org/registry/sles/"> 852Khronos Group OpenSL ES 1.0.1 specification</a> 853<li><a href="http://www.khronos.org/message_boards/viewforum.php?f=15"> 854Khronos Group public message board for OpenSL ES</a> 855(please limit to non-Android questions) 856</ul> 857For convenience, we have included a copy of the OpenSL ES 1.0.1 858specification with the NDK in 859<code>docs/opensles/OpenSL_ES_Specification_1.0.1.pdf</code>. 860 861<p> 862Miscellaneous: 863<ul> 864<li><a href="http://en.wikipedia.org/wiki/Java_Native_Interface">JNI</a> 865<li><a href="http://stackoverflow.com/search?q=android+audio"> 866Stack Overflow</a> 867<li>web search for "interactive audio", "game audio", "sound design", 868"audio programming", "audio content", "audio formats", etc. 869</ul> 870 871</body> 872</html> 873
