Vrs 1.0

Authored by SONiVOX

www.sonivoxrocks.com

1 Introduction

1.1 Overview

This document contains content creation guidelines for composers and sound designers authoring music and sound effects for the SONiVOX JET platform. JET is an interactive music player for small embedded devices, including the Google Android platform. It allows applications to include interactive music soundtracks, in MIDI format, that respond in real-time to game play events and user interaction.

JET works in conjunction with SONiVOX’s Embedded Audio Synthesizer (EAS) which is the MIDI playback device for Android. Both the JET and EAS engines are integrated into the Android embedded platform as well as inherent in the JET Creator application. As such, the JET content author can be sure that the playback will sound exactly the same in both the JET Creator and the final Android application playing back on Android mobile devices.

The JET content author works in up to three different applications to create JET content; a standard MIDI sequencer (Logic, Cubase, etc.), optionally a DLS2 instrument editor (Awave), and the JET Creator application to add and audition JET interactive elements.

The final result is a .jet file that the content author gives to the application programmer for use in the game or application.

1.2 Abbreviations and Common Terms

It is important to use a common set of terms to minimize confusion. Since JET uses MIDI in a unique way, normal industry terms may not always suffice. Here is the definition of terms as they are used in this document and in the JET Creator application:

Channel: MIDI data associated with a specific MIDI channel. Standard MIDI allows for 16 channels of MIDI data each of which are typically associated with a specific instrument.

Controller: A MIDI event consisting of a channel number, controller number, and a controller value. The MIDI spec associates many controller numbers with specific functions, such as volume, expression, sustain pedal, etc. JET also uses controller events as a means of embedding special control information in a MIDI sequence to provide for audio synchronization.

DAW: Digital Audio Workstation. A common term for MIDI and audio sequencing applications such as Logic, SONAR, Cubase and others.

EAS: Embedded MIDI Synthesizer. The name of the SONiVOX MIDI synthesizer engine.

JET: Jet Interactive Engine. The name of the SONiVOX JET interactive music engine.

Segment: A musical section such as a chorus or verse that is a component of the overall composition. In JET, a segment can be an entire MIDI file or a derived from a portion of a MIDI file.

SMF-0: Standard MIDI File Type 0, a MIDI file that contains a single track, but may be made up of multiple channels of MIDI data.

SMF-1: Standard MIDI File Type 1, a MIDI file that contains a one more tracks, and each track may in turn be made up of one or more channels of MIDI data. By convention, each channel is stored on a separate track in an SMF-1 file. However, it is possible to have multiple MIDI channels on a single track, or multiple tracks that contain data for the same MIDI channel.

Track: A single track in a DAW containing a timed sequence of MIDI events. Be careful not to confuse Tracks with Channels. A MIDI file may contain many tracks with several tracks utilizing the same MIDI channel.

1 The JET Interactive Music Concept

Interactive music can be defined as music that changes in real-time according to non-predictable events such as user interaction or game play events. In this way, interactive music is much more engaging as it has the ability to match the energy and mood of a game much closer than a pre-composed composition that never changes. In some applications and games, interactive music is central to the game play. Guitar Hero is one such popular game. When the end user successfully ‘captures’ the musical notes coming down the fret board, the music adapts itself and simultaneously keeps score of successes and failures. JET allows for these types of music driven games as well.

There are several methods for making and controlling interactive music and JET is one such method. This section describes the features of JET and how they might be used in a game or software application. It also describes how JET can be used to save memory in small footprint devices such as Android enabled mobile handsets.

1.1.1 Data Compression

JET supports a flexible music format that can be used to create extended musical sequences with a minimal amount of data. A musical composition is broken up into segments that can be sequenced to create a longer piece. The sequencing can be fixed at the time the music file is authored, or it can be created dynamically under program control.

1.1.2 Linear Music Example

Figure 1: Linear Music Piece

This diagram shows how musical segments are stored. Each segment is authored as a separate MIDI file. A post-processing tool combines the files into a single container file. Each segment can contain alternate music tracks that can be muted or un-muted to create additional interest. An example might be a brass accent in the chorus that is played only the last time through. Also, segments can be transposed up or down.

The bottom part of the diagram shows how the musical segments can be recombined to create a linear music piece. In this example, the bridge might end with a half-step key modulation and the remaining segments could be transposed up a half-step to match.

1.1.3 Non-linear Music Example

Figure 2: Non-linear music piece

In this diagram, we see a non-linear music piece. The scenario is a first-person-shooter (FPS) and JET is providing the background music. The intro plays as the level is loading and then transitions under program control to the Searching segment. This segment is repeated indefinitely, perhaps with small variations (using the mute/un-mute feature) until activity in the game dictates a change.

As the player nears a monster lair, the program starts a synchronized transition to the Danger segment, increasing the tension level in the audio. As the player draws closer to the lair, additional tracks are un-muted to increase the tension.

As the player enters into combat with the monster, the program starts a synchronized transition to the Combat segment. The segment repeats indefinitely as the combat continues. A Bonus Hit temporarily un-mutes a decorative track that notifies the player of a successful attack, and similarly, another track is temporarily un-muted to signify when the player receives Special Damage.

At the end of combat, the music transitions to a victory or defeat segment based on the outcome of battle.

1.1.4 Mute/Un-mute Synchronization

JET can also synchronize the muting and un-muting of tracks to events in the music. For example, in the FPS game, it would probably be desirable to place the musical events relating to bonuses and damage as close to the actual game event as possible. However, simply un-muting a track at the moment the game event occurs might result in a music clip starting in the middle. Alternatively, a clip could be started from the beginning, but then it wouldn’t be synchronized with the other music tracks.

However, with the JET sync engine, a clip can be started at the next opportune moment and maintain synchronization. This can be accomplished by placing a number of short music clips on a decorative track. A MIDI event in the stream signifies the start of a clip and a second event signifies the end of a clip. When the application calls the JET clip function, the next clip in the track is allowed to play fully synchronized to the music. Optionally, the track can be automatically muted by a second MIDI event.

Figure 3: Synchronized Mute/Unmute

1.2 Audio Synchronization

JET provides an audio synchronization API that allows game play to be synchronized to events in the audio. The mechanism relies on data embedded in the MIDI file at the time the content is authored. When the JET engine senses an event during playback it generates a callback into the application program. The timing of the callback can be adjusted to compensate for any latency in the audio playback system so that audio and video can be synchronized. The diagram below shows an example of a simple music game that involves pressing the left and right arrows in time with the music.

Figure 4: Music Game with Synchronization

The arrows represent events in the music sequence where game events need to be synchronized. In this case, the blue arrow represents a time where the player is supposed to press the left button, and the red arrow is for the right button. The yellow arrow tells the game engine that the sequence is complete. The player is allowed a certain time window before and after the event to press the appropriate key.

If an event is received and the player has not pressed a button, a timer is set to half the length of the window. If the player presses the button before the timer expires, the game registers a success, and if not, the game registers a failure.

If the player presses the button before the event is received, a timer is set to half the length of the window. If an event is received before the timer expires, the game registers a success, and if not, the game registers a failure. Game play might also include bonuses for getting close to the timing of the actual event.

2 JET Content Authoring Overview

To author JET files and hear them playback interactively, the content author will work in two or three applications which are designed to work together smoothly. The first is application is any off-the-shelf MIDI sequencing application or Digital Audio Workstation that supports VST (for PC) or AU (for Mac) plugins. (Logic, SONAR, Cubase, etc) Here the author will compose their MIDI music files using the SONiVOX EAS Synth plugin as the playback synthesizer.

Once the composer has completed their MIDI file(s), they import them into the JET Creator application. Here the author will setup and audition the conditions for interactive playback within the JET enabled game.

Optionally, the author may elect to create a custom DLS soundbank. This can be created in any off-the-shelf DLS authoring application, such as Awave from MJSoft, and loaded into JET Creator along with the MIDI files.

Below is an overview of this process. A more detailed explanation of each step follows.

Launch digital audio workstation (DAW)

Assign the SONiVOX EAS Synth plugin as the playback synthesizer
Optionally load a custom DLS2 soundset
Compose and save MIDI file(s)

Launch the JET Creator application

Create segments using the MIDI and DLS2 source files
Add interactive elements
Audition interactive elements
Save and Export JET files for use in the Android application

Launch DAW – Content authors will need to use a third party MIDI authoring application to compose their MIDI files. It is recommended they use a digital audio workstation (DAW) application that supports VST or AU plugins as this will enable them to listen to the EAS MIDI Synthesizer and DLS2 soundsets that will be utilized in the Android application itself. Some examples of popular DAWs include SONAR (PC) and LOGIC (MAC).

Assign SONiVOX EAS Synth plugin as the playback synthesizer – The SONiVOX EAS Synth plugin is a VST and AU compatible virtual instrument that plugs into VST or AU compatible DAWs. This software plugin uses the same SONiVOX EAS MIDI synthesizer engine and default General MIDI wavetable soundset inherent in Android. Using this plugin allows content authors to hear the exact audio rendering of the instruments and MIDI file that will be used in their Android applications.

Optionally Load DLS2 Soundset – The SONiVOX EAS Synth plugin allows for the loading of any DLS2 compatible soundset for playback. These could include a new GM wavetable set, or a small collection of just a few custom instruments for a given application. Note, the DLS file does not replace the internal GM wavetable used by the EAS engine. DLS soundsets play in conjunction with the internal GM wavetable.

Compose MIDI File – Compose MIDI soundtracks for the Android application.

Launch JET Creator – Once all DLS2 and MIDI source files have been authored, the content author should launch the JET Creator and begin creating JET Segments. The segments will reference the MIDI files and any custom DLS2 soundbanks.

Assign JET Segment Attributes – After creating segments the content author interactive elements. Interactive elements include mute and unmute settings of individual tracks in the MIDI file(s) as well as MIDI controller numbers that serve as “events” in the game. These attributes tell the JET engine how and when to play the different musical segments according to the JET API commands in the Android application. See below for more detail on this.

Audition Interactive Playback – After assigning the segment attributes and creating the JET file, the content author can audition all interactive playback elements in the JET Audition window.

Save .jtc File – After the author is satisfied with the result, it is recommended they save the JET Creator .jtc file which will save their settings, references to source files, etc.

Export Files – Exporting the JET Creator file will bundle all source files and their attributes into a single .zip file. The zip file will also contain a .jet file for use by the Android application.

3 EAS Synth Virtual Instrument Plugin

Included in the JET Creator package is the EAS software synthesizer in plug-in format. The EAS plugin synth allows the composer to hear the instruments used in Android as they are composing their MIDI sequence. The EAS Synth plugin allows for the loading of custom DLS2 sounds as well.

3.1 Installing the EAS Synth Plugin

Follow the instructions for your individual DAW to install and utilize the plugin. For Mac users this will typically involve copying the “EAS Synth.componant” file into your plugins folder which is usually located at /Library/Audio/Plug-ins/Components. PC users will want to install the “EAS Synth.dll” into the plugin folder that their DAW requires.

3.2 Requirements and Settings for using the EAS Synth Plugin

The EAS Synth is an embedded synthesizer for small mobile devices. This means it does not have the flexibility of high end synthesizers typically utilized in a professional application such as Logic, Digital Performer, etc. As such, only the following attributes are supported.

Macintosh:

Mac OSX (Intel) Macs

ASIO Supported Soundcards

Sample Rate: 44100 hz

Buffer Size: 256 kbytes

PC:

Windows 2000 or Vista operating systems

ASIO supported soundcards

Sample Rate: 44100 hz

Buffer Size: 256 kbytes

3.3 Assigning MIDI Tracks to use the EAS Synth

Each DAW has its own particular method of assigning MIDI tracks to virtual instrument plugins such as the SONiVOX EAS Synth. Please consult the user manual for your DAW for detailed instructions. Below are some general guidelines for Logic (Mac) and SONAR (PC).

3.3.1 LOGIC 8

The SONiVOX EAS Synth virtual instrument is a multi-timbral synthesizer. (i.e. it plays back multiple instruments on unique MIDI channels in a single instance) In Logic 8, however, you’ll want to set up 16 Logic Instruments, each with their own instance of the EAS Synth. Each Instrument should be assigned its own MIDI channel. Use Channel 10 for Drums. The reason for this is that MIDI controller messages, such as Volume (CC7) and Pan (CC10) will not be channelized if the plugin is assigned to only a single Instrument and all MIDI tracks are set to playback on that Instrument. In order for each MIDI channel to respond to its own controller messages, you must assign 16 different EAS Synth instances to 16 unique Logic Instruments.

A Logic 8 template file has been included in the Android Cupcake release to facilitate the above.

Playback in Logic 8 may require you to be in record enable mode for each track you are auditioning. To record enable multiple tracks hold down the Option key.

To write out a standard MIDI (type 1) file from Logic, you need to use the File Export command. IMPORTANT: Most edits in Logic are non-destructive edits meaning they are not modifying the actual data but rather adding an overlay onto to the data. Quantize is one such non-destructive edit. Therefore when you export a MIDI file, you may not see your quanitization settings.

In addition, the mix parameters for volume, pan and program changes may not appear in the event list and therefore may not write out with the MIDI file. Before exporting a MIDI file in Logic it is recommended you do the following:

Select All and use the “Insert MIDI > Insert MIDI Settings as Events” command.

Select All and use the “Apply Quantization Settings Destructively” command.

3.3.2 Cakewalk SONAR 7

Sonar 7 is a bit easier to set up, use and save than Logic 8. Simply open or start a new MIDI file. Go to the Insert menu and select Insert Soft Synth>SONiVOX>EAS Synth. Then assign each MIDI track’s output to the EAS Synth. There is no need to record enable a track to hear it play back. When saving, be sure to select MIDI Type 1.

SONAR 8 works similarly to SONAR 7.

3.3.3 Digital Performer

We’ve seen some instances when creating content with Digital Performer where notes with a release velocity of non-0 will generate an extra note-on event in the EAS synth. If you are hearing a doubling, editing the release velocity events to zero should fix this problem.

3.4 Using Custom DLS2 Soundsets

The SONiVOX EAS Synthesizer supports two simultaneous soundsets or wavetables. One is the internal General MIDI wavetable inherent to the SONiVOX EAS Synthesizer. The other is a Downloadable Sounds Level 2 (DLS2) soundset. The internal wavetable is a GM Level 1 compliant wavetable with 127 melodic instruments and 1 drumkit. It is in a proprietary SONiVOX format. The DLS2 soundsets are an open format published by the MIDI Manufactures Association.

In the Android Cupcake release, the internal wavetable is only 200 kbytes, very small, in order to be compliant with all Android devices which may not have a lot of memory. DLS2 soundsets can be any size that a particular device supports. Upgraded (larger) internal wavetables as well as custom DLS2 instruments can be licensed from SONiVOX.

3.4.1 Loading a DLS2 Soundset

To load a custom soundset, click on the Load DLS button in the EAS Synth plugin interface. Browse to the DLS2 file you wish to load and say OK. Only DLS Level 2 formatted soundsets are supported.

3.4.2 Using a DLS2 Soundset

Since both the internal EAS GM wavetable and a custom DLS2 soundset are used simultaneously, you must be sure you have your MIDI Program Changes set correctly. DLS2 instruments must be assigned to a Bank other than the default GM bank used by the internal synthesizer.

The internal EAS synthesizer is assigned to Banks 121 (melodic instruments) and 120 (drum instruments). This follows the General MIDI Level 1 specification. Note: Most MIDI sequencers require you to use Bank 0 to select the default wavetable. Custom DLS2 soundsets, therefore, should utilize a different Bank. We recommend Bank 1.

The EAS synth supports MSB (Controller 0), LSB (Controller 32) Bank change messages. There are two places you need to set this Bank and Program Change number. The first is in your DLS2 soundset. Using Bank 1, each Instrument would be assigned MSB 1, LSB 0, then the Instrument Program Change number. The second place to use the Bank and Program Change number is in your MIDI sequence.

In your MIDI track, the MSB should be sent first followed by the LSB and then the Instrument number. For example, if your DLS2 instrument is assigned MSB 1, LSB 0, Program1, you would send CC0, 1 followed by CC32, 0 followed by Program Change Message 1. This might look like the following in an event window:

4 JET Creator Guidelines

JET Creator is the desktop application where you’ll edit and audition the JET interactive music elements. For details on the JET Creator application please see the “JET Creator User Manual”. Below are some additional guidelines to help you out.

4.1 Order of Tasks

As with all projects, its best to discuss and design the interactive music scheme with the game designer and programmer before beginning your composition. An outline and/or specification can go a long way in saving you from having to redo things after the game is in place.

In general you’ll want to first write your music in your DAW of choice the way you’re used to composing, then break up the final MIDI file as needed for the application. Next, move to JET Creator and create all of your music segments in the order easiest to preview them when played in order. Finally, add the JET Events to control the segments via the Android game and Audition them as needed in JET Creator. Finally, save the project in JET Creator and hand off the .jet file to the programmer to integrate it in the game. After previewing there will likely be changes to the MIDI file(s) and JET Creator attributes.

4.2 Conserving Memory

If you’re trying to conserve memory, compose as few MIDI files as possible, and create several segments from that MIDI file. For example a 12 bar MIDI file with three sections of 4 bars, A, B, C, can create a much longer song. Simply create multiple segments that reference the one MIDI file, then order them however you like. For example, A, A, B, A, C, A, B, A, A would create a 36 bar song. Use JET to add repeats, transpose segments, and interactively mute and unmute tracks to keep it even more interesting.

4.3 Replicate

To make adding segments or events faster, use the Replicate command. Replicate can add multiple segments or events at one time and uses an offset parameter and prefix naming convention to keep things easy to read. The MOVE command is also useful for moving multiple events by a set number of measures, beats or ticks.

4.4 Interactive Options

There are several interactive audio concepts possible in JET. Below are a few examples although we hope developers will come up with others we haven’t thought of! These are:

4.4.1 Multiple Segment Triggering

In this method the application is triggering specific segments based on events in the game. For example a hallway with lots of fighting might trigger segment 1 and a hallway with no fighting might trigger segment 2. Using JET TriggerClips in conjunction with this method creates even more diversity.

4.4.2 Mute Arrays

In this method the application is triggering mute and unmute events to specific tracks in a single MIDI sequence. For example a hallway with lots of fighting might play MIDI tracks 1-16 and a hallway with no fighting might play the same midi file but mute tracks 9-16. Using JET TriggerClips in conjunction with this method creates even more diversity.

4.4.3 Music Driven Gameplay

Music driven gaming is similar to what Guitar Hero and JETBOY have done in that the music content determines how graphic events are displayed. The application then queries the user response to the graphic events and interactively modifies the music in response. In this method the game is utilizing JET Application Events, MIDI controllers that are embedded in the MIDI file and read by the game in real-time. Based on the user response, multiple segment triggering and/or mute arrays can be set.