Working with Audio

When building your interactive project, you might want to add and play different sounds, whether it is feedback for user interaction or background music that plays throughout. The audio plugin provides a simple interface for playing sounds (powered by howlerjs) and its API is documented here. Here are a few pointers on how to use it.

Adding sounds

The simplest way to add sounds is to drag an audio file into Eko Studio’s library area. Eko Studio will encode your source audio file, assign it a unique ID and automatically register it with the audio plugin by generating a sounds map object which will be passed on as part of the audio plugin’s init options. To get the unique ID of a sound, right click its file in Eko Studio’s library area and select Get element ID. If you are developing your project locally, you can easily override a sound’s properties by editing your src/config/playerOptions.json file. For example, say you have a sound with ID audio_ding_8427f5 which is too loud, you can use the following JSON to reduce its volume to 50%:

// src/config/playerOptions.json
{
    "plugins": {
        "audio": {
            "sounds": {
                "audio_ding_8427f5": {
                    "volume": 0.5
                }
            }
        }
    }
}

Useful properties and methods

Video Volume

The audio plugin allows you to set volume not only of the audio elements you are playing but also of the video being played using the videoVolume property, for example:

player.audio.videoVolume = 0.5;

This is useful for when you want to play an audio overlay and adjust the relative volume of the video.

Muting

You can easily mute all sounds using

player.audio.muted = true;

Fading

The audio plugin uses the howlerjs library which provides some advanced audio functionality. For example, a nice thing you can do is fade the volume level of a sound.

You can access the underlying Howl instance of the audio element you registered by using the get() method:

let myAudio = player.audio.get('audio_myaudio');

Then using the fade() method, you can change the volume gradually.

myAudio.fade([FROM_VOLUME], [TO_VOLUME], [FADE_DURATION_IN_MSEC]);

For example:

// Fade out from full volume to zero over 2 seconds
myAudio.fade(1.0, 0, 2000);

Memory management

Great care needs to be taken when developing audio rich Eko videos.

Audio files have to be decoded before they can be played back, and as long as a sound is loaded, the decoded buffer is kept in memory at runtime. These decoded buffers can easily get huge - 60 seconds of uncompressed audio translates to 23MB of memory at runtime. If your project contains just 20 minutes worth of audio files (total duration), all loaded at once, that’s over 460MB of memory which is more than enough to cause an iOS Safari tab to instantly crash.

This is why it’s important to manage the loading and unloading of sounds throughout an Eko experience. To help you do that, we introduced the setPreload() method (and the preload property that’s passed into the add() method).

To calculate the runtime memory (bytes) used by an audio file use the following equation: durationInSeconds * 48000 (48k samples-per-second) * 4 (32-bit per sample) * 2 (stereo)

Load sound on init

In some cases, a sound needs to be available for playback throughout the entire experience (for example, a global soundtrack or a button click sound). When Eko Studio generates the audio plugin’s init options, it uses a preload:true value for the soundtrack and any sound associated with a button, which means that they’ll be loaded at init and available for playback at any time.

Associate sound with nodes

The setPreload() method allows you to associate a sound with nodes by passing an array of node IDs as the 2nd argument. By passing an array of node IDs (or node instances), the sound goes into lazy-loading (and greedy unloading) mode. Whenever a node associated with the sound is pushed into the playlist, the sound will automatically be loaded and stay loaded and available for playback for as long as there are associated nodes in the playlist ahead. Once playback advances past the point where the sound is needed (i.e. there are no more nodes associated with the sound in the playlist ahead), the sound will automatically be unloaded and its memory will be released.

Use this when sounds are needed in the context of specific nodes. This allows having lots of audio assets in the project and should meet the needs of most projects.

Load sound on demand

By default, Eko Studio will assign a preload:false value to all sounds that are NOT associated with a button or used as the soundtrack. The setPreload() method allows you to change the preload value of a sound at runtime. Using the values true or false will cause a sound to immediately load or unload respectively.

Use this method if you have many audio assets and need fine grained control over loading and unloading. For most use cases though, the declarative approach of associating a sound with nodes is preferable.

Using node metadata to associate sound with nodes

In order to manage which sounds should be associated with which nodes, we suggest using Eko Studio’s metadata feature. This would allow you to keep your code generic by allowing modifications to node/sound relationships to be made from within Eko Studio without requiring code changes. Let’s say that the sound with ID audio_alarm needs to be available for playback on the node node_goodmorning. We can attach the following metadata to node_goodmorning:

// Metadata on "node_goodmorning"
{
    "soundsToPreload": [
        "audio_alarm"
    ]
}

Similar metadata can be attached to any other nodes that require sounds. In our initialization function, we could use the following code snippet to make sure the sounds will be lazily-loaded on all associated nodes:

// An object that will map sound IDs (keys)
// to array of associated node IDs (values)
let soundIdsToNodes = {};

// Iterate over all nodes to populate the soundIdsToNodes map
player.repository.get('all')

    // Only include nodes that contain
    // a soundsToPreload array in their metadata
    .filter(node => {
        return node.data.studio &&
            Array.isArray(node.data.studio.soundsToPreload);
    })

    // Populate the soundIdsToNodes map
    .forEach(node => {
        node.data.studio.soundsToPreload
            .forEach(soundId => {
                soundIdsToNodes[soundId] = soundIdsToNodes[soundId] || [];
                soundIdsToNodes[soundId].push(node.id);
            });
    });

// Iterate over all sounds and apply their associated nodes
Object.keys(soundIdsToNodes)
    .forEach(soundId => {
        player.audio.setPreload(soundId, soundIdsToNodes[soundId]);
    });

In some cases, loading a sound when an associated node is pushed to the playlist may be too late, as the sound needs to be playable immediately. In these cases we recommend including the parent node as an associated node as well, so that the sound will be preloaded as soon as the parent node is pushed, and available for playback by the time we reach the child node.

Rate this page: X
Tell us more!