When timing code to work in sync with a video experience, you might come across some caveats. Let’s go over some of them:
Resources Readiness
Since video experience should be a seamless running experience, it is important to prepare required resources before the video actually needs them.
If you are preparing to add externals libraries, make sure to load them in time.
Don’t forget that video nodes are also resources which should be prefetched. If a player attempts to play a node which hasn’t been downloaded yet it the video might buffer.
If you are using the decision plugin you can take advantage of its automatic prefetching, if not then you should handle it manually using the Node’s addPrefetch API. See the dynamic streaming guide.
Use a Single Clock
The player time is the clock of the experience.
An interactive video’s main user experience is, well, the video, and so events should be synced to it. However the video experience might not be fluid (due to a video being paused, buffering, etc). Therefore you should avoid using additional clocks (such as setTimeout) as they can get out of sync with the video timing, this is critical when UI component positioning needs to be synced with the underlying video.
Instead, you should rely in the player’s time in the redux state or use cue point-related events sent by the player. The most useful one being the Node’s timeupdate event. You can listen to a specific time on the player time or on a specific node. When listening to a node timeupdate event you can also specify a certain time from the node end as well using the minus sign.
Timing
Time is not 100% accurate to the frame level.
This is because we encode the video to our platform. During this process we insert keyframes to the encoded video and “lose” the original frame accuracy. Attempting to sync to a frame level will not be as accurate.
On the other hand, attempting to sync with time events of the video node (using timeupdate) will also not be accurate to the frame level due to JS event loop internal implementation.
When will this be a problem? Let’s say you want to place a UI component exactly over an object in the video. The object moves around as the video node plays and so attempting to move your component around to match the object’s location using time events can result in a location discrepancy.
A possible solution will be to put the visual aspect of the component in the video itself and make the actual UI component transparent (and slightly larger than its in-video counterpart). The user will not notice if the component is somewhat out of sync with the object’s location in the video.