Video Playback Metrics: New Synthetic Testing Tool StreamOrNot

Video content on the the web is increasing – with such an immersive medium – it is pretty much a . no brainer.  However, a few months ago, I looked into how the web is using video on the web, and found that slow video startup times and stalls are much more common than we would all hope.

Video metrics generally focus on 3 attributes:  How quickly did the video startup, did the video stall, and did the video have high enough quality.  In this post, I’m going to describe StreamOrNot, a tool that helps developers accurately measure the first 2 metrics:  will playback be fast and stall free?

Measuring Video Performance

One of the greatest tools for measuring web performance is webpagetest.org. By understanding what assets are being loaded (in which order), one can work to discover inefficient downloads, and work to optimise – thus speeding up the page.

Here’s a WebPageTest waterfall of a website with a background video:

The big long green line? That’s the video.  It is 4 MB (75% of the total page weight) and takes 8 seconds to download.

This is all great information – we know this page has a video, how large it is, and how long it took to download – but we don’t get any meaningful data about whether the video started playing.  Or if the video had a stall.  In order to test how videos play back – we have Stream or Not!

Measuring Video Playback Stats

To run Stream Or Not, Just enter the url of your video and hit go.  This will show you how fast your video will load, and if it will stall on your internet connection.

Screen Shot 2019-11-15 at 10.10.39 AM

Optional Parameters:

The default test in SON only downloads one file – the video.  Your website probably has a bunch of other files that are being downloaded at the same time, and that will affect the video download.  StreamOrNot allows you to choose between 0-100 images that can be downloaded at the same time as your video.

Note: Video on StreamOrNot is prioritised with priority hints, giving the video high priority over all other files.  This is to ensure that the images and video are downloaded together,  Generally – you should not give video files priority over all other downloads on your website.

Other options

you can play with HTML parameters, and you can also have your computer process some recursive fibonacci sequences in Javascript to slow down the processor – to see how lots of JavaScript might affect playback.

Adjusting The Network Speed

The test as described is based on your internet connection.  Frankly, your connection is probably faster than that of your users. To test your video playback at different speeds, you can use devTools to throttle your connection speed (And yes, I have added custom speeds based on WebPageTest)

Screen Shot 2019-11-15 at 9.29.34 AM

I’d recommend testing on the Cable and Fast 3G connection speeds to see how your video plays on slower connections.

 

Results

As your video plays back, you will receive live information on how long it took for the video to begin playback, how much video is in the buffer (and if it is close to stalling), and a count and time spent in the stall state.

Screen Shot 2019-11-15 at 10.28.56 AM

  1. The size of the video (dimensions and number of bytes)
  2. On Chrome, the connection speed is shown (navigation.connection is Chrome only)
  3. Video playback state. Possible options are
    1. Not Started
    2. Playing
    3. waiting
    4. stalled
    5. empty buffer
    6. ended
  4. Video Start time. Measured from start of JS execution to when the video begins playing. The longer this takes, the text color changes to yellow and then red.
  5. Playback time (in seconds)
  6. Remaining time in buffer (in seconds)

Video buffer chart:

Under 5s – stalls are imminently possible.

5-10s – warning – stalls might occur soon.

10s – green – You’ve got some video stored, and if you keep it up, you’ll be ok.

The chart  is an SVG that is drawn in real time. I think that’s really cool 🙂

 

In the example above, the video starts up very quickly, but it is very high bitrate, and the buffer is never able to fill up, and there is a short stall every second or so, making the video very janky and unpleasant to watch. The stall report (generated at the end of video playback shows this very clearly:

Screen Shot 2019-11-15 at 10.30.19 AM

 

Automating Video Playback testing

To automate this testing, and to ensure similar conditions for all of your tests, you can utilise WebPageTest and Andy Davies’ Google Sheet to automate the tests.  The details on how to do that are all on GitHub.

Resolving issues

Videos that are slow to startup or stall generally have a higher bitrate.  This means that the video plays back faster than the network can download the video.  Lowering the bitrate of your video (lowering the quality, changing the dimensions) will help your video play back faster and with fewer stalls.

Conclusion

There have been no tools to test how well a video plays back on the web.  Stream Or Not is a synthetic testing tool that enables you to quickly determine how well your video will play on different types of networks or devices.   Give it a go, and please provide feedback or suggestions!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.