In previous posts on video streaming, I’ve looked at how video adaptive streaming works and some of the performance impacts on video on the mobile web. In this post, I’d like to combine these two ideas by looking at how websites are streaming video, and see if websites are following the best practices I have proposed.
For my dataset, I’ll be using the HTTPArchive crawl from 12/15/2017 of the mobile web, and I’ll specifically be looking at HLS streams (as HLS is currently the predominant streaming mechanism). In HLS streaming, the first request in a video stream is the manifest (and in HLS streaming, the manifest has with extension m3u8). Every time the bitrate changes, and additional manifest file is requested and downloaded. For that reason, it is very common to see multiple manifest files per website, and the data corroborates this. If I do a query for requests with the m3u8 extension, I find 9464 m3u8 files requested in the dataset, across 1974 different pages.
SELECT pageid, COUNT(ext) AS cnt FROM ( SELECT pageid, ext, FROM httparchive:runs.2017_12_15_requests_mobile, WHERE (ext = "m3u8") ORDER BY requestid ) GROUP BY pageid
The median is 2 manifests per page, but as shown below, there are several outliers with over 68 manifest file requests during page loading:
Now, once the manifest is downloaded – the browser knows what video files to download to commence the stream (as they are enumerated in the manifest file). The player can begin downloading the stream, or it can wait until the user clicks the play button.
In the HTTPArchive data, 1187 sites (60%) that have at least one manifest file request, but 0 video segments. For mobile, this is ideal, saving cellular data by now actually downloading the video files until the video is started by the user. The median site with manifest file downloads, but no video segments has a SpeedIndex of 11233.
The remaining 787 sites (40%) have at least one manifest file, and at least one video segment. The median SpeedIndex of these sites is 12726, indicating about a 1500ms penalty in page load time for these sites.
Comparing Mobile to Desktop
The desktop data is nearly reversed. 67% of sites (1747 of 2616) download both manifests and segments, while 33% (869) only offer the manifests (with no video segments). Again, this makes sense, as desktop computers are more likely to be on a faster, non-metered network.
Identifying Trends in Streaming
To better understand how these sites stream video, I need to look at individual sites, and the manifests/video segments being downloaded. To identify these sites, I ran this query:
SELECT pages.speedindex, pages.rank, pages.url, pages.pageid, manifests.manifstcnt, manifests.segmentcnt FROM httparchive:runs.2017_12_15_pages_mobile pages JOIN ( SELECT pageid, SUM(m3u8) AS manifstcnt, SUM(ts) AS segmentcnt, //count(*) bub FROM ( SELECT pageid, url, respsize, mimeType, type, ext, IF(ext ="m3u8",1,0) m3u8, IF(ext="ts",1,0) ts FROM httparchive:runs.2017_12_15_requests_mobile, WHERE (ext = "ts" || ext = "m3u8") ) GROUP BY pageid ) manifests ON pages.pageid = manifests.pageid WHERE (manifests.manifstcnt > 0 && manifests.segmentcnt >0 ) ORDER BY rank ASC
From the resulting list, I manually looked at 18 of the top websites with multiple manifests and segments to understand how these sites set up their streaming videos for mobile consumption.
Looking at Individual Sites
To understand how these sites download video, I use the pageID to identify all of the manifest and video files downloaded. I then sort them by the request ID, to give a chronological list of requests:
SELECT pageid, requestid, url, respsize, mimeType, type, ext, // if(ext ="m3u8",1,0) m3u8, // if(ext="ts",1,0) ts FROM httparchive:runs.2017_12_15_requests_mobile, WHERE //change the pageid to get a different site pageid =19484828 &&(ext = "ts" || ext = "m3u8") ORDER BY Requested
The results look something like this:
If we look at the url column in the results above, we see in row 1 the index.m3u8. This file lists all the bitrates available for the player to download. If I select this url, and open it in Chrome (using the HLS m3u8 Manifest Viewer extension), I can quickly read the manifest file, and identify all of the available bitrates for the stream.
In row 2, the player requests the /500/prog_index.m3u8, and we see index0 and index1 downloaded at bitrate quality “/500/”. In row 5, the player requests a new manifest: /750/prog_index.m3u8, and we see index2 downloaded at this quality. Finally, there is a request for a 3rd manifest /1000/prog_index.m3u8, and the next request is for index3 at quality “/1000”.
Each app/player/server combo has a different naming convention for manifests and for the different bitrate video files. In order quantify these details, I began manually parsing the files for the top 18 streams in the data set. Let’s see what we can learn!
Compiling the Data
Each WebPageTest run is slightly different, and each of the webpages I tested behaved slightly differently (and downloaded a different number of segments). This is a very small sample set, and each one of the sites has a slightly different behavior. But I think looking at these results can help us understand some basic tenants of video streaming.
For the 18 sites I looked at, I tabulated (from the manifest) the available bitrate streams. I then looked at the files downloaded, and identified which streams were used. I color coded the bitrates used red (first), orange (second) and yellow (third).
Let’s look at the initial bitrates set for each of these streams (the first column). The initial bitrates vary from 0.24 – 3.44 MBPS, indicating that there is a large discrepancy in how initial streaming of video is handled across the web.
A best practice for initial video bitrate is to use an intermediate bitrate for the initial stream – good “enough” quality, but also ensuring a fast download to prevent a long startup delay.
Recall that these HTTPArchive runs use a throughput of 1.5 MBPS for all data traffic. Since the video is being downloaded at the same time as other content, the max bitrate for streaming is likely around 1.3 MBPS. 11 of the 18 sites begin streaming below this threshold, and 6 are above. The median initial bitrate is 0.752 MBPS, a good balance between video quality and ensuring good delivery quality. The videos that start at very low throughputs have a pretty low range of available bitrates (none over 1 MBPS – indicating a desire for the streaming to work on mobile, but not a huge concern on the quality of the stream.
We can also use the data in the manifest to examine the bitrates that are offered by the streams. What I see is there is no real standard for video bitrates being used – the values are different for every site examined. Some of these sites appear to have mobile specific manifests – with appropriately lower bitrates/video qualities. Others have a huge range, likely meaning the same manifest is used on all device formats.
The median bitrate range is 2.18 MBPS (average 3.2 MBPS).
To ensure smooth streams and jumps in quality that are less noticeable, it makes sense to have changes in bitrate that are ‘smooth’ across the range offered (there are fewer large jumps in quality). If we look at the range and the number of bitrates, we can calculate the average bitrate “jump.” For streams with smaller ranges of available bitrates (likely optimized for mobile) we see that these jumps can be as small as 0.15 MBPS (where users are unlikely able to see any change in quality). Most of the other streams cluster their jumps in quality to around 0.5 MBPS or 1 MBPS.
Changes in Bitrate
When the player determines that the video cannot be downloaded fast enough to prevent a stall, it will change the bitrate to a lower quality. Similarly, if the player believes it can increase the quality of the video, it will request a higher quality stream.
Of the 18 streams examined, only 5 never deviate from the initially selected bitrate (28%). 10 (59%) streams change once, and the remaining 4 (22%) change video quality twice. Many changes to video bitrate leads to frequent (and sometimes noticeable) quality changes that degrade the watchability of a movie.
8 of the videos ended at a lower quality stream than initially selected, and 4 jumped to a higher quality video. The largest bitrate increase was 0.65 MBPS, while the largest decrease was -1.94 MBPS. The fact that more videos dropped in quality indicates that the initial video quality is generally being set too high for mobile devices running on 3G.
Mobile sites have around the same frequency of streaming video as desktop, however, the number of streams with downloaded video segments is much smaller. This is great, and reduces the data load on resource constrained mobile devices (and saves your customer’s data bucket!). Sites with video have a higher SpeedIndex than those hat do not.
Digging into specific examples of streaming implementations, I find that there are a wide range of video bitrates presented to mobile customers. 33% of the streams analyzed initiate at a speed over 2 MBPS, leading to a large amount of data consumption (assuming that the device has adequate throughput) , or a delay (as the player must request a new bitrate to start the playback). All videos examined have low bitrates suitable for streaming on a 3G network, even though it may take a bitrate change (or two) to correctly identify it.
Several sites have mobile specific manifests (ensuring that only bitrates appropriate for mobile devices are served), while others offer every possibly bitrate (including high quality videos that would not stream in HTTPArchive tests, but would be too high quality for a mobile device anyway).
Videos that had the fewest bitrate changes tended to start in the 0.6 – 1.2 MBPS range, a nice tradeoff in video quality and mobile data throughput.