In my previous post “What is Video Streaming”, I discussed how video streaming works at: a high level: Videos are broken into small segments that can be played one after another, allowing the video to be played while it is still being downloaded.
Playing the video while it is still downloading is a huge improvement to the customer experience, but we need to further optimize the video being delivered to our customers. In this post, I’ll look at how you can customize content for your customer’s screen size and available network throughput.
Screen Size: One Size Video Does Not Fit All
You’ve build a streaming service, and video is being delivered to your customers. But are you delivering the right bitrate and dimensions of video to the customer? A video destined for a 4K TV in your living room will have 4096 x2160 pixels (8.8M pixels). However, a smartphone may only have 2560 x 1440 (Samsung S7 – 3.6M pixels).
Streaming a 4K video to the phone will force the phone’s CPU/GPU to resize each frame. Since it can only display 3.6M pixels, and the stream is delivering 8.8M pixels, the phone must re-render each frame, and throw away over 50% of the pixels in each frame to get the picture to fit properly on the screen. That not only wastes your customer’s data, but at 30 frames per second – the phone’s processors will be working overtime to do the work, leading to heavy battery drain. Your customers notice heavy battery usage because that causes the battery (and the phone) to get warm to the touch.
Another factor to take into consideration is that not everyone is on the same speed network. Some streaming TV viewers will be on gigabit fiber, while others will be on DSL (or even hotel Wi-Fi <shudder>). Your mobile users might be on LTE, but could also be on a slower 3G network. Sending 4K video to customers on slow connections will lead to a slow and frustrating experience.
Adaptive Bitrate to the Rescue
Adaptive bitrate occurs when multiple resolution/bitrate combinations of the same video are available for download. When a video stream is set up, a video manifest file is delivered to the player, itemizing the available streams. Like a restaurant menu, the manifest tells the player what resolutions and bitrates are available, and the player chooses an appropriate stream. If network conditions change in the middle of the stream – the player can change the version being downloaded to ensure that video continues downloading (it “adapts” the bitrate J).
Here is a list of common adaptive bitrates from a popular streaming provider (I have added the color column to simplify the figures below… and I like rainbows). We can assume the manifest tells the player that these formats of video are available:
We see that as the ID increases (and the color moves form ROY to G to BIV), the video dimensions and bitrate increase.
Adaptive Bitrate Stream Start
When the player reads the manifest, it knows the device’s screen size, but it has no understanding about the network (or network conditions). So the player will pick a video quality that is “middle of the road” to make sure that the video starts playing quickly. In the example above, it might initially select level 2. The first few segments are downloaded at level 2, and if network conditions are too slow, the player may switch to 1, but if the network speed appears to be fast – it may attempt to increase the video quality to 4 or 5.
We can see this occur in AT&T’s Video Optimizer Video tab. In this case, the initial quality is 2 (~600kbps), as the player recognized faster network conditions, it jumped to quality 4 (~1.5 Mbps), and then continued to play at 1.5 Mbps:
If you look carefully at the example above, segment 1 is downloaded twice: at qualities 2 (yellow) and 4 (blue). The player realized that the throughput could hander a higher quality video, and upgraded the quality. It had time to replace the 600 kbps (yellow) version of segment 1 with the 1.5 Mbps (blue) segment. This is called segment replacement:
The player ends up discarding 288KB of data (segment 1 at quality 2), but the user is treated to a higher quality video (segment 1 at quality 4), leading to a better video experience.
Video Quality Downgrade
Unfortunately, in this trace, the network conditions were not adequate to handle video quality 4. The player realizes continuing at quality 4 will consume the buffered video faster than it can be replaced (leading to a stall). It requests a lower quality 0 for segment 7 to keep the video playing, and to avoid a stall.
Segments 7-9 are downloaded at quality 0.
Video Optimizer collects the video on the screen at 30 frames per second. If the DRM in the application allows it, that means the video is recorded as seen on the device. In this case, we can see that at the outset, the video quality is sharp, but then the quality level drops, as seen by the jaggy and pixelated video:
However, the video never stalled, which is paramount.
Picking and Choosing Bitrates
At the outset of this post, a table of selected bitrates (with the rainbow) is listed. I have no doubt that videos utilizing this schema look great on high speed networks – bitrates from 1.5-8 Mbps are well represented. However, for any network under 1.5 Mbps, viewers will have just one option: 280 kbps 320×240 video. Consider that many 3G connections will be below 1.5 Mbps, and you may not be serving the highest quality video to your mobile customers.
Of course, adding additional bitrates for many videos will have a large storage cost, and therefore should be applied carefully. In this case, one option might be to remove one of the two 1.5 Mbps streams, and replace it with 2 streams at 600 kbps and 900 kbps (since they add up to 1.5 Mbps – the storage cost will be essentially the same). Now customers with cellular throughput under 1.5 Mbps will receive a better streaming experience.
In general, smaller jumps in quality will allow the player to make smoother transitions between video quality levels, and makes large degradations in quality less likely. (look at what Tim K said in the London webperf on images).
In addition to streaming files – adapting the quality of the stream to the device type (screen resolution and network type) ensures that the right amount of data is used to present the correct quality video. If network conditions change, the player can adapt the stream to suit the conditions – lowering and raising the video quality to ensure that the stream continues without stalling, but also at the highest possible quality for the device in use.
In future posts, we’ll look into startup delays, stalls, encodings, and other streaming methods like variable bitrate.