So Many 3rd Party Analytics, So Little Time

When it comes to content on the web, be it a video, a webpage or a smartphone app – speed matters.  Recent research has shown that 89% of consumers are likely to recommend an brand after a good experience on mobile, while those who have a negative experience are 62% less likely to purchase again.  A three second delay leads to over 50% of customers abandoning the site.

Getting to Fast

Tools like AT&T’s Video Optimizer can help you test your mobile videos and apps to ensure that they transfer data efficiently – helping to ensure a fast user experience in your services.  Testing with Video Optimizer should be run constantly during development and before launch, whenever changes are made to the way content is delivered to your mobile application.

Once your service is ‘in the wild’ with customers, you are not done worrying about the performance of your site or app.  It is important to monitor how your application is being used by your customer base. Are your customers in Asia getting the same loading experience or video quality as those in the US (or the EU, or South America)?  Are there CDN issues, or content issues that are not easily found with basic testing?

Luckily, there are dozens (if not hundreds) of analytics tools that provide real user measurements (RUM) on crashes, performance, video streaming, and more.  By inserting small libraries or SDKs into your app, you can immediately begin seeing how your customers are interacting with your content, where errors are occurring and if performance is lagging.  Acting on this data is another important step in ensuring that your customers are using and satisfied t=with the experience in your mobile app.

In my testing of mobile apps, I have always seen calls to 3rd party RUM providers (and, again, I think this is awesome!).  But, sometimes, I see applications that appear to have taken the idea that “if one set of RUM data is good, using 6 (or more!) will give us even more insight!”

This is not always the case.

In Quantum Physics, Observations Affect Outcomes

The theoretical physicist Warner Heisenberg is famous for his Uncertainty Principle.  It states that the more accurately you measure one property of a system, other measurements become less precise.  Known outside of quantum physics as the ‘observer effect’ it basically means that the more you study and perturb a system – the more you actually change the system and affect the outcomes.

Can using RUM affect my mobile experience??

In general, no.  The creators of RUM analytics tools have done a great job to ensure that the data they collect is sent in a way that does not block or slow delivery of content to the end user.  But there are rare occurrences where a multitude of analytics calls have blocked application rendering, and here I will present one example:

Mobile Video Delayed by RUM

In the below example screenshot from Video Optimizer, we are looking at the packets and information from a trace where a movie is being selected, downloaded and played.  At ~73.2 seconds, a movie was selected from a list  (the blue bar in the User Input row notes that the screen rotated from portrait to landscape to begin playing the movie):

Screen Shot 2017-08-03 at 10.21.19 PM.png

As the screen rotates, the density of packets transmitted (Packets UP and Packets DL rows) vastly increases to an almost solid bar of gray lines.  This might lead you to believe that the video has begun downloading for playback.  But look at the top row or the chart – the throughput.  When video is being downloaded, you would expect that a LOT of data is being transmitted, and that does not happen immediately – it happens some time after 75 seconds.  So, the user clicked on a video, and there is a lot of data being transmitted  – but it is not a video.  What is happening in the 2 seconds before the video starts downloading?

Let’s take a look at the connections taking place. This data is presented in the diagnostics tab in Video Optimizer below the chart above).  The top row (in dark blue) corresponds to the time the screen rotates.  There are then 14 connections (one per row) established to various servers highlighted in light blue before the video begins downloading at 75.361s (the last row in the chart).

Screen Shot 2017-08-03 at 10.42.31 PM.png

The last connection is clearly the connection with the video file, as it is the only connection in the table with a large amount of data transferred (in the byte count column you can see a transfer of ~4.9 MB, where the other connections are all under 50 KB, and mostly under 8 KB).  Many of the 14 connections that begin prior to the video streaming belong to video RUM measurement providers (I have intentionally obfuscated the domain names and IPs as the issue is NOT the RUM providers, just the quantity of RUM providers in use).  With so many different providers initializing their services – the video is actually delayed nearly 2 seconds before it can begin to download.

Video Optimizer also records the screen at HD, so know that the time from download of the video to playback is 1s.  Normally, a 1 second playback is good but when the 2 seconds of additional delay is added in, we have a performance issue.

Network Conditions Accentuate the Effect

Research has shown that startup delays over 2 seconds lead to video abandonment.  We are seeing a delay of 2-3 seconds on a fast mobile network.  What happens if we turn down the knob, and repeat these experiments on a slower network? The network attenuation feature in Video Optimizer allows us to run the same test, but in different network conditions.  The results are not surprising – as the network speed decreases, the delay to video playback grows – and grows quickly.

Screen Shot 2017-08-03 at 11.55.06 PM.png

What we find is that if the network throughput is under 5 MBPS, the connection delay nearly doubles to 4 seconds, and the video playback time jumps to over 5 seconds (and sometimes over 10 seconds!).

Conclusion

This application is not alone.  More and more applications are adding more and more measurements to ‘stay on top’ of how their customers are using their apps.  And again, I think this is a good thing.  But it is also important to know what impacts all of these SDKs have on the end user experience.

So test your app.  Use RUM tools to measure how your customers interact with your application.  But when choosing the data you collect, also realize that as you add more third party connections to measure – the measurements can actually impact the performance of your service.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s