There's No Such Thing as an "Average" Experience
Previously, we’ve talked about leveraging the power of the device to eliminate last mile issues, and how it’s super important to not break the immersion of the viewing experience for the user. We talked about how Disney is throwing down an extra $0.18 per asset almost across the board, just to make sure their users get the most immersive experience possible. And we talked about the steps they’re taking to make sure their users have the lowest possible chance of being broken out of that experience while they’re having it.
But here we’re asking the question of how do you know if you have a last mile issue? How do you know you need to do something about that? And is it worth it to put effort into bettering that experience?
The most common way to analyze performance and do cost-benefit analyses is looking at averages. i.e.: What is your average user experiencing when using your product?
If your analytics are coming to you in averages, you may think everything is okay. Just get that number as low as possible and everything is great, right? Your numbers may be telling you that you don’t have an issue. For example, your data could be saying that of all your streams, only 0.6% are experiencing re-buffering events... On average.
If every customer experienced a 0.6% re-buffering rate, they are not having an immersive experience at all though. A 0.6% re-buffering rate translates to watching the spinning wheel for 11 seconds on a 30 minute episode or 43 seconds on a 2 hour movie. Talk about breaking the immersion.
Of course, that isn’t how averages work. Averages are an aggregate measure, and datasets that have heavily populated outliers can hide what the majority of people are experiencing. The truth is, you have many customers who aren’t experiencing any re-buffering, and the rest who are experiencing some level of re-buffering events, ranging from a tiny number of occasional issues all the way up to almost unwatchable content. These people are ‘bringing up the average’ but still being largely hidden beneath the weight of the majority who generally have no issues. Imagine being in a room full of people who are all discussing the great experience they’ve had with some vendor or service provider, but your experience with that same provider has only ever been negative, for whatever reason. Do all of the other people’s positive experiences invalidate your negative experience? Do you feel like that company is doing a good job all of a sudden just because everyone else thinks they are?
Now, imagine that provider is serving up your favorite content in their shiny new app. How long are you willing to put up with that spinning wheel in your viewing experience? How many times are you going to be okay with a network issue yanking you out of the immersion of the experience? And is it worth $0.19 instead of $0.18 to ensure your users aren’t seeing those events? Not being pulled out of the experience before the careful world building of the writers wants them to? What if it was only $0.185, or $0.181? How much do you think Disney would pay, without even really thinking about it, to insure their viewers never saw that little spinning wheel?
Averages make data much more consumable, but they don’t tell the story of the individual, and individuals tell stories about their experiences. Often to anyone who’ll listen to them.
So tell me… How do your individual users feel about their experience? Is an average metric obscuring someone’s negative experience from you when you could be making it flawless?
How much would it be worth to you, or more to the point, to them, to never be pulled out of the experience while it was happening, ever again?
In the above image you can see what this looks like on a graph. The dashed line represents what your average is telling you (the median) your re-buffering problem looks like, while the blue line (the mean) is what that average is hiding. This could potentially represent whole categories of user experiences that averages in your reporting are hiding from view.
Certainly not something you’d want to overlook, right?
