Facebook, Internet

Fun with Statistics

There are three kinds of lies: lies, damned lies, and statistics, Mark Twain or Benjamin Disraeli

You know that feeling that’s both amusing and depressing? Is there a word for it? “Demusing?”

Nothing is more demusing than reading a breathless blog post that some new app has gotten “millions of users” in a few weeks time.

This is happening with increased frequency, as more services use the “frictionless” sharing and signup tools offered by Facebook to drive what looks like spectacular growth, which really isn’t. This is not to blame Facebook for their platform; instead, I’m asserting that it’s trivially easy to twist some of the data based on their platform to mislead people; or, more commonly and more innocently, to allow people to mislead themselves using that data.

The canonical demusing blog post used to go like this: “Look at this, this app got 42 million users in 3 weeks. They’re crushing it!” You’d dig and find out that what happened is 42 million people clicked a dialog on Facebook providing authorization to an app. With blogs posts like this one, most readers and writers alike have gotten more sophisticated, and properly skeptical, of these claims.

The newer form of these bemusing posts goes like this: “I now know that what matters is engagement, not signups. But look at this app! I checked out AppData and 50% of their MAUs are DAUs. They’re crushing it!”

Not really.

You see, the way that the Facebook API works, it can appear that a service or app has a lot of daily active users. But in almost every single case where you see a breathless, too-good-to-be-true claim about users or usage, it’s a result of the quirks of how the Facebook API works. What appears to happen is this: if your app or services asks for offline permission to read your stream, or asks for permission to access your news feed, it can appear that you are “active on a daily basis” even if you haven’t touched that app for a week or a month.

For example, a service might ask for permission to check your Facebook newsfeed in order to grab stories being shared by your friends on Facebook. If that service checks your newsfeed every day to see if there are new stories on your behalf, it will appear that you are active every day when you look at the data presented on services like AppData. Even if you haven’t opened up the app. That’s because to Facebook’s API, it looks like you’re active.

I follow a really simple rule: do the stats look better than Instagram’s at a similar point in its history? Instagram is the fastest growing, most successful app or service I’ve seen the last two years. People use it, and people love it. It has taken Instagram two years to get to 20 million monthly users (as reported by AppData). It has taken them two years to get to 7 million daily active users (a phenomenal number).  For those of us who have been following, the growth has really exploded the past 5 months.

If the stats from the too-good-to-be-true app or service are half, or even a tenth, as good as Instagram’s but the service is only a few months old, take a breath and step back. Check out the signup process, look to see if the app or service has asked to access the news feed or for other similar permissions. Do a Twitter search for the app or site URL to see if there is real, frequent use. Look at your own feeds to see if your friends or people you know use the app.

You just can’t and shouldn’t rely on the publicly reported DAUs or MAUs most apps and services using the Facebook; you’ll need to dig deeper to find out it’s as good as Instagram or instead a mirage like this.


Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s