Exploring the differences in social media measurement tools

For the last couple of years, I have used a combination of three different social media measurement services to help me gather information about Bemidji State’s presences on Twitter and Facebook — I started by using CrowdBooster, added Sprout Social a bit later, and also spend some time with Twitter’s own analytics and Facebook Insights.

I always ended up incredibly frustrated by the process, because while each service provided some small interesting element that the other didn’t, the basic data was never consistent across platforms.

Let’s take a look at Twitter, and I’ll show you what I mean. Let’s look back to the week of April 26-May 2, 2015 — Finals Week for BSU, and the week leading up to Commencement on May 8, which meant it was a heavy week for us. So let’s look at what each of these three measurement tools shows for our activity during that week.

CrowdBooster:

You can click to enlarge and see everything, but to hit the high points Crowdbooster claims that from April 26-May 2, we tweeted 127 times with 75 favorites, 68 retweets, and 4 replies.

Sprout Social:

This is from Sprout Social’s brand-new Twitter reports, which they launched on July 30. Again, click to enlarge; it shows that we tweeted 82 times, and elsewhere in the report that we had 123 favorites, 70 retweets and 10 replies.

Twitter Analytics:

I exported Twitter’s data as a .csv and put it into Google Sheets for this report. Twitter’s data shows 82 tweets, 113 favorites, 69 retweets and 7 replies.

It doesn’t take much data analysis to see that these numbers are all over the board. Sprout Social and Twitter agree that we sent 82 tweets; Crowdbooster’s number of 127 isn’t even in the same neighborhood. Favorites are different across all three reports, as are retweets and replies, but the Crowdbooster numbers are in some cases lower in spite of its claim that we sent 45 more tweets — it reports 4 replies, for instance, while Sprout reports 10 and Twitter reports 7.

For a long time I have just written this off as “this is why you cannot take social media metrics at all seriously.” It never occurred to me until recently that at least Crowdbooster and Sprout were likely measuring entirely different things — in ways that on the surface are essentially invisible. It’s a puzzle to be solved, and while Google searches have turned up feature comparisons of Sprout vs Crowdbooster, I haven’t found anything that dives into the differences in what the tools report.

So if this is a puzzle, let’s try to solve it.

I have created three Google Sheets for this examination containing exports of the data each of the three services generated for my April 26-May 2 tweets: one for Crowdbooster, one for Sprout Social and the previously-mentioned one for Twitter Analytics. These will be our starting point.

The first discrepancy I want to explore is the difference in the number of tweets being reported by Crowdbooster, 127, as compared to Sprout and Twitter, which both agreed was 82. The first step there is to make sure Sprout and Twitter really are in agreement about what we sent, and the numbers are not just coincidentally the same. To do this, I simply reviewed both lists of tweets side by side in two Sheets windows. They did, in fact, completely agree; every tweet in the Twitter record was also listed in Sprout, and vice versa. So, to explore why Crowdbooster is reporting so many tweets, I did the same thing and compared that list side-by-side with Twitter’s.

Those lists are more difficult to compare. Twitter’s list starts with the most-recent tweet at the top, then descends to the oldest tweet in the time period you’re measuring. Crowdbooster starts with the oldest tweets at the top. Immediately, a discrepancy – Crowdbooster did not include this tweet in its list which was on both the Twitter and Sprout Social lists. But, that tweet was auto-posted by Sprout Social at 8 p.m. on April 25 — the day before we said we wanted to begin our measurement. It shouldn’t have been on any of the lists, but both Twitter and Sprout had it.

Five tweets into the list, it’s clear why Crowdbooster is reporting a heavier tweet load – it’s including retweets. I can see both sides of the argument on whether retweets count as “our” content – on one hand, it’s not original content posted by us, but content that originated elsewhere. On the other, we’ve made a concious decision to put those tweets into our timeline and share them with our followers. This is a decision that is actively made for every retweet, and retweets are something that we discuss in our overall talks about how we’re using Twitter. By giving someone a retweet, we’re bringing them into our timeline and making them part of our content stream. I believe they should count as “our” tweet for the purposes of counting our monthly activity.

But, Twitter doesn’t; my next step was to count the retweets in Crowdbooster’s list and see if there were 45. If there are, the entire discrepancy between tweet activity can be attributed to our RTs — which would be fantastic, because segmenting out that information could be incredibly useful.

However, only 40 of our tweets started with an RT during that week. Five are still missing. And, I discovered that Crowdbooster is including tweets we deleted (like this one, which had a typo (you’ll have to take my word for that since the tweet no longer exists) and was immediately replaced with this one) in its total. I counted five deleted tweets that were being included in the Crowdbooster report – that means the entirety of the discrepancy between Crowdbooster, Sprout and Twitter has been identified.

Twitter and Sprout Social are including only original tweets that are currently active and sourced from your account in its list of your activity for the month. Crowdbooster is including all of those, plus any retweets and it’s including deleted tweets.

This discovery actually leads to a significantly more-important question – if we deleted a tweet in April, why is Crowdbooster still able to find that tweet and include it in a report that I ran on June 30? Twitter’s own support documents say:

  • When you delete a Tweet, it is removed from your account, the timeline of any accounts you follow, and also Twitter search results.

Obviously, these tweets are not actually deleted in the sense that anyone thinks “deleted” means. Twitter is preventing a deleted tweet from being displayed on your timeline, but it obviously still exists on Twitter’s servers if Crowdbooster can parse and find it. That’s a problem.

I also discovered that Twitter and Sprout don’t necessarily report times of tweets accurately – for example, this tweet was posted at 10 p.m. on April 30. Crowdbooster reports it as having been posted on April 30. Twitter reported it in its analytics as being posted at 3 a.m. on May 1. That’s +5 hours, which indicates a time zone might be set incorrectly somewhere; however, all are well – Twitter is correctly set to GMT-5, as is Crowdbooster, and Sprout Social is correctly set to UTC-6. In some cursory explorations I haven’t yet been able to find out why this is. It bears further exploration.

However, none of the mystery time zone issues or philosophies behind whether RTs should count as “our” content are as important in my mind as discovering the fact that deleted tweets are, in fact, not deleted.