Alternate title:

McFib?

Thanks to Mike Zavarello (@brightmatrix) for the #McFib suggestion. I wish I had thought of that.

Okay. This is not going to be a kind, warm and fuzzy post. Before I start, let me say this: I would much rather have a brilliant Social Media case study to champion. A story about a company doing something right, with data to back up its results, clean metrics to boot, and flawless analysis from start to finish. Wieden + Kennedy’s Old Spice campaign will soon be the subject of such a post, when more sales numbers are made available. But as things stand, this post is not going to be about a company that got things right.

What we are going to talk about today is either a case study in either poor journalism (Mashable reported on the story without questioning and verifying the math), or the origin of questionable claims by a McDonald’s representative about a Foursquare promotion’s alleged outcome. I don’t want to speak ill of the company (I actually like McDonald’s), but I can’t not bring this up.

(UPDATE: 19 September 2010) Look for an update to this story at the end of the post.

Here is how the story begins (via Mashable):

At the Mobile Social Communications conference yesterday, [Rick] Wion shared that McDonald’s was able to increase foot traffic to stores by 33% in one day with a little Foursquare ingenuity. McDonald’s total cost for the successful campaign was a measly $1,000.

Econsultancy reports that McDonald’s, with Wion driving campaign direction and strategy, opted to try and take advantage of Foursquare Day (4/16) to bring in more business. The company used 100 randomly awarded $5 and $10 giftcards as checkin bait to lure in potential diners. The bait also worked to attract the media’s attention and resulted in more than 50 articles covering McDonald’s Foursquare special.

The campaign worked in both digital and real world capacities. Patrons flocked to McDonald’s restaurants for the chance to win giftcards in exchange for checkins, and 600,000 online denizens opted to follow and fan the brand on social media sites.

The Econsultancy piece is here. Sounds great, right? Here is what you just heard: McDonald’s spent $1,000 on a campaign, leveraged Foursquare to get people in the door, and increased foot traffic by 33%, resulting in some unknown but probably decent spike in business (sales.) Except no. That isn’t what happened. There’s also this:

Of course, the metric here was checkins (not sales), and there were likely several other factors contributing to the campaign’s success, but it’s still a story that many an agency should pay heed to.

Let me quote that again, in case you missed it:

“The metric here was checkins (not sales).” Or actual foot traffic, apparently.

Oh. Take the pig off the spit – the celebration might be a little premature. Right idea, wrong execution, and horrendous analysis. Let me explain.

1. In spite of its “conclusion,” did McDonald’s measure actual foot traffic?

This was a test. The premise was this: If we spend $1,000 on a Social Media promotion using Foursquare, can we get more people in stores? The most important metric to measure then, both as a baseline and an ultimate outcome would be what?

Foot traffic. Actual foot traffic. As in… someone with a clicker standing at the door, a laser in the door way recording how many times someone passes through it, or a sensor in the door recording how many times it opens and closes. Unfortunately, unless Mashable forgot to mention it, McDonald’s did not actually measure foot traffic.

Wait… what?

That’s right: Though Rick Wion claims to have increased foot traffic by 33%, nowhere does the piece mention that McDonald’s actually measured foot traffic. The only metric mentioned: Foursquare Check-ins.

The one thing McDonald’s should have measured, the very metric Rick Wion is reporting jumped by 33% in one day, is the one metric that may not in fact have been measured at all.

Here is how you measure deltas/changes in foot traffic:

1. Lock in your baseline (the average daily foot traffic before the campaign).

2. Measure average daily foot traffic during your campaign.

3. Compare the two.

What you measure are actual people actually walking into actual stores, not people checking-in from their phones three blocks away or clicking a “like” button on Facebook from their desk, or clicking “follow” on their twitterberry. You measure people in the stores.

If McDonald’s Wion wants to report on a win (as well he should), it should be this: “With only $1,000 investment, we increased Foursquare check-ins by x%.” That is what actually happened. This is the extent of a win. It isn’t as sexy as claiming a 33% increase in foot traffic, but it’s a start. A proof of concept. Something McDonald’s can now build on and expand.

I now leave Ronald McDonald’s Social Media team and Rick Wion to explain how exactly how check-ins conveniently transformed themselves into foot traffic.

2. Metrics are not randomly interchangeable:

Here’s a simple, basic, no BS fact of Social Media performance measurement: A Foursquare check-in is not foot traffic. Foot traffic is not a Foursquare check-in. Here is the difference between the two:

– Foot traffic is someone coming into your store.

– A Foursquare check-in is someone within five city blocks of your store pushing a button on their phone.

Case in point:

@lizy_dee: I just checked in to some McDonalds up the road while I’m sitting on my couch at home.. #fail

Likewise, mentions are not sales, and new followers aren’t new customers. Metrics are not randomly interchangeable. Metrics are specific. Don’t measure one thing and magically turn it into another to embellish your quarterly report and get some media attention.

3. Contests are not strategies. People will do anything for free stuff and then move on:

“Do this and you might win something. One day only!” Here, McDonald’s has at its disposal the combined power of the Social Web and mobile phones, with a potential reach of hundreds of millions of people. It has a chance to increase net new customers, increase customer loyalty, earn mind-share, impact preferences and customer habits in the long term… But no. McDonald’s big Social Media idea was to attach the typical one-day-only contest carrot – which requires neither Foursquare nor Social Media of any sort – to get people into its stores for one day (which it may not have actually accomplished since it measured the wrong thing).

I will, however, grant McDonald’s Wion this: His experiment proved that promotions in which people might win free stuff if they push on a button will make some of them push the button. Unfortunately, we already knew that. We can get lab rats to do it as well, even without a social media strategist.

The fact is that tying Foursquare check-ins to a promotion doesn’t prove that Foursquare will work for you, or that you know how to leverage it yet. The Foursquare check-in feature is a button people push to get their prize, not the reason why they came to your store. Let me illustrate:

Replace the term “Foursquare check-in” with “funny hat”.

If your promotion had asked people to wear a funny hat to your stores for a chance to win a gift card, would you be saying that funny hats helped you generate 33% in additional foot traffic? Here’s an example:

At the Mobile Social Communications conference yesterday, [Rick] Wion shared that McDonald’s was able to increase foot traffic to stores by 33% in one day with a little funny hat ingenuity. McDonald’s total cost for the successful campaign was a measly $1,000.

The company used 100 randomly awarded $5 and $10 giftcards as checkin bait to lure in potential diners. The bait also worked to attract the media’s attention and resulted in more than 50 articles covering McDonald’s funny hat special.

Patrons flocked to McDonald’s restaurants for the chance to win giftcards in exchange for wearing funny hats.

See what I mean? What’s your Foursquare strategy again? Is it the same as your funny hat strategy or your pink tie strategy or your flip-flop strategy?

Here’s a little dose of reality: This was a promotion just like every other promotion before it. Push the button we ask you to push, give us the magic password, and maybe win something. Foursquare was an accessory, not the catalyst. Even if McDonald’s had actually seen a 33% increase in visits to their restaurants on that day, the gift card promotion, not Foursquare, got people to participate.

4. “It’s still a story that many an agency should pay heed to.” Um, no. Absolutely not.

Unless what you want to do is teach agencies that not measuring relevant campaign outcomes, mistaking one metric for another, making up numbers entirely, not understanding basic Social Media marketing strategy and having no clue how to tie campaign metrics to business metrics is the right path.

There is already plenty of that going on on. We need less of it, not more.

5. Take your case studies seriously: McDonald’s should have measured conversions. Here’s how to do it:

What Rick Wion and his team should have done:

600,000 new fans and followers? That’s “Reach.” That’s your starting point. Here’s your conversion chart:

Reach → Response → Visits → Foursquare Check-ins → Transactions → Revenue (then repeat)

Don’t know how to put these things together? It’s simple:

Reach, we’ve already touched on.

Responses can be anything from RSVPs on Facebook, to registrations or sign-ups on a microsite, to mentions to RTs on Twitter.

Visits are your first real measurable conversion. You actually have to count visitors. Not estimate but count. If the suggestions I made earlier in the post are not feasible, look into software that will allow you to count visitors by analyzing your CCTV feeds. (Your surveillance cameras capture foot traffic in and out of the store, and to the cash register. Put them to good use.)

Foursquare Check-ins are simple. By the way, since these check-ins have location data, you do realize that you can map check-ins nationally and therefore assign check-in volume to specific stores, right? So, you can see what areas are responding well to your campaign, and what areas are not? In terms of targeting, this is gold. I hope McDonald’s Social Media team is paying attention. Rick, your team knows how to do this, right?

Transactions are pretty basic. Now that you have your foot traffic data, not only nation-wide but also per individual store, you can see store by store what the deltas/changes are. Now that you also have Foursquare check-in data store-by store, you can overlay that data with your foot traffic data. This gives you a nice composite of increased foot traffic per store + volume of check-ins per store. See where I am going with this? Okay, now, the last bit of data: Look at the volume of net transactions per store next to these two metrics. What you are looking for is this: Stores in which foot traffic, foursquare check-ins  and net transactions rose together. This is where you begin to see the path from action to reaction to R.O.I.

Revenue is the final metric you need to calculate the R.O.I. of your $1,000 campaign. You’ve already looked at foot traffic, check-ins and net transactions. The “net new” numbers in the data set that show an increase in all 3 is where you can PROVE that your activity had an impact on business performance. There are several ways of assigning a dollar value to your campaign’s net gain, but the fastest is to calculate the corresponding percentage of net $revenue to the percentage of net new transactions in that data set. This is your hard number: The 100% unquestionable net gain (in revenue) from your investment. If your campaign cost $1,000, you can now prove that revenue generated that day by your campaign was $x. With this data, you can present the board (and the general public) with a real case study. One with an actual process, proper measurement, and measurable facts including R.O.I.

Behind closed doors, you can also look at collateral outcomes: Those you suspect but cannot prove 100%. This would be an increase in sales revenue that can be attributed to increased foot traffic, but not Foursquare check-ins, for example.

That is how it is done. I suspect that other people at McDonald’s already know this, which is why the company has been so successful.

Note: The campaign’s cost was not merely $1,000. Also factor in the salaries of McDonald’s Social Media team, the equipment they use, the software they use, the training they have received, the conferences they attend, and other costs associated with their “management” of this campaign to get a more accurate value.

6. Mashable, when someone hands you a cup of Koolaid, check it before drinking it.

Mashable, I love you. I really do. If you were a person, I would hang out with you a lot. In a way, we already do. And it is because I love you that I have to tell you to be careful: You’re the popular kid in the class who can help make reputations. If you like someone, suddenly everyone likes them too. You can make nerds popular with a mere wave of your magic wand. That makes you a target for clever PR people with a story to sell and the talent to sucker you in. Usually, you smell what’s cooking. This time, I think you jumped too soon on the opportunity for a scoop. And yeah, your traffic is pretty solid. You’re going to be generating a lot of advertising dollars from this McDonald’s story. But you didn’t have to buy into it hook, line and sinker:

“Foursquare ingenuity” – Except it wasn’t. There was nothing ingenious about the campaign or its use of Foursquare, not to mention that the measurement piece is a disaster.

“the successful campaign” – No.

“netted the company measurable success” – No again.

Patrons flocked to McDonald’s restaurants” – Nicely written, but still no. Is there proof of “flocking?”

Or is it that you were so impressed with what you heard that you forgot to mention the other metrics used by McDonald’s? Metrics like… actual foot traffic? Or transactions? Anything that might back up Rick’s assertion that foot traffic increased by 33% on Foursquare Day? If McDonald’s has this data and you failed to report it, (even going as far as to say that “the metric” was Foursquare check-ins), you just made Rick look bad.

Whatever the case may be, Mashable, you can do better than this.

What can we learn from this? Several things:

1. With many companies desperate for a Social Media win, questionable “case studies” like this one are bound to start popping up at conferences and on sites like Mashable with increasing regularity.

2. Social Media Directors with little understanding of the space in terms of strategy, program development, Social Business planning, program integration, program management and performance measurement are still the norm. (This will be the topic of my next post.) This may or may not be the case with Rick Wion. I don’t know that yet.

3. What this kind of story illustrates is this: When few people understand how to leverage the Social Web to create specific business outcomes and measure them properly, Social Media “experts” can make a complete fiasco of a campaign sound like a Social Media knockout victory with a few carefully selected words.

4. Headlines and anecdotal “evidence” are so easily consumed that facts tend to get lost in the PR copy, and trusted sources don’t always understand their subject enough to spot red flags. Even eConsultancy and Mashable are vulnerable.

5. A tough job market for Marketing professionals, the promise of fame, influence and perhaps a book deal, and little understanding of proper Social Media measurement from the C-suite combine to create the perfect environment for less than qualified individuals to pose as qualified Social Media Directors.

Many companies have them in their midst and have no idea that their Social Media experiment has been put in the hands of someone with absolutely no idea what they are doing. Whether the McDonald’s/Foursquare experiment turns out to be the result of poor methodology or poor reporting by Mashable and eConsultancy is still unclear, but it does illustrate the danger of operating in a medium that so few executives, marketers and journalists understand. My inbox and DM column are full of questions from “Social Media Directors” and VPs who pass themselves off as experts yet show a disturbing lack of even basic Social Media knowledge. Just because someone’s job title is “Social Media Director” doesn’t mean they even know how people use Twitter. (I am not exaggerating.) The fact that I still have to explain R.O.I. to Social Media “experts” is also indicative of the gap between “perceived” knowledge in the industry and “actual” knowledge in the industry. Someone’s job title does not guarantee their expertise, especially in this field.

This is the kind of BS that drives me ba. na. nas.

UPDATE DETAILS: 19 September 2010

Mashable has changed the title of its story to “McDonald’s Foursquare Day Campaign Increased Checkins by 33%” and added the following caption:

Update: McDonald’s head of social media Rick Wion tweeted that, “the 33% increase was in the number of checkins. We consider checkins the same as a person entering the restaurant.” (Original tweet)

This helps explain this statement from Rick Wion last week:

“I was able to go to some of our marketing people — some of whom had never heard of Foursquare — and say, ‘Guess what. With this one little effort, we were able to get a 33% increase in foot traffic to the stores.'”

Kudos to Mashable for catching the error of their original title and article, which supported Rick’s assertion that a 33% increase in foot traffic had in fact resulted from its Foursquare Day promotion.

No data to support that an actual 33% increase in foot traffic occurred on that day has yet been produced by Rick Wion or McDonald’s.

Yet, from Rick Wion’s own blog:

Note the headline: “33% More Foot Traffic.”

Tip to business owners and C-suite executives: When your “Social Media Director” doesn’t know the difference between an increase in Foursquare Check-ins and actual foot traffic in your stores, consider it a giant red flag.

If you don’t know why, here it is. In order for your business to be able to substitute Foursquare Check-ins and foot traffic, you need all three of these elements to be true:

1. 100% of your customers must be Foursquare users.

2. 100% of your customers must check-in to Foursquare 100% of the time when they visit one of your stores.

3. 100% of Foursquare users are always honest about their check-ins. No one EVER registers a false check-in to game the system.

These are the requirements. If you do not have all three of these elements in place, you cannot ensure a match between Foursquare check-ins and actual foot traffic in your stores.

As it is, McDonald’s (or any business anywhere) would be hard-pressed to establish that even 20% of their customers are Foursquare users, much less 100%.

You can see where Rick’s belief that Foursquare check-ins are the same as a person entering the restaurant,” his statement “with this one little effort, we were able to get a 33% increase in foot traffic to the stores,” and the title of the September 17 post on his own blog are… problematic.

At the very least, this case illustrates the danger of assuming that a Social Media Director is qualified to develop and manage a Social Media program just because s/he holds a Director-level position with a reputable company or major brand.  More often than not, they are not.

For an excellent analysis of the McDonald’s “campaign,” I recommend that you read Thomas Baekdal’s post on the subject, which offers numbers and real data. Click here.

Among the most insightful bits of data:

The underlying problem is that McDonald’s [serves] about 26 million customers per day, meaning a 33% increase equals 7.8 million foursquare users. But, Foursquare has only a total of three million accounts (and only one million active users). You don’t have to be a math wiz to see that something is wrong.

And then this:

In terms of tweets then McDonald’s claim that nearly 600,000 people potentially noticed it via Twitter, and still only 719 people reacted to it. That is a conversion rate of 0.12%, which is below most click-trough rates of website banner ads.

It really doesn’t get any simpler than that.

UPDATE DETAILS: 21 September 2010

In this follow-up piece from eConsultancy (click here), Meghan Keane brings up the same check-in vs. foot traffic questions raised here already, and goes back to Rick Wion for clarification:

McDonald’s, boasted that a special McDonald’s ran on Foursquare Day in April increased foot traffic at McDonald’s 33%.

That’s no small number, and since not all 26 million McDonald’s customers are on Foursquare, it was a bit unclear what, exactly, had increased 33%.

With commenters and bloggers questioning the results, I followed up with Wion. As it turns out that McDonald’s results were not measured in actual foot traffic. According to Wion:

“We measured check-ins, not foot traffic, transactions, or sales, on Foursquare Day.”

That’s an over 20 million customer difference. However, Wion still stands behind his results. And he’s using this case study to justify more social media offers at McDonald’s.

Wion used the term foot traffic because he found it easier to convey the impact of the campaign:

“This kind of program was uncharted waters for us, so we needed to develop a means of measurement that translated to other commonly understood metrics within the McDonald’s system.”

No.

Note the statement “Wion used the term foot traffic because he found it easier to convey the impact of the campaign.”

Except it didn’t. What it did was completely misrepresent the impact of the campaign.  A cardinal rule of project management: You do not misrepresent results and outcomes. Ever. Not accidentally and certainly not deliberately. Everyone would find it “easier” to convey the impact of a campaign by embellishing it. Most of us, however, don’t.

The result of this alleged “decision” to substitute foot traffic for check-ins:

As a result of this breach of trust, the validity of future McDonald’s Social Media studies will unfortunately be received with suspicion, at least when they are initially released.

Rick’s trustworthiness when it comes to accurately reporting on campaign outcomes is now in doubt, especially given his choice to continue to “stand behind his results” rather than acknowledge his error, which he clearly understands.

This constitutes an unfortunate and unnecessary black eye for the fast food giant’s Social Media activities.

Poorly done all around.

Perhaps the medical profession’s “first, do no harm” ethos should be applied to the Social Media profession.

The saddest thing about this whole mess was that on its own, however small it may have been, the campaign was a success. A 33% increase in Foursquare Check-ins was something positive to report on. A basic a proof of concept. Had Rick stuck to the facts, however small, however insignificant outside his organization, he could have built upon this test with ease. In terms of moving things forward for McDonald’s in the Social and LBS space, gaining the trust of decision makers internally, building momentum for his strategies across the McDonald’s organization, the truth would have worked just fine. There was no need to misrepresent the study’s outcome.

On a more promising note, it is good to see eConsultancy and Mashable updating their reporting of this story. With a little luck, when pitched with future claims like this one, both outlets will remember what they learned this past week about verifying a company’s claims before reporting on them, especially in the still shady world of Social Media measurement.

Kudos to them for the quick turnaround.

McDonald’s, I like you a lot, but something isn’t right with your social media team in the US. You might want to take a closer look at what is going on in that department before someone does something embarrassing again. (Let’s hope this was just an accidental hiccup.)