Harvard Smartphone Study Turns Up Bunk

A so-called usability study has been performed by Professor Dennis Gallatta at the Harvard Human Factors in Design lab comparing one device from each of the following categories: BlackBerry, Windows Phone, Android, and iPhone. A video accompanies this test and finds results on 3 "tasks" that users who have supposedly never used the phones at hand before. Watch with us as they fumble through making a call, adding a contact, and sending a text. What we will show in the following text is that this is, and I quote a colleague of mine here and agree, an unscientific half-baked study by morons.

The test below is all in regards to the study seen in the video directly below this paragraph. Our conclusions here assume that this study is a real study as actually done by a Harvard Professor for use in a scientific manner. As you'll find throughout, this isn't so much the case.

The first device up for testing is the Samsung Focus running some version (we're not told) of Windows Phone 7. The tester, supposedly never having used a Windows Phone device before, fumbles trough the process, getting quite close very quickly to the goal of making a call. The study is given a score of 3 stars out of 5 – what they've been scored on though, we're not told. Next is the BlackBerry Storm again running some form of the BlackBerry OS, but which version we're not told. This device gets 4 stars out of 5 – why? We have no idea!

Then comes the kicker – the iPhone 4G. What's that, you say, there's a 4G version of the iPhone? No, not actually – in reality, Gallatta is one of the thousands mistakenly thinking that the "4" in the name actually signifies that the device is running on a 4G network of one kind or another. The iPhone "4G" gets 4 stars out of 5.

Finally there's the HTC ThunderBolt – a real "4G" device, mind you, though they don't say so here, running some version of Android. Both the iPhone and the ThunderBolt we are not told which version of their operating system is in use nor which user interface is in use on said devices though the kind and type have a very large chance of affecting the results. The HTC ThunderBolt gets 5 stars out of 5 in this study.

For this second study we once again get to see the users fumble through adding a contact to their address book in the device they're testing. Again the operating system specifics are not noted. The Windows Phone device gets 4 stars out of 5. The BlackBerry device gets 1 star out of 5 because the user could not, no matter how hard he or she tried, add a contact to their address book.

The iPhone 4, aka 4G, as the professor continues to say, gets 3 stars out of 5 because the user, much like each of the other users, continues to make the same mistake of typing in a number with the keypad before attempting to access the address book first. It appears very much that these users have either been coached or have had much experience using a certain piece of equipment or another so that they'd try the same method for adding a number as one another first.

The Android device again gets the BlackBerry treatment in the video, that being some Benny Hill music, a fast-forward button, and a stop at the end with a single star because the user was not able to add a contact no matter how hard they tried.

In this final study, the Windows Phone user takes a long time compared to previous studies to access the screen where they're able to send a text message, yet the result is a 4 out of 5 stars rating. The BlackBerry device has the fast-forward treatment again, this time with a different song other than the Benny Hill theme, yet gets 3 stars out of 5 for a result.

The iPhone in this test has the user basically instantly able to access text messaging and sending several messages out successfully without a hitch, yet the result is just 4 stars out of 5. Again, without a reference point to speak of.

Finally the ThunderBolt gets a 3 stars out of 5 rating with the same fast-forward treatment in the video because the user has such a difficult time finding the correct app. Note here that the user appears to be the professor himself as he says "message sent – to my cousin" at the end of the video segment.

The "winners" of this study are the iPhone and Windows Phone, complete with fireworks in the background for a fancy effect. After the result, we see some scrolling test that reads: Usability Comparison Study – Human Factors in Information Systems Design – Summer 2011 – Professor Dennis Galletta – Case: Halona Mui, Hasib Zaman, Rita Lam, Thomas Karns, Alek Sudan, Richard J Herstein, and Andres Iriarte.

What we've got here is either one of the worst experimental trials we've ever seen, or this video is simply to serve as an example for students to work with in the class the professor is teaching. After having done some digging on the subject, your humble narrator uncovered a syllabus for "Human Factors in Information System Design", only one letter off of the title listed in the video – aka the same class. In this syllabus there's an "Optional YouTube Project" worth 25 points which describes the project point by point.

What do you think? Terribly unbalanced and unscientific study, or a simple example video?