LSC & UPA March Meeting Review:
Usability & Web Analytics
by James W. Korth, Member, PR committee, &
Adam Polansky, Member, DFW UPA
LSC Review
You’re getting close to rolling out a new Web site and you want to ensure that the site will appeal to users as well as effectively produce quantifiable results. Is usability testing the way to go, or should you use web analytics?
Jim Machajewski of Perot Systems has fifteen years of experience in usability and design. He recently shared his insights as the featured presenter at the February LSC meeting.
Usability testing and Web analytics are very different in approach and in the applicability of their results. Usability examines how people use material, what motivates them, and what they find interesting. It is valuable in observing how people use an interface, the what and why that underlie their actions. Usability testing is prospective and provides insights into future directions. Correct evaluation of results can lead to valuable enhancements before development, discovery, and resolution of problems before they become disruptive after rollout.
Web analytics is more quantitative and retrospective. It looks at the where and when of user actions. It tells us what has happened and may provide a large amount of data to analyze. Web analytics rarely provide output that give immediate, obvious results. Time-consuming analysis is usually necessary. Web analytics is based on data and not on behaviors. It is useful in proving whether something succeeded or failed.
The big difference between the two approaches is that usability testing looks for targeted audiences. Who do we expect to visit and what are their motivations? We get direct input on how real users use the system. But it does not measure persuasive momentum, solve usability problems, or lead to quantitative results that might be useful in supporting discussions with decision makers.
Web analytics, by contrast, looks at who actually did visit. It looks at the entire population that came to the site. It validates interface design decisions and can provide objective guidance on design decisions that may otherwise be subjective. It can identify gaps between user, business, and technical requirements. Finally, Web analytics is scalable across small and large sites, and multiple sites, within a system.
Varied human behavior versus quantitative data. Subjective versus objective. What and why versus where and when. Prospective versus retrospective. Jim Machajewski demonstrated how usability testing and Web analytics each have significant advantages and drawbacks. When properly applied, each approach brings value to the designer.
UPA Review (Adam Polansky)
While attending the recent joint meeting of the STC and the UPA, I had the opportunity to listen to Jim Machajewski give a presentation entitled; “Usability Testing and Web Analytics – Two Sides of the Same Coin”.
I’ve been acquainted with Jim professionally for a few years so I was already aware of his impressive background and experience so I was looking forward to hearing him. Jim is currently the Director of Global Digital Initiatives at Perot Systems where he’s been for about 5 years. All together, he’s spent about 15 years in the Internet space focusing on User Experience (UX) in different capacities, which makes him one of the original pieces of equipment in the UX world.
The talk’s premise was that while there is fuss being made over the need for Usability testing in Web environments, there are only so many things you can observe with your analysis still being qualitative at best. Analytics, on the other hand, compliments that analysis by providing quantitative data that can more accurately target inefficiencies in a Web application.
With Usability testing, the effort is essentially an attempt to hedge your bets (short of looking in a crystal ball) to predict what will happen when a site goes live. That’s not necessarily a bad thing as we discussed later. The fact is that there are Web sites where a qualitative experience is what you’re aiming for and obviously there are sites out there that do a great job of engaging their customers along those lines. That said, even if you have the optimal number of participants (whatever that might be—they’re still debating that one.) and you have the most comprehensive prototype and the most unbiased, objective testing guidelines, you’re still basing your findings on trends that can be skewed from what you might observe in actual use.
Analytics provides a glimpse of what actually has happened with a site in production. Jim quickly pointed out that no analytics program can provide you with analysis—only data. Someone still needs to do the analysis.
So, if analytics are so reliable, why doesn’t everyone use it to prove/disprove the assumptions made when developing a site? The primary reason, as Jim acknowledged, is the steep learning curve, yeut he asserts the reward is worth the effort.
Jim gave us a unique view into one of his projects that went live roughly 24 hours before he came to present. One of Perot Systems’ major stakes is in the healthcare industry with impact on both the organizational aspects of large healthcare providers and individual practitioners. The project —a daunting effort to capture and distill the implications of the new government stimulus package, had to go from scratch to production in a wildly aggressive time frame. As you can imagine, a number of people were waiting to learn how the site would perform given that much of the information comprised Perot’s proposals on how best to take advantage of those government funds.
In preparation for the launch, the only lead-in was an e-mail blast to a qualified list of current and potential clients which invited recipients to a webinar on the subject. Within that presentation, the Perot Systems’ Chief Medical Officer referenced the site. Within 12 hours, Jim provided a quick analysis to his executives of the traffic on the site which included the good news that not only had many people visited the site, but that more than 70% actually made their way to its interior. When you have less than a 30% bail-out rate, you’re doing something right. Furthermore, Jim demonstrated that the traffic was most active in the content areas where they had hoped would draw viewers. One hiccup occurred when the data showed that a certain percentage of visitors were coming to the site from a 404 error page—something to investigate, yet by no means a show-stopper of the site’s success.
Jim went on to identify some of the different analytics applications available and to propose how the typical Usability cycle of “test-design-test-etc” can be revised to take advantage of the qualitative observations that come from the laboratory environment and marry them to the quantitative data that comes from analyzing unvarnished usage.
As I expected, the presentation was informative and thought-provoking along with a rare glimpse at a project launch so fresh, the pixels hadn’t yet cooled off. Nobody has the secret sauce for launching successful Web applications. Too many variables exist from how diverse your customer base is to the expectations (read “funding”) of your executives. You have to weigh those and many other things to decide what approach you’ll take to ensure success. However, if you need a good 1-2 punch that lets you launch with some foresight and review with some accuracy, Jim offers up a solid combination.
Adam Polansky is the Information Architecture Manager at Travelocity.com. He is a past-president of the DF/W chapter of the Usability Professionals Association and a contributing author to the book “Usability Success Stories – How Organizations Improve by Making Easier-To-Use Software and Web Sites”.