We specialise in online usability testing.

Is All Remote Usability Testing The Same?

February 24th 2009 by Posted In: Research methods and approaches

Tags: , ,

Every time we talk about remote usability testing, many people have many different ideas of what remote testing is.  And there are many different remote testing methods.

Rex Hartson in 1996, defined remote usability testing as the method “wherein the evaluator, performing observation and analysis, is separated in space and/or time from the user”.

Two simple distinguishing factors, which are now commonly accepted, are that remote testing falls into two main categories:

  1. Synchronous / moderated remote testing, where the evaluator has direct communication with the participant during the evaluation. This is normally carried out using screen sharing software, or talking over the phone.
  2. Asynchronous / automated remote testing/ automated remote testing, where the evaluator has no contact with test participants. Asynchronous remote testing is the method used by Webnographer.

Asynchronous remote testing falls into three sub categories.

The simplest is analog remote testing, such as critical incident reporting, where the participant is sent a paper questionnaire which is filled in after carrying out the test. While this may seem like an easy method for evaluators, this is a method which can be very time consuming for the participant.

A bit more advanced are task-based online questionnaires, which are online tools that query the user for feedback after carrying out the task. Yet, those do not track any interaction on the page which makes it difficult to infer why problems are occurred. They therefore can only offer summative results such as success rates, time on task and satisfaction.

The most advanced tools use the CARUT methodology (Combined Asynchronous Remote Testing methodology), which combine logs of the user interaction during the task (hovering, clicks, key press, etc.) with questionnaires, which record rich qualitative feedback. This allows inferences about where and why problems occurred and offers both summative and formative results. Webnographer uses the CARUT methodology.

Yet, a challenge with most of the remote tracking tools which use the CARUT methodology is that they require the test participant to download software for it to work. We have found most people, worried about viruses and trojans, do not want to download software, especially one that tracks them. The other disadvantage of downloaded software is that it normally only works with certain types of browsers and operating systems. We have found that being able to work with different combinations of computers and browsers is important as we found the test participant’s configuration of their computers has a major impact on how they use a site.

Webnographer does not hold the above challenges. It works with nearly all computers, and with the most browsers. The user does not have to download anything and the website does not need to be modified. Webnographer gives therefore full freedom to carry out any test, including tests on your competitor’s website for competitive analysis.