Observations are an integral part of summative evaluations, the final validation that a device’s user interfaces are safe to use. During a summative evaluation, the manufacturer sets up a set of scenarios that capture all of the tasks that the device is expected to perform. The manufacturer creates environments that mimic the real use, capturing the finest details, all the way down to playing recordings of expected sounds within that given environment.

The biggest distraction within simulated use scenarios are the test personnel. Aside from distracting the participants, it is quite easy to influence them, prompting answers and workflow through tone of voice and subconscious body language. Below, I share tricks and tips to aid in being as unobtrusive as possible to prevent influencing the outcome of the summative usability evaluations.

If you don’t have a one way mirror, wear all black.

In a summative evaluation you want the participant to feel like they are in a real life scenario. You should set up the environment, lighting, noise and clutter to represent the specific use scenario. Having someone stand near the participant watching and “judging” is not representative. In the ideal situation you would have a one-way mirror so as to give the participant the full immersive experience. This is not always feasible. When it is not possible to use a one way mirror, you should be as inconspicuous as possible. Wear all black, stand still, and position yourself in a location where you can see everything. Be as far back and out of the way as possible without losing your ability to properly observe.

Use written directions

Observations are a two way street. You want to eliminate the ability of the participant to pick up subtle nuances that you may unknowingly generate which direct the participant to the correct answer.  When we speak, we tend to highlight the points we want the listener to remember. Accentuating those words and phrases help us make our point. This is done purposefully, but it is also subconscious.

In a summative evaluation you want to eliminate the risk of influencing the participant. To do this, use directions written on a card. For example; you have a use scenario you would like the participant to perform. Write it out and have them read it, so as not to identify which aspect of the scenario you are paying particular attention to. Or, if you have a multiple choice question, write the questions and answer choices on a card. Instead of asking them the questions, have the participant read and choose the right answers from the card.

Ask one follow up question

In a summative evaluation you cannot ask questions during testing about observations you have made. This would constitute a test failure, as it is not representative of real life and considered leading and/or  training the participant. Instead, make notes and ask all of your follow up questions after completion of the testing in order to discover the root cause(s).

The participant shouldn’t take notes during summative testing. In some instances there can be multiple use scenarios to simulate that last many hours. In those cases, there may be events that the participant found difficult that were not obvious to the note taker. The events may be forgotten before the interview section is finally conducted after all use scenarios and knowledge tasks have been completed.

A way to remind participants of these use-error events is to ask one open-ended question after each tested scenario. For example, a question such as, “How did that go for you?” gives the participant the opportunity to expand on anything they found difficult or unexpected. The observer should not ask any follow up questions at that time. Instead they should note the answer and follow up at the end of the study for root cause.

When observing a summative evaluation it is crucial that the observers do not influence the participants. By following the above tips, you will create a more representative and successful form of testing. This will lead to better, safer devices in our hospital and homes in the future.