A critical component of Human Factors input during product design is evidence – based, user-centered feedback, gathered from conducting formative usability activities. We use two main approaches to gathering such feedback: Formative Evaluations and Formative Testing.  This blog details the differences between these methods, and the appropriate circumstances for their application.

Formative Evaluations

Formative evaluations provide a basis for evaluating system components at a smaller scale, to provide user based feedback early in design development. They are conducted casually, and can happen either internally, with employees, or externally with Key Opinion Leaders of the respective device. These evaluations can be planned and budgeted, but are more often completed as spontaneous reviews to validate a particular design concept. These design evaluations would follow preliminary research, including heuristic reviews.

When performing a formative evaluation, the facilitator is trying to answer a particular question surrounding the system design, such as “Is this handle shape comfortable and easy to use for our target user population” or “can a user follow this quick reference guide easily and effectively”.

The evaluation should be planned beforehand, but be casual and flexible in nature. The facilitator should have activities and interview questions planned and prepared, but be open to deviations in expected evaluation activities if the deviations are likely to provide value and insight.

During the early stages of product development, when completing iterative prototyping, formative evaluations are a great way to capture user feedback directly, quickly and efficiently. The documentation for this activity is short and direct, highlighting the objective of the evaluation, the method in which it was conducted, and key findings from the activity. Findings are usually directed towards specific sub-system components, based on the scope of the evaluation.

Results and findings from formative evaluations are usually kept as internally documented activities, unless the activity is specified as a project deliverable. When thinking of formative evaluations, think of “flexible, casual, and usually internal”.

Formative Tests

Formative Testing is the next, more formal stage of usability testing. This activity should be budgeted and considered within the project’s statement of work, and its completion will always provide a document highlighting key findings and design recommendations.

When performing a formative test, the facilitator is validating designs, as well as identifying any possible use errors at either a sub-system or system level. A formative test is conducted externally with either Key Opinion Leaders (KOLs) or intended users of the device. It is important to ensure the right participants are chosen for formative testing and accurately represent the intended user of the device. Criteria should be considered when selecting prospective participants, including their level of training, experience, and familiarity with similar and analogous devices. It is important that these individuals do not have previous experience or knowledge specific to the design being tested.

A test plan is developed beforehand, including the participants, the testing content, and method of data collection. Formative testing content consists of tasks the participant is required to complete that would be representative of tasks that the intended user would be asked to do and provide the information required to satisfy the scope of the testing. For example, they could include asking the participant to “assemble to device in preparation for use” or “start a therapy using the screen on the device”.

The facilitation of a formative test is more rigidly structured, and should follow the facilitator’s script more carefully. Because of the effort required to organize and execute this testing, this practice is best suited for more developed design content, such as a Graphical User Interface (GUI) wireframe, or a higher fidelity prototype that has at least some level of limited function.

When formative testing is executed, it is recommended that previous formative evaluation activities have been completed. In essence, formative evaluations are the building blocks towards completing a formative test. Because of the previous evaluations of the sub-system level components, formative testing can be structured to test the design at a system level, which is a good opportunity to test that the sub-system level user interface components work well together. When formative testing at a system level has been completed, and no new major findings identified, the system is ready for the Summative Usability Validation. Formative testing should be thought of as “planned, detailed, and structured”.

At StarFish Medical, we emphasize the importance of including both formative evaluations, and formative testing, as part of medical device product development. Both forms of usability studies provide unique insight into the usability of the system, and drive the development of the product with user-centric feedback.