Our design team discuss their top insights from time constrained, critical ventilator project in Canada’s Response to COVID-19
Paul Charlebois, Industrial Design and Human Factors Manager, asks team members. Hannah Rusak-Gillrie, Human Factors Engineer, Niall Redmond, Senior Human Factors Design Engineer, and Tim Park, Industrial Designer, about tactics and examples of adapting KOL, formative, and summative research for COVID-19 guidelines as they recount experiences from the Canadian Emergency Ventilators project.
What was the biggest human factors challenge with the Ventilator project?
Niall COVID-19 was in progress and we couldn’t do our usual preliminary analysis routine. This involves collecting contextual analysis through on site KOL interviews, understanding workflows, gathering and analyzing ergonomics and anthropometric data. All those are things that we would only do with analogous devices to develop workflows and background information required to start a project.
Hannah For me, it was understanding how mechanical ventilation works in a short amount of time to a level where we were able to sufficiently understand and appreciate the user interface requirements of the ventilator, in order to develop the appropriate ventilation parameter controls and interface components.
Paul This is one of the differentiators to many projects; in order to be successful from a usability perspective, you really need an innate understanding of how the device works externally.
Tim The ventilator has a fairly unusual user interface by today’s standards. With knobs, instead of a touch screen, there weren’t too many designs to draw inspiration from. This choice to use knobs was made due to time constraints and the mandate to use parts that could be sourced easily. Another challenge was taking a large number of controls and indicators that would normally be on a bunch of separate screens, and laying everything out on a single physical plane in a way that was really easy to understand.
Paul You really needed an efficient interface design similar to decades ago before modern GUIs came out. One thing I’d add is the lack of time to do a proper vetting in finding people to talk to about this project. Classically our projects are heavily supported with KOLs and SMEs from our clients. Getting as much input as possible before deciding what to do from a design perspective added some challenges for us.
How does COVID change the way you conduct your tests?
Niall The biggest challenge with COVID was that we didn’t get to meet face to face with the participants, particularly at the beginning. We had to envision new ways to conduct formative testing, which ended up being quite informative. Typically, we would meet the participant at a physical test location but due to COVID restrictions we met participants virtually through an online meeting room. It was discovered that testing in this remote nature accommodated more naturalistic use environments for user interfaces particularly graphical user interfaces. It’s as good, if not better than travelling onsite. You don’t have the test personnel in the room, which kind of distracts a little bit, making participants feel uneasy or nervous. Remote testing removes that element. For physical components, on the other hand, it’s much more difficult.
Hannah Being able to organize the formative studies remotely through video conferencing added a bit more flexibility. It shortened the amount of time to organize studies and execute them. That being said, it did pose some challenges with identifying potential use errors through observing participants interact with the device. Sometimes for informative studies, we’ll have a rough prototype for participants to interact with physically. For the ventilator, we didn’t have this capability. The interaction they used was through a computer interface. For the ventilator control interface, we had the main portion of the ventilator present on the screen as an image and asked participants questions about the layout and the control parameters. It was challenging in that we weren’t able to observe them interact because half of the time we were gathering information by watching them, not necessarily capturing the feedback, like the verbal feedback that they were giving us.
Niall A great example was when we created the readout for the pressure and PEEP in the graphical user interface in the patient data section. The pressure was on the left, and PEEP was on the right. And then with the alarms, we had PEEP on the left and pressure on the right. From a visual perspective, it made very little difference until you interact physically with the ventilator later. Then all of a sudden, it was clear that they were backwards.
Paul That is an excellent example of where usability testing trumps the heuristics we put in as we’re designing. It was originally the other way because the first thing to do is adjust the pressure. But what we discovered that wasn’t how they were interacting with it. They weren’t interacting serially. They were looking at it as a full tableau of things. It made a lot more sense when things were more lined up, organized so that the hand didn’t have to move all over the place.
Tim We weighted our decision making more heavily based on what our KOLs and SMEs had to say near the beginning of the project. We Normally would have had more in person site visits to see things with our own eyes.
What was the biggest hurdle of working from home on the project from an HF or UX perspective?
Niall Working from home I don’t have all of the same tools and access directly to the people within the office. Not being able to do a very quick heuristic evaluations and talking to team members was an initial challenge. The hurdle is working from home and not Human Factors and Usability. It is more about reorganizing the way in which you work. Time management, setting meetings, planning events and all of those things that usually happen within an office environment now need to be planned, organized and have additional time allocated. We’ve already talk about COVID challenges from the human factors perspective.
Hannah As a visual learner, I found the biggest hurdle to be not having access to physical components and prototypes as well as the current products that are out there, to play with and to understand. Instead, my learning had to shift from relying on a physical interaction to understanding it through images and information available to me from my home office.
Paul Having worked on a multitude of devices over the years, I found, in general, a ventilator is a very organic device in terms of how users use it. It’s not like you just put something on or flick it off. There’s a natural amount of latitude that clinicians need to pull from in order to make a successful therapy. Not being able to be there and feel that out was definitely one of the challenges. If you were designing something a bit more hard lined like a plastic tube or a surgical tool it would be different. I think that was one of the biggest hurdles for this particular project, the sophistication of the whole usability process.
Tim It was difficult to get all the same physical samples to every member of the team. One person would have a sample of the breathing circuit while a couple of other team members would have access to the prototype of the control panel. We had to be cognizant of that and make sure we were communicating what we learned about these physical samples with the other team members who didn’t have access to them.
A couple of us took it on ourselves to make paper and cardboard print outs and little models of the layout until we had access to the real thing. That gave us a sense of the scale of the controls.
Paul I think that actually getting that sense of scale right as you were doing rendering, I always thought it was a lot more compact. This way brought a bit more realism to the whole thing. You can’t underestimate the value of modeling.
How did you overcome getting input from users on the project?
Niall Additional planning was required for getting user input. We contacted local hospitals and got in touch with actual users. I then set up online meetings where we would run through scenarios and ask them detailed questions about the layout of the user interface. And then very quickly, iterate on the feedback that we got. Rather than having them come in, or meeting them in person, we did everything virtual.
Paul The logistical in time constraints meant that we wouldn’t usually have a large one swath of feedback that we usually would. How did you deal with the different levels of experience involved?
Niall We wanted the less experienced users. We wanted the situation where the person using the device had little, or even maybe no, experience using a ventilator. So in a pandemic situation where a registered respiratory therapist says to a nurse or a very junior or other respiratory therapist, “can you make some quick adjustments” because they’re busy dealing with another pandemic situation. The user interface is simple enough and intuitive enough that those changes can be made without supervision. For that reason, we choose Junior RTs, some of whom hadn’t even left university. We got their inputs. If they understood the UI, then anyone could have done it. We contacted the local hospital and then asked them if they knew anybody. They put us in touch with some junior RTs who put us in touch with some of their friends who were first years. It was sourcing continuously, like phone calls after phone calls until we got the right people.
Hannah In parallel, we also made sure to test respiratory therapists who had more senior experience. We did include participants who were senior in their field with 10, 20 years of experience. Even with the amount of individuals we tested, we were able to see clearly that there are biases with respect to the technology that senior RTs had used vs. those who were new to the field. We ensured the ventilator interface is tailored in a way to accommodate both senior RTs and younger individuals.
Tim Compared to other projects, at least in the early stages, we definitely had more frequent calls with users and a lot more design iterations. Maybe we were able to do that because we didn’t have to be there in person and didn’t need to spend time on travel. Although there are disadvantages to having meeting over conference calls versus in-person, one of the benefits is that there is more time left over to collect a greater quantity of user input.
What was the most rewarding part of the project for you?
Niall Seeing everyone come together, the camaraderie and the extended hours that people were putting in. Just the overall knowledge that what we were developing was for the greater good of Canada. Solving a problem that is critical within the health care system. One of things that I enjoy about working on medical devices is that I feel like I’m making a difference, a positive difference. With this project it was really at the forefront of all of that. This was something that was really important. It really showed through everyone’s camaraderie.
Paul It felt like StarFish spent 20 years training for this project.
Hannah It was extremely rewarding to take part in, and witness a medical device move from such an early stage of development through manufacturing in the span of weeks to months. That’s pretty rare in the medical device world. It’s a once in a lifetime opportunity to be a part of a project that moves along so quickly for such an urgent and substantial cause.
Paul It’s a testament to the company’s processes and how they’ve been refined over the years. It went reasonably well considering the chaos of trying to compress a five-year program in to three to five months.
Tim Just seeing our internal team mobilize so quickly to get things prototyped and functional, considering how quickly we did put everything together. It was pretty rewarding to see a video of some of the first prototype panels light up and the mechanical components inside starting to move within only weeks of working on them.
What’s is your biggest design lesson from this project?
Niall It’s really important when working this fast to set boundaries on what you’re designing and who is in charge of those requirements. And then keeping constant communication between all parties to ensure that the requirements are being met and that any changes are being communicated to the whole team. That’s true of every project, but is particularly true in working at this speed.
Paul Absolutely. Setting those requirements as soon as possible and then getting any changes, that happens with requirements, because, we’re sort of almost designing and manufacturing at the same time are all really valid points with the program.
Hannah How fast this project moved really emphasized the importance of sufficiently and properly documenting design decisions along the way. When you’re moving forward at such a fast pace, decisions get made, especially with respect to user preferences that are important, and they impact the way that the user interacts with the device. If these decisions aren’t documented sufficiently, the reasoning might get lost as to why they were included in the first place.
Paul You make a great point there, particularly because of the more isolated aspects of people working from home and not everybody is in the same room. Sometimes certain requirements just come out of discussions. Having the ability to understand the rationale behind decisions when we get to the point where we’re reviewing as a group all the different requirements, we really have a better understanding. We can more quickly get to conclusions as to whether or not we want to convert them into specs.
Tim I was pretty surprised by the effectiveness of this new way of working. It worked out pretty well and can be effective depending on what is important for a project – especially things like tight time constraints. While we normally try to apply a fail fast, fail often style of working in our hardware development, this project pushed it to another level. We definitely learned some lessons there.
Paul The fact that this was one of the bigger programs in a very long time and the fact that we had not really had a chance to try this whole working from home beforehand. All these things sort of came together very quickly. From a design perspective, having a solid architecture and getting buy-in from the multidisciplinary team from the outset is a really important and critical aspect of getting more successes and having a reduction in the amount of missteps that you don’t have a lot of time to course correct with this time line.
You can find background on the ventilator project at: https://ceventilators.ca/.
Astero StarFish is the attributed author of StarFish Medical team blogs. We value teamwork and collaborate on all of our medical device development projects.