We’re all probably familiar with the following scenarios from sometime in our careers. Maybe we’ve just powered up the first populated sample of our new prototype PCB; it all seems to work fine. Two weeks later something smokes and we subsequently discover that one of the boost converters is unstable and killing downstream parts.
Maybe we’ve just tested our firmware implementation of a communication protocol and control commands are clearly getting through. Weeks later someone complains about the control latency and we discover that 9 out of 10 packets are failing the CRC. Maybe we’ve just tuned our motion control system and done a few quick tests: perhaps checked a few step responses and checked the following error. Sometime later the system goes wildly unstable and crashes into the end stop, destroying the prototype.
I see these examples in medical device design – and I can think of many more too – as symptoms of things going too well, things working the first time. If they had not gone so smoothly we might not have been so ready to assume that everything was working and therefore would have tested it all more thoroughly. So why does this happen? Maybe we were focused on some other system functionality. We were probably under time pressure. Maybe we were at the end our budget. Maybe it was Friday afternoon and by Monday our heads were in a different place.
So how do we prevent this initial burst of luck and confidence from causing us to prematurely believe our work is done? Well, we can’t really try to sabotage our efforts so that things go poorly. A pretty reliable method is one that we all know and have probably been told many times: Plan out our testing in advance. The extra trick after that is to actually work through that plan, even if things are going spectacularly well. This is easier than it sounds, mind you; the same pressures still apply. The difference though is that if we actually have a plan, we’ll be less likely to deviate from it.
While we’re planning, it helps to pay particular attention to those parts of the system which are novel or were technically quite difficult; these items take the longest, leaving less time for proper testing both discretely and after integration into the system. Also try to be conscious of any parts of the system that you are assuming will work, especially if it’s something that hasn’t been proven previously. This has the added benefit of crystallizing in our minds ahead of time what the pass-criteria are, which helps keep focus during a tricky development. It will mean that once we have finished our design and it’s dangerously late in our schedule we will still have a bunch of testing to be done. At least at that point the testing is well understood and therefore harder to gloss over or skip altogether.
Finally, even when we decide that we’re really, really done there are always some remaining assumptions that we’ve made about how done a particular prototype really is. Let’s make sure that everyone is aware of these assumptions. That way people, hopefully, will be wary of any functionality that may have limitations or has not been fully tested.
The idea is to be a little more superstitious so that when things seem too good to be true, we recognize that they probably are!