Unit testing in embedded systems: 3 myths and an automated tip
The vast majority of medical devices with electronics contain some sort of embedded processor running firmware. There are a range of modern techniques that can be used to facilitate better reliability of embedded firmware.
One that I’m a big fan of is unit testing. Embedded engineers opposed to unit testing will raise a few common objections. After seeing firsthand how unit testing can improve medical device firmware quality, I decided to write this blog debunking three common myths and sharing my experience using an automated unit testing tool.
3 Common Myths of Unit Testing
- Our firmware is very simple, we don’t need unit testing.
- We can’t unit test microcontroller code, it’s too close to the hardware.
- Unit testing will add unnecessary cost burden to our project.
What is unit testing?
Unit testing is now a widely accepted software engineering practice. However, unit testing is severely under-utilized in the world of embedded firmware because of a few myths. Unit testing involves (usually automated) testing of small software “units” in a much larger program. The benefits of unit testing are well-known: looser coupling between software modules, simplified regression testing, and greatly improved code quality. Most software engineers working in high-level languages accept that automated unit tests are a critical part of software development.
Our firmware is very simple, we don’t need unit testing.
Medical devices firmware is rarely “simple.” It’s true that there are very simple applications for embedded microcontrollers: as timing elements, for example, or as devices which execute very basic logic operations. In such a case, tests are usually unnecessary. That said, truly “simple” firmware, in my experience, is rare. Most embedded firmware actually serves a large range of requirements.
If your firmware is doing anything more complex than sequencing LEDs or reading an analog value to raise a warning indicator, you can almost certainly benefit from having good automated test coverage. Finally, simple firmware often does not stay simple. Scope creep affects firmware just as much as high-level software and new requirements discovered later in the project can dramatically increase code complexity.
We can’t unit test microcontroller code, it’s too close to the hardware.
I would argue that if your hardware interface code is so tightly coupled to your application logic that they are inseparable, you have failed to ensure loose coupling between modules. At minimum, most hardware register read/writes should be wrapped in some sort of abstraction layer. This allows them to be correctly mocked by a test framework. If performance is a concern, these functions can be marked as inline which removes calling overhead while still keeping the code clean and separate. All that said, there are methods to make unit testing frameworks mock register interaction correctly if required.
Unit testing will add unnecessary cost burden to our project.
The cost burden of unit testing is a matter of debate. Industry consensus on the issue is that unit testing has a higher up-front cost yet it results in much fewer defects in the finished software. Shipping with fewer defects is very desirable in embedded firmware. Unlike a web application where you can quickly push to production with a few keystrokes, updating software on embedded processors often requires technician visits, or a trip back to the factory. Recalling an order to re-flash units and fix a critical bug can be very expensive, both in dollar terms and in damage to reputation.
Unit testing also results in software which is much easier to maintain and add features to throughout the software lifecycle. There are situations with “simple” medical device firmware where unit testing is an unnecessary cost burden. However, in most cases unit testing provides a tangible benefit in the form of cleaner software and cost savings as new requirements are discovered and integrated into the existing codebase.
How to Unit Test Effectively: Ceedling
So, now that we know that unit testing is a valuable activity for embedded firmware, how do we go about it? One of the best automated unit testing tools I have found to date is called Ceedling, from ThrowTheSwitch.org. It is a free open-source build system designed for embedded projects written in C that integrates the Unity unit test framework, the CMock interface mocking utility, and the CException exception handling library.
If you have good separation between modules (defined as a C header file), Ceedling makes developing and running tests a breeze. CMock integration allows it to scan the header files of all dependent modules and generate appropriate mock functions for your tests, completely decoupling your modules from each other and allowing you to test them in isolation. You can run it either on your host development machine, or in a hardware simulator. It is highly configurable and does not require integration with your IDE. You simply point it at your source files and start writing tests. Its command-line nature makes it easy to integrate with continuous integration systems. If you also use continuous integration systems, you can integrate tests into your build workflow.
Despite common objections, embedded firmware benefits from unit testing just as much or more than high-level software applications. Modern firmware is increasingly complex, especially with the proliferation of connected medical devices. Unit testing helps medical device manufacturers’ ship firmware with fewer defects, reducing the risk of field visits and expensive recalls. It’s never been easier to add unit testing to embedded firmware, and there is a clear case for doing so in medical devices. Why not start today?
Image: StarFish Medical
Peter Kazakoff is a StarFish Medical Electrical Engineer. A recent graduate from the University of Victoria. Peter uses unit testing on a variety of medical device projects.
Nice article, and I strongly agree! As unit testing bare metal systems, even with abstraction, can be a challenge, a few years ago I wrote a white paper that uses some C++11, threading, and template concepts to deal with components like registers and primitive communication buses like SPI or I2C. Our MCU is an MSP430 and test tool is Google Test, but the concept can be applied to any bare metal system written in C.
Paper via: https://e2e.ti.com/blogs_/b/msp430blog/archive/2016/09/27/unit-testing-the-msp430-within-a-desktop-environment
Nice article! Unit testing is highly underrated by most embedded programmers, unfortunately. I now consider myself fortunate that my first job was as a software test engineer for medical devices. We used IBM’s Rational testing tool, but it is essentially just a much more complicated version of Ceedling.
I think the biggest benefit is that it makes the inevitable bugs much easier to find. Without a set of tests the bug could be in many places, and might be caused by a dumb typo, assignment as comparison, a logical error, some integration issue with another module, or pretty much any type of error. With high coverage unit tests I *know* that my code works as intended in all tested scenarios, so I know where *not* to look when hunting down a bug.
If you were hunting for a fugitive in the wilderness, would you want to use a paper map with some pins marking your best guess at his location, or would you want to have a comprehensive set of satellite images of the area from five minutes ago?
Ceedling has allowed me to integrate a complex new feature into one product with no feature-related bugs found in system testing. My initial integration errors were simple to track down since I knew that my code worked if it was “plugged in” correctly, so to speak. The other system integration bugs that popped up were also trivial to find, because I knew my internal logic was sound.
I also can’t tell you how many times I’ve found stupid coding mistakes that would have resulted in a *lot* of debugging to solve had I not found them from the start with a unit test. I like to think of unit tests as an averaging filter for debugging time. I spend at least some time testing every module instead of spending hours hunting down a single mistake.
Sometimes, the firmware function is to control some devices through I2C or other buses. In this case, how to do the unit test? For example, if I have a Power Supply control function, I set the voltage to 3.3v. How to check if the Power Supply has been set to the correct voltage?
Hi Stephen, thanks for the question. You’ve touched on a circumstance that is tricky for automated unit testing. In some cases a simulated test bench can be setup to fake the interaction. Even that is not a substitute to actually testing with hardware.
I do not believe – I don’t think I’m alone here – that automated unit testing can ever fully replace all forms of testing, especially as you get closer to the system level. Arguably, a power supply control algorithm that communicates through I2C probably includes a number of software units as well as hardware components and therefore is not really a candidate for unit testing. Instead, you will likely have to test this either on the bench or in the final system. It may be possible to automatically test individual units in this subsystem, but mostly only those that don’t interact with hardware. Yes, you can add abstraction layers to minimize this code but be careful to avoid making the overall code tricky to follow and troubleshoot.
My firm belief is that a significant portion of firmware testing is on a bench with oscilloscopes and digging through component datasheets. That’s one of the big differentiators of firmware from software in my books.