EmbeddedRelated.com
Blogs

Unit Tests for Embedded Code

I originate from an electrical engineering background and my first industry experience was in a large, staid defense contractor. Both of these experiences contributed to a significant lack of knowledge with regards to software development best practices. Electrical engineers often have a backwards view of software in general; large defense contractors have similar views of software and couple it with a general disdain for any sort of automation or ‘immature’ practices.  While there is significant value in the more traditional methods of the more established engineering disciplines the software industry is pioneering many useful practices that should be adopted as much as possible by other disciplines.  One very useful tool I’ve used in developing embedded software is the unit test.

What are Unit Tests?

Unit tests are functional tests of individual program elements. That’s a lot of high-minded words. A more straightforward way of putting it is to say that unit tests are to verify that individual functions work the way they’re supposed to. They fit into an overall test strategy and work best when used with a variety of testing methods:

  • Unit tests verify that individual functions work the way they’re supposed to. They work best when they test only one function at a time - not the entirety of your codebase.
  • High-level tests verify that the entirety of your code (the sum of the functions) does what it’s supposed to do - typically this is evaluated at the boundaries of your devices (i.e., for most microcontrollers, at the device pins).
  • Integration tests ensure that your device works properly within the context of the system as a whole: does it play nice with the other components of the system? Does it respond to serial messages as it’s supposed to, when it’s supposed to?

My experience of larger defense-focused engineering firms has been that everything other than integration testing is generally ignored. My frustration with those practices, subsequent experience with testing and personal research into good software development practices has convinced me that while it may be time-consuming to implement all of these levels of testing it’s rarely a waste of time.

This article is available in PDF format for easy printing

Why Unit Test?

Unit testing helps to identify bugs that may be hidden or difficult to find when all of your code is run at once. One of the most frustrating aspects of debugging software is determining where exactly a bug is within your code. If you’re doing high-level testing only (i.e., testing your whole program at once) you’ll be reduced to single-stepping through a debugger, toggling port pins or using a plethora of debug printf statements to track down where an error may be. Unit tests give you the certainty that individual functions are operating as they should be, freeing up your debugging time to examine the less-certain areas of your code.

Unit testing also acts as an insurance policy against future code changes. Every project goes through multiple iterations, hardware changes, coding standards changes, feature creep, etc. Even if your requirements don’t change you’ll always be refactoring code for greater readability, lower memory usage or greater speed. Do you need to implement a quick sort instead of a bubble sort in a function that returns the maximum value in an array? No matter which sorting algorithm you use the function should still return the maximum value from the array - your unit tests don’t change when you change algorithms. No matter what is under the hood in your functions, unit tests verify that they still operate as they’re supposed to.

Unit testing is also important because it allows you test your code in non-ideal situations. Defensive coding practices dictate that preconditions should always be checked and error conditions should be handled. In the real world these situations may never arise: a parsing function may never encounter a malformed data stream if it’s tucked deep within your program, guarded by multiple levels of functions performing their own checks. However, if you reuse that code in a different application with fewer safeguards you may start exercising your error handling functionality and find that your functions are doing something foolish like indexing an array out of bounds, dividing by zero, or silently ignoring errors rather than reporting them up the chain. Unit testing lets you exercise these portions of your code without having to find ways to produce complex error conditions in the real world.

When to Unit Test

Ideally you would generate unit tests for every function in your program. Generally though this gets tedious - especially if you’re the only developer on the project. While I can’t tell give you an exact formula for determining which functions you should test and shouldn’t there are a few good rules of thumb that I can give you:

  • Your main loop - main() shouldn’t have a significant amount of logic or processing in it and for embedded applications it will involve an infinite loop. Generally infinite loops don’t make for good, bounded unit tests.
  • Functions involving hardware - Any function that relies on hardware to operate is in general going to be difficult to generate a unit test for: either you’d have to test with the hardware or find a way to fake out the hardware. Either way, most of the difficulties you’re going to have with these sorts of functions generally involve the hardware itself, not the logic inside of them. Unit tests often don’t tell you anything useful about these sorts of functions, so you can profitably avoid testing them.
  • Functions with little or no logic - Not all functions are created equal. For example, in object-oriented languages like C++ there exist get and set functions whose sole purpose is to read and write to a variable hidden within an object. You may have similar functions in C whose only purpose is increase code clarity or service some particularly difficult design decision. These functions are not important to test - if anything do them last!

Generally this is going to leave several classes of functions:

  • Functions that implement complex mathematics - It’s very easy make mistakes when implementing these sorts of functions which makes it critical to verify they work before integrating them into a larger program.
  • Functions that implement complex logic - It’s important to unit test function that make a lot of decisions for several reasons: to ensure that the decisions operate correctly, and to ensure that all of the code paths are exercised and tested.
  • Functions with significant failure modes - This includes anything that will process raw data or signal faults within your system. This functionality is typically very important to get right: you always want your system to fail safe.

Approaches to Implementing Unit Tests

Although the idea of unit tests is fairly straightforward there are often many difficulties in actually implementing them. I’ll discuss two general approaches.

On-Target, No (or minimal) Framework

The simplest way to test your functions individually would be to create special programs that run on your microcontroller (often called the target) that implement your tests without utilizing any special frameworks, libraries or other software packages. There are many benefits to this approach. One of the main ones is that your code is running on the target itself so if your test passes you can be pretty certain your code will work once it’s integrated into the rest of the program. It also doesn’t require you to learn or buy any additional software to perform testing - you can get started right away with only the tools you need to develop code. And finally, you have the added benefit of being able to test hardware at the same time as software if you want.   At first glance there doesn’t seem to be anything that would preclude this approach and it can be effective but there are several wrinkles that introduce difficulties:

  • Reporting results can be ungainly - Several approaches are possible: use a serial port to report results to a desktop PC, or use a debugger to halt on errors and inspect the state of the program. The serial port has the issue that it can be less than straightforward report anything more complex than a pass or fail result. Debuggers are always nice, but the hardware can be expensive and debuggers can sometimes modify the way that code executes. This can lead to questions as to whether a failure is real or simply induced by the test environment.
  • If any aspect of the test compromises the integrity of the overall program (accidentally wiping the processor state, overwriting important parts of global memory, inadvertent infinite loop, etc.) you may not get a result that’s more informative than ‘it failed somehow’.
  • Running on the target might allow error conditions that aren’t related to the code under test. If your device initialization routines don’t configure the clock correctly, or you fail to clear the watchdog timer it will cause spurious failures in your tests. You could spend a long time trying to track down these ‘bugs’ only to find the issue is in another part of your code.

Hosted with Framework

Some of the issues associated with running on the target without a framework can be remedied by moving the test environment to a desktop PC (a hosted environment) and utilizing a unit test framework. Unit test frameworks are very nice pieces of software that smooth out the entire process of writing tests, running them and generating reports. They offer a lot more options than rolling your own framework. Some of these include:

  • Report generation - Printf may be your friend, but there’s a lot to be said for an HTML file that tells you percentage pass/fail, which tests failed, execution times, etc.
  • Overflow and timeout checking - One of the problems with not using a framework on the target processor was the inability to diagnose whether memory was overwritten or an infinite loop occurred. Unit test frameworks in a hosted environment can typically tell you if the code is trying to access memory it shouldn’t, or cause a test failure if it doesn’t finish within a certain amount of time. Catching these sorts of mistakes on your desktop PC (where you can debug them more easily) will save you a lot of time
  • Coverage tools and reports - One issue that I haven’t talked about yet is generating coverage results. When writing a test you’ll want to make sure you’ve exercised all of the execution paths in the function. Coverage tools tell you which code has run and which wasn’t. While there exist coverage tools for some microcontrollers you can never be sure that yours will and you may not want to pay for it when it does exist. For your desktop PC there are free coverage tools available such as the GNU coverage utility - GCov.
  • Smoother and faster testing - Desktop PCs are fast nowadays and will have no problem running your tests in the blink of an eye; microcontrollers are much more limited. On a desktop all you have to do is compile, run and get your results. For a microcontroller you have the added step of programming the device (which can be a frustrating addition to the process when you’re trying to debug). You also have more automation options on your desktop than you do on a target device.

Some people may cry foul: it’s unreasonable and difficult to test code meant for an embedded device on a desktop PC. Generally, there are two main complaints.  

The first is that embedded code won’t easily compile for a desktop PC; the code is too dependent on the target’s compiler, libraries and overall environment to allow compiling on a desktop PC. This complaint is generally true, but it can be mitigated through good code design and practices. If a majority of your functions depend explicitly on the hardware or software libraries present only on your target it means that very likely your code isn’t organized properly. For the sake of testing and reuse, functions which are limited in their use to a very specific set of circumstances (compiler, libraries, hardware, etc.) should be distinct from functions which have duties separate from them. If you follow this paradigm a majority of your functionality will require only minor stubbing to operate on the desktop PC or on the target itself. For example, if the compiler you’re using for your target is a GCC-derived compiler you probably have the option of using standard integer types (uint8_t, int32_t, etc.) rather than the built-in types unsigned char, int, long, etc. which may be slightly different for your target architecture. Library-specific functions can be overridden

The second is that even if you can compile and test your code on a desktop PC it doesn’t tell you enough about how the code is going to act on the embedded target itself. There is some truth to this statement - testing on a desktop PC is definitely not the same as testing on the actual device. However, this is not the goal. The vast majority of bugs you’ll find in your code have nothing to do with the hardware it’s running on. More likely you’ll find your typical off-by-one issues, minor gaps in logic and other mundane errors in your code. True, there’s nothing saying with 100% certainty that if your tests pass on the desktop PC that your code will function correctly on your target. That’s not the point of these tests: the point is to debug the simple, foolish mistakes that we always make in a more capable debugging environment than we often find on embedded devices.

Other Approaches

There are other potential twists on both of these approaches. I’ve worked with large software packages which combine unit tests frameworks which run the tests on the target. While these packages are expensive and sometimes painful to use there’s no denying the benefits of combining the features of a unit test framework with the assurance of testing done on the target itself. Generally this isn’t going to be an option for hobbyists or small companies due to the complexity and cost.

There are lightweight unit test frameworks that don’t rely on heavy external libraries or facilities available only on a desktop PC. Generally these frameworks will either be included in your project as a single header file or a pair of header and source files with few or no dependencies. Because of this they can easily be compiled for your target framework so your tests can run in the native environment of your target. Despite this, these frameworks are generally less feature-filled than the more extensive hosted frameworks, but they definitely have their advantages.

If you like the convenience of testing on a desktop PC but want the assurance of testing on the actual hardware, you might look into whether a simulator is available for your target architecture. A simulator allows you to compile your tests for your target architecture and run them as if they were on the actual hardware, but with the benefits of your desktop PC: namely, increased speed and a much more user-friendly debug interface. Simulators are generally not universally available, nor are they universally free - but there are exceptions. AVR Studio is free and comes with a simulator for all AVR chips. Cursory investigation seems to indicate that there might also be solutions for the MSP430, ARM and maybe others.

A Simple Unit Test

My preferred method of implementing unit tests is to use a hosted approach with my favorite unit test framework: Check

I won’t go into the full spiel about it (that’s what the website is for) but I will mention that it’s a unit test framework for C that is simple to use, has some nice features I appreciate and is targeted towards Unix-like operating systems (which means you’ll need Cygwin and/or MinGW to run it on Windows). It’s certainly not the only unit test framework available for C, nor perhaps is it the most appropriate framework to use for embedded code. You can examine a large list of unit test frameworks on Wikipedia and do some more research for yourself, or find a framework for an alternative language.

Due to the relative complexity of installing and configuring Check on Windows I won’t discuss how that is accomplished, and instead focus on what you’ll want to do with it to test your code when you finally have it installed. This is the function we’re going to test:

uint8_t test_function(uint8_t a, uint8_t b)
{
    if(a>b)
    {
        return a-b;
    }
    else
    {
        return b-a;
    }
}

What does this function do? Nothing really special, but it has some logic, has some math, and is just complex enough that my sleep-deprived brain has difficulty with it. That means it’s perfect for writing a quick set of unit tests against.

Before you start worrying about how the unit test framework works or how you’re going to compile and build your tests you should spend a second thinking about what sort of inputs you’re going to test your function with. I could write a whole article on selection of inputs for unit tests to maximize test validity, but for this specific situation in which all of the inputs and outputs are integers and there’s one decision point there are a few quick guidelines I can share:

  • For integer inputs always test 0 and 1 - these are considered special cases for integers and often produce interesting behavior
  • Always test the maximum and minimum values for the integers - in this case since we have 0 (the minimum value) taken care of it means we should make sure to test 255 (the max for an 8-bit unsigned integer)
  • Test around your decision points - test a slightly greater than b, a=b, and a slightly less than b. This ensures that your logic is correctly implemented at the decision point.
  • Test a uniform distribution of values within the range - you always want to make sure that your function works correctly for a wide range of values within your normal operating range. In this case the range is 0-255 and I’ve decided to test five uniformly distributed sets of values (they’re not actually uniformly distributed - I’m winging it). Generally, the more sets of inputs you test the merrier (with exhaustive testing of every value in the range being the most merry) you’ll spend lots of time writing and running the test, so don’t go overboard.

The paradigm that Check uses to organize all of its testing is as follows:

  • A test case contains multiple unit tests for an individual function. In this example, each unit test is a set of inputs passed to the function and the expected result.
  • A suite contains multiple test cases - typically all of the cases associated with a set of functionality (called a module in the Check parlance)
  • A suite runner executes a suite of test cases

This is a fair amount overhead for a unit test framework, but it’s not completely over-bearing. A fully-implemented unit test for the above function is seen below:

#include <check.h>
#include <stdint.h>
//Top-level: a suite
//Suites contain a test case
//Test cases contain unit tests
//Unit tests are equivalent to test cases discussed above
//Function under test
uint8_t test_function(uint8_t a, uint8_t b)
{
    if(a>b)
    {
        return a-b;
    }
    else
    {
        return b-a;
    }
}
//One unit test
START_TEST(basic_test)
{
    fail_unless(test_function(0,0)==0,"Case 0,0");
    fail_unless(test_function(1,0)==1,"Case 1,0");
    fail_unless(test_function(0,1)==1,"Case 0,1");
    fail_unless(test_function(1,1)==0,"Case 1,1");
    fail_unless(test_function(255,0)==255,"Case 255,0");
    fail_unless(test_function(0,255)==255,"Case 0,255");
    fail_unless(test_function(255,255)==0,"Case 255,255");
    fail_unless(test_function(4,3)==1,"Case 4,3");
    fail_unless(test_function(3,4)==1,"Case 3,4");
    fail_unless(test_function(100,54)==46,"Case 100,54");
    fail_unless(test_function(54,100)==46,"Case 54,100");
    fail_unless(test_function(27,36)==9,"Case 27,36");
    fail_unless(test_function(15,4)==0,"Case 15,4");
}
END_TEST
int main (void)
{
    int number_failed;
    //Create the test suite for the test function 
    Suite *s = suite_create("Test");
    TCase *tc_core = tcase_create("Core");
    tcase_add_test(tc_core,basic_test);
    suite_add_tcase(s,tc_core);
    SRunner *sr = srunner_create (s);
    srunner_run_all (sr, CK_NORMAL);
        number_failed = srunner_ntests_failed (sr);
        srunner_free (sr);
        return (number_failed == 0) ? 0 : -1;
}

Needless to say this isn’t the exact structure you would use when you unit test your code - the function under test would be in its own source file for one. This is just an easy way of showing you all of the code necessary to get your unit test framework up and running. The structure of the code is as follows:

The test_function is the function under test. There are no special modifications you’ll need to make to it in order to unit test it.

The basic_test test is the unit test for the test_function. There are many tests incorporated into this one unit test but that may not always be how your unit tests are organized. With Check, every test runs in its own memory space under its own process, so that will figure in to how you break up your tests. For simple functions like this it doesn’t really hurt anything to group them like this. Each specific input/output combination is tested via the fail_unless statement. Check offers a variety of possible ways to implement tests: fail_unless, fail_if, ck_assert_int, ck_assert_str, etc. You have a lot of options. I like fail_unless because it offers the ability to write a string describing the test that failed. I chose the specific tests for the function based on the criteria I discussed earlier and you can see many of the items I discussed (i.e., testing 0,1, a range of nominal values, limits of the type, etc.). Each specific case is labeled with a string that is reported in case of any failures - this helps you track down any issues (especially since you can print variable data from these calls just like printf).

Inside of main all of the test overhead is initialized: The suite is created via the suite_create call (obvious, I know). The “Core” test case is initialized via the tcase_create call and the unit test basic_test is added to the test case by the tcase_add_test call. To execute the suite requires the calls to create the runner (srunner_create) which then runs the suite (srunner_run_all) and the number of failed tests is retrieved (srunner_ntest_failed) before destroying the runner (srunner_free).  If any tests failed, the program returns a status of -1, otherwise 0 if everything passed. It’s all a bit complex for a simple application, but once you’ve addressed all of the overhead you probably won’t have to deal with it too much afterwards.

This file can be compiled (on Cygwin or any Unix-like operating system) by using the command ‘gcc -lcheck -o ’. When run, it produces the following output:

Running suite(s): Test

0%: Checks: 1, Failures: 1, Errors: 0

basic_unit_test.c:38:F:Core:basic_test:0: Case 15,4

You may be wondering why there are any failures. In fact, I inserted a failure into one of my unit tests just to show you what it would look like. The failure is reported with the string passed to the fail_unless function and is identified explicitly with the line in the source file that produced the error. That’s really slick and represent one of the benefits of using a hosted unit test framework as opposed to rolling your own. This isn’t the only reporting option though - Check offers a variety of output formats from which you can choose.

I hope I’ve swayed you to start writing unit tests for your code and that my brief tutorial has been helpful in giving you direction to implement the tests. Over a long period of time practices such as this can be very beneficial and save you a large number of headaches, so I hope you take some of these lessons to hear.


[ - ]
Comment by Nikita65July 3, 2014
Could you please provide an example of unit tests of the real embedded device on the PC platform? For example for MSP430G2xxx devices? The difficulty with it is that all the platform specific headers (msp430.h, for example) doesn't work when compiling code for the PC target. How to workaround that?

Common error:

gcc -O -Wall -I /opt/local/include -I src -I /opt/local/msp430/include -L /opt/local/lib -l check -g -o testme test/test.c
Undefined symbols for architecture x86_64:
"__WDTCTL", referenced from:
_init in test-1a8736.o
_test_can_create in test-1a8736.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

And I just tried to reset the watchdog in my code.
[ - ]
Comment by SFriederichsJuly 6, 2014
Unit testing code that interacts with hardware is not nearly as easy as a function that only manipulates data and doesn't access hardware. Still, there are some techniques that can be used to allow any microcontroller-specific code to compile with a standard GCC compiler. It will probably require totally replacing the device-specific headers completely and faking out the registers. If you provide me with the code you're trying to test I can try to create a harness that will allow it to be tested on a hosted environment on the PC.
[ - ]
Comment by Tomasz49October 19, 2014
Is there a way to simulate interrupts when testing hosted with framework? Creating a simple thread is not enough because during thread execution content can be switched back to main thread and ISR on microcontrollers doesn't work this way - ISR is atomic. Mutexes won't solve all the problems.
[ - ]
Comment by SFriederichsOctober 19, 2014
I'm not sure what aspect of ISRs you're having difficulty with from your question. It may not be an issue I've encountered before, but I haven't had any difficulties with switching contexts during unit test.
[ - ]
Comment by Tomasz49October 20, 2014
In the microcontrollers you can be sure that code of an interrupt will not be interrupted with execution of main code. In some cases other interrupts may interrupt it but not main code. On the PC the thread's function can be interrupted in any time and execution can be switched back to the main function (content can be switched). I experienced problem with this fact when trying to test my code under PC.

For example, If I have code:

fun()
{
protectFlagSet(); //Information for the IRQ that orginal list can't be changed
handleOrginalList();

disableIrqs();
protectFlagClear();
replaceOrginalListWithShadow();
enableIrqs();
}

IRQ()
{
if (protectFlagGet())
{
writeToShadowList();
}
else
{

writeToOrginalList();
}
}

On the PC you would need to disableIrqs (set mutex) before protectFlagSet to make to safe, but that would slow down code on the uC.
Do you know how to handle it in other way? This is not the only issue I try to solve about testing the uC's code on the PC.
[ - ]
Comment by SFriederichsOctober 20, 2014
I think I see - you spawn multiple threads for each separate execution context in your microcontroller: main and interrupt. However, you don't necessarily have control over when one thread will be interrupted by the other.

My unit tests don't operate that way: instead of testing all of the code at the same time I test only individual functions. They aren't software integration tests, so there aren't multiple contexts running at the same time to be interrupted.

You might get around this by specifying priorities for each thread - giving the ISR thread the higher priority so it's never interrupted.
[ - ]
Comment by Tomasz49October 21, 2014
So maybe my approach is wrong. I have module that allows my to register a function that will be called after X miliseconds (addFun(...)). Elapsed time is handled in an interrupt (time_irq()). Registered function is called from main thread (check()). What would be correct approach to test these three functions that provide one functionality?
[ - ]
Comment by sidekickSeptember 15, 2015
I liked your article, but it's unfortunate that you have not covered unit testing a function in a typical embedded software project. I've similar trouble as another user (Nikita) mentioned. For example, I've a project for ARM Cortex series target and typically I do the cross compilation on my Linux box and then flash it on the target board. Let's assume that I've one function (say, ArrayToInteger(uint8_t a[], int len) ) that does not invoke any platform specific code such as accessing GPIO register etc and is defined in a source file, with other platform specific functions and hence it also # include lot of headers, so how should I go about testing that function. The simplest way I can think of is copy pasting that function (ArrayToInteger) in the test file(say test01.c) and invoking my test pattern against that function. But this is a rather lame approach. Your suggestion would be highly helpful to me to get going with the Unit testing that my embedded system project is desperately looking for.
[ - ]
Comment by SFriederichsSeptember 15, 2015
This strikes me as a situation where improved organization of your source could benefit you. Why is the ArrayToInteger function (which is non-platform-specific) included in a source file that contains platform-specific functionality? If platform-specific code was separated from non-platform-specific code then unit testing your code would be much easier.

If that's not an option you can perhaps wrap the platform-specific code inside of #ifdefs within the source file and only compile it when you're generating code for your target rather than all the time.
[ - ]
Comment by sidekickSeptember 16, 2015
@SFriederichs, First of all thank you for your reply. Yes I agree with you on code segregation, however It is not always possible, For example, what If my debug routine that prints messages invoke a UART method, then I need to have those UART specific function calls and corresponding headers. Another example, calling a sleep routine that ultimately invokes platform specific code also needs similar function calls and headers. I tried your suggested method of wrapping the code around #define and I was able to unit test the function.

$ gcc -c utils.c -o util <--- bunch of embedded project specific routines
$ gcc -c test001.c -o test001 <--- test case using Check framework
$ gcc utils test001 -lcheck -o test <-- linking together
$ ./test <-- Test invocation on hosted (Linux) platform

However ideally I'd like to have a test harness (say a bunch of test001,c test002.c etc) that can be invoked to unit test different functions, with minimal changes (For example, these #defines) in the project code. 'make' should help, and I'll try that soon. Any other suggestions are very welcome.
[ - ]
Comment by sidekickSeptember 17, 2015
my previous comment got messed up due to weird auto formatting on this website :(

To post reply to a comment, click on the 'reply' button attached to each comment. To post a new comment (not a reply to a comment) check out the 'Write a Comment' tab at the top of the comments.

Please login (on the right) if you already have an account on this platform.

Otherwise, please use this form to register (free) an join one of the largest online community for Electrical/Embedded/DSP/FPGA/ML engineers: