search the lab

Embedded Testing with Unity & CMock (by Mark)

Embedded Testing with Unity & CMock

James Grenning's book (features Unity)

Test Driven Development for Embedded C

Topics

Wednesday
Jul272011

Unity Release Notes

Unity 2.0

  • Added optional 64-bit support
  • Added optional parameterized test support
  • Added optional verbose floating point support
  • Added pointer asserts and size options
  • Added array asserts for all numerical types
  • Added fallback memory asserts
  • Added more sample targets (like gcc64 and Hi-Tech PICC)
  • RUN_TEST now has default implementation, so users who don't want to use the scripts can more easily use Unity
  • Significant updates to API and output for consistency
  • Added optional "extras" folder for non-standard uses
    • (like 'fixture' which makes Unity act a bit like CppUTest, originally submitted by James Grenning)
  • Scripts now compatible with Ruby 1.8.6 through 1.9.2

Unity 1.9

  • Added optional floating point support
  • Added configuration options for size of int, etc.
  • Standardized _MESSAGE variants
  • Added YAML files for sample targets to more easily run self-tests on multiple platforms
  • Added example project
  • Started using setjmp / longjmp instead of wrap and return
Wednesday
Jul272011

CMock Release Notes

CMock 2.0

  • Even Better Parser than 1.9 (someday maybe it'll be a full-blown C parser)
    • strange arrays containing arithmetic operators, etc.
    • more difficult anonymous function pointers
    • loads of little detailed fixes here and there.
  • Smart internal memory management
    • CMock 2 has a much smarter built-in memory manager for storing all those expected and returnable values. You can even teach it to understand your processor's alignment and packing needs.
    • It's fast... WAY faster than CMock 1.
    • It's efficient... WAY more compact than CMock 1.
  • Pointers and Arrays
  • The Actual Line Number
    • How embarrassing. When CMock was an internal project it was acceptable to report failures as the line in the Mock file... but the test writer cares about the line in the TEST that failed. We've fixed that.
  • Better variable naming
    • CMock 2 chooses names for internal storage that you're unlikely to run into conflicts with... finally!
  • Supports all the latest Unity assertions internally.

CMock 1.9

  • Better Parser
    • CMock 1 was dandy for developing new code, but when you pointed it at some nasty legacy code or complicated libraries, it cried and hid its head in shame. No longer! CMock handles all sorts of craziness. Unnamed arguments, custom types, function pointers... it happily crunches through all of it and gives you your mocked goodies.
  • Easier Configuration
    • We admit it, the original CMock was kinda painful sometimes. We hope you'll find the latest version a more pleasant experience.
  • Pointers and Arrays
    • CMock 1 was a little lazy sometimes. When you passed a pointer as an argument, it just assumed you wanted to compare pointers. Often this wasn't the desired behavior. Now, you're in control. With the array plugin, you can specify if you want the pointer compared, the object being pointed to, or if that pointer is actually an array that should be compared.
    • Arguments can actually be function pointers now. How cool is that?
  • Stubs and Callbacks
    • Sometimes you want to fill data into an object being pointed to
    • Sometimes you want to perform really complicated comparisons in your mocks
    • Sometimes you want to trigger custom events during your mock calls
    • The new callback plugin gives you the power to do all that
Wednesday
Jul272011

CMock Roadmap

This is a rough idea of where we are going with CMock. Feel free to add your thoughts and comments. We'd love to hear them!

CMock 2.5

  • This is a performance milestone. It's all about efficiency. We want the generated C code to be smaller and faster. We want it to handle memory even more awesomely than it does now. Also, CMock itself will get faster so that you get the goods as soon as possible.

CMock 3.0

  • We're exploring the idea of making CMock handle C and C++. We haven't seen a tool out there that does this well at the moment... and we figure why not us?
  • What else are you interested in?
Thursday
Jul072011

When Bad Code Runs Green

Contributed by Matt Wilbur

This is a cautionary tale.

I was recently test-driving a UART module. Nothing earth-shattering, I know. For my current project, I have adopted the "DH" pattern for my driver work, where the Driver provides useful services and interfaces to higher level code and relies on a separate Hardware class (well, in C, a compilation unit), to hide niggly hardware details. I had a simple task for my UART: Receive a character. Pretty basic stuff, eh? That's what I thought.

As I test drove my driver code, I started to add more funky stuff, like allowing my imaginatively-named string receiving function UartDriver_ReceiveString to return before an entire string was read in the case that it received a UART_RX_NOT_READY value from the hardware. That might happen if some clever user asked my driver to always return immediately. Here's the code:

s32 UartDriver_ReceiveString(UartDriver driver, s8* stringToReceive, u32 lengthOfString)
{
  int i;
  s8 theChar;

  if ( stringToReceive == 0) 
    return 0;
  if ( driver == (UartDriver) 0 )
    return 0;

  for (i=0; i<lengthOfString; i++)
  {
    theChar = getByteFromHardware(driver);
    stringToReceive[i] = theChar;
    if (theChar == UART_RX_NOT_READY)
      break;
  }
  return i;
}

Pretty straightforward, right? The helper function getByteFromHardware called one other function, which I'll replicate here as it was when it was first written.

s8 getByteFromHardware( UartDriver driver )
{
  if( driver->timeout == UART_DRIVER_TIMEOUT_INFINITE )
    return getByteFromHardwareWhenItsReady( driver->hardware );
  else
    return getByteFromHardwareAndReturnImmediately( driver->hardware );
}

s8 getByteFromHardwareWhenItsReady( UartHardware hardware )
{
  volatile s8 theCharacter;

  do 
  { 
    theCharacter = getByteFromHardwareAndReturnImmediately( hardware );
  } while ( theCharacter == UART_RX_NOT_READY );
}

s8 getByteFromHardwareAndReturnImmediately( UartHardware hardware )
{
  return UartHardware_ReceiveCharacter( hardware );
}

"Hey!", you say. "How dare you write code without a test!"

Don't worry. I didn't. Here was my simple test:

void test_UartDriver_ReceiveString_ReceiveGetsNegOneIfTimeoutIsNoWait(void)
{
  s8 stringToReceive[5];

  UartHardware_ReceiveCharacter_ExpectAndReturn(mockHardware, 
                                                UART_RX_NOT_READY);

  UartDriver_SetReceiveTimeout(mockDriver, 
                               UART_DRIVER_TIMEOUT_NO_WAIT);

  TEST_ASSERT_EQUAL(0, UartDriver_ReceiveString(mockDriver, 
                                                stringToReceive, 5));
  TEST_ASSERT_EQUAL(-1, stringToReceive[0]);
}

"Hey!", you say (stop shouting, please, I can hear you fine). Why so many friggn' calls? We're embedded software designers. We don't like the overhead of such things. True enough. But I will invoke a few items from my defensive bag of tricks.

  1. I'm striving for clarity in the code. As long as I'm not actually running out of ram or failing some sort of timinig requirement, I'm quite comfortable with these calls. In this case, I don't have a timing requirement.
  2. One can inline things later. A la C/C++ (with inline), not a la Java with Eclipse.
  3. Don't try to out-optimize your compiler. You won't win.

If you still don't like it, I'm okay with that. TDD has made me confident enough in my code that I'll still sleep at night. I'll also refer you to Test Driven Development For Embedded C, 12.5, page. 267. Don't have the book? Go buy it.

As any sane person would, I use Ceedling to do my TDD. I felt very clever when all tests ran green.

Test 'TestUartDriver.c'
-----------------------
Generating dependencies for UartDriver.c...
Compiling UartDriver.c...
Linking TestUartDriver.out...
Running TestUartDriver.out...

-------------------------
OVERALL UNIT TEST SUMMARY
-------------------------
TESTED:  13
PASSED:  13
FAILED:   0
IGNORED:  0

Awesome. So, I took this code, made a library and linked it with my application. Badness. Bad badness. A simple, simple CLI (linenoise), was not receiving characters. How dare it! I tested this code! It's good! I figured I misunderstood the hardware (mocking only buys you so much). I spent several hours scratching my head during DOH (debugging on hardware). Finally, I realized what happened. My helper getByteFromHardwareWhenItsReady was not helping. It was hindering. It kept the received character for itself and did not share it. Very rude.

So how did this slip through my safety net? Bad luck. In assembly, here is how UartDriver_ReceiveString breaks down around the call to getByteFromHardware.

0x804acd5 <UartDriver_ReceiveString+47> call 0x804ad03 <getByteFromHardware>
0x804acda <UartDriver_ReceiveString+52> mov  %al,-0x9(%ebp)

Okay, so we know getByteFromHardware puts the received char in al. So, what does getByteFromHardware look like?

0x804ad1c <getByteFromHardware+25>          call   0x804ad32 <getByteFromHardwareWhenItsReady>
0x804ad21 <getByteFromHardware+30>          jmp    0x804ad30 <getByteFromHardware+45>
0x804ad23 <getByteFromHardware+32>          mov    0x8(%ebp,%eax
0x804ad26 <getByteFromHardware+35>          mov    (%eax),%eax
0x804ad28 <getByteFromHardware+37>          mov    %eax,(%esp)
0x804ad2b <getByteFromHardware+40>          call   0x804ad50 <getByteFromHardwareAndReturnImmediately>
0x804ad30 <getByteFromHardware+45>          leave

What a clever compiler. It simply calls getByteFromHardwareWhenItsReady or getByteFromHardwareAndReturnImmediately and then leaves eax in a pristine state so that getByteFromHardware can use it. But this is where it ought to break, no? I am not actually returning the character from getByteFromHardwareWhenItsReady like I should. In C, I am not. But that clever compiler was too clever for me.

0x804ad3e <getByteFromHardwareWhenItsReady+12>    call   0x804ad50 <getByteFromHardwareAndReturnImmediately> 
0x804ad43 <getByteFromHardwareWhenItsReady+17>    mov    %al,-0x9(%ebp)
0x804ad46 <getByteFromHardwareWhenItsReady+20>    movzbl -0x9(%ebp),%eax
0x804ad4a <getByteFromHardwareWhenItsReady+24>    cmp    $0xff,%al
0x804ad4c <getByteFromHardwareWhenItsReady+26>    je     0x804ad38 <getByteFromHardwareWhenItsReady+6>
0x804ad4e <getByteFromHardwareWhenItsReady+28>    leave
0x804ad4f <getByteFromHardwareWhenItsReady+29>    ret

See what's going on? You don't? Come on! It's staring you in the face. Nothing happens to eax between the call to getByteFromHardwareAndReturnImmediately and the ret. So, the return value from getByteFromHardwareAndReturnImmediately (theCharacter) is being returned up to UartDriver_ReceiveString, as far as my x86 computer is concerned. Suppose I force another call to a function before returning so eax is modified.

s32 foo() { return -1; }

s8 getByteFromHardware( UartDriver driver )
{
  if( driver->timeout == UART_DRIVER_TIMEOUT_INFINITE )
    return getByteFromHardwareWhenItsReady( driver->hardware );
  else
    return getByteFromHardwareAndReturnImmediately( driver->hardware );

  s32 bar = foo();
}

Sure enough. Failures galore. If we return theCharacter after that call, all is well again.

As an epilogue, it was (rightly) pointed out to me that I should really have some dedicated tests fore these helper functions. That is, in addition to test_UartDriver_ReceiveString_ReceiveGetsNegativeOneIfTimeoutIsNoWait, I should have a test like this:

void test_UartDriver_Helper_getByteFromHardwareAndReturnImmediatelyReturnsCorrectByte(void)
{
  UartHardware_ReceiveCharacter_ExpectAndReturn( mockHardware, UART_RX_NOT_READY );
  TEST_ASSERT_EQUAL(-1, getByteFromHardwareAndReturnImmediately(mockDriver));
}

I did add that. Same problem, but I feel better having that test in place.

And that, my friends, is how your compiler can screw you and why sometimes tests can't always save your ass from dumb mistakes.

As postscript (which differs from an epilogue, I hope) I realized a fundamental flaw in my zeal to get a working UART driver. In fact, I'd be surprised if Kent Beck weren't getting out a ruler at some point during this note so he could smack me on the knuckles (the Kent-Beck-in-my-head that is. I'm not delusional enough to think Kent will read this). In case you're as dim as I was, I'll confess my sins in the hope that it will save my knuckles. Typing verbatim from my Test Driven Development by Example, by (the real person) Kent Beck, here is the TDD micro-cycle:

  1. Quickly add a test.
  2. Run all tests and see the new one fail.
  3. Make a little change.
  4. Run all the tests and see them all succeed.
  5. Refactor to remove duplication.

Okay, maybe there was more than one flaw (pobody's nefect), but the biggest one that could have avoided this whole ordeal is number 2. I didn't see the test fail. I was so happy with myself and my ace coding skills that the test passed the first time, I didn't test the test. Which, to me, is one of the functions of step 2. Bad me, but lesson learned. Hopefully, you've learned it too.

Wednesday
Apr202011

YAML Primer

We, the developers of CMock, Unity, and Ceedling, like YAML. YAML is a markup language... or... I guess it's not since it stands for YAML Ain't Markup Language. In any case it's a handy way to serialize configuration data in a human-readable and machine-readable format. So we use it to configure all those fancy-pants Ruby tools like generate_test_runner and CMock and Ceedling.

So if you're thinking you'd like to use these tools but are perplexed by what you see when you crack those .yml files open, maybe this tutorial will help. I'm not planning to show you everything YAML-related. Instead, I'm going to concentrate on the features we use the most to get you up and running. If you want to see the full YAML specification, go to yaml.org

The Basics

YAML is stored in text files with a .yml extension (or occasionally .yaml). These files care quite a bit about white space, how you indent things, and have all sorts of fun with dashes and semi-colons. Let's talk about those things. If you want to be a proper mad scientist, you'll just have to remember a few rules.

Here's a preview of what kind of stuff we're creating:

:cmock:
  :when_no_prototypes: :warn
  :enforce_strict_ordering: TRUE
  :unity_helper_path: "../unity/helpers/"
  :plugins:
    - :ignore
    - :cexception
    - :arrays
  :treat_as:
    uint8:    HEX8
    uint16:   HEX16
    uint32:   UINT32
    int8:     INT8
    bool:     UINT8

We use two spaces for each level of indent. That's not official YAML, but consistency is nice, right? Avoid using tabs. Don't put any white space at the end of a line. White space in the middle of a value is fine, but it's best if you use quotes then.

YAML nests hashes and arrays as deep as it wants. Let's try picking this apart to see what we can learn.

:cmock:

In the example above, the first key you find is the line that says :cmock: This tells us two things. The colon at the end says that :cmock is a key, and whatever is on the right or indented below is the value for that key. The colon on the left tells us that :cmock is a symbol. If you know anything about Ruby, this means something to you. If you don't just understand that we use symbols for almost all of the keys because it's way faster and it flags them as something special.

  :when_no_prototypes: :warn

Let's look at the second line in the example. Here we see another key value, right? We have a key :when_no_prototypes (we know it's a key because there is a : after it). It's value is :warn. Both :when_no_prototypes and :warn are symbols again. The key isn't a surprise, as mentioned. The value, however, is usually just a string (or number, or whatever). Sometimes, though, there are a limited number of options that apply. Often we use symbols for those situations too. These will be spelled out in the documentation.

  :enforce_strict_ordering: TRUE

Ah. Another key-value pair. This time, the value is TRUE. The Boolean types TRUE and FALSE are built into the language too. For those of you that spend a lot of time in C, you might expect to be able to use 0 and 1 to the same effect. You'd be surprised when you find that both of these evaluate to TRUE. So let's just stick with TRUE and FALSE.

  :unity_helper_path: "../unity/helpers/"

Here we see a key-value pair where the value is a string. There are a number of ways to add strings, but let's concentrate on two of them. One is to just write the text (making sure it doesn't start with a colon!). Another is to put double-quotes around it. We tend to use the latter for long strings or strings that contain spaces. Better safe than sorry, right?

  :plugins:
    - :ignore
    - :cexception
    - :arrays

Hey! Now we're getting fancier! Our key :plugins has a value that is an array! We know it's an array because it's a bunch of lines starting with a dash and a space. For this array, all the members are symbols. For some (like :includes) they could be text. We could go really crazy and have them be hashes or sub arrays or other crazy stuff... but we try to avoid that sort of thing so that you don't hate us.

  :treat_as:
    uint8:    HEX8
    uint16:   HEX16
    uint32:   UINT32
    int8:     INT8
    bool:     UINT8

Instead this is just about as fancy as we get. This is a key :treat_as whose value is another hash! Note that the other hash is indented further. Once you stop indenting, you're back out of this sub-hash. Here you can see that keys aren't always symbols. These ones are strings. Their values are also strings. This is because we don't know ahead of time all the options for either... instead it depends a lot on what the user has configured. Strings are the best for open-ended information.

Also, note that the white space between the key and value doesn't matter.

Fancy Stuff

Check this out:

:defines:
  :commmon: &common_defines
    - F_CPU=16000000UL
  :release:
    - *common_defines
  :test:
    - *common_defines
    - TEST
  :test_preprocess:
    - *common_defines
    - TEST

See that &common_defines ? That creates a name "common_defines" for everything that is in the value of :common. In this case, it's a one-element array containing the string "F_CPU=1600000UL". It could have been a huge array, a hash, or whatever.

So why name it?

We see almost immediately below. The *common_defines injects the contents of that named section where ever it occurs. So no we've duplicated the contents of :common under :release and :test: and :test_preprocess. How handy is that!?

There are some other tricks that we use too... but we'll get into some of those when we talk about Ceedling configuration specifically. The stuff in this primer will get you going.