CppUnitLite2 1.1

At this point, we have been using CppUnitLite2 for a year at High Moon Studios doing test-driven development on Windows, Xbox 360, and some PS3. It has been used to unit test libraries of an engine, pipeline tools, GUI applications, and production game code.

CppUnitLite2_1_1.tar.gz

About a year ago, I wrote an article comparing unit test frameworks that went to become one of the most popular in this site. The article concluded with the recommendation of one of the less well known frameworks: CxxTest.

Shortly after writing that article, it came time to roll out test-driven development at work. Interestingly, once I weighted in all the requirements, I ended up not following my own advice. Instead, I decided to go with a customized version of CppUnitLite, which was listed in the article as the second choice. The idea of an extremely simple unit testing framework was too attractive to pass up in an environment where we have to support a large variety of environments and target platforms.

At this point, we have been using CppUnitLite2 for a year at High Moon Studios to do TDD on Windows, Xbox 360, and some PS3 (we even have a stripped-down version that runs on a PS3 SPU). It has been used to unit test libraries of an engine, pipeline tools, GUI applications, and production game code.

CppUnitLite2 was originally released along with my article a year ago. It was a heavily modified version of the original CppUnitLite by Michael Feathers. This version introduces the changes we made at work as different issues came up and we needed more functionality from the framework. In particular, Jim Tilander and Charles Nicholson were responsible for a lot of those changes. By now, I doubt there’s a single line left from the original code. Additionally, CppUnitLite2 itself has a set of tests using itself in an interestingly recursive way.

What’s new

Fixtures created and destroyed around each test.

I can’t believe that the original version of CppUnitLite2 didn’t have this feature. It was the biggest glaring problem with the framework (I even pointed it out in the article). Each fixture used to be created at static initialization time, which, apart from being a really bad idea, caused a lot of objects to be created and live in memory at the same time. Considering that some of those objects would initialize/shutdown some important game systems, we clearly needed to manage their lifetime a bit better.

The test macro was changed to register a temporary test with the TestRegistry, which in turn will create the fixture at execution time. Now fixtures and tests are created just before each test, and destroyed right after, just as you would expect from any sane framework.

For those masochists out there, here’s the ugly macro for a test with fixture in all its glory. Notice how the TestRegistrar registers the dummy test class that we created as part of the macro.

 #define TEST_F(fixture, test_name)                                                      \ struct fixture##test_name : public fixture {                                        \ fixture##test_name(const char * name_) : m_name(name_) {}                       \ void test_name(TestResult& result_);                                            \ const char * m_name;                                                            \ };                                                                                  \ class fixture##test_name##Test : public Test                                        \ {                                                                                   \ public:                                                                         \ fixture##test_name##Test ()                                                 \ : Test (#test_name "Test", __FILE__, __LINE__) {}                       \ protected:                                                                      \ virtual void RunTest (TestResult& result_);                                 \ } fixture##test_name##Instance;                                                     \ TestRegistrar fixture##test_name##_registrar (&fixture##test_name##Instance);       \ void fixture##test_name##Test::RunTest (TestResult& result_) {                      \ fixture##test_name mt(m_name);                                                  \ mt.test_name(result_);                                                          \ }                                                                                   \ void fixture ## test_name::test_name(TestResult& result_)

Improved check statements

Another one of the major weaknesses of CppUnitLite as it stood was the check statements. You needed to have a custom check macro for each data type you cared to check. The generic CHECK_EQUAL macro will now work on any data type as long as it can be compared with operator == and output to a stream.

This version of CppUnitLite2 also includes some special macros to check arrays with one and two dimensions. Again, this works on any data type that supports operator == and operator[]. It came it very handy for checking equality of vectors and matrices in a game engine. For example, the following code checks that two matrices are equal:

Ironically, in some heavily restricted environments (like the PS3 cell processors), we can’t use streams, so in that case we’re forced to go back to the old specialized check macros.

Invariant statements in checks

Another really annoying feature of the previous version of CppUnitLite2 was that the code in a check statement was called multiple times, which could cause strange side effects.

Consider the following code:

The way the check statement was expanded out, both those functions could be evaluated multiple times. If the functions had side effects, they would be executed several times, causing unexpected results (for example, the check could fail but the printed result could be different). The fix wasn’t trivial because we couldn’t save off the value of each statement because they could be of any type and we have no way of finding the type from within the macro.

The check macro now expands out to a templated function, which is able to evaluate both statements only once and work with the saved value to avoid any surprising side effects.

Optional fixture initialization

As we continued hammering on the unit test framework for all our development, one clear pattern appeared that wasn’t handled by the framework: Being able to create fixtures with parameters defined in the test. For example, the fixture might do ten different steps, but we want to set a particular value in one of those steps, and by the time control reaches the test code itself it’s too late.

We started getting around that by creating different fixtures, and then by inhereting from other fixtures and changing some parameters. Both of those solutions were less than ideal (lots of code duplication and added complexity). Things got out of hand when we made the fixtures a templated class on a value, and created them by specifying the value as part of the texture name. Now it was a pain to debug and things were even less clear.

So we finally implemented it natively in CppUnitLite2 as fixtures with initialization parameters. We introduced a new macro, TEST_FP (test with fixture and parameters), which takes any parameters you want to pass to the fixture constuctor.

Now you can declare tests like this:

We should prob
ably simplify that to avoid repeating the fixture name, but that works fine for now.

Include cleanup

The original version of CppUnitLite was pretty casual about what headers it included. Unfortunately, that means that all test code were automatically exposed to a variety of standard headers, whether you wanted it or not. This version is a lot more strict, and the only header that will get pulled in from using it will be iosfwd, which is even a fairly lightweight one.

Memory allocations

We have very strict memory allocation wrappers around our tests, and if any memory leaks, it is reported as a failed test. Since checking for memory allocations is very platform specific, it is left out of the framework (although it could be added pretty easily in the platform-specific sections). In any case, some of the static-initialization time allocations were confusing the memory tracker, so we removed them completely. All it means is that we use an array to register tests instead of a std::vector and we cap the maximum number of tests to a fixed amount. If you need more than the default 5000 tests, go right ahead and add a few zeros.

New license

Previous versions of CppUnitLite were not released with an explicit license. To make things clear, I explicitly added the license text for the MIT license with an added restriction. The MIT license allows you to use the code in any project, commercial or otherwise, without any restrictions. The added restriction to the license prohibits its use in any military applications or projects with any military funding.

Ratings

How does this release of the CppUnitLite2 compare with respect to my initial set of features I used to rate the different unit tests frameworks?

• Minimal amount of work needed to add new tests. Nothing has changed there. Still passes with flying colors.
• Easy to modify and port. That has always been the strong suite of this framework.
• Supports setup/teardown steps (fixtures). Much improved now by creating them correctly around each test.
• Handles exceptions and crashes well. No changes there. Still one of the best I’ve seen.
• Good assert functionality. The check macros have been much improved to match the best frameworks out there.
• Supports different outputs. Yup. Still there.
• Supports suites. It still doesn’t. It just shows that I really haven’t needed this feature yet.

Future work

So, what’s next for CppUnitLite2? It’s really quite usable right now, but I could see a few changes happening in the next year, and maybe a few others will come up and surprise me.

Test registration is based on static initialization of global variables, which is not very reliable across compliers. For example, if you try to put the tests in a static library, MSVC will strip out the registration code leaving you with no tests. To get around that, we could introduce a custom build step in a scripting language that would generate a simple file with explicit registration of tests. This could also be used to collect tests from many different libraries into a single executable.

A couple of times it would have been convenient to have parameterized tests (I think that’s the correct term for it). Basically, to have tests that could run with different data. Something like this:

 TEST_PARAM(MyCustomTest, int a, int b) { CHECK (something); CHECK(something else); CHECK(yadda yadda); }  TEST(MyTest) { CALL_TEST(MyCustomTest, 10, 5); } TEST(MyTest) { CALL_TEST(MyCustomTest, 12, 7); }

The key point is that a test fails in the parameterized test, it should report the error correctly indicating where it came from.

This is actually pretty easy to implement with a couple of macros. We simply haven’t needed it more than a couple of times, so we haven’t bothered yet. But I imagine next time we run up against this we’ll bite the bullet and implement it.

Wrap-up

Charles had so much fun working with this framework, he decided to write his own from scratch. He made it available on his web site, so go check it out.

Thanks to High Moon Studios for being open and letting us release the changes we made and share them back with everybody. If you end up using the framework and you make any changes, drop me an email and I’ll add them for the next release.