Exploring the C++ Unit Testing Framework Jungle over on Games from Within is a really good look at all of the C++ unit testing frameworks out there. It compares the following frameworks: * CppUnit * Boost.Test * CppUnitLite * NanoCppUnit…

Years of work on various large-scale software systems made me realize a few simple things:   Software testing is hard. Great advances in Test Driven Development made testing more productive. Still, to achieve full test coverage for a system,…

2005-05-16 links

* Games from Within: Exploring the C++ Unit Testing Framework Jungle

* SCons: A software construction tool

Taken from Jim’s del.icio.us links

Feet Up!

If you’re shopping around for a C++ unit test framework, make sure to check out this analysis of C++ unit test frameworks. It’s a good idea to pay attention to the conclusions made by the author of such a…

Great review. I did spent some time comparing unit test frameworks and asking people for something simple.

For now i’m using TUT http://tut-framework.sourceforge.net . Its favourite feature for me – fixtures. Once you wrote struct containing data setup(ctor) and teardown(dtor) for test it gets constructed and destructed for each test in a group. And fixture members are referenced implicitly through “this” pointer. I choosed it over a boost because i dont like libraries that need to be built in some special way only. Major drawback of TUT is in its compiler requirements – i just cant use it in the old project built by MSVC6.

I also find book reviews by people who actually do a lot of programming very useful. Sometimes it’s really hard to find books that don’t talk about the basics for half the book. So I’m really happy to see your reviews, keep it up

BTW, would you mind mentioning what MT plugin you are using for the currently reading section? I’m new to the MT world.

Thanks

GUI applications are the most difficult kind to do with TDD (and, to a certain extent, the ones you get least amount of benefit from, especially if they don’t have much real “logic” inside).

Some resources for TDD with GUIs are the Yahoo mailing list (very little traffic) http://groups.yahoo.com/group/TestFirstUserInterfaces/, and Phlip’s book (http://www.amazon.com/exec/obidos/tg/detail/-/0321227328), which should be available in a few months.

The main problem with GUI apps is that you often end up having to twist the framework around to be able to do TDD comfortably.

Games on the other hand, are *not* mostly GUIs. There’s a layer that deals with rendering graphics or creating sound, but most of it is the game logic inside, which can easily be done in a TDD way (especially for anything below game scripts, I’m not convinced that TDD is the right way to go for really high level gameplay code which is changing all the time and nothing depends on it).

More about all this soon. Very soon

http://arrizza.com/unittesters/jutasserter/jutasserter.html

for a very simple unittester

# Small:

* UtAsserter.h: 55 lines

* UtAsserter.cpp: 343 lines

* main.cpp: 2 lines

# Simple:

* Test cases are created by declaring them: TEST(x)

* main() requires only one line TestSuite::Run()

# don’t have to maintain the test suite, done automatically

# easy to add more assert types

# format works with MSVCs next_error (F4)

# works with VC6 & VC7; partially works with g++

http://article.gmane.org/gmane.comp.lib.boost.devel/116244

> to the error location whether or not you use one

> assertion per test case. One thing I am sure though is

> that one assertion per test case will definitely lead to

> longer run time ( with couple hundred assertions in

> test – 10 times and more), more time spend on tests

> development

It not so much a matter of style, but of maintainability. Doing TDD, I expect to write a lot of unit tests. Usually about as many lines of code as production code, or even more. I also expect to refactor my code heavily, which means refactoring my unit tests as well.

When I used to write “large” unit tests (>50 lines containing >5 assert statements), I found it very difficult to refactor anything. The causes for a failure were not clear, and it often required me to jump in the debugger and step through the code.

Having really simple test cases (I rather use that term than “one assertion per test case”), wich means test cases <10 lines and at most 1-2 asserts, I find that I can refactor any test case trivially because it’s immediatelly obvious what they’re trying to accomplish.

Yes, it’ll be slower to run the tests, but hopefully not much slower (I usually want a unit test to take less than 1ms to execute). When the running time goes over 3-4 seconds then it’s when I have to start using suites. Also, that’s when having per-suite setup/teardown can really help with some of those performance issues.

But you hit in the nail when you said that the main problem of working in a completely generic library is trying to support everybody. I’m sure there are people who want exactly the opposite.

I’m sure you found some of these links already, but here is some reading on the whole “one assert per test approach”: http://www.artima.com/weblogs/viewpost.jsp?thread=35578 and http://www.theserverside.com/news/thread.tss?thread_id=24136.

I will definitely write about it in the near future

> have applied that in real life. In practice a

> few test cases end up with two assertions, but

> the majority of them only have one. When I

> started using unit tests, I would cram a lot

> of asserts in a single test case, but that

> lead to tests that were difficult to refactor.

> When you’re doing TDD, you need to refactor

> your tests as much as your production code,

> and simplicity is essential.

After you mention this principle I read a couple of articles and discussions about it and it still doesn’t make too much sense to me. In general I am in favor in TDD ideas but I never like extremes in any aspects of life. Solid unit tests written pre/during/post class implementation is a cornerstone of reliable development. But what you advocate is just a matter of style IMO. I prefer test cases organization by feature/area: test constructors. test destructors, test access method, test comparison, test search, validations e.t.c. With proper test framework you will get pointed exactly to the error location whether or not you use one assertion per test case. One thing I am sure though is that one assertion per test case will definitely lead to longer run time ( with couple hundred assertions in test – 10 times and more), more time spend on tests development (just compare how many lines more you will need to type) and more time spend on fixures preparation and managing (imagine simple case where you just refactor common code into fixure. But then you found that one assertion require slightly different one, the same with another. Now you write second fixure that reuse first one e.t.c). All in all I don’t see how writing several related assertions together make my unit test code less “requirement like”.

As a side note I would like to mention that in design and implementation on test framework the biggest challenge I found is not code portability (even though Boost.Test works on several dozens of configurations – essentially most that I know about – which should say something about portability) or support some specific features (like test case dependency – which BTW could be useful, believe it or not, specifically in a cases when you know that one test case will run very long and fail if another test case is failing). The biggest challenge is huge diversity of requirements to/expectations from unit test framework. And in many case they contradict each other. You yourself managed to state both requirements independency from exception handling and complete exception handling support for your ideal framework. For this reason Boost.Test is presented not as one monolithic framework, but rather as a collection of many components used in different environments/circumstances: from simplest one header minimal testing component to the full featured Unit Test Framework. I did not try to enforce any specific testing style and I am trying to support features that could be useful.