Tuesday 29 May 2012

The Unit Testing Catch 22

I am unit testing. Unit testing falls closer to the second of the two types of tests I enumerated here and is, I think everyone agrees, a 'good thing.'

The problem with unit testing is that, like everything short of full functional testing, it relies on domain knowledge. In the days when everything was specified in documents, about which architects had meetings and then bestowed like blessings upon the rest of us, this was not a big deal; the document described exactly what stimulus the unit would receive and if it was wrong it was not your problem. These days we are all agile and stuff, and there are no documents, and we are forced to rely on our own understanding of the wider system to know what stimulus our unit will receive, which in a large system is almost certainly incomplete.

The catch 22 here is that the majority of bugs are due to incomplete domain knowledge. I simply didn't realise that the system would, in some rare occasions, provide me with two notifications before actually doing anything. I assumed that when I received the second notification the first operation was complete. No amount of unit testing would have caught this bug because I would no more have tested it than I would have tested how it coped receiving references to deleted objects.

This is why I am suspicious of mocking frameworks. Mocking is useful, obviously, but it forces you to rely even more on domain knowledge, which is a point of weakness.