Effective Units Tests and Integration Tests

A unit is the smallest testable part of an application.

More often than not, I see unit tests that forget this core tenet. Picture a method or function as a sandbox. When you are writing a unit test, never let a call get outside of your sandbox.

You unit test should be testing the conditional logic and correctness of the code in your sandbox. Let another test worry about all that external/dependent code. When writing a suite of unit tests we're interested in code coverage. We want to exercise as much of our code base as possible. That's a lot of tests. We need to make sure these tests are not fragile, difficult to write or hard to maintain. And we need to make sure they're quick to execute.

The code in your sandbox really only has one or two purposes:

• To execute some external operation, and/or
• To compute some result

Your unit test should be testing all possible code paths through your sandbox. What can cause your code path to change?

• If-Then statements
• Ternary operators
• Switch statements
• Boolean expressions (x = y or z)
• Try-catch blocks
• Goto statements (if applicable)

Each of these constructs can make one pass through your code different from the next one. That's the stuff your unit tests should be exercising.

Computing a result includes:

• Checking return values
• Checking thrown exceptions
• Checking set member variables and/or globals

Your unit test should be throwing all kinds of input at your code to ensure the return values are correct. Also that your code throws the appropriate exceptions on bad input. If your code doesn't return any values you may need to check that any global or member variables were correctly set.

How to stay in your sandbox
Nearly all programming languages have some sort of Fake, Mock or Stub library available for it. The purpose of these libraries is to short-circuit the external code and replace it with your test code. Something that:

1. Always returns an expected value
2. Has no state
3. Has no external state dependencies (i.e. a database initialized to known values)
4. Returns nearly instantly
5. Has no conditional logic in it

In "XUnit Test Patterns", Gerard Meszaros has a nice breakdown of Fakes, Mocks, Stubs and Dummies:

• Dummy objects are passed around but never actually used. Usually they are just used to fill parameter lists.
• Fake objects actually have working implementations, but usually take some shortcut which makes them not suitable for production (an in memory database is a good example).
• Stubs provide canned answers to calls made during the test, usually not responding at all to anything outside what's programmed in for the test. Stubs may also record information about calls, such as an email gateway stub that remembers the messages it 'sent', or maybe only how many messages it 'sent'.
• Mocks are objects pre-programmed with expectations which form a specification of the calls they are expected to receive.

I try to only use Dummies and Mock objects for unit tests. I don't like storing state in my test doubles ... after a while you spend so much time making an effective stub you don't know if you're testing the stub or the code in the sandbox. That's why I like mock objects. They usually just consist of "return value" or "raise exception" ... simple, maintainable and readable.

If you find yourself putting an If statement in a Mock, you're doing something wrong.

How can you verify an If-Then statement?
Most of the time you just want to be sure that an external method or function was actually called. Do you need to actually call that method? Hell no. Not in a unit test. Many of these Mock libraries also have provisions for "Was Called" semantics. Essentially a flag is set if a Mock method was called. If your library doesn't have one, it's pretty darn simple to simple to set one up:

So, rather than let your code call outside of your sandbox, replace the external method (via monkey-patching, dependency injection or some other mechanism) to use your mock instead.

Does all code need a Unit Test?
No. Consider the following gist:


What would a unit test accomplish here? Nothing. Just make sure you have tests around each of the methods within foo().

When Unit Tests aren't sufficient
Let's say we did write a unit test for foo() in the above example. Would it accomplish anything? No, because we aren't checking the semantic ordering of the calls within foo(). Putting logic in to ensure that do_this() was called before do_that() and do_something_else() would just be a waste of time.

What if foo() was changed to this:


Our unit test would still pass just fine. But it would be wrong. What we need here is an Integration Test. Integration tests do check the semantic ordering of statements. With an integration test we allow ourselves to step outside of our sandbox. We may even use test doubles to help us build our integration tests (including Fakes and Stubs).

But now our mandate for these tests have changed: No longer are we concerned with code coverage, but instead we're interested in testing specific usage scenarios that are critical to our customers acceptance of the software. Other than manual testing, integration testing is the only way to ensure we're building software that will work for the customer.

We don't need 100% code coverage, instead we want the 80% of the every day scenarios the user will be experiencing when they use our software. If we were making a word processor, do we need integration tests for mail merge, form builder and the math editor? Probably not. Sure, it would be nice to have, but it's not critical. What we absolutely need are integration tests for: launch, enter some text, setting basic styles like bold, italics, etc, printing, saving and loading, page flow, etc.

Do we need to do integration testing for every edge case where an error might occur? No (again, it would be nice to have) ... but realistically we need to test: disk full, disk failed, out of memory, out of paper, etc. The most common scenarios.


But, if integration testing is so good and so important, why not just do integration testing?

Because integration testing is hard, slow, brittle, hard to read and hard to maintain. Everything unit tests aren't. In other words ... they're a pain in the ass. Integration testing is sometimes frowned upon because of these limitations, but they're being compared to unit tests. Integration tests and unit tests are very different animals. They serve two different purposes and give two different levels of comfort to the developer. I don't even think they're comparable at all. The worst mistake a development team can make (regarding testing) is to mix and match their integration tests with their unit tests. You get the worst of both worlds. Keep them clearly separate.

Also, treat your integration tests like your core code base. Unit tests can be hacky since they're so small. But integration tests are complicated and need to be carefully maintained. They need to be documented properly. They need fantastic logging capabilities with rich output. They need to follow the same coding styles as your core code base. They need to be structured, refactored and updated so that they're always easily readable. Unit tests are rarely refactored, since they're so small and atomic, there's usually nothing to change.

So, to grow an effective body of Unit and Integration tests for your application, remember these rules:

• Don't step outside your sandbox for unit tests
• Use Mocks and Dummies for your unit tests
• Check all branching logic in your unit tests
• Go for very high code coverage for your unit tests
• Integration tests are difficult beasts. Go for high-impact user stories in your integration testing. Mostly Happy Day scenarios.
• Don't spend a lot of time on the edge-case failure conditions in your integration tests
• Keep your integration test code clean and maintainable. Refactor as frequently as your core code base. 
• Set up a dedicated integration test server that runs on every commit to trunk (they're slow and hard to set up remember)