Tuesday, December 1, 2009

Inside Out

When we as testers approach a system, we often attempt to do so through analysis and exercise. We break down inputs, do identify load characteristics, define interesting use cases, create performance paths for throughput testing, etc. These tests are all great, but they all share one major characteristic: they test from the outside in.

The net effect of testing outside in is that you work the components that are exposed to the user (UI, API, etc) fairly well, but you're not actually exercising the internal components much. The classic answer is that components are (or darn well should be!) unit tested. So we're fine. Except the ways in which we're not fine.

We still have a hole in our approach. We're exercising an internal component in isolation with our unit tests. And we're exercising externally visible components with our system tests. And we're hitting some aspects of the internal components with our system tests, but not particularly well. Think if it like a soccer game: if your team always has the ball, your forwards (your externally visible components) will get a lot of work, but your defenders aren't going to get a great workout. Sure, they're on the field, and sure they have roles and do things, but that's hardly exercising everything they should be able to do.

We need to exercise our defenders. Unit tests are great, but (to continue the analogy), they're more like drills than like an actual soccer game. We need to put it together. We need a system test of our internal components.

We need to design a test from the inside out.

Just like we did for our externally visible components, we need to take a hard look at our internal components and break them down. What kinds of inputs do they take? What can we do to those inputs (boundary value analysis, data type mismatching)? What about interactions between internal components, or with externally visible components? Then take that knowledge and design a system test to show it. You'll still get at it through external components, but you're manipulating the components in a way specifically designed to exercise the internal components in the ways you've identified.

For example, let's say your system has a UI and a database component. In this case, the UI is the externally visible component, and the database is an internal component. Certain external tests you do will validate some things about the database - like putting in a too long login name. However, the database is probably not stressed. So we design an internal test. We may notice when studying the internal component (our database) that it uses a stored procedure that locks a table when creating a user. Fine. So we're going to stress that and make sure it doesn't cause problems. So we design a load test that makes a lot of users, to see if that table lock causes any problems. The test will run through the UI using your preferred tool, but it's designed to exercise the internal component. It's a test we've designed from the inside out.

What exactly you wind up testing is going to be highly specific to your system and to the internal components of your system. The general rule is that you can analyze your system from the inside as well as from the outside in order to identify interesting system tests.

No comments:

Post a Comment