Augmenting Exploratory Testing with Automation

My friend Brian spends his vacations exploring and mapping caves all over the world. He is part of a global community of like-minded enthusiasts who plan and execute multi-day trips miles underground, in the hopes that they might be the first people in history to lay eyes on a new cavern, pool, or rock formation. They take sophisticated mapping equipment that records their progress and the shape and size of the passages they traverse. Back at the surface, they upload this data for the benefit of future expeditions, along with notes about where they found water and where they slept. They also share updates on the state of improvements to the route: ladders, ropes, and other cached equipment and supplies. These include improvements they themselves added to the route.

rope-descent
Copyright 2017 by Brian Louden. All rights reserved. Used by permission.

Every improvement to the route gets the next expedition to the frontier faster. They can travel lighter by relying on cached equipment and supplies, and faster by knowing which paths present the fewest obstacles. By leaving their own improvements to the route and documenting their steps, they extend the exploratory reach of those to follow, and push the frontier deeper.

Brian is also my colleague at KSM Technology Partners. He writes software to analyze telemetry data from the power grid, paired with the known locations and output of generators for a given point in time, to drive the settlement of complex electricity markets. The state of the system at any given point on the grid comprises hundreds of variables: the proximity of generation and load, line ratings, circuit breakers, and known outages, to name a few. Exploratory testing requires reproducing the state of that system over some subset of variables the tester deems important to control, then fiddling with additional variables to see what breaks. Reproducing the state of the system – or test setup – to enable meaningful exploration takes a long time to perform manually.

Exploratory testing without automation is like exploring caves the first time every time, without benefit of route improvements.  We testers waste a lot of time if we manually execute setup drudgery to get to the frontier, where discovery requires the insight and judgment of a human tester.

And make no mistake, human judgment is required: just as ladders and ropes can’t map out the next unexplored cavern, automation cannot teach us anything about the (mis)behavior of software under unexplored conditions. As testers, exploratory learning is still our job. But by consigning test setup to automatons, we can extend our exploratory reach and push the frontier farther, faster.

The test doesn’t find the bug. A human finds the bug, and the test plays a role in helping the human find it.

Pradeep Soundararajan, H/T to Michael Bolton

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s