Exploring exploratory testing

Recently I have come across a question regarding exploratory testing in one of the software testing textbooks.

Ralph was the project manager of a small telephone switching system development. He started out as an electrical engineer, specifically as a logic designer. As his career progressed,he acquired solid domain knowledge of telephone switching systems. When the project prototype was completed, and when the first increment of software was loaded, Ralph signed up for three hours of scarce system test time. At the end of his session, he called the whole project team together and announced that the system was “full of holes” and that much work remained. When asked for more details, all Ralph could say was that he tried “a bunch of things” and most of them did not work. There was no record of faults found, no indication of the tests that were executed, and nothing was repeatable to help isolate the faults.

Discuss the ways in which Ralph’s testing conforms to and differs from exploratory testing.

(Software Testing – A Craftsman’s Approach 4th edition by Paul C. Jorgensen)

After reading this, I have no idea that how it differs from exploratory testing. Because my idea about exploratory testing mimicked what Ralph was doing.

So I have done little research on the topic “Exploratory testing”. I read couple of text books and online tutorials which details about exploratory testing. Here is what I have found:

I have misunderstood Exploratory testing with ad hoc testing. Even though Ralph was able to find bugs in the system through this method. However it is,

  • Highly unstable
  • difficult to manage and schedule
  • Unable to reproduce the faults
  • No documentation

If this is not Exploratory testing, then what is it ? How it differs from ad hoc testing ?

I found the below definition in Standard Glossary of Terms used in Software Testing :International Software Testing Qualifications Board.

Exploratory test is an informal test design technique where the tester actively controls the design of the tests as those tests are performed and uses information gained while testing to design new and better tests.

In exploratory testing, testers learn about the system under test and uses his/her knowledge to explore the system more with more focused tests.

Also while testing if the tester happens to encounter an abnormal system behavior, he/she will be designing and executing more focused tests in order to isolate the issue.This way exploratory testing is more or less like simultaneous learning, design and execution. Also it is highly creative and interactive approach.

It still lacks the important part which is nothing but the documentation part. This can undermine the whole purpose of doing testing if we can’t state how much or what we have tested/ not tested in a system.

One way people have come up with to reduce this problem with exploratory testing is to use test charters. A test charter specifies the tasks, objectives, and deliverables. The charter may also identify where to focus the testing effort,what is in and out of scope of the testing session, and what resources should be committed to complete the planned tests. Exploratory sessions are then planned to achieve those objectives. A session used to focus on particular defect types and other potentially problematic areas in a system.

Then this question pops up in my head. To what degree is our testing exploratory?

I assume most of of the testers have been doing pretty much the same thing as mentioned above, such as  “simultaneous learning, design and execution”, in their day to day testing activities.Because of that reason, for me now the distinction between testing and exploratory testing appeared as a blur.

 I was delighted to quote few words here from James Bach’s blog regarding exploratory testing.

It is time to begin assuming that testing means exploratory testing, instead of assuming that it means scripted testing that has also has exploration in it to some degree.

Are you doing testing?Then you are already doing exploratory testing. Are you doing scripted testing? If you’re doing it responsibly, you are doing exploratory testing with scripting(and perhaps with checking). If you’re only doing “scripted testing,” then you are just doing unmotivated checking,and we would say that you are not really testing. You are trying to behave like a machine, not a responsible tester.”

Hope you also got an idea why our Project manager Ralph’s testing is different from exploratory testing.

Or, Would you even consider that as testing ? Don’t you think he was just doing an unmotivated checking ?

References:

  • Software Testing – A Craftsman’s Approach 4th edition by Paul C. Jorgensen
  • Advanced Software Testing—Vol. 1
  • The Software Test Engineer’s Handbook, 2nd Edition