Exploratory Testing with the team

I use Exploratory Testing (will call it ET from now on) every day at work. I like to use it, because it requires focus, creativity, and deep thinking skills.

Now, even though my team is doing Scrum, we are now close to the beta-release of our product and the business starts to panic a little (I know, scope and time shouldn’t be fixed in agile, that’s a whole different story). We have waaaaaaay more points in the Sprint than we can ever commit to normally, and a sudden explosion of bugs happened. This left me feeling not so confident about our product and I also noted that the developers didn’t test that often. I then (finally!) formed the idea of getting everybody in the team together and do a couple of ET sessions.

I told the PO and Scrummaster about the plan and they liked it. (a small voice in my head said: ‘why didn’t we do this before?’). The original plan was to do an ET session on Thursday, with a time box of 2 hours. Not 2 hours of testing, but 15 minutes explanation and 2 time boxes of 30 minutes of testing. In between a little break for ppl to run around, get coffee etc.

The plan changed when the PO wanted to do it earlier! So suddenly on Tuesday I had 30 minutes to prepare myself….ok here goes. I quickly made a presentation about ET in a nutshell. With only a couple minutes of explanation at hand I thought about the essentials of ET. I thought the most important thing was to explain what ET is NOT. Many ‘n00bs’ think it’s somewhat monkey-testing. “You just go around in the application, and you’ll find something”. No, ET is about focussing on the charter you’ve just made. So, I had to explain about charters (the mission of the ET session). Also, documenting is very important during the session. Developers, who mostly write automated tests, are maybe not used to remembering what they’ve done. And you don’t want to come across a defect and then not remember how you’ve produced it. So, document what you’re doing. I also had to explain that I don’t need their whole documented journey, we just want to find nasty defects and that’s the only outcome of the session: defects and steps to reproduce.

In those 30 minutes prep, I also created some charters. I thought it would take too much time if we would create those with everybody involved. Time was sparse. The charters involved testing risky area’s in the product. We are making a tablet app, so one of the charters said “Explore rotating in the app with an Android tablet, to discover unexpected behaviour”.

For this first session, we got together with a part of the team (the iOS developers and the back-end developers). I wanted to let the iOS developers test the Android app and vice versa. I explained the bare basics of ET and we divided the charters among those involved, only focussing on Android this time. Some people paired up, but some preferred to test alone. And off we went! We only had time for one time box of testing, but after 30 minutes we found a lot already. During the debrief I wrote down 14 defects. The Android developers came to me and we discussed the defects. Some would be solved (like a crash we found after rotating) and some were dismissed as ‘won’t fix’.

But wait a second, 30 minutes of testing with 6 people uncovered 14 defects. Wow. When I test alone, I cannot find that many. Plus, the input of other people is invaluable. To turn around the idea that ‘testing is done by the tester’ into ‘testing is done by the team’ was the best outcome. The developers liked the testing (I guess because it was only a short amount of time :P) and also that it was focussed on a clear goal. Of course, they weren’t expert testers at once. I noticed that they were distracted so easily! I wonder if they really did ET or that it was a bit of monkey testing after all. But for the first time, I don’t really care.

On Thursday we had another ET session, with more people. Again, I explained the idea of ET, and with more people on board we could do more charters. We had so much fun again this time. Many nasty defects were found and I saw a lot of ‘testing potential’ in some developers. One of the iOS developers was very good at it; very methodical in his documenting skills and approach and able to focus very well.

Observations this time included:

• Pairs of people are able to focus better, even more so when they are alone in a room.
• Some developers are very easily distracted (again). They find a bug and start debugging (I had to remind them of the mission, focus and timebox). 

And then, when the time box was nearing its end….the test environment died. Frustration all over the place. It was down for the next 2 hours so we couldn’t do another session. But in the end, with designers, PO, testers (even one from another team) and developers participating (!) we found twenty something defects. More importantly, there was good enegery, everybody liked the approach and talked about doing this every sprint. This made me very happy. I really hope this was a turning point for the team and we will see (‘manual’) testing more as a team activity instead of something that ‘only testers do’.

And now a voice in my head screamed: “Why didn’t we do this before!!????”.

And: Exploratory Testing approach all came from “Explore It!”, by Elisabeth Hendrickson.

Next time I’m going to try and introduce more ET stuff like the different test techniques of ET and the Test Heuristics Cheat Sheet. Hopefully, we can progress the ET skills of our whole team.