Exploratory Testing with the Team: The Sequel!

As luck would have it, one of the developers I worked with during the time I was doing experiments with getting my whole team hooked on exploratory testing is now also working at my current client. He came to my desk and asked if I was interested in teaching his team about exploratory testing, because he felt it would be very useful. Great to hear, of course! I was faced with a new challenge though. How do you set up an exploratory test session without any domain knowledge? Would it even be an issue? In this blogpost, I’ll walk you through what I did to make the session a successful event.

Preparation

First of all, what really helped was the enthusiasm this particular developer displayed for exploratory testing. He knew what strings to pull and with which people we needed to have a chat to get this thing off the ground. We started by getting some people together and explain to them why we wanted to do a test session that was more structured than they are used to (they are used to doing ‘break the app-sessions’, which translates mostly to ad-hoc testing). I made the case for giving people a mission (a focus area) during testing (the mission/charter), so they would end up doing a targeted deeper coverage test session instead of superficial coverage on a broad area. After the people who had a say about how the team spends its time were convinced, we planned the location, time and duration of the session.

Next, we looked up a domain expert. I knew next to nothing about the application under test this time, which made me slightly uncomfortable. I needed the domain expert to tell me where she thinks we should focus our test efforts. This conversation was kind of hard. It was clear the domain expert knew a lot about the application, but she was definitely no tester. We had communication problems, basically. The way she tests the application seemed quite random to me, she described it as she ‘just knew where the problems would be’. Intuition based on experience, I guess? But for other people to get value out the session we would need to help them. I needed to get her tacit knowledge into the explicit domain (the trickiest part of testing! How to communicate effectively about it). I started asking her questions, like “what are the risky parts of the application in your opinion?”, “are there any returning bugs on production”, “where do you go first when you start testing the app”. After half an hour, we had quite a lot of test missions written down.

More conversations with developers gave me some ideas about where technical risks might be, so we wrote a couple of charters on that too. And finally, my own expertise in certain areas also helped. We wrote a couple of generic charters that focus more on quality attributes and general areas I always focus on: UX, error flows, looking for time-outs etcetera.

The session

The room and lunch were all arranged by the developers. We also made sure there were plenty of devices prepared with the right build of the application. People literally just had to show up and test. I started the session by giving an introduction to Exploratory Testing again, that took about 10 minutes.

After that, it was testing time! In about an hour, we found sixty bugs. Of course, some developers fell into the pitfall of not documenting their steps enough and weren’t able to reproduce some bugs. Other developers got a bit into despair mode when they saw how many bugs were found. Jokes like ‘Toss this app in the bin’, ‘Who the f wrote this crappy app?!’, ‘After the test session I am going to change my rating for this app!’ could be heard. I walked around and gave hints and tips wherever possible. Mostly I said “have you documented anything??”.

It was all good fun. The atmosphere was relaxed, people were into it and we got value out of it. They want to do a session like this again. I think it is better if they would incorporate exploratory testing in their everyday work, instead of waiting to do a big session before a release. Another developer suggested that they could do it during code reviews and it makes sense to me. A code review is a form of exploratory testing in my eyes, so why not add a little bit of extra testing to that?

Next steps

This session was only with Android developers. I’m probably going to do another session with the iOS developers. I’m also curious to the reception of the event. Right now, all the bugs have been logged, but no decision has been made yet which ones are going to be solved. Since six teams are involved, I am foreseeing some difficulties in that regard (defect ping-pong, anyone?). I’m especially curious as to how the enthusiasm of the developers will fluctuate over time. I will keep you posted!