TestBash Utrecht 2018 Liveblog

This TestBash is really special to me. It is the first testing event that I went to since my 10-month hiatus. I’ve started working as a tester again in January this year, but going to this edition of TestBash feels like it makes my return to testing 100% complete.

Day 1

My original plan was to only go to the conference day, but two weeks ago I was reading the descriptions of the workshops. Coincidentally, I had been reading up on the Cynefin framework so when I saw that there would be a workshop about Cynefin at TestBash that serendipity moment was too hard to pass up on and I bought a ticket.

Cynefin workshop, by Martin Hynie & Ben Kelly

As I’m writing this, it’s the evening of day 1 and my brain sort of exploded today. Learning more about Cynefin is kind of hard to describe, but it’s not a linear process. I think everybody gets to learn more about this framework in their own way and at their own pace. I’m saying ‘learn more about the framework’ as opposed to ‘learn the framework’ for a reason. It’s not something like a skill you can learn to apply, it’s more about learning when or if you have data that might fit somewhere in the framework. I think this might not make sense to you and I’m not even going to attempt to summarise this workshop for you, because I would fail, miserably (Marianne made a sketchnote/summary, that might help you). I can only say that if you are someone who enjoys philosophy, complexity theory and problem solving, you might enjoy learning about Cynefin. As I’m only a beginning Cynefist myself, I would suggest reading the Wikipedia page to get a general idea and then to find video’s of people talking about it and go to a workshop yourself. Immersing myself into this framework for a day has been interesting, exhaustive and I ended up with more questions about it then I got answers. This is a good thing, in my book.

After the day was over, we still had drinks, food and a meetup happening at the venue. It was so great to chat with people I hadn’t seen for well over a year. It felt as though I had never left the community and people were very curious as to how my time off was, and what I was doing right now. A heartwarming experience.

Right now I’m satisfied, tired, and ready to call it a night. See you soon for the live blog of the conference day! 

Day 2

It’s a beautiful sunny day here in Utrecht and we are ready for Day 2 of TestBash Netherlands! I arrived at 8:30 to have some time for a much-needed coffee and chats with people. I’m here in the front row, trying to capture little nuggets of wisdom for you all. If you aren’t here at the conference, I hope you can at least experience some things of it through my blog. The day is being hosted by Bart Knaack and he always brings a smile to my face. Bring it on! 

Holding Space: Making Things Better by Doing Less – Maaret Pyhäjärvi

The first talk of the day is by Maaret! We start off with a radical idea: do less in your daily work and feel better by doing less and make things better.

She starts off by telling us that she works at F-Secure now, but she never stays in one job for too long (think, 2 to 3 years). She feels that as an internal employee, she gets to achieve more than a hired consultant, but only if she manages to make a real connection with management. That means that you really have to have a vision of what you want to achieve and change at the company.

So, how do you get to that ‘right place’? How do you get to be a catalyst for change?

Maaret has been in testing for 25 years. Even though she sometimes dabbles in other roles like project manager, or test manager, she always comes back to being a tester. She loves doing what she does in testing. After 25 years, she even has a feeling that an application speaks to her when she tests it. She feels where she has to go in her testing sessions.

She once had an assignment to teach 19 developers how to test. On one day, she sat together with the junior developer. They were pairing, but Maaret decided to not say anything, to not guide the testing. She actively didn’t say anything. The developer was testing by himself and suddenly he saw that he had to do a certain test. “You’d want me to do this test, don’t you?”. She didn’t say anything. It took a lot of energy to not say anything, and it was funny that he kept looking at her for validation. Maaret: “So, why are you doing all these awesome test cases when I’m next to you? Don’t they come to mind, unless I look at you?”. She realised that she was the developers ‘external imagination’, by being there when they tested. She entertained the idea of putting a voodoo doll of herself on the desks to remind the developers to test but quickly discarded the idea.

So, how could she (and we) do less and make the developers think of testing more? We are obsessed with contributing, we want to do things because it makes us feel good. It’s a way to feel proud of yourself and feel accomplishment. Managers, especially, are obsessed with contributions. There are testers being fired because they didn’t put in enough pull requests and JIRA tickets. Crazy, right?!

She shows a contribution of her own that looks silly now. In her former job, she reported and got fixed 2261 bugs. Now, she’s more proud of her achievements like coaching others, helping others improve their testing work and collaborating.

There’s a big difference in productivity versus generativity. Productivity is about ‘doing stuff’, whereas generativity is about ‘enabling stuff’. Sometimes, things happen when we (testers) are there.

We also care about getting credit too much. Maaret tries to care less about getting the credit because it can get in the way of caring about the work.

“Be the change you want to see”. I feel like this is the core of her message. What is your vision? What do you want to achieve, together with others? Do you want to be a catalyst (to change in your team, your company), a conscience (for the team, for the company), a cheerleader (for the team), a critical thinker? Someone has to step up to make things visible. This can be scary because you are leading the way to change, and that might scare some other people in your company. Maaret was once asked to lie in her test report and she decided to quit that job.

Maaret never thought she would be calling herself ‘a cheerleader’, she used to ‘play a role’ that was the counter of the person she was working with. So, if a person was very positive, she would mirror a negative role. But, now she is a positive influence on others. She notices that by saying good things, others would also take over that spirit and spread good things too. As a tester, you don’t have to be negative all the time. Maaret advises us to acknowledge when other people around you do good work and to say that to them! Give positive feedback and brighten their day. You might actually help their career when this news reaches their manager as well.

So, to summarise: hold space to enable others to do more. Recognise when you have a privilege (for example, when your boss likes you, you can get your message across more easily than others). Get your environment to work for you. Try to be brave when others are not. Teach and learn without having explicit permission (you can always ask for forgiveness later).

Pair with other people to catalyse your learning and the learning for the other person.

Building your wizard: make sure you invest in making your team a safe space, a space where people enjoy working together. A space where experience is spread. Learning is such an important part. Make sure you are learning every day on your job. Learning trumps everything!

An expert is not the one who knows the most, but who learns the fastest. Also, try to amplify the good behaviour in yourself and in others and speak up when you see bad behaviour.

A great talk by Maaret, I always love to see her speak!

Go Old School, Why Test Techniques Aren’t Dead – Sue Atkins

Sue is passionate about testing techniques and I’ve got to say, it’s refreshing to hear a talk about this topic. For some reason, most testers ignore ever talking about this?

Her father got her started on techniques and on testing. He didn’t ever stop her from asking questions. He gave her innate ability to feel when something wasn’t quite right.

What’s special about techniques? Using the techniques can help us clarify our thoughts, and that’s needed because there are a lot of techniques out there!

Her favourite is state transition testing. You can use the technique to help build a model, to think about the application and the states it has. In this particular example, the developers said that there will be three states. Sue thought there would be more. In a workshop, the team found out that there would be 15 states instead of three! I personally found the rest of the example a bit hard to follow because it was very context-specific. What I got out of it, is that there are always more error-states than you initially think. It’s when things go wrong that you get states that you didn’t expect.

[props for the first 2 speakers for enduring the technical problems. The beamer is a b*tch, it keeps turning off. Really annoying.]

Pairwise testing: with a test technique like this you can get a smaller set of test cases from a huge number. And it’s got tool support! Putting things together in a combination there’s more chance of catching a bug. Tools: www.pairwise.org. Allpairs tool. Orthogonal arrays to get your pairwise combinations.

Combining pairwise testing with equivalence partitions makes for even better testing.

[I do feel that this talk is a bit rushed. There’s a lot of text on the slides that’s being skipped over and it creates a bit of confusion on my part. My tip for Sue would be to make her slides less text-based.]

Double modal defects: that’s when you think X + X might find a problem.

So, why use different techniques? They find different types of defects. The techniques should complement one another: flow problems, data problems, states, combinations. The more angles, the better! [I think you should always consider risk].

And that’s the talk! I truly think she rushed a bit too much because there are still 10 minutes left. I would love a workshop on this topic because I think practising with test techniques never hurts and is more effective than talking about them.

Bart asked the best question: does exploratory testing mean you can let go of test techniques? NO, ABSOLUTELY NOT. Don’t be an idiot and think exploratory testing is just ‘doing random stuff’. It’s making test techniques work in real time. And now, I’m going to get a coffee.

Coffee break!

During the coffee break, I talked to some familiar faces and set out on a quest to get myself a pillow to sit on.  You can really tell this is a church because the chairs are frigging uncomfortable! I was already getting pain in my you know where during the first 2 talks, so I need this. I’m getting old, ya’ll.

Kill the Mutants – Simon & Nico

I have no idea what to expect from this talk, so I’ll let them surprise me. They both are developers and I think it’s a good thing to have developers speak at a conference. Same goes for getting testers to speak at developer conferences, btw!

They are going to talk about Stryker, a tool they made, which is a JavaScript (and TypeScript) mutation testing tool. It’s an open source tool. Mutation testing can tell you something about how good your tests are.

We had to stand for a little while and play a game: they asked us a couple of questions and if the answer was ‘no’ for you, you had to sit down. Almost everyone but Del and Johan sat down. So, although most people here had heard of mutation testing, almost no one in the audience is applying it.

Now, they show us a very old-school version of the test pyramid with that ugly cloud of ‘manual testing’ at the top. Where mutation testing comes in, is to test if your tests are of good quality.

They dare to talk about code coverage. I’m an absolute opponent of believing too much in code coverage. I think that metric gives you less information then you think, so I’m sceptical. I do like that they use real code and actually run the code to show what they’re doing. They brush over which kind of coverage types are important, but only briefly.

With mutation testing, you actually measure the effectiveness your tests. You start with your normal set of tests and the mutation testing framework creates mutants that will alter the normal code to see if the test now fails. If your tests aren’t of good quality, for example, you don’t test for boundary values, you’ll find that some tests give you false positives. The mutation testing report will tell you which tests give you false positives so you can improve your tests.

You can also integrate the mutation testing reports in your SonarQube or your build pipeline to force yourself to look into the results.

Example frameworks: Stryker (JavaScript), PITest (Java), Mutant (Ruby), Cosmic Ray (Python).

What are disadvantages of mutation testing? It can be slow and not every language has support for it.

Now we are getting a demo of the tool they made, Stryker. They show us live how to get Stryker up and running in a JavaScript project. Not really possible to live blog this, but you can click on the link above here to go to the Github page.

Summary: don’t use mutation testing to bash your developers, but see it as a way to learn more about how you can improve and strengthen the quality of your code.

Really good talk! I really appreciate the live coding parts and demo parts of this talk. The project is open source too, so I hope people start using it and trying it out. Sadly, I cannot use it in my project because Swift is not yet supported.

Encouraging Engagement – Marianne Duijst

Marianne has known the feeling of belonging to a community because she has been a girl scout since the age of 7. Now, she experiences the same in the testing community.

The story of changing and encouraging engagement started in the summer of 2016 when she took a course relating to her scouting activities. The course was focussed on personal growth and after 8 intensive days of physical activities (climbing in the mountains) she then had to work on her goals for the next 6 months. All the people in the group were happy to challenge themselves. Within 24 hours the people in the group got to know each other and focussed on helping each other to get up the mountain. Because Marianne had the bad luck of being ill, she really benefitted from getting help. That gave her a great sense of ‘community feeling’.

At the same time at work, there was a big change coming. The roles of tester and developer were being put together: everyone is now a software engineer. We all tend to identify with our role, so it was quite a change. The team now had to ‘own’ all the tasks.

We all have rituals at work: we go to lunch with the same people, we go for a coffee break with the same people. We connect with other people on an emotional level.

How often do you dare to connect with people you don’t know yet? At a conference, there’s a huge opportunity to do so. But also at work, there are options to do that. Grow your web of connections, you will never know what it can bring you!

What is your behaviour like, usually? Are you an owl, a hawk, a tortoise? This model (on the picture) can help you to assess how you are usually behaving. I know that I can be a goat sometimes, but I would like to think of myself as a more giving person usually. Being a peacock doesn’t sound half bad.

Realise that you have options to shift your role. It’s okay to sometimes take a more passive role, but it won’t be productive in some situations. If you want to influence your work culture, you’ll have to shift into the giving role.

At work, Marianne helped the culture and community forward by visualising success at her department and the connections between people. They also had regular lunch sessions to talk about how the atmosphere was and how they wanted the work culture to be like.

Marianne, just like Maaret, stresses this: don’t ask for permission, forgiveness can be acquired later. Learning and collaboration should not be done after getting permission but should be natural.

[Technical difficulties are back, but Marianne is handling them like a boss.]

Core message: Focus on people. You may disagree on the content, but you can still connect on an emotional level.

Let your light shine, so other people can do the same.

Positivity, folks, it works. [Although I sometimes enjoy being grumpy too].

Lunch break!

Extra long lunch break of 90 minutes. Great sandwiches, great people, great chats. The weather outside was very good, so I enjoyed that for a bit. Richard was so kind to lend me his power adapter because my laptop needed juice.

Mind Your Language – David Evans

I’ve already seen this talk once, so I’m curious if I remember any of it.

What makes a great tester? We shout the answers to David, but he just went to the internet. The internet says passion, attention to detail, curiosity, common sense….and more. The internet says many more things, analytical, orderly, tolerant of ambiguity…The list is huge!

We are asked to chose from 4 lists of attributes which ones we like the most as testers. Turns out, David fooled us because none of the lists actually describe the tester role, but in fact: developer, BA, copywriter, leader!? So, the attributes testers have are shared with many other roles.

The further an opinion is from your own, the harder it is to relate and accept. But this is what we have to work with during our daily work. [It is GOOD that people have a different opinion, that’s helping diversity.] You think you’re passionate, someone else might think you are impetuous. They might even think you’re crazy! Any characteristic you have can be exaggerated (positive or negative).

We create categories and put people in them. It’s easy and it helps us understand our social environment. We confirm and behave according to the unwritten rules of our own group. We compare ourselves with other groups.

You have many different groups you might feel you belong to. Those can be based on where you live, what your gender is, what your profession is. In that way, you almost guaranteed to feel at home in more than one group. Or, do you reject all those notions? You might think that those things (the social constructs) should not matter. You are a person, voila.

Is there even a problem here? Let’s take for a given that we identify as testers. Even there, there is more than one group (context-driven, automation engineer….). We meet each other at conferences and this strengthens our feeling of belonging to that group. A problem arises if we stay in our group, our inner circle. That enforces our biases about others and other groups. We risk talking in an echo-chamber, where you actually already know that everyone will agree with you.

So, mind your language and make sure you don’t limit possibilities for testing for other team members. Help them learn more about testing and don’t think you are the only special snowflake who can do testing. 

By saying “I’m a tester”, you can make people feel “I’m not a tester”, or “you don’t know how to test”. Instead, say “I specialise in testing”, “I can help you with testing”. He went over a few more examples. Great stuff, David! This is practical advice that people can apply.

Turns out, I didn’t remember ANYTHING about this talk so I enjoyed it 100%. Might be Korsakov or something, I dunno.

Lessons From Famous Detectives – Geoffrey van der Tas

Geoffrey comes on stage wearing a detective hat. We start off with a little clip from Sherlock, where he shows his amazing deductive skills from just a tiny bit of information.

Detectives and testers do have things in common though.

We start with detection. Finding the facts about the software, the process and people by listening, reading, talking, using and observing.

Geoffrey is trolling us a little. We had to pick out from 3 photo’s who is a serial killer. Turns out, some of us accused Geoffrey’s colleagues. Whoops! And the secret choice was obvious: don’t choose because you don’t know anything about the people in the photos.

Activate your mind by focussing! Don’t do too many things at once and don’t switch contexts all the time because you will miss information.

Absorbing. We make mind maps, we make visuals. There are techniques to help us absorb information better. Motivation to remember is a technique that helps your brain help information better. Explain the information to yourself and to others. Chunking is also helpful. You break down a blob of information into smaller parts that make sense to you.

Knowledge base. Saving information is a difficult thing. We simply cannot store a lot of tiny details. Sherlock can though. He remembers the crazy things, that no one else notices. He saves information by subject, not by date. You can use the mind palace (memory palace) technique to remember things. [I recognise a lot of techniques from the free course Learning how to Learn. I recommend that course for you all].

Deduction. It’s an art and not an exact science. For testers, it can be helpful to think about effects before the cause. Backwards reasoning can help here. When a defect pops up, you use this technique to figure out where the defect is coming from.

[after lunch dip kicks in for me, ughhhh.]

This talk was really fun, though. I enjoyed the many interactions with the audience!

Ginger beer & chocolate break!

Bought a TestSphere deck to try out with my team, directly from the creator Beren 🙂

Solving Complex Problems – Martijn

We like to solve complex problems, but the pitfall is that you might jump to conclusions. When confronted with a problem, it’s usually wise to spend more time defining what the problem is before you start solving it.

How do you define problems: asking the right questions and make things visible.

How do you ask the right questions? Aks OPEN questions. (Questions that can’t be answered with yes or no). Hints: who, what, where, why, when, how are good words in open questions. They are aimed at gathering more information. People who ask closed questions are often experts on the topic. They know a lot and that actually hinders them in their quest to solve the problem. Closed questions converge, whereas open questions diverge. That’s not to say that closed questions are bad, but you might get stuck if you only ask closed questions.

[This pace of this talk is very quick and it’s packed with good information, I’m not even capturing all the good stuff here.]

5 x why: used by many people to get to the root cause of a problem. But it’s not fail-safe. Why 5 why’s? Why not more? What about multiple causes? Mono-cause problems don’t exist! An alternative: event map. You write down a problem and create an event map around that.

With an event map, you can branch out the problem and look at multiple facets of what might have caused it.

What causes the growing number of problems that are complex? Big data. Automation. Workforce ageing. “Is the problem complex or is the solving complex?”. The problems itself haven’t really changed, but the amount of data and stuff you have to know to solve it becomes more complex.

[Geez, this guy is going FAST. The single-threaded processor that is my brain cannot parse all the information that is being thrown at me. I really have to look this one up in the Dojo when the videos get published.]

Unclear problems –> gather & order information

Incomprehensible problems –> Structured analysis

Unpredictable –> Controlled experimentation

[We are being tested with a puzzle and I feel like a total idiot because my brain is going blank. I blame it on the live blogging.]

Often, figuring out what is NOT the problem is very helpful.

This talk is just insanely packed with information, oh my god, I cannot get over it.

Summary: Define the problem before fixing it. Complex problems are often unclear problems. Apply an effective strategy to solve incomprehensible or unpredictable problems.

[I’m going to take a short break from blogging.]

Last ginger beer break!

Yay, Alex and I won with the #tbtestmatch Twitter thing that was going on and we got Ministrystroy of Testing swag!

Agile Test Management with CI – Angie Jones

Finally, I get to see Angie!! Omg, her accent, it brings a smile to my face. It’s so….American 😀

Once upon a time, she was working at a company where they had thousands of test cases and each release took forever because the tests took so long to run. The regression backlog had to be automated and that’s when Angie came in. There was no way they were going to automate 25k tests so they got it down to about 2k valuable test cases. The automation train got rolling, with a couple of famous tools etc. The automation was now a replacement for the regression phase. BUT.  Running these tests only at the regression phase was TOO LATE.

The process had to be made more efficient. Continuous integration, would that be something cool?

What about fast feedback? When the developer checked in a piece of code, they would benefit from getting fast feedback. But the tests took hours to run, so that wasn’t helpful. And what about test failures? They’d had to investigate. And this also doesn’t help the team owning the quality.

They differentiated the tests: build tests, smoke tests, all other tests.

Maybe divide & conquer would be a better approach? Divide the responsibility among the teams. If a team’s responsibility is ‘registration’, they would have the responsibility to make sure the tests in their work area would run. Adding tags to the tests also helped with putting the responsibility in the right teams.

The huge test build would also still run, but it ran every 6 hours.

Angie is now going through some specific problems you might run into when you are managing a very large automated test suite. Adding tags also gave some problems, for example, inflexibility. They solved that with runtime configuration.

Lessons learned: Bugs can be missed [I think the probability of bugs being missed is 100% in every software project, but yeah.] Tests have to be reliable, no one likes to have false negatives. Make sure you clean up the tests and make them more reliable.

The end of the official programme!

99-second talks. I think they’re a great addition to the TestBash conference days, but I’m not going to live blog them. I’m spent! My neck and shoulders are killing me from typing on the laptop the entire day so I’m going to lean back and enjoy the short talks.

Thank you so much for reading this live blog and I truly hope this has helped you follow along a little.