Mapping Biases to Testing - The Framing Effect
Hello dear reader, welcome back to another blog post in de Mapping Biases to Testing series. Today, I’m going to write about the Framing Effect. This effect can potentially affect your work greatly. What’s even scarier is that you can never be sure, because the real world differs from a controlled scientific experiment. I’ll first go over how the framing effect is explained in scientific literature and then consider how it can be a risk for testing activities.
In the book Thinking Fast and Slow the workings of this effect are explained by an experiment conducted by Kahneman himself. Daniel Kahneman and his colleague Amos Tversky asked two groups of people to make a choice about a treatment for a fictional disease that will affect 600 people. Group A got the positive scenario: “Choose the treatment and save 200 out of 600 lives” while group B got the negative scenario: “Choose the treatment and have 400 certain deaths out of 600 lives”. Effectively, this is the same choice, but because it was framed differently you can guess what happened: 72% of the participants in group A were in favour of using the treatment, but out of the people in group B only 22% chose to do the treatment.
Keep in mind though, that the two groups only saw their own scenario. A later replication of the experiment by James N. Druckman gave groups access to both scenarios to see if the framing effect would still take place. It did, to a small extent. A more important message I took from that paper is the question “are people risk-neutral at the moment they are exposed to a frame?” (p. 96, Journal of Economic Psychology 22).
The Link to Testing
What is your view towards risk? That is super contextual. If you are working in an industry where competition is fierce, you might be inclined to take risks and release without testing thoroughly. If you are working on healthcare software and products, risk-averse behaviour seems more appropriate. And let’s not even talk about legal matters, standards and the law and their effects on the software development process. The point is, no one is ever risk-neutral. Because individuals are subconsciously influenced by their context, it is more likely that they are risk-seeking or risk-averse.
This attitude towards risk starts to matter when we communicate. An important part of software development is communication. We seem to forget it sometimes, but if we don’t convey information to one another we can never truly succeed. As a tester, an important goal should be to communicate findings to others. This is starting to relate to the framing effect. Why? Because how you tell others about your findings is very important. This is when you start asking yourself some questions: “What is my main message and what would I like the outcome to be? Am I going to frame my message in a positive or negative way? Who is going to receive my message? Are the receivers going to share my opinion? Do we have the same goal?”
Realising that you have options is paramount.
Let’s go over this with a concrete example. Once upon a time, I was testing an input field in a mobile app, with the tool Charles running in the background to analyse the calls that were going from the app to the backend system. The backend system was not developed in our team. I noticed that as I typed in one character, a service call was fired to the backend. Since I was typing in a set of 8 characters to make the input valid, I saw 8 calls going to the backend. What was even worse, the calls were fired twice. In the app, you didn’t notice this effect at all, was it a bug? I thought it was a performance risk, but when I offhandedly made a remark about this to the developer, he wasn’t really interested. How I was telling it right now was not effective. Not his backend, not his party.
I still thought this effect of sending useless calls to the backend needed to be solved so I had to change my strategy and frame my message differently. I went to the Product Owner with some more information. I had gone to an architect to ask how expensive backend calls were for this company. Turns out, every call costs a few cents and the volume of calls per day is quite large (millions), so fixing this would have a great benefit. Now that the story was complete, I explained it and it didn’t take much to convince the Product Owner that this needed to be solved. I’d like to think I succeeded because I framed the problem in a way that took company interest at heart. Of course, I can never be sure because I cannot turn back time to test an alternative. That is how our world differs from the scientific experiments of the framing effect, we can never truly test the alternative.
Communication moments
After reading this little story, I hope that you start thinking about your own context and when you encounter framing choices. What are important communication moments that relate to testing for you? How do you report bugs? Matter-of-factly, or do you try to frame them in a way that explains their risk for the company? You can imagine that saying ‘does not meet requirements’ might not automatically get people to jump with excitement. Go one step further and think about the questions I presented earlier.
How do you handle test reporting? Do you think about who reads (or hears) your reports? In this case, I want to make a special mention for KPIs here. Before you think, “what on earth does that have to do with test reporting”, hear me out. Often, testers are not the ones deciding what happens with the information that testing uncovered (bugs, risks for the project, etc). Ideally, the team would have a say in that, but in many companies, it is decided by the stakeholders of the project. What’s funny is that those individual stakeholders might have different KPIs to reach. So your test report is going to be interpreted differently because the person reading it is affected by the frame of his KPI. One person might have a target of reaching an X amount of sales and will benefit from releasing new features. Another person might have a target of keeping the performance within a certain margin and is concerned that new releases decrease performance. You can see the problem here. Those different receivers of your test reports have different risk preferences; they aren’t neutral. They are likely to seek information in your report that supports their frame and goal (confirmation bias). Company politics are a fact of life and they will matter in your work. Learn to recognise the framing options and how they might influence the people who you communicate with.
On a smaller scale, all your communications can be linked to the framing effect. You always have a choice in how you say things. Thinking about that all the time is not possible, but when you are trying to say something very important, you might want to go over some possible consequences of different ways of framing what you want to say.
In the end, however, remember that “Every significant choice we make in life comes with some uncertainty” (p. 270, Thinking, Fast and Slow). And with that quote from Kahneman, I’ll leave you now and I hope I made you think about the framing effect. See you next time!
Older posts in this series:
Sources for this post:
Thinking, Fast and Slow – Daniel Kahneman
Evaluating Framing Effects – James N. Druckman (Journal of Economic Psychology 22, 2001. p. 91-101)
The Framing of Decisions and the Psychology of Choice – Amos Tversky, Daniel Kahneman (Science 211, 1981. p. 453-458)
Comments ()