The Thing with Quality at Speed

For the first time ever I’ve had to start writing a blog for the third time over. A new experience to me, because usually the words come flowing. Not this time! The topic is a difficult one in the sense that I have a hard time finding the right words to express myself. It’s so easy for people to misunderstand what you mean.

I already asked a question on Twitter about this topic: “What, according to you, are prerequisites for making Continuous Delivery (CD) successful for a company?”. I got a variety of different responses, which was great to expand my own thought process.

As you can probably guess, I have an ‘issue’ with how CD is implemented in its current form. As always, I feel that people are quick to jump on the ‘tool as solution’ bandwagon, without really assessing what problem they are trying to solve and what the downsides of said solution are.

I will list a couple of the Twitter responses I got to my question first, with my comments:

“What, according to you, are prerequisites for making Continuous Delivery (CD) successful for a company?”

• Seeing CD not as an end state to achieve, but as a change that will challenge you to solve problems that were less visible before CD (Joep Schuurkes, @j19sh).

A very Zen way of thinking, I could jokingly say: the way towards the goal is more important than the end goal. A very insightful point that I hadn’t considered.

• there’s a real business need for it and little risk for backfiring? (Dave von Stein, @Dave_von_S)

The only person who mentioned risk! Something we often forget to think about.

• Start doing it is one of the best ways to find out if you are ready for it or not. Imho. Admitting you need a culture of trust and learning (Merlijn Tishauer, @mtishauer)

I love the pragmatism of this response and the mentioning of culture and trust!

• People! It’s all about people. (Robert Strauch, @shrubbytester)

Couldn’t agree more, it’s still people who are making this work.

• Feedback – learning cycle. People who don’t learn from internal feedback will struggle with external that leaks more. (Maaret Pyhäjärvi, @maaretp)

CD is indeed all about fast feedback, and improving on things you’ve learned.

• if CI doesn’t work (including not rushing to fix a broken build) don’t bother with CD. (Gil Zilberfeld, @gil_zilberfeld)

This repsons hints to what I think about CD as well. It’s easy to see it as a silver bullet, but don’t skip a few steps.

Thanks for all your replies, I appreciated all of them.

Now, onto the hard part of this blog post: what exactly is my issue? In the last couple of years, I have observed a shift in the IT landscape with regards to how we deal with quality. A lot of the (management) focus was about cutting costs, going for shorter release cycles, getting away from long-term software projects. But were we really shifting away from the old approaches and ways of thinking?

I have only worked at big corporate clients and they have made me very, very sceptical. Sure, the software development teams work ‘agile’, but what does that mean when around those teams a big, logging enterprise is still imposing its rules and mistrust upon those teams? What does it mean when they say ‘quality is important to us’, yet they don’t give teams the time or mandate to adhere to the Definition of Done? What does it mean to work ‘agile’ with User Stories and stuff like that, but still have a long term planning with deadlines and milestones set months in advance (even though we call it a ‘roadmap’)?

In short: I think a lot of companies are faking it. They don’t realise they’re faking it, but you can’t implement agile. You have to be agile. I am personally not sure if this is even possible in big companies. (I might be too negative for your taste here, excusez-moi). Of course, you might have a counter example and if you do, please share it. Aside from the Google’s and Facebook’s of this world, I have yet to see another company using CD with a DevOps way of working successfully.

In any case, quite a few big, logging companies have now discovered DevOps and Continuous Delivery. Doing a shitty job at working agile wasn’t enough, they want to push heaps of shit to production faster. Oh, why are we constantly making the same mistake thinking a tool will solve problems for us? If a company has not grasped the essence of agile, why are they even thinking of going for Continuous Delivery?

I have seen companies go for the option: just implement it (features) and see what happens. What were the consequences that I observed? The Ops engineers were flooded with incidents at times, because the part before production was rushed. Testing, ain’t nobody got time for that!? The engineers had a blind faith in automation, manual work was looked at with contempt. Moreover, I didn’t see a culture of really questioning what tests were written. Testers were sacked, because they were slowing down the process and replaced with rules like “minimum of 70% unit test coverage” and “every flow should be covered with a happy path and alternative path test”. Testing was seen only as an artefact creation process and the whole investigative practices and risk assessing practices were forgotten.

Am I completely dismissive of Continuous Delivery though? Not at all!

I think CD can work wonderfully if you have mastered the art of cutting down big problems into small experiments. You also need a good software engineering culture and a culture of trust in general. Then, you truly focus on getting one thing done at a time and done really should mean: well tested, using good coding practices, deployed in production and you have verified whether the feature has added value. Until a new feature is in production it is nothing more than a hypothesis (or experiment), only real usage can prove whether it was worth making. If the experiments are small enough, then you can speed up this verification process. This is something most big companies have not yet understood: they are busy with large features that they assume will add value. Then they want to push those big features to production as quickly as possible, and skip a few steps along the way (testing, more often than not) to save time. CD will enable those companies to do that.

So, instead of looking at Continuous Delivery as a tool to go faster, look at it as a paradigm shift for a new way of thinking. You can go faster, but you’ll have to make things small. Really small.

Thanks for reading. I found this blog post very hard to write so some parts might not be clear enough. If you disagree with me or don’t get some parts of what I am saying, please leave a comment so we can have a respectful exchange of views, thanks!