Confident uncertainty

Philosophy

Let’s get a bit philosophical in the beginning, because the topic pretty much calls for it. From our early ages we are taught to rationalize and explain world around us. Everything we see can be explained. If you see a rainbow, it is not just “beautiful” or “magical”. It is a dispersion of light and there is probably no pot of gold at the end of it. There are clouds in the skies, that are not just giant soft pillows, but they are a visible mass of liquid droplets or frozen crystals. We believe in Santa Claus, but sooner or later we understand that the resemblance between him and someone from the relatives is not occasional. By the way, if you still believed in all that... sorry for ruining your childhood! :) In any case, eventually we stop believing in magic. And I think, that is a huge tragedy. We cannot look at the world through the child eyes anymore, naively, without any pre-judgements. Watching and accepting the events just as they are.

Of course, that is what we call learning and education. In any case I am not advocating against that. Actually, vice versa, we should become more professional and guarantee great result. It is very sad when so-called programmers do not understand the basics and rely on “magic”. However, the habit to rationalize the world around us becomes so ingrained into us so we apply it to everything. If a human can explain truly amazing phenomenons, fly to space, surely we can explain the dependencies between two events. That gives us a lot of overconfidence that we are so powerful and almighty.

How good are we in assuming and predicting? Honestly - not so good. People are not computers or machines. We get tired, our decisions depend on many surrounding factors. People tend to be affected by anchoring. Same people behave differently in a different groups of people. We tend to groupthink and under no circumstances we are rational. We even cannot predict how our life will look like a year later, we cannot predict what the customer will want in a few weeks and I am not even speaking about predicting the weather for tomorrow. If you will study that area of psychology, the only question you will eventually have - how the hell did we survive all that long? Seriously!

You might argue, that there are people who have predicted the future, that the man will fly, the financial crisis and basically everything else. In many cases I believe we confuse wisdom with pure luck. Under no circumstances I do not want to question their genius, we do not know how those people made those predictions, maybe they truly are indeed prophets. However, we should not exclude that it is statistically possible those people just got lucky. From the amount of all the people who make prophecies it is quite possible that one or two will be precisely right. If I will predict today that tomorrow it will be snowing or a completely new trend in technology that will change our lives, does it mean I am genius, visionary or plain lucky?

At the same time, the indication that someone was right with predictions before, together with ingrained habit of rationalizing events around us is a dangerous mixture. That gives us the “proof” that this can be done. That we indeed are almighty. How much “proof” do we need?

Overconfidence and mistreated empirical evidence that someone accomplished that before leads us building the wrong thing. Because we do not see the need to validate our ideas. They are perfect they way they are already.

Stories

The habit to rationalize the surroundings is an interesting game for your mind. As long as it stays a game. But then at some point we start believing that we actually can be right and sometimes we are. It plays dirty tricks with us. Here are a few stories, that suffered from that.

Let’s do it the old way

We noticed that we are spending a lot of time estimating, specifying and documenting before the project start. However, when in a project things were starting to change, we had real difficulties adjusting. That affected not only project schedule, quite often is was going up to the legal issues. Things like scope and deadlines were clearly specified in the contract. That in turn involved negotiations, lawyers and jeopardized normal working process. People in a team did not know if they should continue working or stop right now. Spice it up with shortcoming deadlines, that, surprisingly, did not care about anything above.

Obviously, the problem did not want to go away itself and we could not ignore it anymore either. Each time we were going through the same story all over again as a bad dream. Besides, it was painful not only to us. Finally, we realised we need to work differently with our customer. That was our incentive to change our cooperation to make it more bearable and productive.

We tried introducing shorter lifecycles, less specifying upfront and making our cooperation more flexible from contract perspective. Of course, it did not go that smooth either. Every once in awhile it was tempting to work the old and familiar way. We knew it was problematic, but this road was very well known to us. On the other hand, the path we were going was full of discoveries and unknown. No one could say what exactly we are stepping into.

Finally, one of the customer’s representatives suggested we will do the next short project the old way. They have been thinking about it for a while, domain is also very familiar, nothing will change there. They actually promised, that nothing will change there. It did sound suspicious at a time, but we thought it would be a good way to check if what we are doing is right. That was the way to compare two different approaches. So, we gave it a try, but still kept an eye on it. And everything went surprisingly fine... for the first week or so. And then, suddenly we get a call. Meanwhile, the requirements have changed and they realized that the way we are doing it right now won’t work for them. But they were so confident it won’t happen!

Of course, it is all known truth:

“Anything that can possibly go wrong, does”

The moral of this story: even if you are expecting things to change, you still can be very much surprised. However, there is a huge difference if you are expecting the things go wrong or it catches you as a surprise. And if it catches you just like that, it means everything you planned for, everything you have prepared, that everything is completely wrong and wasted.

Illusion of control

We have been working on a project that had really high discipline. Existing, so called, "agile" companies can be seriously jealous of what we had back then. Regular releases on production every two weeks, organized quality control, attention to the codebase and great people working together.

Of course, there was a lot of work ahead and that should be managed somehow. There was a common repository for all the improvements that needs to be done in the system. It was strictly organized and prioritized. After each release some items were disappearing from there and some new items, based on the received feedback, were taking its place according to the prioritization. 

However, there was one problem with that picture. Do you want to guess which one? Each release contained around fifty items and the whole list was more than one thousand items. If the company would close its doors and did not accept any new items, there was already enough items to work more than a year. Of course the list wasn’t decreasing either.

Every time it took significant time to go through the list and set the priorities. Also, the turnaround for the items was very slow. In many cases by the time the item reached the development it was either somehow implemented, not needed or the person responsible for adding it left the company already. So there went a second round of finding what needs to be done.

“When you think you control something, you’re wrong”

It feels really busy and highly productive, organized. People gather for meetings. Make decisions. Discuss business priorities. Set goals. However, the reality is something different. No one in that project would tell that something is not productive. In the contrary, most of them would vote for more control and organisation, because it feels natural. That in turn would probably make the situation even worse. Because the problem is not in the process, but in doing too much pointless work and being confident you are on it.

Good enough

We had a request to develop the whole system from the scratch. That is always fun and quite challenging exercise. In addition to that, customer already knew what the system they would like to get. They have been doing the same thing with other means and they knew exactly who and how will use the system.

A lot of preparations were done. Analysis documents describing the functionality, possible user groups of the system appeared. Before the beginning of a project it really seemed as we know everything we need to know to make it happen. We all felt confident that this is exactly how the system will look like. Of course, that kind of confidence is contagious and gives you an impression that you can complete everything in very short timeframe.

As we started moving along with implementation we started noticing holes and ambiguities in the analysis. It was still possible to follow it, however, as we were ready to accept changes, we gave that option to the customer. Moreover, if you add something into the scope of the project that is bigger than the original idea, something has to leave the scope of the project.

“A diamond is a chunk of coal that is made good under pressure”

If you start with a great idea. You think it will definitely work for you. Why would you stop there? Why do you think it is the best you can achieve? Why you think that idea actually is a good idea? As long as you try something new and verify your assumptions, you cannot go worse. You can discover even better ideas, that are maybe even easier. Choosing the easy path and sticking to the thing you know is plain lazy.

What can we do?

Confident uncertainty? In my opinion, the only way to be confident is to be confidently uncertain. Always verify what are you doing right now if it is right or not. Thankfully, there are several tricks that can help you with that.

PDCA

First one is quite straightforward. It is called PDCA that stands for plan–do–check–act or plan–do–check–adjust. It is a management method used for the control and continuous improvement of processes and products. It has been around  for a long time already and it was made popular by Deming, therefore, sometimes this method is called the Deming circle.

The fundamental principle of PDCA is iterating towards an improved system. With each iteration the hypothesis is either confirmed or negated and the further executions of the method will extend the knowledge. That is in turn - continuous improvement.

The key factor here is to iterate fast. If you make a full cycle in six months, that means, that probably you are working for six months on a totally wrong thing. The hypotheses should be broken down to the size when spending time investigating it and realizing that idea is wrong does not cost you too much.

Put your goals into business context

Second trick is to put your requirements into business context. One way to do it is to use the Effect/Impact Mapping technique.

http://angner.se/services/effect-mapping/

Let’s take for an example a simple and a widespread requirement - a user should be able to login into the system. It is almost perfect requirement - simple, straightforward, clearly understood who is affected party and quite easy to see which part of the system it affects. However, it lacks completely the business context - why would your user like to have that?

From the user perspective, this step - login into the system - is completely pointless and cumberstone. The user does not feel any great joy by pressing the “Login” button and does not feel happy about yet another user account in yet another system with yet another password. Seriously, I cannot imagine a person who would actually love that process.

However, if the reasoning is to keep user information secure, provide history of user’s actions or being able to digitally identify and sign documents, then that is a totally different story. And there are several reasons why.

This requirement jumps immediately to the solution or implementation details. User login is pointless standalone and it supports other functionalities. Consequently, if you have a huge backlog of items, the original requirements why you needed that may disappear as you move along, leaving you with pointless login functionality at the end. Knowing what you want to achieve with this feature may help you to discover alternative and better ways to achieve it. Maybe there is something that is less complicated and more pleasant for the users and you can still reach your business goal. And of course, you will always know whom to ask about any details of that requirement if it identifies business context and stakeholder.

Placing your requirements into the business context will help you prioritize items. Your discussions about the items will not focus only on how complicated it is to implement it, but also on the business value a requirement will bring.

Finally

To sum up, we must embrace the human factor in such exact disciplines as software development. We tend to over rationalize the surroundings and have too much faith that we are always right. This behaviour helps us discover new ideas, it moves us forward, but it also plays dirty tricks on us when we rely a lot on our assumptions.

The natural thing seems to be adding more control over what we do. Finally we end up with perfectly working pieces of the process, but the whole process makes the situation only worse. Sometimes we try to leave everything as it is and struggle day after day with the same problems. Eventually the size of complexity and organization will make it impossible to comprehend. Even when it seems we have a perfect solution, that just needs implementing, we are not sure if it really is a great solution and maybe we can achieve even more.

Whatever we do, what we need is an immediate feedback for our actions. And the smaller our action is, the easier it is to validate it.