Confident uncertainty

Philosophy

Let’s get a bit philosophical in the beginning, because the topic pretty much calls for it. From our early ages we are taught to rationalize and explain world around us. Everything we see can be explained. If you see a rainbow, it is not just “beautiful” or “magical”. It is a dispersion of light and there is probably no pot of gold at the end of it. There are clouds in the skies, that are not just giant soft pillows, but they are a visible mass of liquid droplets or frozen crystals. We believe in Santa Claus, but sooner or later we understand that the resemblance between him and someone from the relatives is not occasional. By the way, if you still believed in all that... sorry for ruining your childhood! :) In any case, eventually we stop believing in magic. And I think, that is a huge tragedy. We cannot look at the world through the child eyes anymore, naively, without any pre-judgements. Watching and accepting the events just as they are.

Of course, that is what we call learning and education. In any case I am not advocating against that. Actually, vice versa, we should become more professional and guarantee great result. It is very sad when so-called programmers do not understand the basics and rely on “magic”. However, the habit to rationalize the world around us becomes so ingrained into us so we apply it to everything. If a human can explain truly amazing phenomenons, fly to space, surely we can explain the dependencies between two events. That gives us a lot of overconfidence that we are so powerful and almighty.

How good are we in assuming and predicting? Honestly - not so good. People are not computers or machines. We get tired, our decisions depend on many surrounding factors. People tend to be affected by anchoring. Same people behave differently in a different groups of people. We tend to groupthink and under no circumstances we are rational. We even cannot predict how our life will look like a year later, we cannot predict what the customer will want in a few weeks and I am not even speaking about predicting the weather for tomorrow. If you will study that area of psychology, the only question you will eventually have - how the hell did we survive all that long? Seriously!

You might argue, that there are people who have predicted the future, that the man will fly, the financial crisis and basically everything else. In many cases I believe we confuse wisdom with pure luck. Under no circumstances I do not want to question their genius, we do not know how those people made those predictions, maybe they truly are indeed prophets. However, we should not exclude that it is statistically possible those people just got lucky. From the amount of all the people who make prophecies it is quite possible that one or two will be precisely right. If I will predict today that tomorrow it will be snowing or a completely new trend in technology that will change our lives, does it mean I am genius, visionary or plain lucky?

At the same time, the indication that someone was right with predictions before, together with ingrained habit of rationalizing events around us is a dangerous mixture. That gives us the “proof” that this can be done. That we indeed are almighty. How much “proof” do we need?

Overconfidence and mistreated empirical evidence that someone accomplished that before leads us building the wrong thing. Because we do not see the need to validate our ideas. They are perfect they way they are already.

Stories

The habit to rationalize the surroundings is an interesting game for your mind. As long as it stays a game. But then at some point we start believing that we actually can be right and sometimes we are. It plays dirty tricks with us. Here are a few stories, that suffered from that.

Let’s do it the old way

We noticed that we are spending a lot of time estimating, specifying and documenting before the project start. However, when in a project things were starting to change, we had real difficulties adjusting. That affected not only project schedule, quite often is was going up to the legal issues. Things like scope and deadlines were clearly specified in the contract. That in turn involved negotiations, lawyers and jeopardized normal working process. People in a team did not know if they should continue working or stop right now. Spice it up with shortcoming deadlines, that, surprisingly, did not care about anything above.

Obviously, the problem did not want to go away itself and we could not ignore it anymore either. Each time we were going through the same story all over again as a bad dream. Besides, it was painful not only to us. Finally, we realised we need to work differently with our customer. That was our incentive to change our cooperation to make it more bearable and productive.

We tried introducing shorter lifecycles, less specifying upfront and making our cooperation more flexible from contract perspective. Of course, it did not go that smooth either. Every once in awhile it was tempting to work the old and familiar way. We knew it was problematic, but this road was very well known to us. On the other hand, the path we were going was full of discoveries and unknown. No one could say what exactly we are stepping into.

Finally, one of the customer’s representatives suggested we will do the next short project the old way. They have been thinking about it for a while, domain is also very familiar, nothing will change there. They actually promised, that nothing will change there. It did sound suspicious at a time, but we thought it would be a good way to check if what we are doing is right. That was the way to compare two different approaches. So, we gave it a try, but still kept an eye on it. And everything went surprisingly fine... for the first week or so. And then, suddenly we get a call. Meanwhile, the requirements have changed and they realized that the way we are doing it right now won’t work for them. But they were so confident it won’t happen!

Of course, it is all known truth:

“Anything that can possibly go wrong, does”

The moral of this story: even if you are expecting things to change, you still can be very much surprised. However, there is a huge difference if you are expecting the things go wrong or it catches you as a surprise. And if it catches you just like that, it means everything you planned for, everything you have prepared, that everything is completely wrong and wasted.

Illusion of control

We have been working on a project that had really high discipline. Existing, so called, "agile" companies can be seriously jealous of what we had back then. Regular releases on production every two weeks, organized quality control, attention to the codebase and great people working together.

Of course, there was a lot of work ahead and that should be managed somehow. There was a common repository for all the improvements that needs to be done in the system. It was strictly organized and prioritized. After each release some items were disappearing from there and some new items, based on the received feedback, were taking its place according to the prioritization. 

However, there was one problem with that picture. Do you want to guess which one? Each release contained around fifty items and the whole list was more than one thousand items. If the company would close its doors and did not accept any new items, there was already enough items to work more than a year. Of course the list wasn’t decreasing either.

Every time it took significant time to go through the list and set the priorities. Also, the turnaround for the items was very slow. In many cases by the time the item reached the development it was either somehow implemented, not needed or the person responsible for adding it left the company already. So there went a second round of finding what needs to be done.

“When you think you control something, you’re wrong”

It feels really busy and highly productive, organized. People gather for meetings. Make decisions. Discuss business priorities. Set goals. However, the reality is something different. No one in that project would tell that something is not productive. In the contrary, most of them would vote for more control and organisation, because it feels natural. That in turn would probably make the situation even worse. Because the problem is not in the process, but in doing too much pointless work and being confident you are on it.

Good enough

We had a request to develop the whole system from the scratch. That is always fun and quite challenging exercise. In addition to that, customer already knew what the system they would like to get. They have been doing the same thing with other means and they knew exactly who and how will use the system.

A lot of preparations were done. Analysis documents describing the functionality, possible user groups of the system appeared. Before the beginning of a project it really seemed as we know everything we need to know to make it happen. We all felt confident that this is exactly how the system will look like. Of course, that kind of confidence is contagious and gives you an impression that you can complete everything in very short timeframe.

As we started moving along with implementation we started noticing holes and ambiguities in the analysis. It was still possible to follow it, however, as we were ready to accept changes, we gave that option to the customer. Moreover, if you add something into the scope of the project that is bigger than the original idea, something has to leave the scope of the project.

“A diamond is a chunk of coal that is made good under pressure”

If you start with a great idea. You think it will definitely work for you. Why would you stop there? Why do you think it is the best you can achieve? Why you think that idea actually is a good idea? As long as you try something new and verify your assumptions, you cannot go worse. You can discover even better ideas, that are maybe even easier. Choosing the easy path and sticking to the thing you know is plain lazy.

What can we do?

Confident uncertainty? In my opinion, the only way to be confident is to be confidently uncertain. Always verify what are you doing right now if it is right or not. Thankfully, there are several tricks that can help you with that.

PDCA

First one is quite straightforward. It is called PDCA that stands for plan–do–check–act or plan–do–check–adjust. It is a management method used for the control and continuous improvement of processes and products. It has been around  for a long time already and it was made popular by Deming, therefore, sometimes this method is called the Deming circle.

The fundamental principle of PDCA is iterating towards an improved system. With each iteration the hypothesis is either confirmed or negated and the further executions of the method will extend the knowledge. That is in turn - continuous improvement.

The key factor here is to iterate fast. If you make a full cycle in six months, that means, that probably you are working for six months on a totally wrong thing. The hypotheses should be broken down to the size when spending time investigating it and realizing that idea is wrong does not cost you too much.

Put your goals into business context

Second trick is to put your requirements into business context. One way to do it is to use the Effect/Impact Mapping technique.

http://angner.se/services/effect-mapping/

Let’s take for an example a simple and a widespread requirement - a user should be able to login into the system. It is almost perfect requirement - simple, straightforward, clearly understood who is affected party and quite easy to see which part of the system it affects. However, it lacks completely the business context - why would your user like to have that?

From the user perspective, this step - login into the system - is completely pointless and cumberstone. The user does not feel any great joy by pressing the “Login” button and does not feel happy about yet another user account in yet another system with yet another password. Seriously, I cannot imagine a person who would actually love that process.

However, if the reasoning is to keep user information secure, provide history of user’s actions or being able to digitally identify and sign documents, then that is a totally different story. And there are several reasons why.

This requirement jumps immediately to the solution or implementation details. User login is pointless standalone and it supports other functionalities. Consequently, if you have a huge backlog of items, the original requirements why you needed that may disappear as you move along, leaving you with pointless login functionality at the end. Knowing what you want to achieve with this feature may help you to discover alternative and better ways to achieve it. Maybe there is something that is less complicated and more pleasant for the users and you can still reach your business goal. And of course, you will always know whom to ask about any details of that requirement if it identifies business context and stakeholder.

Placing your requirements into the business context will help you prioritize items. Your discussions about the items will not focus only on how complicated it is to implement it, but also on the business value a requirement will bring.

Finally

To sum up, we must embrace the human factor in such exact disciplines as software development. We tend to over rationalize the surroundings and have too much faith that we are always right. This behaviour helps us discover new ideas, it moves us forward, but it also plays dirty tricks on us when we rely a lot on our assumptions.

The natural thing seems to be adding more control over what we do. Finally we end up with perfectly working pieces of the process, but the whole process makes the situation only worse. Sometimes we try to leave everything as it is and struggle day after day with the same problems. Eventually the size of complexity and organization will make it impossible to comprehend. Even when it seems we have a perfect solution, that just needs implementing, we are not sure if it really is a great solution and maybe we can achieve even more.

Whatever we do, what we need is an immediate feedback for our actions. And the smaller our action is, the easier it is to validate it.

Anonymous Analyst

Let’s start with the confession. It comes a bit harsh and straightforward, will try to elaborate it later on. Do not get offended (at least yet). Okay, I warned you, so here it comes - I do hate software analysts. More specifically, I have serious difficulties understanding what they exactly do and how-come the result is valuable to anyone. What I do like is when people understand the domain they are working on, they are clear on what they are going to build and who needs it. So the problem is not in a position itself, but in what people understand they should do and the value they produce.

In a way too-many organizations analysts serve just an additional link in a long chain between developers and the customer. The idea that someone will analyze the problem, chew it up and then developers would have to “just” code it, looks great on paper but simply does not work. It is a bit chocking and too obvious, but the thing is, developers do the actual implementation. They materialize thoughts into the program logic. So they have to understand the problem and represent it in the form of source code.

What does it mean in practise? Developers should do the same job that analysts just did all over again. But it gets even worse. Program logic is very formal. If it is Boolean - it is either true or false, but never both. We use that very formal language to describe problems and domains we do not fully understand or just not meant to be described. We try to imagine ourselves in a future using this new system, operating with all the stakeholders and all being perfect.

Let’s consider this small mental exercise. Try to put into simple if-then-else logic how do you select what you will eat for lunch. It should work for any given day and person. Probably you won’t even start doing that. Even if you will narrow the problem to selecting either you will have a meat or fish today, it does not really help much. You can draw some abstract analysis, that will look great, but you will fail implementing the actual logic.

Another problem with software analysts - they never fail. Their part of the job is done before the development start, all the blame is laid on the developers who have to rediscover what analysis did not foresee. Just think about it! Whatever you draw is genuinely correct. If you separate the fantasy drawing from the actual doing and you have no understanding what programming is - you can never fail. Developers, on the other hand, are perceived as dream-crashers, incompetent and lazy.

structure of organizations

The root of the problem lies deeper. Every organization is structured to better suit its processes. The structure is secondary, it represents a model how things work right now. The paradox is that later on structure prevents organizations from bringing in new and radical ideas. And the more complex structure is, much harder it is to make a change.

Organizations that have analysts, developers, testers, project managers separated from each other mirror classical Waterfall Model. Yes, currently it is very trendy to be agile, so everybody does Scrum. Others who does not want to be related with agile for one reason or another do tell that they use RUP (firstly, usually it is an excuse not to mention Waterfall; secondly, they demonstrate rude misunderstanding what RUP is).  And such a model has its own downsides.

I keep on saying, that there is nothing wrong with one development methodology or another. As long as companies understand the consequences and are ready to pay the price. And none of the methodologies come free. If you are doing the waterfall - you are slow to react and have a lot of paperwork to impress your customer with, instead of software. If you are agile and/or value oriented - then you live a fast life of constant changes and having a trouble growing your organization as a culture.

However, it is much worse to believe that you do well something what is very trendy right now, but actually you are doing something else. And this is the moment I personally fail to understand. The problems that organizations face applying the waterfall process happen all the time - low quality of a product, no testing, late delivery, non-existing or wrong analysis prior development. Those problems do not happen once or twice. They reoccur constantly for many-many years in a row. Why? Why don’t they learn something from that?

The best part is, that in a sequentially structured organization it never feels like you are not producing anything. You are always delivering something, there is a lot of firefighting and a lot of movement. There is always something happens and someone need answers right this moment, or better to have them yesterday. And if you compare their process with something different, for instance - you actually do not need a test-and-fix churn at the end of each project, you might be better off without detailed upfront design by learning by doing. It is perceived as slacking off and non-professionalism. Their process is perfect. There is nothing to improve. There is no another way.

There is another way! No process is perfect, you always improve and always learn more about your people, your business and the environment you are in. If you believe Refactoring is good for the code, why don’t you believe the same refactoring will be good for organization? Chances are that you do not believe in refactoring, but then it is a topic for a totally separate blog post!

promise and failure of a bright future

Currently you might wonder, why do I express all my anger on analysts, while it is not only them to blame? Well, because they are in the sweetest spot - they cannot fail. Project managers get pushed for being over the budget, developers for missing the deadlines, but analysts are in the middle. Therefore, they do not feel responsible for what happens before and after. Their part of the job is done. They never fail.

Besides writing overly fuzzy and too long blog-posts I do teaching at the Tallinn University of Technology. In the autumn I participate in a course “Software quality and standards”. It is a obligatory subject if you are doing MSc in Informatics. From the whole course less than 10% have ever written a line of code, vast majority wants to become [drum droll] an analyst.

Same goes with the people who are coming for the job interview. Okay, many of them, who are actually invited to an interview, have written some code. However, quite many believe that when you start your career, you start in a Quality Assurance field. Then you study and grow and become a programmer. Then you have a chance to become a project manager. And here is the shortcut, if you start as an analyst, you can skip the boring stuff of actually doing the job and become a PM right away.

The result is, that projects are lead by the people who were avoiding software development per se from the start. And in such organizations, those people are responsible for negotiations with the stakeholders about the scope and deadlines, in some cases even for estimations. Then you wonder, why so many projects fail to be delivered in time and budget. (Not mentioning that the specializations above are not in one and same career tree)

When we talk about great project managers or analysts with great track record, we should not forget that people understand success differently. If there is a thousand of project managers in the country, the chances are quite high, that ten of them got lucky on every project and delivered in schedule. Or, maybe, organization just considers that situation when projects go over the line or completely redone as part of the business and able to pay that price.

And once again, this is what it always comes to. Is an organization able to pay the price that always comes with the methodology it is using? Probably, the better question is - what happens if suddenly it cannot? In any case, it is very depressing to work in a company where someone is constantly glorified and cannot ever fail and someone just has to make the work done, usually at nights.

flip of a coin

At this point you are probably convinced that analysts in organizations - might be a problem. It seems to be the perfect spot to make the twist in a script. But, let’s have another cup of coffee first.

The thing is, as surprising as it sounds - we do need software analysts. It’s just, the ones we have are doing the wrong thing.

Imagine we will let only developers to design and create a system for the customer without his involvement. Most probably, it will look like the visual representation of a database structure. All fields grouped together in a strict logical order. Massive search options to allow searching on every possible combination. No mixed views or additional functionality. Everything is on its place and nothing else is there. You can see People, Addresses, Phones separately, but you cannot see them all together.

But this is not what customer wants. He might say he wants it like that (because it makes logical sense), but then he will wonder if everything could be redone. The real use of the systems has nothing in common how the domain model or the database table structure looks like. People think about the usability, where we can do an action with a single click without opening ten-thousand other views before that.

That knowledge lies in practice and hands-on experience that developers do not poses and no one is able to specify it and write it down. That knowledge constantly changes as people try it out on practice.

This is where we need someone technical to think like a customer. Analysts should study the domain and understand it thoroughly. It should be crystal clear to them how the business works. They should help business to discover new ways how to optimize the existing process. They are technical enough to say that this or that solution is not possible with current technology.

Of course, ideally every member of the whole cross-functional team should be able to do it. In one way or another they should finally know as much as analysts do before writing any line of code. However, having a dedicated analyst in a team, who simply has more dedicated time (and probably skills) to do exactly that is always better.

It can be compared to the researcher’s or scientist’s job. Where people work the magic, create theories, study and then try the idea out on practice together with the development team.

Another case is when a company does not have an IT department and orders the development from the outside. Having an analyst there, who understands the nature and complexity of the domain and how existing systems work - is just a blessing. He can keep an integrity among different systems and able to talk technical language with the developers. In many cases that person can represent the departments of an organization and become a Product Owner for the developers. Who can make the decisions and justify them among others.

I cannot stress it strong enough here, but in a nutshell. Analysis is ongoing as long as development is ongoing. And during that, you do not design software, you design knowledge and make it available to all members of a team.

what if you are an anonymous analyst
The constant firefighting and crises in projects happen because people do not try their ideas on practice. If you add there over-confidence that somehow things will be different just exactly this time during this project - and we have a recipe for disaster. The best part - it is an endless loop and no one dares to break it. But this is exactly what needs to be done.

Here are a few points that analysts should do. And if you are not doing them, it is actually quite easy to change.

• First things first, stop thinking that your job is only to analyze things. No one is interested in drawings or crunched knowledge if you keep it only to yourself. That does not make you smart or more valuable if you cannot send the message across to your colleagues. Your primary objective is to share knowledge and make it easy and accessible by all members. Writing a document - is not a knowledge sharing. It creates a document that probably no one will have a look. It gives an impression of shared knowledge, but only makes things worse. The right question is - what do you want to achieve with that document?

• Leave software and database design to developers or create the common view together. Software development is an evolution. We begin with an assumption that we know how to start. Then we develop it as simple as possible and grow it bigger. You cannot consider what you do not know yet, before you actually tried it. Normalized table structures might look great on paper, but sooner or later have nothing in common with the reality, or performance requirements.

• Forget UML diagrams. Firstly, just a few analysts know how to draw them. Secondly, if you have something to draw - engage whole team into discussion. Come to the common ground and, if needed, document it then. Use whiteboard, markers, pens, it is a way faster than drawing boxes in some weird program. Do not be a hero, it does not mean you cannot design it alone by yourself, it means you will do it better together.

• Become part of the team. You cannot finish your job before development start. You should sit together, share the knowledge, communicate. If you do not know what developers are doing, you know way less than you should. Help the team to get information, help Scrum Master to remove impediments, or even act as a Scrum Master.

• Generate and design test scenarios. Learn how to make your scenarios executable. The best way to explain how the system should work is by a good test case. Tests are there not to find bugs, but to ensure that our stories keep on working over time.

• Forget emails. If you have something to share, just say it out loud. Communicate face-to-face with the customer. The human language is full of ambiguity, why shall we make it even more complex and add another level of misunderstanding.

• Share the success and sorrow together with a team. If you screw-up - you did it all together. Everyone let it to happen. If you succeed - celebrate it also together. Everyone has participated equally to get the result.

No organization will prevent you from doing that if you just start. Those things seem trivial, but really... should it be complicated?

Do not forget to share your experiences!