Tuesday, February 14, 2012

Anonymous Analyst

Let’s start with the confession. It comes a bit harsh and straightforward, will try to elaborate it later on. Do not get offended (at least yet). Okay, I warned you, so here it comes - I do hate software analysts. More specifically, I have serious difficulties understanding what they exactly do and how-come the result is valuable to anyone. What I do like is when people understand the domain they are working on, they are clear on what they are going to build and who needs it. So the problem is not in a position itself, but in what people understand they should do and the value they produce.

In a way too-many organizations analysts serve just an additional link in a long chain between developers and the customer. The idea that someone will analyze the problem, chew it up and then developers would have to “just” code it, looks great on paper but simply does not work. It is a bit chocking and too obvious, but the thing is, developers do the actual implementation. They materialize thoughts into the program logic. So they have to understand the problem and represent it in the form of source code.

What does it mean in practise? Developers should do the same job that analysts just did all over again. But it gets even worse. Program logic is very formal. If it is Boolean - it is either true or false, but never both. We use that very formal language to describe problems and domains we do not fully understand or just not meant to be described. We try to imagine ourselves in a future using this new system, operating with all the stakeholders and all being perfect.

Let’s consider this small mental exercise. Try to put into simple if-then-else logic how do you select what you will eat for lunch. It should work for any given day and person. Probably you won’t even start doing that. Even if you will narrow the problem to selecting either you will have a meat or fish today, it does not really help much. You can draw some abstract analysis, that will look great, but you will fail implementing the actual logic.

Another problem with software analysts - they never fail. Their part of the job is done before the development start, all the blame is laid on the developers who have to rediscover what analysis did not foresee. Just think about it! Whatever you draw is genuinely correct. If you separate the fantasy drawing from the actual doing and you have no understanding what programming is - you can never fail. Developers, on the other hand, are perceived as dream-crashers, incompetent and lazy.

structure of organizations

The root of the problem lies deeper. Every organization is structured to better suit its processes. The structure is secondary, it represents a model how things work right now. The paradox is that later on structure prevents organizations from bringing in new and radical ideas. And the more complex structure is, much harder it is to make a change.

Organizations that have analysts, developers, testers, project managers separated from each other mirror classical Waterfall Model. Yes, currently it is very trendy to be agile, so everybody does Scrum. Others who does not want to be related with agile for one reason or another do tell that they use RUP (firstly, usually it is an excuse not to mention Waterfall; secondly, they demonstrate rude misunderstanding what RUP is).  And such a model has its own downsides.

I keep on saying, that there is nothing wrong with one development methodology or another. As long as companies understand the consequences and are ready to pay the price. And none of the methodologies come free. If you are doing the waterfall - you are slow to react and have a lot of paperwork to impress your customer with, instead of software. If you are agile and/or value oriented - then you live a fast life of constant changes and having a trouble growing your organization as a culture.

However, it is much worse to believe that you do well something what is very trendy right now, but actually you are doing something else. And this is the moment I personally fail to understand. The problems that organizations face applying the waterfall process happen all the time - low quality of a product, no testing, late delivery, non-existing or wrong analysis prior development. Those problems do not happen once or twice. They reoccur constantly for many-many years in a row. Why? Why don’t they learn something from that?

The best part is, that in a sequentially structured organization it never feels like you are not producing anything. You are always delivering something, there is a lot of firefighting and a lot of movement. There is always something happens and someone need answers right this moment, or better to have them yesterday. And if you compare their process with something different, for instance - you actually do not need a test-and-fix churn at the end of each project, you might be better off without detailed upfront design by learning by doing. It is perceived as slacking off and non-professionalism. Their process is perfect. There is nothing to improve. There is no another way.

There is another way! No process is perfect, you always improve and always learn more about your people, your business and the environment you are in. If you believe Refactoring is good for the code, why don’t you believe the same refactoring will be good for organization? Chances are that you do not believe in refactoring, but then it is a topic for a totally separate blog post!

promise and failure of a bright future

Currently you might wonder, why do I express all my anger on analysts, while it is not only them to blame? Well, because they are in the sweetest spot - they cannot fail. Project managers get pushed for being over the budget, developers for missing the deadlines, but analysts are in the middle. Therefore, they do not feel responsible for what happens before and after. Their part of the job is done. They never fail.
Besides writing overly fuzzy and too long blog-posts I do teaching at the Tallinn University of Technology. In the autumn I participate in a course “Software quality and standards”. It is a obligatory subject if you are doing MSc in Informatics. From the whole course less than 10% have ever written a line of code, vast majority wants to become [drum droll] an analyst.

Same goes with the people who are coming for the job interview. Okay, many of them, who are actually invited to an interview, have written some code. However, quite many believe that when you start your career, you start in a Quality Assurance field. Then you study and grow and become a programmer. Then you have a chance to become a project manager. And here is the shortcut, if you start as an analyst, you can skip the boring stuff of actually doing the job and become a PM right away.

The result is, that projects are lead by the people who were avoiding software development per se from the start. And in such organizations, those people are responsible for negotiations with the stakeholders about the scope and deadlines, in some cases even for estimations. Then you wonder, why so many projects fail to be delivered in time and budget. (Not mentioning that the specializations above are not in one and same career tree)

When we talk about great project managers or analysts with great track record, we should not forget that people understand success differently. If there is a thousand of project managers in the country, the chances are quite high, that ten of them got lucky on every project and delivered in schedule. Or, maybe, organization just considers that situation when projects go over the line or completely redone as part of the business and able to pay that price.

And once again, this is what it always comes to. Is an organization able to pay the price that always comes with the methodology it is using? Probably, the better question is - what happens if suddenly it cannot? In any case, it is very depressing to work in a company where someone is constantly glorified and cannot ever fail and someone just has to make the work done, usually at nights.

flip of a coin

At this point you are probably convinced that analysts in organizations - might be a problem. It seems to be the perfect spot to make the twist in a script. But, let’s have another cup of coffee first.

The thing is, as surprising as it sounds - we do need software analysts. It’s just, the ones we have are doing the wrong thing.

Imagine we will let only developers to design and create a system for the customer without his involvement. Most probably, it will look like the visual representation of a database structure. All fields grouped together in a strict logical order. Massive search options to allow searching on every possible combination. No mixed views or additional functionality. Everything is on its place and nothing else is there. You can see People, Addresses, Phones separately, but you cannot see them all together.

But this is not what customer wants. He might say he wants it like that (because it makes logical sense), but then he will wonder if everything could be redone. The real use of the systems has nothing in common how the domain model or the database table structure looks like. People think about the usability, where we can do an action with a single click without opening ten-thousand other views before that.

That knowledge lies in practice and hands-on experience that developers do not poses and no one is able to specify it and write it down. That knowledge constantly changes as people try it out on practice.

This is where we need someone technical to think like a customer. Analysts should study the domain and understand it thoroughly. It should be crystal clear to them how the business works. They should help business to discover new ways how to optimize the existing process. They are technical enough to say that this or that solution is not possible with current technology.

Of course, ideally every member of the whole cross-functional team should be able to do it. In one way or another they should finally know as much as analysts do before writing any line of code. However, having a dedicated analyst in a team, who simply has more dedicated time (and probably skills) to do exactly that is always better.

It can be compared to the researcher’s or scientist’s job. Where people work the magic, create theories, study and then try the idea out on practice together with the development team.

Another case is when a company does not have an IT department and orders the development from the outside. Having an analyst there, who understands the nature and complexity of the domain and how existing systems work - is just a blessing. He can keep an integrity among different systems and able to talk technical language with the developers. In many cases that person can represent the departments of an organization and become a Product Owner for the developers. Who can make the decisions and justify them among others.

I cannot stress it strong enough here, but in a nutshell. Analysis is ongoing as long as development is ongoing. And during that, you do not design software, you design knowledge and make it available to all members of a team.

what if you are an anonymous analyst
The constant firefighting and crises in projects happen because people do not try their ideas on practice. If you add there over-confidence that somehow things will be different just exactly this time during this project - and we have a recipe for disaster. The best part - it is an endless loop and no one dares to break it. But this is exactly what needs to be done.

Here are a few points that analysts should do. And if you are not doing them, it is actually quite easy to change.
  • First things first, stop thinking that your job is only to analyze things. No one is interested in drawings or crunched knowledge if you keep it only to yourself. That does not make you smart or more valuable if you cannot send the message across to your colleagues. Your primary objective is to share knowledge and make it easy and accessible by all members. Writing a document - is not a knowledge sharing. It creates a document that probably no one will have a look. It gives an impression of shared knowledge, but only makes things worse. The right question is - what do you want to achieve with that document?
  • Leave software and database design to developers or create the common view together. Software development is an evolution. We begin with an assumption that we know how to start. Then we develop it as simple as possible and grow it bigger. You cannot consider what you do not know yet, before you actually tried it. Normalized table structures might look great on paper, but sooner or later have nothing in common with the reality, or performance requirements.
  • Forget UML diagrams. Firstly, just a few analysts know how to draw them. Secondly, if you have something to draw - engage whole team into discussion. Come to the common ground and, if needed, document it then. Use whiteboard, markers, pens, it is a way faster than drawing boxes in some weird program. Do not be a hero, it does not mean you cannot design it alone by yourself, it means you will do it better together.
  • Become part of the team. You cannot finish your job before development start. You should sit together, share the knowledge, communicate. If you do not know what developers are doing, you know way less than you should. Help the team to get information, help Scrum Master to remove impediments, or even act as a Scrum Master.
  • Generate and design test scenarios. Learn how to make your scenarios executable. The best way to explain how the system should work is by a good test case. Tests are there not to find bugs, but to ensure that our stories keep on working over time.
  • Forget emails. If you have something to share, just say it out loud. Communicate face-to-face with the customer. The human language is full of ambiguity, why shall we make it even more complex and add another level of misunderstanding.
  • Share the success and sorrow together with a team. If you screw-up - you did it all together. Everyone let it to happen. If you succeed - celebrate it also together. Everyone has participated equally to get the result.

No organization will prevent you from doing that if you just start. Those things seem trivial, but really... should it be complicated?

Do not forget to share your experiences!


  1. Nice post and great tips in the end!

  2. Eventually the proposed 'Should do' list is coming close towards the good Product Owner' 'should do list'. Sad to mention but I'm seeing bad examples of POs which suffer with basically same child's diseases as mentioned here above.

  3. Nice post.

    One thing I have noticed is that sometimes developers get used to such anonymous analysts and they forget that software development is not about coding algorithms but it is about understanding what customer needs (not always same what customer wants).

    I think that much better software can be built if we stop expecting someone else (be it customer or analyst, PO or whoever) to tell us what is needed and instead collaboratively immerse ourselves in the problem domain.

  4. Thanks for the great insight, but is an anonymous analyst a reality in a business setting? Let's just have a quick overview of what an analyst could do and the specifics associated, shall we?

    managing stakeholders: the reality is that there's usually just too much information, and what's very important, unstructured information that has to be processed, cleaned etc. all this is a changing input from people who all have their agendas and thus the situation gets a bit or a lot political. The skills and time needed to navigate this unstructured and political environment building consensus and aligning everyone to a common goal is - a Lot.

    writing requirements: well someone has to do it, and it better be someone who can write text that is readable and can be at least partially understood by both parties. It could be a developer, but interpreting business requirements and translating them into other type of requirements and making sure they are approved by the stakeholders, well, takes time, and probably won't be done during lunch time or a brown back session

    analyzing: kind of obvious, BUT the domain and specifics ARE important and the amount is usually just, well, just to big.

    supporting business with their software development initiatives: excel kung-fu and modelling know-how is a must, but even more important is - can you build trust with the business owners and related stakeholders for it all to be a success? Possible, but, you've probably already guessed it - takes time.

    The work of an analyst doesn't end when development begins, this implies that all the requirements, business cases and all inputs are already known beforehand, which sometimes is the case, but this is almost unique and very rare for sure. There's less work after the analysis is finished, but it doesn't mean that there shouldn't be control as part of the feedback mechanism (in the case of SCRUM this role is played in part by the SCRUM master, in that she defends the team and the product backlog and works through the incoming changes with the business owner).

    Can a programmer that has the skills mentioned above be a good business analyst? Probably yes. Can a programmer doing all that is mentioned above be a good programmer? Probably not.
    A hybrid of a BA-programmer/engineer is maybe a reality in a small scale project, but doesn't make sense in a bigger setting. By becoming a BAgrammer, if you allow me to use that ugly word, the person in question will ultimately become a mermaid - where you want it to be a fish - it's a woman and where you want it to be a woman - you have a fish.

    With that said, any soft skills gained by an engineer/programmer will greatly benefit her in daily work and this is actually what separates great programmers from good programmers, the touchy-feely, ability to work with business, wear different hats (e.g. programmer, analyst, functional person, call center operator etc.), have the ability to analyse the bigger picture and ultimately becoming a leader in her field - adding value to clients, partners and co-workers not only through programming but through any activity.

  5. Thanks for interesting reading.
    I think you should first define what you mean when saying developer or programmer.
    All that article applies when readers understand developer same way as you do :-).
    The problem will pop up when you say developer but actually mean programmer.
    So I'll try to define those two words, as I understand them.

    Developer - the person who masters programming languages and sometimes even uses it for writing program code. Most of the time developer spends for thinking and for communication with client helping to build right solution for the client.
    Programmer - the person who knows some programming language. Main task for this kind of person is to produce program code based on the spec, produced by analysts.

    So, coming back to the question raised in the beginning of article - for who is the result of analysts work valuable?
    I'd answer that the target group is the programmers. They desperately need to have some paper in front of them to convert into program code.
    Programmers usually do not find problems in specs, they just convert it into program language. And their work is usually evaluated by the amount of code lines they have produced, not by the amount of client problems the have solved.

    There are quite many companies that actually need this additional link as analysts in the chain because they haven't found or met yet good enough developers and have only programmers available.

  6. Creating software is great! For all Product owners, BAs, Developers and Programmers. We all love to do it!
    I have been working under every single one of those "job descriptions".
    Still I see the things mainly from the perspective of a Product owner or BA as that's what I'm doing.
    I'm not doing it because I would know it better than dev or programming, but it has always been my reason to do software.
    Programming to me has always been a secondary objective and delivering software the primary objective (under my domain).
    I have never been a good programmer though I have been doing it for many years. My passion has always been somewhere else:
    it is on the side of doing and having some great stuff to utilize on a specific domain.

    What I have got to witness on my career are very different expirences from love to not caring with the people how have been involved in the project where I have been.
    That has made me to think of what are the results from different approaches towards the achievement. What is actually the achievement?!
    For me it is: we should aim to do the best software suitable for the user(s) of it.

    In order to achieve that many pieces need to fall in place.
    One of the key ingredients has been discussed here: the one who translates the business need language to technologically understood format.
    Usually Business Analysts do that.
    In my understanding it is meaningless if a person who doesn't have a real want to understand what a user wants tries to interpret something
    to a person who's only passion is to see how easy it is to create a platform using Ruby on Rails.
    I have needed a glue. People who are interested about actual software i.e. users and developers doing the software has have to got closer to
    each other. This has usually meant to introduce people to each other. This has always meant to introduce domain and software on the domain
    to be introduced and tried by developers.