Notes from Agile 2011 - first two days »
Tuesday
Sep062011

We just can't afford Taylorism anymore. Now what?

Let’s go straight to the point: in the IT industry we made a major mistake about 60 years ago.

An understandable mistake, but nevertheless something that costed us a great amount of money and energy, something that we are still paying dearly for.

That mistake was to think that we could build software in the same way as we had built anything else since, especially mass-produced tangible goods.

That was a bad idea, simply because it doesn’t work; or, at the very least, it’s a highly inefficient way of building software.

The good news is, of course, that we are now aware of that – and the Agile approach is one clear and solid proof of that awareness, along with other new developments in the software industry.

However, let’s start from the beginning.

An Easy Mistake To Make

As a species, we learned to build stuff.

And what we learned about building stuff we pass on from generation to generation, in a process of continuous education.

Let's play a little game for a second: let's pretend that all that mankind has learned about building stuff you could learn in a one-day course, from 9 am to 5 pm.

Next, let's define a couple of reference points in our training program: let's arbitrarily say that we have been seriously building stuff since the pyramids, about 3000 years ago; and let's say that we have been seriously building software for the last 60 years, roughly.

If we set the building of the pyramids at the beginning of the course (9 am) and we set the present day at the end of the course (5 pm), with a little math you'll find out that, in an entire day of training, you have been taught about building software only from 4:51 pm to 5:00 pm (the 9 minutes are 60 years on a scale of 3100 years) and that’s assuming there were no breaks in our course!.

Everything else you learned during that day had very little to do with building software, or nothing at all.

So it's not surprising that, when we started building software a few decades ago, we applied the same practices that we have used for centuries to build other things.

The problem, of course, is that we didn't realize that software was a completely new and different story, something that mankind had never created before.

I'm not talking about the technical components of software, what I'm saying is that never before in our history there was an industry that would build and sell, on planetary scale, a totally intangible product.

How Software Is Different

Software is not the first intangible thing that we ever produced: music is certainly another and it dates way back than software.

There are in fact similarities between music and software: music exists only in the instant when it’s played, software exists only in the instant when it’s run.

After that instant, both go back into the void whence they came; anything else is just a static representation of theirs in an arbitrary form – a music score or lines of code – that our mind needs, to be able to manipulate that intangible product.

Another characteristic of software is that, mainly, it’s produced by the collaborative effort of a group of people.

Considering the complexity of most problems that we create software to solve, that’s reasonable: it would be almost impossible for a single person to develop a very complex application (one that solves a complex problem) in such a short amount of time to provide actual business value.

In other words, if a single person spent five years to develop a product, it’s very likely that after that time the potential customers would have lost interest in the product, at least in that form.

So we need teams that work together to develop something intangible for which there is only an arbitrary representation.

Finally, software development is intrinsically heuristic, that is, there is an inevitable amount of exploration and learning that comes with developing software.

After all, the main reason why someone would spend time, money and energy to develop new software is because there isn’t already a piece of software that does exactly the same thing: nobody ever wrote it.

So there is always a certain amount, big or small, of “unknownness” in developing software, which is why we need time to explore that unknown, learn about it and learn how to deal with it – and the amount of time we need is very hard to predict precisely.

For this reason, software development is not a deterministic activity, but an empirical one, at least in part.

So, the act of developing software is the collaborative effort of a group of people that heuristically develop a completely intangible product of the human mind.

And we’ve made a global industry out of that.

Now, don’t tell me that this isn’t quite new for us humans!

We Are Influential

At the point where we are in the history of mankind, the software industry is one of the most influential activities on the planet, meaning that we touch and change, from a minor to a major degree, the lives of billions of people.

There is software in pacemakers, just to give you an example of how much we may affect one person’s life.

To mention a few other examples (but you can grow this list by yourself) software is used:

  • to make airplanes fly, ships sail and cars run;
  • to move huge amount of money on the planet everyday;
  • to assist virtually all science researchers;
  • to create movies, music and other art forms;
  • to reconnect distant people;
  • to give freedom of speech – and freedom to read – to everyone who has an Internet connection (in most countries)

And of course, there is software in many personal appliances including your washing machine, your TV set and your telephone.

This list is far from being exhaustive; it’s just to give you an idea of what I mean when I say that our industry is influential: we directly and indirectly affect the lives of billions of people – and, in truth, we try to make some areas of their lives easier.

We Are Inefficient

Interestingly enough, we are also one of most inefficient industries on the planet, especially considering the big responsibility that comes with our enormous influence.

When making such a bold statement, one could mention the Standish Group’s Chaos Report to support it, although there is quite a lot of controversy about the figures in the report and, more specifically, about the analysis and data collection methods used to produce it.

If you have been in IT long enough, though, you don’t need figures to know how frequently a project may suffer from one or more of the following issues:

  • project runs over time (product doesn’t meet the delivery date)
  • project runs over budget
  • the product has serious functional defects, up to the point that customers aren’t keen to buy it or use it (diminished business value)
  • the product doesn’t meet expectations, doesn’t have some of the expected functionalities or, in general, doesn’t solve the problems that users were hoping for it to solve (diminished business value again)

Sometimes – too frequently, actually – these issues are so serious that the project gets cancelled before anything is delivered; and if you like figures, the Chaos Report states that about 30% of projects fail completely, while about 50% are challenged by the issues listed above.

Even ignoring the figures and relying only on my years of experience in IT, I can’t think of another industry on the planet that is to ingrained into the lives of billions of people and, at the same time, scores so badly in terms of quality and performance.

And I say this with all the respect I have for the industry I’ve been working in for the last twenty-some years, and most of all with the respect I have for its people.

But They Need Us

In my opinion, the only reason why we could get along all this time – using an inadequate production model, being so inefficient and delivering products that are so distant from what people needed – is just because we happen to live in times when people on this planet crave for software.

In other words, our industry survives because there is far more demand for software than we are able to supply and, regardless of how inefficient and sloppy we are, there will always be someone asking us to create new software.

Of course it’s the industry that survives, not the individual companies: with such a high project failure rate, some companies are forced to shut operations down or to lay off quite a few people and enter a slow-paced or fast-paced death march.

And now for the good news: the fact that people crave for software means that there is a lot of room, a lot of business opportunities, for IT companies that are going to be efficient and are able to provide their customers with quality products in reasonable time and at a affordable price.

Basically, there is a lot of room for IT companies that will be able to stand out and move away from the old production model.

How We Got Here

As I said earlier, when we started building software a few decades ago, we didn’t know or realize that software was a completely different story, something that no industry had ever produced in the history of mankind.

So we started building it just like we used to build other things, such as mass-produced tangible goods.

Which brings us to Frederick Winslow Taylor and his management theory of Scientific Management (also referred to as Taylorism).

Taylor defined his theory at the end of the 19th century, when industrialism and the Industrial Revolution required labor to operate the factories and many people left their occupation as craftsmen or farmers to go to work in assembly lines.

As such, Taylorism is focused on optimizing labor and processes in a repetitive, deterministic and measurable workflow.

There would be a lot to say about Taylor's theory to make it justice, but for the sake of this discussion there are two aspects that are of interest (from Wikipedia):

  • the process should be enforced by management

"It is only through enforced standardization of methods, enforced adoption of the best implements and working conditions, and enforced cooperation that this faster work can be assured. And the duty of enforcing the adoption of standards and enforcing this cooperation rests with management alone."

  • workers are incapable of understanding what they are doing

"'I can say, without the slightest hesitation,' Taylor told a congressional committee, 'that the science of handling pig-iron is so great that the man who is ... physically able to handle pig-iron and is sufficiently phlegmatic and stupid to choose this for his occupation is rarely able to comprehend the science of handling pig-iron.'"

In practice, Taylor was suggesting that all the thinking and planning should be done by well-educated managers, and then the plan is passed down to uneducated workers for its actualization.

Since the produced goods are tangible and the process is algorithmic and deterministic, it's then possible to compare the output from workers (product) with the input from managers (plan).

Of course, Taylor realized that working many hours a day on a production line, performing a repetitive task and without ever seeing the final result of your labor (you work only on a portion of the product) would provide no intrinsic motivation for workers to do their job well, so the solution was to measure performance and provide monetary incentives or deterrents.

Also, at this point workers become interchangeable and so they are frequently referred to as "resources" rather than "persons".

For an assembly line, the model envisioned by Taylor may be understandable and, in fact, it's been used in countless factories and companies in the last two centuries.

It's been around for so long that it became ingrained in our society and in our very lifestyle, up to the point that it's been imported into companies that don't even have an assembly line, as if this were the only way we have to produce something (that's why the lines above may have rang a bell in you, even if you don't work in an assembly line).

Probably more unconsciously than not, this approach was also adopted by IT companies; but as we have seen, software development is not a deterministic, algorithmic, repetitive process to mass-produce tangible goods.

We who work in IT are knowledge workers who create intangible goods; and we do that empirically, not algorithmically, because of the very nature of the products we create; and we do it in teams, not in sequential assembly lines.

Aside from common sense, there is plenty of scientific evidence that shows how negatively knowledge workers are affected by a factory-like approach.

I won't delve into this subject here but, as a starter, I'd like to refer you to Daniel Pink's book "Drive" and to the related, very enjoyable video on YouTube.

Here We Call It Waterfall

A very concrete and dramatic incarnation of a Taylor-like approach to software development is what ended up to be called the "Waterfall model", still dominant in the IT industry.

Waterfall mirrors, for software, many of the principles of Taylorism, including the fact that the process is sequential, predictable and all design happens upfront, eventually delegating implementation to labor workers (interchangeable programmers or "resources").

The Waterfall model is attributed to Winston Royce, but the truth is that Royce never believed in a sequential production model applied to software development.

In his paper "Managing the Development of Large Software Systems", when talking about a sequential model for software development, he says:

"I believe in this concept, but the implementation [...] is risky and invites failure."

Risky and prone to failure – that depicts pretty well the fate of so many IT projects developed worldwide.

What Royce suggested in his paper was, instead, an iterative model where feedback loops in the production process would play a major role, which is exactly what you need to do when you work in an empirical context: constantly measure and adapt, as fast as you can.

So, going back to Waterfall, why on earth would a manager or an entrepreneur want to use such an approach to create their software products, when studies and empirical evidence would suggest otherwise?

In my opinion, because of the three thousand years we spent building stuff and because of the two centuries of mass-producing tangible goods since the industrial revolution.

On Command and Control

One of the byproducts of a Taylor-like model, including the Waterfall one, is the development of a "Command and Control" culture.

In its military definition, Command and Control is "The exercise of authority and direction by a properly designated commander over assigned and attached forces in the accomplishment of the mission".

You can see its resemblance with Taylor's idea that management should think and impart orders, while labor workers should execute without understanding or questioning what they are asked to do.

I've seen two major problems when a culture of that kind is brought into software development organizations:

  1. technical team members show a decreased willingness in taking responsibility for what they do, as well as a decreased level of initiative and creativity (as explained by Daniel Pink) which would instead be quite valuable in our field
  2. this culture is self-reinforcing and it becomes therefore a major obstacle to change in favor of a better production environment

To elaborate a bit more on the second point, if one of the founding ideas of Taylor's production model is that management should enforce the process on labor workers (and that idea somehow stuck, even when not explicitly stated today) that means that the kind of personality that is more likely to climb up to a managing position is a dominant one.

In other words, the reason why we tend to see dominant managers is because the production model that we use calls for a dominant personality in managers.

Since the model also measures performance to assign incentives and people in managing position have a bigger impact on production (because they manage many other people) they get bigger incentives and tend to be less keen of challenging the status quo.

Unfortunately, in IT being a dominant manager has a negative influence on the productivity.

So we have an industry based on knowledge workers, which is in dire need of creativity, initiative, collaboration and knowledge sharing to perform at its best.

At the same time, that industry is designed to have dominant people float up more easily than others to managing positions, where they frequently (and most of the time involuntarily) squander the exact resources they are in dire need of.

Let’s Wrap It Up

A Taylor-like approach, being designed for algorithmic mass-production of tangible goods, is an inadequate model for knowledge workers and, specifically, for software development, where people produce intangible goods working in a collaborative and heuristic context.

The attempt to use a Taylor-like approach (which we call Waterfall in IT) may in fact be the main reason behind the high level of inefficiency and the low quality frequently associated with the IT industry, something that many companies (not to mention users and customers) just can't afford anymore.

Also, the command-and-control culture that comes with such an approach invites people to play roles, to focus on incentives rather than quality, to assume dominant positions (managers) or to avoid taking responsibility (teams).

Such a culture doesn't motivate nor inspire knowledge workers and prevents people from developing systemic awareness, which is the basis for self-organization.

The Agile Manifesto nailed all this down pretty well, ten years ago, when its signatories wrote its four values:

  • Individuals and Interactions over Processes and Tools
  • Working software over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan

Such values are much more adequate to an industry whose people collaboratively and heuristically develop intangible products of the human intellect.

What the Manifesto doesn't specify is how to actually implement those values, especially in organizations that come from many years of Taylor-like processes and where the cultural change is substantial.

Fear not, this and much more will be the subject of upcoming posts.

PrintView Printer Friendly Version

EmailEmail Article to Friend

References (31)

References allow you to track sources for this article, as well as articles that were written in response to this article.
  • Response
    Response: Jaylynn Roop
    Thanks again for the blog.Really looking forward to read more. Keep writing.
  • Response
    Response: Sidney Knudson
    I cannot thank you enough for the article. Awesome.
  • Response
    Response: Mekhi Hamby
    Awesome blog article. Want more.
  • Response
    Beyond Agile - Beyond Agile - We just can't afford Taylorism anymore. Now what?
  • Response
    Beyond Agile - Beyond Agile - We just can't afford Taylorism anymore. Now what?
  • Response
    Response: click for source
    Beyond Agile - Beyond Agile - We just can't afford Taylorism anymore. Now what?
  • Response
    Response: Added Analysis
    I’ve recently started a site, the info you offer on this site has helped me greatly. Thank you for all of your time & work.
  • Response
    Thanks - Enjoyed this article, is there any way I can receive an email every time you write a new post?
  • Response
    Response: more data
    You have brought up a very good points , appreciate it for the post.
  • Response
    This site is known as a stroll-through for all the information you wanted about this and didn’t know who to ask. Glimpse right here, and also you’ll undoubtedly discover it.
  • Response
    Response: Blog for money
    you're actually a excellent webmaster. The site loading speed is amazing. It seems that you're doing any distinctive trick. Furthermore, The contents are masterwork. you have performed a magnificent process in this subject!
  • Response
    Response: Blog for money
    Just wanna say that this is very helpful , Thanks for taking your time to write this.
  • Response
    Beyond Agile - Beyond Agile - We just can't afford Taylorism anymore. Now what?
  • Response
    Response: have a peek here
    Good Web-site, Keep up the very good job. With thanks.
  • Response
    Beyond Agile - Beyond Agile - We just can't afford Taylorism anymore. Now what?
  • Response
    Response: gameshark
    Beyond Agile - Beyond Agile - We just can't afford Taylorism anymore. Now what?
  • Response
    Response: would've
    Beyond Agile - Beyond Agile - We just can't afford Taylorism anymore. Now what?
  • Response
    Beyond Agile - Beyond Agile - We just can't afford Taylorism anymore. Now what?
  • Response
    Response: bekijk de video
    Beyond Agile - Beyond Agile - We just can't afford Taylorism anymore. Now what?
  • Response
    Response: enjoy online
    Beyond Agile - Beyond Agile - We just can't afford Taylorism anymore. Now what?
  • Response
    Response: reduce fine lines
    Beyond Agile - Beyond Agile - We just can't afford Taylorism anymore. Now what?
  • Response
    Beyond Agile - Beyond Agile - We just can't afford Taylorism anymore. Now what?
  • Response
    Beyond Agile - Beyond Agile - We just can't afford Taylorism anymore. Now what?
  • Response
    Response: dumpsters
    Beyond Agile - Beyond Agile - We just can't afford Taylorism anymore. Now what?
  • Response
    Response: D D Photographics
    Beyond Agile - Beyond Agile - We just can't afford Taylorism anymore. Now what?
  • Response
    Response: xovilichter
    Beyond Agile - Beyond Agile - We just can't afford Taylorism anymore. Now what?
  • Response
    Response: wordpress.com
    Beyond Agile - Beyond Agile - We just can't afford Taylorism anymore. Now what?
  • Response
    Beyond Agile - Beyond Agile - We just can't afford Taylorism anymore. Now what?
  • Response
    Beyond Agile - Beyond Agile - We just can't afford Taylorism anymore. Now what?
  • Response
    Beyond Agile - Beyond Agile - We just can't afford Taylorism anymore. Now what?
  • Response
    Response: Gegen Haarausfall
    Beyond Agile - Beyond Agile - We just can't afford Taylorism anymore. Now what?

Reader Comments (12)

Awesome post. Thanks for that, it summarises the issue very well.

September 30, 2011 | Unregistered CommenterHass

When you have so much of you career invested in something like Agile, XP etc... it can be hard to see the forest for the trees. I had a consulting job in The Haag many years ago. IBM was the incumbent contractor at the customer site (a bank) but after 5 years on the job they had not written a single line of functioning code. In the office there were two teams of software people... both behind closed doors. The first team was the Data team and the second team was Functional. They rarely spoke and they never shared information. I was there for a week, introduced the client to OO and we had a functioning prototype. Smart people do smart things, You cannot make an underachiever exceptional by using Agile. Either they get "it" or they don't.

September 30, 2011 | Unregistered Commenterrbucker

Hi Andrea :)
What a fabulous post you have here. I read it already 3 times and still amazed how you get to explain so clearly why we still do this silly things.

Getting to your last comment, you say Agile Manifesto does not specify how to implement those values... my question is: should it?
I don't think so.. that way Agile would be as limited as all other non-inspect/adapt approaches.
When we specify something we simply create a space for the whole thing to fail... it's like if this approach is an elastic.. if you take from it the possibility to stretch or shrink..it will become useless.

October 7, 2011 | Unregistered CommenterCatoliv

Hi Andrea,

I like how you're attacking taylorism, I'm sure your article will spark a lot of debates if it gets enough views (waterfallists believe that agile projects are never done!), and that's why I would like to republish it on PM Hut ( http://www.pmhut.com ) where many Project Managers will be able to read it!

Please either email me or contact me through the "Contact Us" form on the PM Hut website in case you're OK with this.

October 8, 2011 | Unregistered CommenterPM Hut

Technology failed me. The iPad app that I use to monitor this blog didn't notify me of new comments.

My apologies to my readers, to whom I'll reply instantly.

November 1, 2011 | Registered CommenterAndrea Provaglio

@Catoliv Hello Catia, good to hear from you. Yes, I also appreciate the wisdom of the Agile Manifesto signatories, who focused on values and principles and not on the actual implementation.

By doing so, they let the community, the industry and the individuals to come up with different and wonderful ways to change the industrial approach to software development, which is why now we have a range of approaches we can choose from.

November 1, 2011 | Registered CommenterAndrea Provaglio

@PM Hut: thanks for your appreciation and for the opportunity. I will contact you right away.

November 1, 2011 | Registered CommenterAndrea Provaglio

@Hass Thank you for taking the time to read it and for your feedback!

November 1, 2011 | Registered CommenterAndrea Provaglio

I came across to this post reading another smart one Please, Stop Saying Software Is An Intangible Product and I must say it may be right: comparing software with music may be really a misleading analogy.

Finance or economics are intangible as well, but unfortunately they wold have lead this article to a completely different result, since finance and economics do use maths and non-heuristic logic.

Comparing software with music might be captious, because it subreptitiously communicates that software developing is an art. Having this eventually weak and wrong working hypothesis, all consequent speculations come easy. Actually, it seems to me that this comparison is fundamental in your logic, but I find it really weak. I might be wrong.

I must admin that post I mentioned made me think

May 16, 2012 | Unregistered CommenterAnne Cecilia

Hello Anne,

thank you for your comment. I must say that I have no thesis to defend. All analogies are inaccurate. They work only to some extent and, in the short part of my post where I was making that analogy, I was referring to the product, not to the production. Other ideas are expressed elsewhere.

Best,

Andrea

May 19, 2012 | Registered CommenterAndrea Provaglio

Hello Andrea. Thank you very much for your reply.

Analogies are much more than a dialectical tool. They are very useful: once you see a phenomena is analog to another, you can start taking into consideration the idea that techniques previously and successfully used with the former can be used with the latter as well.

Hence, the accuracy of an analogy can be very important, indeed.

In this sense, I believe that your analogy music-software development could be even misleading.

Are you sure music development has something to teach to software development? Or was it just an opportunistic analogy you chose in order to strengthen your thesis "with software development, heuristic logic should be used"?

This is not my hypothesis, this is arialdo's one. But it seems sharable.

May 21, 2012 | Unregistered CommenterAnne Cecilia

There seem to have been quite a lot of misunderstanding in the last comments about this post.

For one thing, the subject here is whether or not a classic industrial approach is fit for the software development industry as a whole.

Second, the commenter put words in my mouth that I have never said, such as:

* developing software is like developing music; and

* in software development, heuristic logic should be used

Not only I've never said such things, they are also completely off-topic with regard to the subject of this post.

Now, I don't find it a good use of my time -- and most of all, of the time of the readers of this blog -- to have to point out to people that they completely missed the point and that they are, to some extent, fantasizing about things I've never said.

So I'm sure you'll understand if I will not continue on this thread. To my readers, I apologize if somehow I wasted their time. I'll definitely be more careful with moderation in the future.

However, if we want to bring the discussion back to its real intent, I'll be pleased to follow up.

And what I'm really interested about is: how can we improve the software development industry so that it provides more value to the society? how can we create better products, more efficiently and with more quality? how can we, to some degree, make a positive impact on the lives of the people using our software products, and on the one of those creating them? (and that includes you too, Arialdo/Anne).

May 23, 2012 | Registered CommenterAndrea Provaglio

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>