Why Are Software Projects So Difficult?

As a Business Analyst, you're bound to participate in one or more software projects. If you ever find yourself wondering why software projects are so difficult, there’s probably no paper that does a better job of providing answers to this question than the 1986 paper by Frederick P. Brooks, “No Silver Bullet – Essence and Accidents of Software Engineering”. Though the piece dates back many years, most of his ideas are still relevant to today's projects.

Using the analogy of werewolves, the monsters of folklore who transform from the familiar to the horrific, Brooks describes software projects. The familiar software project can easily be transformed into a monster of missed schedules, overrun budgets and defective software. Unlike werewolves that can be laid to rest with a silver bullet, software projects are a different ball game.

According to Brooks,

"Building software will always be hard – there is inherently no silver bullet".

The author points out that there are two difficulties associated with software: Essences & Accidents.

Essences are the difficulties inherent in the nature of software itself - this is related to specification, design (mapping specification to software) and testing of software. Accidents are the difficulties encountered when building software - this is related to languages, tools and programming techniques. Brooks argues that most techniques and software tools attack the accidents of software engineering and not the essence.

Back to the original question, what makes software projects so difficult? Brooks explains:

Complexity 
Software is naturally complex. No two parts of a software entity are alike. Expanding the functionality of software often results in multiple lines of code with different elements interacting in a non-linear fashion. From this complexity comes a difficulty in communication among team members which could lead to product defects, cost overruns and delays. How do you list all the possible states of a software program? How do you predict all the possible ways in which a system can respond to external events? As a manager, how do you find and control all the loose ends?

Conformity
Software is usually designed to conform to user specifications and other interfaces with which it must interact. These interfaces and user specifications are never static but evolve with time. For example, when the key user of a software application is transferred to another location, the successor may decide he wants the application to work in a different way. Similarly, when regulations change or new regulations are imposed on a business, software must conform. Changes in the environment usually necessitate software conformity - that is the nature of the game.

Changeability
Unlike buildings or cars after construction, software can always be changed. As Brooks explained, it is “infinitely malleable”.  Even if software is rolled out and used to the satisfaction of users, they'll only be content for so long. After a while, they will start inventing new uses for it, outside what it was intended to do, prompting the need for change. Another reason for changeability is that software life span usually extends beyond the life of hardware. When users buy new equipment or upgrade to more advanced hardware platforms (Displays, gadgets and the like), software usually needs to be upgraded to run successfully on these platforms. Software is, as can be expected, embedded in an unpredictable and complex mix of regulations, user requirements and dynamic platforms which constantly impose change.

Invisibility
Software is invisible and unvisualizable. How do you represent all the software components geometrically to convey meaning? How do you map out all the parts and interacting components to promote discussion?

How is this relevant in Business Analysis?

Brooks highlights potential silver bullets that can improve the chances of success and this is where the BA comes in:

1. Requirements Refinement and Rapid Prototyping

Software development should not rest upon the faulty assumption that one can specify requirements clearly from the onset of the project, then buy or build the software and install it without the need for iterative refinement. This he argues, is a fallacy.

According to Brooks:

"The hardest single part of building a software system is deciding precisely what to build. No other part of the conceptual work is as difficult as establishing the detailed technical requirements, including all the interfaces to people, machines and to other software systems. No other part of the work so cripples the resulting system if done wrong. No other part is more difficult to rectify later."

An extensive iteration between developers and clients is necessary. Brooks argues that there’s also no way for users to completely and correctly specify their exact requirements without trying out a version of the product. As such, the abundance of rapid application tools creates room for iterative development, which is necessary for managing the changeability of software applications.

Business Analysts should employ the use of prototypes wherever they can, to facilitate iterative requirements gathering. 

2.  Incremental Development – Grow, don’t build software. Grow software as you need to. Expand from a single functionality to multiple functionalities. This methodology lends itself to the development of early prototypes, which can then be amended to deliver what stakeholders want. Stakeholders become more enthusiastic when they see a working system.

Business Analysts should ensure that they show a working system to stakeholders constantly and as early as possible. Having a physical system at every point in the project not only promotes the much-needed discussions but also renews the enthusiasm and commitment of all stakeholders to the project.

3. Great Designers – Good design practice can be learnt. Organizations should invest in recruiting, training and motivating great designers by providing all the support they need. 

Any software design project should have great designers on board. Having the right team does go a long way in improving the chances of success.