Will there be a silver bullet in software development? If yes, what will it be? If not, how can we defend against the Software Project Werewolf?

There doesn’t seem a silver bullet in the software engineering yet.

According to F. Brooks, there are accidental and essential difficulties in software engineering. Developments so far have solved the accidental difficulties and made some contributions to the development of software but these contributions are not “order-of-magnitude” contributions.

The essential difficulties in software:

1.      Complexity:

Why software systems are complex:

a.       No two parts are alike.

b.      There are massive number of states in software.

c.       A scaling up of a software entity results in nonlinear increase in complexity.

                                                              i.      It’s not merely a repetition of the same elements in larger sizes. It is necessarily an increase in the number of different elements. In most cases, the elements interacts with each other in some nonlinear fashion, and the complexity of the whole increases much more than linearly.

d.      Complexity of

                                                              i.      Software structure and behavior

                                                            ii.      Communication among team members

2.      Conformity:

a.       Software is produced by and run within an organization. It is always important to conform organizational policies.

                                                              i.      introduces additional complexity to software.

                                                            ii.      “Arbitrary complexity” without solid policies, principles and reasons by human institutions and systems.

                                                          iii.      Organizational policies often change.

                                                          iv.      Changes in law, management, etc.

3.      Changeability:

a.       Unlike software, manufactured things are not often changed after they are built.

                                                              i.      They will be replaced with other things, not changed often.

                                                            ii.      Involves high cost for changes

b.      Successful software will often get changed.

                                                              i.      Users who like the software will want to use it in another similar domains.

                                                            ii.      Successful software survives beyond the longevity of a platform for which it was build.

                                                          iii.      Changes in organizational structures, laws, policies, etc.

4.      Invisibility:

a.       Software is invisible and un-visualizable.

                                                              i.      Visualization for software structure and behavior

                                                            ii.      Visualization of big picture

1.      Communication overhead among engineers, customers and mangers.

b.      The fact it is hard to build a solid model of software systems gives customers the illusion that the systems should be flexible and easy to change.

Brook says that past breakthroughs solved accidental difficulties that are not inherent in software itself.

1.      High level languages –free a program from much of its accidental complexity.

2.      Timesharing - preserves immediacy, and hence enables us to maintain an overview of complexity

3.      Unified programming environments - They attack the accidental difficulties of using programs together, by providing integrated libraries, unified file formats, and pipes and filters.

Other developments in the software engineering didn’t prove to be silver bullets too. All these made contributions to software, but they didn’t solve the essential difficulties of software.

1.      ADA – another high level language, greatest  contribution will  be  that  switching  to it occasioned  training  programmers in modern  software  design  techniques.

2.      Object Oriented Programming - removes a higher-order sort of accidental difficulty and allow a higher-order expression of design.

3.      Artificial intelligence

4.      Expert systems

5.      Automatic programming

6.      Graphical programming

7.      Program verification

8.      Environments and tools

So what should be done to defend against the software werewolf?

·         Exploiting  the mass market to avoid constructing  what can be bought.

·         Using rapid prototyping  as part of a planned iteration in establishing software requirements.

·         Growing  software  organically,  adding more and more function to systems as they are run, used, and tested.

·         Identifying  and  developing   the  great  conceptual   designers of the rising generation.

Why is “adding manpower to a late software project makes it later”?

Should we add manpower to the project if we discover that our software project is behind schedule? How can we catch up in a late project?

According to Brooks, failure to meet schedule is the reason for most software project failures. The reasons for that:

1.      We don’t know how to estimate (overly optimistic).

2.      We confuse effort with progress.

3.      Software managers give in to pressure to reduce time estimates because they are uncertain of their estimate.

4.      We don’t monitor schedule progress properly.

5.      We tend to handle schedule slips by adding people.  This tends to magnify the problem…

Progress vs. Cost, 1

When there is no dependency among people, the amount of time to do a task diminishes with each new person.  Note that the cost (people * months) is constant.

Progress vs. Cost, 2

When there is a dependency and the task cannot be partitioned, adding people has no effect on time required, but it has a big effect on cost.

Progress vs. Cost, 3

If task can be partitioned, but requires communication, must handle training and communication as each person is added.  This can cause project to be later.

Example:

Suppose that at 2 months, we achieve milestone A.  We must deliver on time. What can we do?

Assume the project will go according to plan from here on (optimism!)

So 9 person-months must be accomplished in 2 monthsà Add 2 people.

Assume the project estimate is off by a factor of 2.

So 18 person-months must be accomplished in 2 monthsà Add 6 people.

n  Let’s say we add 2 people at month 2.

n  Must train them – assume this takes 1 month and requires 1 of the other 3 people.

n  During month 3:

n  3 person-months of training work

n  2 person-months of actual work.

n  Still have 9-2=7 person-months of work, but only 1 month left!

Brooks’s law:  Adding people to a late software project makes it later.

Should we add manpower to the project if we discover that our software project is behind schedule? How can we catch up in a late project?

The number of months of a project depends upon its sequential constraints. The maximum number of men depends upon the number of independent subtasks. From these two quantities one can derive schedules using fewer men and more months. We cannot, get workable schedules using more men and fewer months.

Exceptions – What should be Done

·         Projects that are not really late, especially those with unrealistic or otherwise flawed timetables.

·         Over-hiring: We can add more staff than are needed. For example, if we think that our project will require 3 people for 6 months, we can hire 4 people at the beginning of the project.

·         Good project segmentation: The notion here is that if a team is broken up into sub-teams, the addition of a new member will drain fewer time, talent and energy resources from the project, since fewer team members will have to be involved in the ramping-up of a new player whose responsibilities and tasks are compartmentalized and relatively isolated—in terms of inputs, outputs and impact on the “big picture” or project as a whole.

Rlationship between System Engineering and Software Engineering

Introduction:

Systems engineers and software engineers work together in the development of modern complex systems.

The two engineering cultures, the concepts, and the best practices have developed independently over four decades.

The present record of success for complex computer intensive systems is that for every six systems put in operation two are canceled; on the average projects are 50% over schedule; and three quarters are failures that do not function as intended or are not used at all, (Gibbs 1994). Incomplete specifications, ambiguous specifications, and misunderstood specifications are a major contributor to these problems.

Applying system engineering principles specifically to the development of large, complex software systems provides a powerful tool for process and product management.

The application of system engineering principles to the development of a computer software system produces activities, tasks, and procedures called software system engineering, or SwSE. Many practitioners consider SwSE to be a special case of system engineering, and others consider it to be part of software engineering. However, we can argue that SwSE is a distinct and powerful tool for managing the technical development of large software projects.

The need for synergism between systems engineering which develops specifications to meet user need and software engineering is particularly important because software portions of systems are increasingly complex and are often being coded in countries far from the country where the system is defined and utilized.

1.       Definitions:

Software Engineering:

Software engineering is the application of science and mathematics by which the capabilities of computer equipment are made useful to man via computer programs, procedures, and associated documentation. 

Software engineering is the systematic application of methods, tools, and knowledge to achieve stated technical, economic, and human objectives for a software-intensive system.

System Engineering:

Systems engineering is the application of scientific and engineering efforts to

(a) transform operational need into a description of system performance parameters and a system configuration through the use of an iterating process of definition, synthesis, analysis, design, test, and evaluation;

(b) integrate related technical parameters and ensure compatibility of all physical, functional, and program interfaces in a manner that optimizes the total system definition and design;

(c) integrate reliability, maintainability, safety, survivability, human, and other such factors into the total engineering effort to meet cost, schedule, and technical performance objectives.

Another definition: An iterative process of top down synthesis, development, and operation of a real-world system that satisfies, in a near optimal manner, the full range of requirements for the system. In fact, most definitions of systems engineering focus on the process by which operational needs and specific requirements are converted into working systems against a backdrop of cost, time and talent constraints.

Software System Engineering:

Software engineering is

1. The practical application of computer science, management, and other sciences to the analysis, design, construction, and maintenance of software and its associated documentation;

2. An engineering science that applies the concepts of analysis, design, coding, testing, documentation, and management to the successful completion of large, custom-built computer programs under time and budget constraints; and

3. The systematic application of methods, tools, and techniques that achieve a stated requirement or objective for an effective and efficient software system.

4. The establishment and use of sound engineering principles in order to obtain economically software that is reliable and works efficiently on real machines.

5. Software engineering is the application of science and mathematics by which the capabilities of computer equipment are made useful to man via computer programs, procedures, and associated documentation.

Both SwSE and SwE are technical and management processes, but SwE produces software components and their supporting documentation.

Figure 2 illustrates the engineering relationships between system engineering, SwSE, and SwE.

Traditional system engineering does initial analysis and design as well as final system integration and testing.

During the initial stage of software development, SwSE is responsible for software requirements analysis and architectural design. SwSE also manages the final testing of the software system.

Finally, SwE manages what system engineers call component engineering.

2.       My Thoughts on the Relationship Between Software Engineering and System Engineering

In the modern world, complex systems are commonly required. Software intensive systems are among the most complex systems and systems engineering is commonly used in those systems. Both system engineering and software engineering address the creating of complex software intensive systems from two different perspectives.

Software engineering is not a deep discipline in some aspects and system engineering can make some contributions for them. Creating software intensive systems require both fields to cooperate.

Software systems are usually designed, developed and maintained by professionals with narrow disciplinary perspectives. Complex systems cannot be designed by one person. System engineering is a horizontal field, not deep in one field but wide in many fields. System engineering borrows from a wide range of areas and formal disciplines. Without these contributions, the field cannot exist.  Without the organizing systems engineering life cycle, they remain uncoordinated expertise fields. Systems engineering is also domain independent. The design and development principles are generic. They can be applied to any problem.

So a system engineer is probably the right person for creating complex systems. Software engineering is deep in its field but not wide in other disciplines. In this aspect, software systems engineering comes to play which joins systems engineering and software engineering in creating complex software systems.

Comparison Involvement in a System’s Life Cycle

In the lifecycle of a software system, Systems engineering places relatively more interest on the front end requirements analysis, modeling and validation processes. System analysis spend more time spend much more time and effort on requirements than software engineers. But it is certainly true that software engineers

have begun to appreciate the importance of needs analysis, user requirements modeling, and organizational requirements profiling.

1.       Comparative advantages

Usually software engineers does not spent a great deal of time on front-end system and user requirements identification, definition, or validation. Instead, there is usually a desire to ‘get coding’ as soon as possible.

Consequently, there are a large number of ‘failures’ that can be traced directly to inadequate requirements analyses. Software engineers usually avoid the systematic development and evaluation of alternative definitions and designs throughout the software engineering process. Criteria-based alternative design evaluation, for example, is seldom conducted by software engineers; nor is there much emphasis on prioritising requirements with reference to constraints in formal (or even informal) trade-off analyses. All such activity is automatic to systems engineering; without such activity, software engineers risk proceeding prematurely to design and coding (as has often been the case).

Software engineering needs an infusion from systems engineering of effectiveness measures, early formalized trade-off applied throughout the work, and sequential build and test. It needs consistent application of hierarchy.

Systems engineering needs to receive from software engineering an infusion of rigorous modeling and information hiding, the use of generators, and application of the classification relationship in modeling. It is critical for reuse, for consistent definitions, and for rigorous communication with software engineering.

The mutual infusion of best practices from the other discipline strengthens both, and ameliorates existing problems of providing specifications to software engineering on large projects.