There are a lot of methods for developing software out there now.  Choosing the best one for your company or team can be difficult – and that’s before you start adapting it to suit your particular situation.  Here’s a brief history and overview of some of the more common ones you’ll come across.  Each of them have good elements that you can make use of or learn from.

In the beginning, there was Waterfall…
Not that long ago, almost all IT companies based their software development on what was known as the waterfall cycle.  To deliver a project you moved through phases in order.  Requirements Specification, Design, Construction, Integration, Testing, Installation and finally Maintenance.  This style of development works well where the requirements are very clear and not going to change, but that rarely happens in practice.  For dealing with a typical software project, the method is rigid and closed.  The software to be delivered has to be fully described in detail months or years in advance of delivery.  The cycles are long with low visibility of what’s being done.  It’s hard to find out if deadlines will be met until the dates have already gone past.  Despite the abundance of planning and documentation, the customer may be unhappy with the delivered product or it no longer matches what the customer actually needs.  This can be a costly and restrictive way of developing software which can lead to people to shifting blame onto others as the final deadline approaches if things are going poorly.

In more recent years there has been a move away from the traditional style of developing software to address some of the issues mentioned above and allow for projects where the requirements aren’t fully known at the start.  Many of these are incremental and/or iterative.  Incremental development is when the work is divded into parts (or increments) which are integrated when they’re completed.  At the end of each phase there is a fully completed unit of work or increment.  Iterative development uses multiple iterations to progress to a final solution.  The important distinction is that each iteration can include refactoring or expanding on a piece of work – rather than fully completing a unit of work at a time.  The first iteration may provide a basic piece of functionality and later iterations will develop it further until it grows into the final product.

The Unified Process…
The Unified Process (and also the Rational Unified Process) is an iterative framework.  Each time-boxed iteration provides an increment of work which is the input for the next iteration.  It’s still quite similar to waterfall.  The architect/designer is a vital role as each iteration is driven by UML use cases.  Roles and responsibilities are very clearly defined with each level in the organisation delivering particular artifacts to the next.  The Unified Process gives better observability than the waterfall method, but using tools like Rational Rose add overhead and the process creates big divisions between design and development.  At each point where an artifact is handed over, responsibility is also effectively handed over which is not necessarily a good thing.

Enter Agile…
Agile is a group of software development methodologies with the agile principles at the core of each.  The Agile Manifesto is a statement of these principles.  The four main principles are:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
You can view the full Agile Manifesto and history here.  Three of the Agile methodologies are XP, Scrum and Lean Software Development which we will take a quick look at.

XP / Extreme Programming…
XP replaces use cases with user stories.  These are less formal and usually less detailed than use cases as there’s more interaction between the developers and the person giving the requirements.  Instead of handing the requirements over at the start of the project, regular feedback is an important ingredient.  XP aims to have fast iterations with frequent planning and estimating.  Rather than using layering, software is built in slices so that feedback can be given early on.  Teams provide regular increments of software that could be delivered to the client using test-first approaches, continuous integration and regular customer feedback.  This allows for changing requirements as the project progresses.

Eliminating Waste with Lean…
Another Agile method is Lean Software Development which takes its inspiration from methods used in the manufacturing industry.  Waste is reduced or eliminated by removing anything that doesn’t contribute to the overall goal.  Decisions are deferred until as late as possible or until there’s no advantage to holding off.  This means decisions don’t have to be made at the very start of the project, but can be made when more information is available.  Teams try to eliminate time spent waiting (for another developer or a 3rd party to deliver something, or for more detailed requirements) and to deliver only what’s required as fast as possible.  More communication is encouraged and documentation is produced on-demand.  Teams are encouraged to see the bigger picture rather than focusing only on the slice of work they are doing.

Scrum…
Scrum progresses on from XP, but takes a wider viewpoint including focusing on how an agile team can function well within a large organisation.  Scrum was developed independently, but at a similar time as Lean and they have similar goals.   Many teams using the scrum practices also use XP and Lean practices.  Each team includes a Product Owner and Scrum Master.  Teams are small with 5-7 members and cross-functional.  The team as a whole commits to delivering functionality.  The Product Owner decides what functionality they want, and what order they would like it in and the team will decide what it can commit to in each iteration (called a sprint).  The scrum master is there to ensure the process is followed and assist the team in meeting their goals. Communication is a key factor here with daily meetings to keep all the team members in synch and share knowledge.

SOA – the latest buzzword…
Moving on from the Agile methods, one of the things you will commonly hear about now is Service Oriented Architecture.  SOA started in the larger companies – ones that may have merged with other companies on a regular basis, inheriting legacy systems and duplicating functionality as they went along.  As the complexity of working with completely different software systems increases, so does the cost.  SOA is a stategy for managing this complexity.  It can be used to reduce dependencies on 3rd party vendors and help ‘clean up’ the systems.  By reducing coupling between systems you gain the ability to swap out a component and replace it with an improved one.

It’s common to associate SOA with adding a web services layer over different systems but SOA should be seen more as a long-term strategy.  To reduce language coupling you can use XML, to reduce protocol coupling you can use web services.  Network coupling can be reduced using ESB and component coupling is reduced by making components replaceable.  The primary goal is to bring back simplicity.  Not everything should be a service and reuse of components isn’t always a good thing.  If communication is costly then the overhead of communicating that a service is available may be higher than if the component wasn’t reused.

General Strategies…
Agile methods tend to promote a low tech approach that avoids putting distance between the team and the state of the project.  In an ideal situation, an agile team is located together and communication is easier.  The client or product owner is heavily involved in refining requirements and providing feedback.  MS Project and Gantt charts aren’t used and updates are made directly by the people doing the work. Documentation goes out of date fast, so instead of documenting everything extensively, teams are kept aware of each others activities.  Documentation can be as simple as a photo of a whiteboard instead of using software tools.  Teams avoid making vague long-term predictions by assessing the risk of something happening, and then making it observable so changes can be made when they’re actually needed, rather than catering for something that never actually happens.

All of the methods above can work really well, or fail spectacularly.  Regardless of what method a company uses for developing software, the one thing that applies across the board is that there’s always room for improvement. And even a good development process can’t manage teams of individuals well for you.  For that you have to rely on having at least one good team lead – whatever process they follow or job title they hold.