Waterfall to Agile: A Necessary Mindset Shift For Business Analysts

I tend to describe the agile approach as a way of working; A targeted way of working that allows us to make changes, respond to customers’ needs and manage uncertainty with minimal delays, and without needing to wade through “red tape”. Agile manifesto encourages: 

- Customer collaboration over contract negotiation

- Individuals and interactions over processes and tools

- Responding to change over following a plan

- Working software over full documentation

This does not however imply that the agile approach does not involve processes and tools, following a plan, documentation or contract negotiation - but it implies they are less valued in comparison to activities that encourage collaboration, frequent delivery of working software, interaction and responding to change. 

Moving from the traditional/waterfall approach to an agile approach certainly involves a mindset shift that may not always come naturally or even easily for Business Analysts. This mental shift comes with implicit obligations of adopting new ways of working.

The agile approach described in this article is based on “scrum”, the most widely-used framework aligned to agile. I’ll also add that this article has been written within the context of software development, where an actual working product is expected to be delivered at the end of each sprint, though the ideas may be applicable to other contexts.

This article serves as a guide on the key differences between what would be expected of a BA in an agile-driven environment compared to a traditional environment, especially for those BAs who have to make a “shift” abruptly or for new BAs just stepping into the agile mode of working.

This is the first of a series of articles on this subject.

From Direct Interaction With Multiple End Users To Product Owners.

With the traditional/waterfall methodology, BAs are expected and encouraged to interact with stakeholders from multiple domains. There’s no official “Business Analyst” role in the scrum guide however, as scrum identifies only the Product Owner, Scrum Master and the Development Team.

The Product Owner is the authoritative source of requirements. The scrum guide specifically refers to the Product Owner as “Responsible for the product backlog, its content, availability and ordering”. 

So, why on earth do we have or even need BAs on scrum teams? 

The reality is, it’s not always feasible for product owners to be available to specify or even provide information on every requirement across multiple user groups and product features; Neither can we expect them to know all the “specialist” needs of each user group, especially when the end product is expected to cater to a large and diverse user base. BAs may thus be needed on scrum teams to liaise with stakeholders other than the Product Owner (PO) to gain clarity on requirements which extend beyond the expertise or capacity of the PO to clarify. Where the PO is not available to follow up on the details of these “specialist needs”, the BA can always step in to facilitate this process or speed up the development of user stories, where it is not necessary to introduce another PO.

Also, the larger the scope of the development effort, the more the stakeholders that can potentially be affected and the more a BA may be needed to interact with stakeholders beyond the core development team.

There’s also the need to elicit non-functional requirements, transition requirements as well as the design of mock-ups that add clarity to user stories. Business Analysts do have a key role to play on scrum teams in support of Product Owner-type activities or standing in as proxy for the Product Owner.

From Requirements Specification Documents (RSDs) To Product Backlogs

RSDs have their merit in the traditional software development lifecycle. These often lengthy and detailed documents or descriptions provide an avenue for BAs to describe in detail what is required in terms of system functionality. Typically, the BA would elaborate on each requirement with input from relevant stakeholder groups, specify each requirement to as much detail as possible before handling this “deliverable” to the development team, all done within pre-defined project timelines. An RSD would typically contain a list of “System shall...” statements, specified to a level of clarity that is suitable for implementation.

In the scrum context however, user stories are defined via a collaborative process (refinement) that involves the product owner and the development team. User stories are expected to be defined by the Product Owner, with the BA (where there’s such a role). While the requirements in an RSD may be written in the form of “user stories”, all the user stories in the product backlog are not expected to be completely elaborated unlike in the RSD; elaboration is only required for the user stories that are needed for the next sprint, with most scrum teams having a two-week sprint cadence. The idea of defining every single detail upfront before development can begin flies in the face of the scrum framework.

From Status Updates To Daily Stand-ups

Typical waterfall approaches require the BA to provide updates on tasks accomplished either to the project manager or to management directly. This may be in the form of weekly or monthly status reports depending on the organisation.

While scrum in principle does not dabble into project management or governance activities, BAs in addition to any obligations they have under the project management umbrella may be required to participate in scrum ceremonies, one of which is daily stand-ups.

The purpose of daily stand-ups is essentially to provide feedback to the development team in this sample format:

- What was achieved yesterday

- What will be achieved today

- Blockers/Impediments

BAs who do not have any official roles defined in the scrum guide may struggle with the idea of participating in stand-ups as their tasks do not “always” contribute to the achievement of the sprint goal as defined at the beginning of the sprint.

Though project teams still operate within a project wrapper in terms of governance, funding and reporting; ceremonies that the team are expected to engage in typically include daily stand-ups, sprint planning, reviews and retrospectives, and BAs would need to be a part of this, in addition to other engagements.

From Tasks to Epics, Features, Stories and Sub-tasks

With the waterfall approach, requirements elicitation work such as interviews and workshops would be represented as a series of tasks or activities that BAs have to carry out with the aim of identifying requirements, as part of the project plan. 

With scrum however, these tasks are not just tasks on their own to be managed within the BA’s private to-do list, but tasks that need to be linked to an epic or contribute to the achievement of the sprint goal. Epics are large enough items of work that are too big to deliver value to the customers, which implies they are also less likely to be completed within a single sprint. The items that fall under an “epic" categorisation can thus be broken down into features/user stories as they near the top of the product backlog.

Having a clear idea of how each BA task contributes to the sprint goal or an epic can also help the BA in providing updates during stand-ups, for teams that choose to include them in stand-ups.

In summary, it’s not how a company chooses to implement agile or scrum that determines success but the ability to identify a way of working that offers the necessary balance between agility, and the rigour that is characteristic of waterfall approaches so that the benefits of both worlds can be achieved. This mindset shift is particularly important for business analysts as it involves embracing key principles of agile, which are different from conventional principles characteristic of traditional organizations.