Effective Agile sprint cycles for data consulting projects

These 4 Agile Sprint Ceremonies play a key role in any data consulting approach.

Effective Agile sprint cycles for data consulting projects

Data consulting projects can, by their nature, be highly complex and dynamic. Dealing with legacy systems, complex data structures and even changing third-party data source organizations can make data consulting projects highly dynamic. A traditional “waterfall” driven project approach (where one set of tasks are completed before another set of tasks can begin) can be highly frustrating for the client and the project team as data changes and unknowns are uncovered after the project has begun.

Neal Analytics simplifies the project planning and execution process in data consulting projects by employing the Agile Methodology to its project approach. The Agile Methodology employs a process where requirements and solutions can evolve during the project. Agile at its core anticipates change and is therefore structured to accommodate and embrace solutions to problems, changing requirements, or changing environmental conditions by empowering self-organized, cross-functional teams working in collaboration with the customer and end-users.

agile development cycle

The Agile Methodology was envisioned in the Agile Manifesto written in 2001 and is centered around 12 principles:

12 principles behind the Agile Manifesto

We follow these principles:

  1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
  2. Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.
  3. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
  4. Business people and developers must work together daily throughout the project.
  5. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
  6. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
  7. Working software is the primary measure of progress.
  8. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
  9. Continuous attention to technical excellence and good design enhances agility.
  10. Simplicity–the art of maximizing the amount of work not done–is essential.
  11. The best architectures, requirements, and designs emerge from self-organizing teams.
  12. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

A core concept of Agile Methodology is Scrum.

Scrum allows the project team to focus on delivering business value in the shortest time. It  enables self-organized project teams to quickly solve problems in unpredictable and/or evolving environments.

Scrum has three main roles associated with it:

  1. Product Owner – Defines the product’s overall design and breaks the design down into a backlog of User Stories. User Stories further describe desired features and functions in a more bite-sized manner. The Product Owner prioritizes and ultimately approves the completion of the User Stories.
  2. Scrum Master – Is the facilitator of the Project Team, referred to as the Scrum Team. The Scrum Master keeps daily track of progress, manages the exchange of information between the Product Owner and Scrum Team, and is responsible to clear any blockages holding back progress at the User Story level.
  3. Scrum Team – The Scrum Team are the individuals (typically between 5 – 9 people) who are responsible for delivering the individual User Stories, that add up to the overall “Product” or Project Deliverables.

The Agile Methodology requires a high level of communication and collaboration between all three of the character types identified above. You will note, I didn’t identify which role is played by the customer and which is played by the consultant. This is because while the roles are distinctly defined in what they are responsible for, they are meant to be flexible to allow client/consultant participation that best accomplishes the project goals.

Consulting companies often engage a variety of clients who have a wide range of experience working with the Agile Methodology. Many clients will have no experience with Agile at the start of the project. Consulting companies that have Agile deeply embedded into their business processes and approach, such as Neal Analytics, are well suited to step into each of the Agile roles to deliver quality output.

Agile Ceremonies

The Agile Methodology is broken into time-boxed periods called Sprints.

A Sprint is a period of time at least one week and not more than four weeks in which the Scum Team works to accomplish a set of predefined User Stories. These User Stories represent the Sprint Goal.

Note, that the Sprint Goal is not absolute. Things can be discovered or blockages occur that prevent a User Story from being completed in the Sprint, and that is okay – as long as it is not the norm. What is absolute is that once the Sprint begins, the only way that new User Stories are added to the Sprint is if the Sprint Team gains some unexpected efficiency and has the capacity to do more. In this case, User Stories can be added to a Sprint one by one.

In the Agile Sprint, there are four Ceremonies…. Don’t worry, you don’t have to know any secret handshakes or wear a special outfit. While always welcome, refreshments are also not required!

The Agile Sprint Ceremonies are actually meetings with very specific agendas and a required set of participants.

4 agile sprint ceremonies

The Agile Ceremonies are as follows:

  1. Sprint Planning: This ceremony is where the Sprint Team meets and sets the goal (chooses the stories) that will be completed in the upcoming Sprint.
  2. Daily Scrum: This is a standup meeting, a short, 15-minute meeting designed to keep the team moving forward.
  3. Sprint Review: In this meeting, the Sprint Team demonstrates what they accomplished in the Sprint that has just ended.
  4. Sprint Retrospective: This meeting is where the team reviews what worked and what didn’t work in the Sprint that just ended.

Since we are not part of a secret society, let’s demystify what happens and who should be involved in the ceremonies described above.

1.Sprint Planning

Who is required: Product Owner, Scrum Master, Scrum Team

Optional participants: Project Stakeholders

Purpose of the meeting: This ceremony should be held the last week of the current Sprint, in anticipation of the next Sprint. In this Ceremony, the Scrum team goes through the User Story backlog and selects the User Stories that will be accomplished during the next Sprint. During this meeting, the Scrum Team has the opportunity to discuss the requirements of each User Story under consideration and clarify or identify that the User Story may represent too large of a scope of work and needs to be broken into smaller User Stories. In this meeting, it is also the responsibility of the Scrum Team to identify a “scope of work” for each story, by use of Story Points, to determine how much time the User Story is estimated to take in the Sprint. Remember, a Sprint is defined by a finite duration of time, so a Sprint can only contain so many Story Points.

Duration of meeting: Sprint Planning should happen within a one-hour meeting, however, if further refinement is required secondary meetings can be scheduled.

2. Daily Scrum

Who is required: Product Owner, Scrum Master, Scrum Team

Optional participants: Project Stakeholders

Purpose of the meeting: In this ceremony, each of the Project Team members are restricted to describe the following:

  • What did you accomplish yesterday?
  • What are you working on today?
  • What, if anything, is blocking your progress?

ONLY SCRUM TEAM MEMBERS are allowed to talk. Others can attend, but they are only there as witnesses. Any sidebar meetings MUST occur outside of the daily stand up. This particular Ceremony is sacred to the Agile Methodology. Purists will tell you that all participants must… well, stand up!

Duration of meeting: Again, sacred to the Agile Methodology… This meeting should last no more than 15 minutes. If there is a need to discuss project related items to the project, they should occur outside of the stand-up.

3. Sprint Review

Who is required: Product Owner, Scrum Master, Scrum Team, Customer Representative(s)

Purpose of the meeting: This ceremony is all about the Scrum Team showing off what they have worked on in the previous Sprint. The team goes through each major item they have delivered to demonstrate that they have successfully delivered on the User Stories, but also that they have made measurable progress. These demos may not be flashy, may not show major completed states of progress, but rather should be demonstrative of what was actually accomplished during the completed Sprint.

Duration of meeting: The Sprint Review or Sprint Demo should last at least 30 minutes, 60 minutes depending on the volume of deliverables completed in the Sprint.

4. Sprint Retrospective

Who is required: Product Owner, Scrum Master, Scrum Team

Optional participants: Project Stakeholders

Purpose of the meeting: This ceremony is all about continual improvement. During a Sprint, well, bad things happen, team members get sick, problems arise, User Story assumptions are wrong. Good things happen too… the work is easier than expected, technology access happens sooner than expected, a resource has more capacity to work than originally planned. What worked? What didn’t work? Answering these questions will influence the existing process or add/subtract to the process. It’s all about making things better next time… You know, continuous improvement!

Duration of meeting: The Sprint Retrospective should last at least 30 minutes, 60 minutes depending on how large the team is and/or if the Project Stakeholders are in the review. All attendees should provide information.


Data consulting projects – like software development or process evolution projects – need to be flexible in scenarios where there is the potential for many unknowns and a desire for quick demonstrations of measurable progress. That’s why the Agile Methodology is the way to go.

While the underlying systems, data, processes, or technology architecture may be complex, the approach to modernization and execution of related projects are extremely simple and straight forward.

As I wrote at the beginning of this post, data consultants work on highly complex projects with many moving parts, and often with unknowns (the level of data cleanliness, unfamiliar vendor technologies, etc.) Consultants who are expert practitioners of the Agile Methodology are able to skillfully manage multiple workstreams, adjust to changing conditions, and even changing client priorities in real-time – without any delay on the project as a whole.

Agile allows firms like Neal Analytics to apply all of its consulting expertise in tandem so that the client is not only able to see marked progress at the end of a Sprint, but they can see the entirety of the project develop quickly at the same time. If the client is seeing something develop that isn’t exactly what they thought, or are inspired by what they are seeing and what to change direction or add scope, this can easily and quickly be accommodated.


One note of caution both for the consulting practitioners of Agile and the clients of a consultancy like Neal Analytics that deliver projects or solutions using the Agile Methodology:

A Sprint is, simply stated, a predefined period of time. It is not a guarantee that a specific set of work will be completed. Sometimes people revert to Waterfall thinking and assume that the end of the Sprint is a milestone and the gateway for the next phase of work.

While the Scrum Team commits to the work they will do in the Sprint, unknowns or insurmountable circumstances can arise during a Scrum that blocks a user story from completion. When this occurs, the Sprint is not a failure and the user story simply is added to the next or a future Sprint. An expert Scrum Master will account for these situations as subsequent Sprints are planned.

Effective Agile Sprints play a key role in the modern data consulting approach. Working with partners and vendors who have the expertise and a well-defined Agile process will help ensure your success and the success of your delivery teams on your next project.

This article can also be found on LinkedIn.