Within a Big Data project, topics such as building a Data Warehouse or Datalake, data integration, implementation of a BI tool, or an AI/DL model often occur. Several project management approaches and tools are available for the course of a project. In this relation, one differentiates mostly between these three methods: agile, classic, and hybrid.
In the past, projects like the creation of data warehouses was a large, monolithic, quarterly or multi-year effort, and subject to the traditional “waterfall” process. In the modern era, this is no longer the norm, with many organizations opting to use a more flexible and iterative or agile approach.
With business needs changing faster than ever and new data sources being added all the time, the solution is usually to adopt an incremental and agile manner. With an iterative approach, tasks/sub-projects can thus be broken down into smaller stories and business value can be created faster. As a result, the following three approaches have emerged and become the most common methods for implementing a data science project:
Method 1: Scrum
This approach is the most widely used process framework for agile development processes. Scrum emphasizes daily communication and flexible reassessment of plans that are executed in a short period of time. The agile Data Science Manifesto is organized around the following principles:
- Iteration
- Ship intermediate output
- Prototype experiments over implementing tasks
- Innovative product management
- Strong customer focus
Method 2: Kanban
This approach features managing/improving products with a focus on continuous delivery without overloading the development team.
Like Scrum, Kanban is a process to help teams work together by visualizing workflows, WIP boundaries, workflow management, and creating guidelines, it aims to improve collaboration and process.
Method 2: Kanban
This approach features managing/improving products with a focus on continuous delivery without overloading the development team.
Like Scrum, Kanban is a process to help teams work together by visualizing workflows, WIP boundaries, workflow management, and creating guidelines, it aims to improve collaboration and process.
Method 3: BEAM
BEAM aims at Agile Dimensional Modeling, with the goal of aligning requirements analysis with business processes rather than reports. The basic principles of this approach are:
Individuals and Interactions: Business intelligence is driven by what users ask about their business. The technical setting is secondary.
Business Driven: Well-documented data warehouses that take years to deploy will always be out of date.
Customer Collaboration: The end users’ knowledge of their business is your greatest resource.
Responding to Change: If you take all of the above actions, change will come naturally and results in weekly delivery cycles.
The author of BEAM, Lawrence Corr, describes, among other things, a design pattern for the structuring of “data stories”.
Example: Business Event — Customer buys product:
- When was the order date?
- Where was the product purchased, where will it be delivered?
- How many quantities were bought?
- What was the reason behind the purchase?
- Which channel was used for the purchase of the product?
Through these important questions, the business process comes to light and so technical conditions can be derived.
Conclusion
In this article, the three most commonly known project management approaches were mentioned and further described. However, it is important to realize that in addition to finding the best methodology, understanding that a shift to the agile methodology or automated code development is not just a shift in skills. It is rather a shift in mindset. In the field of data science, agile approaches are promising because it is often impossible to assess in advance, whether the project can be achieved at all using the available data. The extent to which the results are successful can only be judged once they are available.