Elección del modelo de flujo multifase para cada etapa

A common misconception among Agile critics is that Agile development involves zero design up front and therefore has a high risk of resulting in a poorly designed product. Conversely, agilists dislike the BDUF nature of plan-driven development, preferring instead to begin building something sooner that customers can evaluate and to which they can react. Uncertainty early in a project makes BDUF too costly and risky. However, experienced Agile developers also know that no up-front design leads to poor quality and high technical debt. What is needed is sufficient design up front (SDUF)— enough to galvanize developers around a shared understanding of problem domain, architecture, user experience, and data. Agile development doesn’t require a whole new set of modeling techniques. What is required is a new way of applying good modeling methods in an incremental, iterative, and evolutionary manner. Establishing a minimally sufficient conceptual model up front, and then incrementally evolving the physical model as the system is built, helps limit technical debt and increase design quality.

However, good evolutionary design requires team discipline, design exper- tise, and technical excellence. In other words, Agile Analytics is not a magic alternative to proper training, techniques, and experience. In practice, evo- lutionary design looks something like this:

Scenario

The FlixBuster DW/BI system was first deployed into production at the end of iteration six with a couple of high-value BI capabilities. The users were amazed at how quickly the development team delivered the first release and were delighted at how useful those first simple BI features were. Now, after 28 iterations, the development team has established a steady rhythm of releasing enhancements plus a few new BI capabilities into production every month and a half. The user community is ecstatic at how fast the development team is able to build useful BI applications for them to use.

As the lead data modeler, Prakash has helped the data warehousing team follow two key principles: First, never implement anything that isn’t necessary to support the current work in progress. Second, all data model modifications must be consis- tent with the reference (conceptual) data model.

ptg6843605 WHAT IS EVOLUTIONARY DESIGN? 145

With Prakash facilitating during iteration zero, the technical team collabora- tively developed a relational reference data model for the integration tier of the warehouse and a multidimensional reference data model for the presentation tier. Originally these were simple whiteboard sketches that gave the team a shared understanding of the database designs. Even though there were lots of unan- swered questions, the team had learned enough during project chartering and planning to feel confident in these high-level conceptual models.

Prakash initially documented these reference data models and published them on the project wiki so that team members could easily reference them during devel- opment. As aspects of the data model get implemented during iterative develop- ment, Prakash updates the wiki documentation to show this detail. This is how conceptual models gradually evolve into logical models. Also, the documentation always matches the physical implementation of the data models.

Each iteration, as soon as the iteration plan is finalized, the team collaboratively reviews the reference models to evaluate how the new user stories will affect the design. Sometimes the new stories don’t require any new data. Other times the new stories require new data that has already been anticipated but is not yet populated in the model. And every now and then a new story requires data that Prakash and the team did not anticipate. When that happens, the team modifies the reference data model to handle the new data requirements, and it evaluates possible side effects or other impacts that the changes will have on the version of the warehouse that is in production.

As the team plans iteration 29, it runs into the third, more complex scenario. This iteration includes the story “As a FlixBuster financial analyst I need to determine cost of sales down to individual transactions so that I can more accurately calcu- late profit.”

The FlixBuster team faces three problems. First, the formula for calculating cost of sales (CoS) is not well defined by the business but is expected to include ele- ments such as studio royalty, handling costs, shipping costs, inventory overhead, loss and damage costs, among other variables. Some of these components are complex by themselves, such as the handling costs. Second, the components that constitute CoS come from a variety of sources, including some syndicated third- party sources. Finally, CoS is an aggregate value covering all sales for some time period. The business logic defining how to allocate CoS down to singular transac- tions must be developed by business experts.

Additionally, the team expects to add the CoS measure to the already populated

F_Transaction fact table, which includes transaction revenue and net profit. This

poses two challenges: how to backfill historical facts with values for the CoS mea- sure (the production fact table contains billions of records) and how the new CoS measure affects the previously developed net profit measure. The team agrees that there may be other issues in addition to these.

The development team shares these challenges with Dieter, the product owner, who brings business users Javier, Beulah, Kari, and Andy, as well as Bob, the busi- ness analyst, and Pete, the VP of finance, into the conversation. The group quickly

ptg6843605

recognizes that completely maturing the CoS story will take a few iterations. So, to simplify the first iteration they agree upon a rudimentary formula for CoS that will give developers a chance to collect the necessary data while business experts develop a more accurate and permanent formula before the start of iteration 30. They also agree upon a simplistic CoS allocation scheme for developers to use until the business experts can agree on a better one. Finally, the group agrees that integrating CoS into the net profit formula is a separate user story. So they write that story and turn it over to the product owner to add onto the backlog. For the time being, CoS will not be included in net profit calculations.

The development team reviews the reference data models, working backward from the star schema in the presentation tier to the relational schema in the integra- tion tier. The F_Transaction fact table already exists, and the team agrees that

this is where the CoS measure belongs. So, a task card is written to reflect this modification of the data definition language (DDL) script for F_Transaction. The

integration schema is more complicated because it involves multiple values, some of which are calculated. However, after some discussion the team agrees to add studio royalty as a new column in the Product _{table; add a} loss_ damage _{field to}

theTransactions table; create a new Studio table, which will be used to popu-

late the D_Studio _{dimension in the presentation schema; and make some other}

data model refinements. The team creates task cards for each of these implemen- tation decisions, and Prakash commits to updating the reference models to reflect the team’s decisions.

Because there is already a version of the data warehouse in production with live data, the team must consider the impact of these new database changes on the existing warehouse and all BI applications that rely on it. The team recommits to using disciplined database refactoring techniques (Ambler and Sadalage 2006). Henry, the database developer with the most experience in database refactoring, commits to reviewing the other developers’ refactoring plans and corresponding code.

The development team quickly reviews the expected work for iteration 29 and agrees that it is reasonable and fits within the team’s capacity. The team formally commits to the iteration plan and begins working with the knowledge that there will be more changes in the upcoming iterations.

This example offers a glimpse of how an Agile data warehousing team takes a highly disciplined approach to evolutionary design to avoid overbuilding the data models while also limiting technical debt and continuously improv- ing the design through careful refactoring. Effective evolutionary data ware- house design has the following benefits (Ambler and Sadalage 2006):

 Minimal waste. By evolving the warehouse design in a just-in-time fashion, you build what is needed, adapt to requirements changes as they arise, and avoid working on irrelevant elements.

ptg6843605 WHAT IS EVOLUTIONARY DESIGN? 147

 Minimal rework. By making small incremental changes in the ware- house design, you avoid sweeping overhauls of the design. Rework efforts serve the purpose of making improvements to, rather than replacements of, existing elements.

 Continuous confidence. An evolutionary approach results in a working system early and the frequent addition of new working fea- tures and enhancements, giving you continued confidence that you are building the right system and are building the system right.  High quality. Refactoring is the discipline of improving your ware-

house design a little bit at a time, continuously.

 Reduced effort. By working only on what you need today, you elimi- nate unnecessary efforts.

Evolutionary design involves the following key developer practices (Ambler 2003):

 Database refactoring to make safe changes that improve quality a little at a time without changing the semantics

 Evolutionary data modeling to ensure that the data model provides exactly what is needed to support the BI applications

 Database regression testing to ensure that new changes don’t break preexisting components of the system

 Configuration management to manage the version history of the entire system as well as the change history of every artifact that makes up the system

 Developer sandboxes to give developers a place to safely experiment with ideas and develop and test their work before integrating it into the system

The following sections offer insight into practices that, when taken together, enable teams to effectively evolve the design of their DW/BI solution. Evo- lutionary design begins with making a series of decisions about the bal- ance between up-front design and evolving design and how a design evolves toward excellence through the use of Agile Modeling, database refactoring, and design patterns. A key constraint to keep in mind during the evolution- ary design process is minimizing technical debt in the design and imple- mentation. Finally, the adaptive architecture section presents an in-depth example of how many of these practices were used to build a complex, hosted DW/BI product for enterprise customers. References are made in these sections to topics such as regression testing, developer sandboxes, and configuration management that are covered in detail in later chapters of this book.

ptg6843605