Cross layer travelling is important, because it highlights complexity uncertainties arising from inter-dependent layers: requirements traveled between hardware and software in virtuous (or vicious) cycles (Figure 8–3). At GridCo, this complexity uncertainty manifested as schedule volatility for dependent components. Adding support for new utility meters to the C&C software system was a great deal of work. For example, adding support for only a few new meters caused the single largest portion of work and uncertainty in the release cycle. Coordinating the timing of completion between layers was problematic. In some cases, support for new HW was contractually obligated, but the HW itself was still being developed. First releasing the HW and then later releasing updated C&C SW to support it was an untenable option. As a
development manger explained, “software and hardware releases coincide because of customer certification requirements.”
Figure 8–3: Cross-Layer Traveling of Requirements
Coordinating development between new HW, FW, and the supporting SW was a major challenge; the resulting complex interdependencies affected requirements at each of these layers. A change in a processor, or the data stored at the HW level necessitated a change in FW that would almost certainly affect the SW layer. However, these dependencies were cyclical: one manager explained, “for FW to complete, [it’s] dependent on SW; for SW to complete [it’s] dependent on FW.” No matter which was completed first, the additional rework was frequently assigned to different teams, possibly in different sprints or even different release cycles. Given this cross-layer inter- dependence, one manager speculated GridCo might “be better off with cross-functional teams.”
For the product development group, reliance on the central information system as well as coordinating documents and standards was essential, but insufficient. For example, at one point a requirements change was made in the C&C SW that required
assumptions underlying SW at an intermediate network device to be challenged. The original intermediate SW had been written as an application, and after the requirements changed at a different level, it needed to be rewritten as a daemon.6
Dependencies were tracked in the central IS, but there was insufficient assignment of responsibility for cross-layer coordination. One participant indicated—in one of the rare times any participant was openly critical of the company—a major frustration with the lack of coordinating project management across hardware, firmware, and software layers, in that there was no orchestration of the critical path between them. This participant requested special care when making these statements, so as not to be seen as attacking any particular individual, indicating, “I’ve said enough to get me into trouble.”
The coordination of HW, FW, and SW was a constant frustration. Development of the SW layer required access to HW. Due to the mismatch between HW, FW, and SW development schedules, the necessary HW was not scheduled to be delivered to the SW team until late in the cycle. The SW managers’ preferred approach was to have a small number of teams work for a longer period building domain knowledge, but volatility in the HW and FW schedules necessitated utilizing more teams over fewer sprints. This created an additional whiplash effect, as the addition of development teams meant not only less productive teams (due to lack of focus and domain knowledge), but also that either teams shared HW prototypes, resulting in slowdowns, or teams being delayed
6 In this case, “application” means a windowed program or executable with the potential for user
interaction. Conversely, a “daemon”, also called a “service” on some operating systems, is a program that runs in the background without interaction from the user.
further waiting for their own hardware. The situation was further complicated by multiple HW iterations during the SW development cycle: to properly certify software for use by customers, it needed to be written and tested against production versions of hardware identical to what customers would receive. In another instance, requirements were changed after they were decomposed to accommodate the unavailability of HW.
Even with the use of documented standards as a vertical coordination mechanism, one firmware manager indicated that when the supporting SW is written before the FW is official completed, it is typically necessary to go back and redo the FW to compensate for instances when the documented interfaces were unclear.
During sprint 6 (about one month after NPD-1), development fell sharply behind schedule. During a heated status meeting, a manager indicated that
dependencies in the FW (complexity uncertainty) necessitated additional resources be allocated to development. Hardware resources that were expected to be available were announced as delayed until at least 60% through the release cycle. A development manager explained in the meeting that if the necessary HW was incrementally delivered from sprints 9 through 12, as anticipated, the necessary SW development work could be accomplished, but there would be no opportunity to find or correct potential critical defects. He continued, “I estimate about 80% confidence of full [development completion] by [NPD-7] if firmware and [hardware] is complete by sprint 9. If that slips one sprint, it’s closer to 50% confidence.” (These resources were not actually delivered until after development on the cycle was complete; work on the dependent requirement sets was done in a minor release, out of cycle.) A different manager indicated that the