INDUSTRIA DE TRANSFORMACIÓN

Following Cameron’s method, I multiplied the “satisfaction/regret” and “source importance” scores for each value flow to produce a value flow score, as shown below in Table 6.

Table 6. Table for determining value flow scores based on attribute scores

Satisfaction/Regret Score A = 0.11 0.19 B = 0.33 C = 0.57 D = 0.98 E = 1 = 0. 11 _{0.01 0.02 0.04 0.06 0.11} 2 = 0. 33 _{0.04 0.06 0.11 0.19 0.32} 3 = 0. 55 _{0.06 0.10 0.18 0.31 0.54} 4 = 0. 78 _{0.09 0.15 0.26 0.44 0.76} So ur ce I m po rt an ce S co re 5 = 0. 98 _{0.11 0.19 0.32 0.56 0.96}

To obtain scores for each value flow, I created Satisfaction/Regret and Source Importance Questionnaires, attached in Appendix C, for every value flow in the model and asked the following individuals acutely familiar with the decadal survey to assign scores to each flow:

• Deputy Director for the Sciences and Exploration Directorate (NASA) • Chief Engineer of Earth Sciences Division (NASA)

• Associate Director for Flight Programs in Earth Science Division (NASA) • Professor of Aeronautics & Astronautics and Engineering Systems (MIT)

• Graduate students in the System Architecture group involved with the project (MIT) Completing the questionnaires for the entire list of value flows requires approximately one hour of time. When completing this exercise, I asked the scorers to think of themselves as the receiving stakeholder for each value flow. Each scorer first answered the Satisfaction/Regret Questionnaire, followed by the Source Importance Questionnaire. We preferred this method, rather than asking them to assign both satisfaction/regret and source importance scores simultaneously for each value flow, for two reasons: (1) assigning all the satisfaction/regret scores together helps keep the scorer’s mind focused on one scoring rubric rather than alternating back and forth between the satisfaction/regret and source importance scales; and (2) if the satisfaction/regret and source importance scores are assigned simultaneously, we found that the

scorer tends to couple the two responses together, as indicated in Figure 31 below. Equally valid, however, are uncoupled scores, as indicated in Figure 32 below. Coupled responses produce less variation among the value flow scores, which removes some of the useful texture in the final results of the value network analysis.

Figure 31. Coupled responses to value flow scoring questionnaires

Figure 32. Uncoupled responses to value flow scoring questionnaires

After the completed questionnaires from each individual were tallied, we used a modified Delphi method with one round of revision to reconcile major differences among scores for particular value flows (Rowe and Wright 1999). In most cases, there were still small discrepancies between the five scores for each value flow, so the final value flow score was determined by taking the average of the five individual scores. Table 7 below shows the combined value flow scores for the Scientists, which include any revisions made after the one round of discussion. The combined scores for all the value flows in the model are presented in Appendix D.

Some of the value flow scores were ignored if they differed significantly from the average. In most cases, this was the result of a misunderstanding of the definition of the value flow. Other flows, such as “science policy reports” from S&T Advisory Bodies, were added to the model after the initial questionnaires were distributed, and only a subset of the initial scorers were able to provide scores for these flows.

Table 7. Combined value flow scores for Scientists

Final Scores from Scorers To:

Stakeholder Value Flow Stakeholder From: _{#1 #2 #3 #4 #5} Combined Value Flow Score Scientists Access to space systems Int’l Partners 0.18 0.31 0.31 0.56 0.54 0.38 Scientists Access to space systems NASA/NOAA 0.44 0.44 0.44 0.56 0.96 0.57 Scientists Funding NASA/NOAA 0.56 0.56 0.18 0.56 0.18 0.47 Scientists Future plans information NASA/NOAA 0.44 0.44 0.31 0.44 0.32 0.37 Scientists Informative content Media 0.18 0.18 0.02 0.10 0.06 0.10 Scientists Skilled workforce Educators 0.56 0.44 - 0.56 0.56 0.62 Scientists Space-acquired data Int’l Partners 0.76 0.54 0.31 0.44 0.76 0.57 Scientists Space-acquired data NASA/NOAA 0.96 0.76 0.44 0.56 0.96 0.74 Scientists Science policy reports S&T Advisory 0.44 0.44 - - - 0.44

We used a separate process to assign scores to the science-related value flows that were split into the six science categories, such as the example shown previously in Figure 17. Rather than ask the questionnaire scorers to evaluate each science category separately for each value flow, we asked them to assign a single score based on the most important type of science to the receiving stakeholder. This prevented the questionnaire scorers from having to assign scores to dozens of additional value flows. For example, there are six types of “science knowledge” that flow from NASA/NOAA to Scientists. We asked each scorer to assign a score for “science knowledge” based on the type of science knowledge that NASA/NOAA would find most important. I call this score the “maximum science” score, as shown below in Table 8.

Table 8. Example of assigning scores to science-related value flows

Final Scores from Scorers To:

Stakeholder Value Flow

From:

Stakeholder _{#1 #2 #3 #4 #5}

Combined Value Flow Score NASA/NOAA Science knowledge Scientists 0.44 0.44 0.44 0.96 0.98 0.65 (max)

For those stakeholders in the model who receive science-related value flows, we assigned each stakeholder a preference of High, Medium, or Low for each science category. These rankings were inferred from policy documents, evidence provided in the literature, and analyses of various information sources, as described in detail in Section 3.2.2. Scores for science-related value flows of medium importance were reduced by a factor of (1/1.7) = 0.59, which corresponds to a one- step drop in the satisfaction/regret attribute scale shown previously in Table 4. Thus, using the example above in Table 8, the medium importance science-related value flows received a score of 0.65(max) x 0.59 = 0.38. Scores for science-related value flows of low importance were reduced

by a factor of (0.592_{) = 0.35, corresponding to a two-step drop in the satisfaction/regret attribute} scale.

The rationale for using this method to reduce the scores is the following: If NASA/NOAA were to assign climate-related science knowledge a score of D (“Its presence is necessary, and I would regret its absence”), it would likely assign a score of C (“I would be satisfied by its presence, and I would regret its absence”) to land-use, solid Earth, weather, and water-related climate knowledge. Likewise, it would likely assign a score of B (“I would be satisfied by its presence, and I would somewhat regret its absence”) to human health-related climate knowledge. This method seemed to work well as an alternative to asking the questionnaire scorers to evaluate each individual science category for every science-related value flow in the model. Table 9 below shows an example of the technique used to assign value flow scores to the individual science- related value flows for Scientists using the maximum science score from Table 8.

Table 9. Technique used to assign value flow scores to science-related value flows

Science Category Stakeholder

Preference Science-related Value Flow

Value Flow Score Human health Low Science knowledge – human health 0.22 Land use & ecosystems Med Science knowledge – land use / ecosystems 0.38 Solid Earth hazards & resources Med Science knowledge – solid earth 0.38 Climate change High Science knowledge – climate change 0.65

Weather Med Science knowledge – weather 0.38

Water resources Med Science knowledge – water 0.38

In this example, the five questionnaire scorers assigned value flow scores for “science knowledge” to NASA/NOAA from Scientists, considering the type of science that NASA/NOAA needs the most. The average of these scores was 0.65, which was designated the “maximum science” score. Separately, stakeholder preference rankings of High, Medium, and Low were assigned to each type of science knowledge for NASA/NOAA. These were obtained from NASA policy documents, as described further in Section 3.2.2. The type of science knowledge that NASA/NOAA requires the most is climate change knowledge, so the value flow of “science knowledge – climate change” received the maximum science score of 0.65.

As mentioned previously, the combined scores for all the value flows in the model are presented in Appendix D. The following section describes the techniques that were used to validate the relative rankings of the some of the value flows within the model.