Findings emerged throughout the research project about the nature of systems thinking and how to measure it. Results supported the findings of Sweeney and Sterman (2000) that stated few individuals, even within high educational levels of math and science, have the capacity to utilize basic systems thinking skills. Initial results also supported the Richmond (1993) statement that systems thinking is difficult to teach and requires a culture of learning to be fully grasped. Students that showed a clear grasp of systems thinking stood out among their fellow classmates. They were able to completely break down a complex system, evaluate how it behaves, and make decisions about what they saw. The typical student understood and utilized pieces of systems thinking skills, however they had difficulty merging these skillsets to evaluate complex questions.
The research instrument, intervention, and design were found to influence the evaluation of systems thinking. The theme of the instrument proved to be important in determining the size of the answer pool, and changed students’ background
understanding. An ideal theme has clear spatial constraints to limit the breadth of proposed answers, and should be a topic the participants have experienced but does not give an advantage to expert knowledge.
The time between the pre-survey and post-survey also influenced instrument results. Taking the instrument twice within a short time frame caused students to copy responses verbatim or reduce the effort they gave on filling out the questions in the post-survey. It is recommended to maximize the time between surveys, and in short time windows provide two versions of the instrument each with a different system to evaluate.
Constraining the instrument questions was one of the most significant changes made to the research instrument. Unconstrained questions placed emphasis on quantity rather than quality of responses. This compounded the issue of effort, giving students the advantage for longer answers. Providing constrained questions results in a smaller answer pool, equity between responses, and shortens the instrument completion time.
3.5 Conclusion
This research project investigated using the systems thinking research instrument developed by Evagorou’s et al. (2009) to measure changes in undergraduate student systems thinking capacity with regards to identifying system elements, spatial
boundaries, temporal boundaries, and subsystems. Tests occurred in three undergraduate courses at the University of Nebraska – Lincoln. The preliminary results showed that the research instrument was able to measure undergraduate students’ systems thinking capacity. Three iterations of changes were made to the instruments question format with the most notable changes being adding constraints to the responses and reducing the time requirement.
Several factors were identified to influence test results including the time between the pre-survey and post-survey, question theme, and participant background information.
The intervention given to teach students about systems thinking was not extensive
enough to see any change. The third course was still in progress and greatly expanded the intervention length. The results from this course will provide evidence on how long it takes to see changes in student systems thinking capacity, and will potentially set the stage for a longitudinal research study on student systems thinking understanding.
References
Arnold, R. D., & Wade, J. P. (2015). A definition of systems thinking: A systems approach. Procedia Computer Science, 44, 669–678.
https://doi.org/10.1016/j.procs.2015.03.050
Briscoe, P. (2016). Global Systems Thinking in Education to End Poverty: Systems Leaders with Concerted Push. CCEAM, 43(3), 5-19.
Castelle, K. M., & Jaradat, R. M. (2016). Development of an instrument to assess capacity for systems thinking. Procedia Computer Science, 95, 80-86.
https://doi.org/10.1016/j.procs.2016.09.296
Davidson, K.M. and Venning, J. (2011). Sustainability decision-making frameworks and the application of systems thinking: an urban context. Local Environment, 16(3), 213-228. https://doi.org/10.1080/13549839.2011.565464
DeVane, B., Durga, S., and Squire, K. (2010). ‘Economists who think like ecologists’:
Reframing systems thinking in games for learning. E-Learning and Digital Media, 7(1). http://dx.doi.org/10.2304/elea.2010.7.1.3
Evagorou, M., Korfiatis, K., Nicolaou, C., & Constantinou, C. (2009). An investigation of the potential of interactive simulations for developing system thinking skills in elementary school: a case study with fifth‐graders and sixth‐graders. International Journal of Science Education, 31(5), 655-674.
https://doi.org/10.1080/09500690701749313
Frank, M. (2006). Knowledge, abilities, cognitive characteristics, and behavioral competences of engineers with high capacity for engineering systems thinking (CEST). Systems Engineering, 9(2), 91-103. doi:10.1002/sys.20048
Godfray, H. C. J., Beddington, J. R., Crute, I.R., Haddad, L., Lawrence, D., Muir, J. F., . . . Toulmin, C. (2010). Food security: the challenge of feeding 9 billion people.
Science, 327(5967), 812-818. doi:10.1126/science.1185383
Hmelo-Silver, C. E., & Azevedo, R. (2006). Understanding complex systems: Some core challenges. The Journal of the Learning Sciences, 15(1), 53–61.
https://doi.org/10.1207/s15327809jls1501_7
Hopper, M., & Stave, K. A. (2008.) Assessing the effectiveness of systems thinking interventions in the classroom. Proceedings of the 26th International Conference of the System Dynamics Society. July 20-24, 2008, Athens, Greece. Retrieved from https://www.systemdynamics.org/conferences/2008/proceed/index.htm Kay, J.J. and Foster, J. (1999). About Teaching Systems Thinking. In Savage, G. and
Roe, P (Eds.), Proceedings of the HKK conference, 14-16 June, 1999, University of Waterloo, Ontario. 165-172.
Kunsch, P.L., Theys, M. and Brans, J.P. (2007). The importance of systems thinking in ethical and sustainable decision-making. Central European Journal of Operations Research, 15(3), 253-269. DOI 10.1007/s10100-007-0029-8
Levine, D., Smidt, R. K., & Ramsey, P. P. (2001). Applied statistics for engineering and scientists: Using Microsoft Excel and MINITAB. Prentice Hall.
Mingers, J. and White, L. (2010). A review of the recent contribution of systems thinking to operational research and management science. European Journal of
Operational Research, 207(3), 1147-1161.
https://doi.org/10.1016/j.ejor.2009.12.019
Mitchell, G. (2005). Bertalanffy’s General Systems Theory. Retrieved from http://www.mind-development.eu/systems.html
Richmond, B. (1993). Systems thinking: critical thinking skills for the 1990s and beyond.
Systems Dynamics Review, 9(2), 113-133. https://doi.org/10.1002/sdr.4260090203 Sweeney, L. B., & Sterman, J. D. (2000). Bathtub dynamics: initial results of a systems
thinking inventory. System Dynamics Review, 16(4), 249-286.
Doi:10.1002/sdr.198
Appendix A
Systems Thinking Instrument Version 1 – Course A
Systems Thinking Survey
The following survey is designed to help you understand and build skills in Systems Thinking.
Let’s start with a definition of a system. A system is a set of elements that interact with each other. For example, a forest may be considered as a system and it may have
elements such as plants, insects, animals, reptiles, soil, rocks, water, etc. A sub-system is a smaller system that is part of a larger system. For example, a tree, pond, etc. may be considered as a sub-systems of the forest.
Instructions: Answer the following questions or problems to the best of your knowledge.
Feel free to draw diagrams if it helps you in answering a particular question. There is no right or wrong answer. Try your best!
Q1: Name the various elements that make up a food system.
Q2: I would like to study the food system and how it operates. On the paper, draw a food system and mark the area that you think I should study.
Q3: Look at the graph below that shows the cost of climate disasters in the US on a yearly basis. The X axis on the graph shows the time in years and the Y axis shows the damage amount in billions of dollars. Notice how the damage amount varies with time
Q3 (i): How often will you like to assess the change in damage amount due to climatic events, knowing that each time you collect data costs you 1 million dollars and that damages above 25 billion influences the profitability of your regions farming operations?
Q3 (ii): Explain why?
Q4 (i): In your own words explain what do you understand by the term “system”?
Q4 (ii): In your own words explain what do you understand by the term “sub-system”?
Q4 (iii): Can you name any other smaller subsystems within the food system?
0 25 50 75 100 125 150 175 200
1980 1985 1990 1995 2000 2005 2010
Damage Amounts (Biilions $)
Time (Years)
Systems Thinking Instrument Version 2 – Course B
Systems Thinking Survey
Instructions: Answer the following problems to the best of your knowledge. Feel free to draw diagrams if it helps you in answering a particular question. There is no right or wrong answer. Try your best!
Think of your favorite Pizza Place and answer the following questions.
Q1: Name the various elements that make up a pizza place.
Q2: I would like to study the pizza place as a system and how it operates. On the paper, draw a pizza place and MARK the area that you think I should study.
Q3: Look at the tables below that shows the fluctuation in price of cheese. Notice how the price of cheese and pizza varies with the time.
Q3 (i): Which table would you choose to use in order to decide if the price of the pizza is influenced by the price of cheese?
Q3 (ii): Explain why?
Q4 (i): In your own words explain what do you understand by the term “system”?
Q4 (ii): In your own words explain what do you understand by the term “sub-system”?
Q4 (iii): Can you name any other smaller subsystems within the Pizza system?
Systems Thinking Instrument Version 3 – Course C
Systems Thinking Survey
Instructions: Answer the following problems to the best of your knowledge. Feel free to draw diagrams if it helps you in answering a particular question. There is no right or wrong answer. Try your best!
Think of your favorite Pizza Place and answer the following questions.
Q1: Using 6 words, name the primary system elements that make up a pizza place.
1. ___________________ 2. ___________________
3. ___________________ 4. ___________________
5. ___________________ 6. ___________________
Q2: I would like to study the pizza place as a system and how it operates. Circle which elements you listed in Q1 you think I should study. *Note* you can choose more than one.
Q3: Name 3 sub-systems within the pizza place?
1. _________________ 2. __________________ 3. _________________
Briefly describe why these are sub-systems:
Q4: Look at the graph below that shows the cost of climate disasters in the US on a yearly
basis. The X axis on the graph shows the time in years and the Y axis shows the damage amount in billions of dollars. Notice how the damage amount varies with time
Q3 (i): How often will you like to assess the change in damage amount due to climatic events, knowing that each time you collect data costs you 1 million dollars and that damages above 25 billion influences the profitability of your regions farming operations?
Q3 (ii): Explain why?
0 25 50 75 100 125 150 175 200
1980 1985 1990 1995 2000 2005 2010
Damage Amounts (Biilions $)
Time (Years)