(1) Effectiveness
The effectiveness is simply the ability to evaluate exercise participants’ performance with respect to the FEMA Qualification System and core capabilities. The ideal is a program that provides individuals and teams both immediate feedback when appropriate, as well as an overall performance assessment. The game should also provide different levels of challenges for teamwork as a whole to improve performance. For example, a catastrophic flood may result in 10,000 survivors needing to be rescued within 72 hours. The game could provide feedback regarding attainment of the goal. Each time the game is played, participants can be challenged to rescue more people, or in different locations, or under different levels of damage than in the last event. In short, games provide an ideal platform for performance effectiveness feedback. In order to improve the effectiveness of the cognitive transfer, the prototype game is based to the maximum extent possible on actual disaster planning factors. For example, FEMA estimates each survivor needs three liters of drinking water and two Meals Ready to Eat (MRE) per day. Even unpredictable factors such as the cost to rent material handling equipment (MHE) have been calculated based on currently advertised rates. Beyond improving teamwork and leadership under pressure, players will internalize actual FEMA planning factors by playing the game.
(2) Cost
The cost of implementing a gaming solution could be as low as $100,000 and take approximately six months. This is a fraction of the $800,000 that FEMA’s National Level Exercise Program was cut in Fiscal Year (FY) 2013 over FY 2012.65 While it is unknown whether the downward trend in exercise budget will continue, any policy decision will obviously have to maximize the resources. A serious game should provide
65 Department of Homeland Security, “Federal Emergency Management Agency Salaries and
not only greater effectiveness, but also greater fiscal efficiency. This criterion is largely one of judging if a MMO game for FEMA exercises is economically feasible.
(3) Realism
The game requires a high degree of realism in order to provide the stressful environment required to prepare people for large-scale disasters. However, the real world does not have to be replicated precisely as long as the requisite cues are available. Since it is a command and control environment, the players do not need to maneuver through disaster neighborhoods like in a first person shooter game. Similarly, the telephone doesn’t necessarily need to ring. If players wear headsets, they may receive calls through the computer system rather than actual telephone. The game environment does need to present authentic cues for operations in terms of communication, coordination, and information gathering difficulties. The game must also accurately model the time element in attempting to quickly support disaster survivors and responders. If it would take 48 hours and $100,000 to move a search and rescue across country, then it should take that amount of game time and reflect costs, transportation challenges, etc. The intent is to stimulate the players through information overload. If those cues can be provided through a reasonable simulation, then a great deal of cost and complexity can be eliminated. D.H. Andrews, Colonel Lynn A Carroll, and Herbert H. Bell referred to this as the 60% solution in, “The Future of Selective Fidelity in Training Devices.”66 Andrews, Carroll, and Bell found that in such simulations as for tank crews, about 60% fidelity was sufficient to achieve the training objectives for multi-crew coordination.67 The bottom line is even if 100% fidelity is possible, it would likely be cost prohibitive and unnecessary. FEMA would need to balance cost and fidelity to arrive at a suitable solution.
66 D.H. Andrews, Colonel Lynn A. Carroll, and Herbert H. Bell, Armstrong Laboratories, “The Future
of Selective Fidelity in Training Devices,” (Final Technical Report, Air Force Material Command, Brooks AFB, TX: 1996), 2, accessed February 7, 2015.
http://oai.dtic.mil/oai/oai?verb=getRecord&metadataPrefix=html&identifier=ADA316902.
(4) Is the Solution a “Good Game”?
Ultimately, the proposed game must be good. FEMA employees should want to play the game for it to be effective. Lewis Pulsipher provides some criteria for what constitutes a good game.68 The overriding concern is gameplay. Other criteria relevant to FEMA are:
1. The game is “player centric.”69 The game design must focus on the players and not the designer. This will be a challenge since the design will seek to impart training defined by guidance while keeping players in mind. Due to the limited scope of this thesis, this factor would have to occur later during player testing.
2. The game has “interesting challenges.”70 Since regional-level disaster
response occurs in a command and coordination center, there are no exciting physical challenges as in a first-person shooter game. The challenge will be more cerebral. The concept will be to stimulate
excitement by placing players in a time-constrained environment, just like the real world. Players will, in fact, either help or leave stranded tens of thousands of disasters survivors in 72 Hours.
3. The players should have a “choice”71 and ability to affect the outcome of the game. Fortunately, this is the very essence of training decision makers. The game will ideally be a “sandbox” that will allow decision makers to try different strategies with each game playing experience. In 72-Hours, players will quickly see the results of their decisions on the population in need.
4. The game will have to foster “interaction with others.”72 Since this is the
essence of disaster response, the game will need to have a realistic collaborative environment. Each role will have different capabilities and authorities that align with actual Regional Incident Support Manual roles and responsibilities. This will foster a collaborative environment.
5. The game should have a high “activity”73 level so players are not
spending a great deal of time waiting for their turn. In order to accomplish this, the game will be a sandbox that will allow simultaneous play while 68 Ibid. 69 Ibid. 70 Ibid. 71 Ibid. 72 Ibid. 73 Ibid.
following 10-minute turns based on an actual RRCC operational tempo. Players will actually simultaneously engage in their respective roles and then come together at the end of each turn. There is minimal to no condition of players sitting around watching others.
6. The game should have great “replayability.”74 This occurs when the game presents new challenges and outcomes each time it is played. A game that feels the same every time you play it will not be successful. 72- Hours is intended to be a game that will be extremely difficult to win. This is similar to the immense challenge of actually stabilizing a disaster within 72 hours. In the game, players can use different strategies and actions to obtain a different outcome with each round of play.
7. The game should have “memorability.”75 Memorability occurs when the play is significant enough that players talk about it after the fact. Per the literature review, a serious game actually has the ability to stimulate similar emotions as real world events.
8. The game should have good “play balance.”76 A good game is fair and
rewards players as they increase their skills. A fair game is one in which players can win in a predictable manner. For example, many children’s games, like Candy Land, are based on pure chance and luck of the draw. Games like chess are both fair and rewards players who improve their skills. Due to the extreme time constraints levied on players in 72-Hours, they will undoubtedly improve their score as they increase their positional knowledge and skill.
9. The game should have “multiple ways to win.”77 Games like tic-tac-toe have a winning strategy. The player who goes first can take the center square and always win or draw. A good game, on the other hand, provides for multiple strategies to win. As stated earlier, 72-Hours would have a sandbox feel and allow players to try different strategies. While there may be definite wrong ways to respond to a disaster, there really are no clear right ways. Each is unique and players can respond accordingly.
10. The game should have a good “control scheme/user interface.”78 How easy is the game to play? The proposed game will rely on players knowing their actual real-world roles. However, it should provide an easy interface. Furthermore, the game should be easy for game masters (exercise control specialists) to tailor and provide exercise injects. Every disaster is a new, 74 Ibid. 75 Ibid. 76 Ibid. 77 Ibid. 78 Ibid.
unique experience. Therefore, a training environment repeatedly focusing on the same narrow skills would not be sufficient. Fortunately, C.S. Green and D. Bavelier found that, “the true effect of action video game playing may be to enhance the ability to learn new tasks.”79 In other words, action video games actually teach people to learn. Game-based training can provide benefits well beyond simply learning how to win the specific game.
(5) Adoptability
The game must be designed for use by typical players based on their information technology. For example, a game based on consoles or specific devices would be difficult to adopt across the spectrum. However, a PC based game that uses standard Internet interface software will enable good adoptability.
Table 3 is a matrix of matrix of the five evaluation criteria that should be considered for 72-Hours. In order to be successful, the game should score favorably across the five criteria.
Table 3. Evaluation Matrix
Effectiveness Cost Realism Good Game Adoptability Ability to evaluate performance Feasible within budgetary constraints Sufficiently stimulate players similar to an actual disaster A “good game” as defined by sub-criteria Acceptable training platform for leaders and players