Mercados

A concept evaluation is conducted in the form of a workshop during a class in a master’s program of IT Management at an Irish university. This course consists of eight participants that have a mixed background in the field of business and IT, both in industry and academia. Five of our participants have over ten years of professional experience in the field of IT. This evaluation takes place at the start of the DSR ABC evaluation phase.

6.4.1. Evaluation Outline

The aim of this evaluation is to explain the EABV AM concept on a high level to a group of potential stakeholders and find out their opinion in terms of certain criteria. Each of the participants has to take on the role of each of the stakeholder types, i.e. EA manager, EA practitioner, and EA customer, and evaluate the following criteria:

 Expected stakeholder satisfaction: This should indicate if stakeholders expect to be satisfied with the EABV AM if it is adopted.

 Utility: Utility, as most prominent DSR evaluation criterion, informs us about how useful our solution is in regard to the business needs.

 Adoption: This criterion is an indicator that informs us about the acceptance and willingness to employ our approach.

 Improvement: With this criterion, we want to capture potential improvements that could be addressed throughout the evolution of the artefacts.

As this is a concept evaluation on a high level, we are interested in different criteria (apart from utility) then we described before. For expected stakeholder satisfaction, utility, and adoption, we ask workshop participants to assign a rating from 1 to 5. The meaning of this rating is outlined in Table 6-3.

Rating Expected Stakeholder

Satisfaction Utility Adoption

1 Dissatisfied Useless No adoption

2 Slightly dissatisfied Slightly useless Unlikely adoption

3 Moderately satisfied Moderately useful Likely adoption

4 Satisfied Useful Very likely adoption

5 Very satisfied Very useful Complete adoption

Table 6-3: Concept evaluation criteria ratings

Moreover, we asked participants to write down three keywords that they would associate with those criteria in a brain storming manner. The aim here is to get an unbiased and spontaneous opinion in terms of how each of the participants understood the context, the EABV AM, every criterion, and the stakeholder type each of them had to assume. For expected stakeholder satisfaction, this means keywords that either increase or decrease it. For utility, keywords that either make it more useful or render it more useless. Finally, for adoption, keywords that either increase the chance or willingness of adoption, or decrease it. Accordingly, this puts more substance and clarification on how that rating was assigned.

In addition, we allow considerable time for questions and answers after the concept presentation and before the actual exercise in order to receive higher quality feedback. We choose to conduct this concept evaluation in such a way because assessment approaches like this are not very well known and are generally perceived as something new by the majority of stakeholders as we found out during numerous meetings and interviews with managers, practitioners, and customers. Consequently, we need to introduce the concept in a short period of time and receive immediate feedback so we can conclude if the concept is valuable and if, for whom.

6.4.2. Results

The results of this concept evaluation workshop reveals the not surprisingly the high approval rate for EA managers in terms of expected stakeholder satisfaction, utility, and adoption. Thereby, expected stakeholder satisfaction accounts on average for 80 percent, for utility and adoption 77.5 percent. This reflects our intent to target the EA managers as prime focus group that benefit from our approach. Consequently, the concept is perceived as satisfying, useful, and very likely to be adopted. The by far most common keyword regarding expected stakeholder satisfaction is “valuable”. Moreover, participants consider our approach as “detailed”. Other keywords stated are “flexible”, “improved decision making”, “focused”, and “improved observability”. For utility, the most common keyword is “useful”. Again, we can find “detailed” on this list. Further keywords are “simplicity”, “applicable”, “improved strategies”, “improved visibility”, “relevance”, and “improved BITA”. Finally, adoption

receives “better strategic support” as most common keyword. Others are “desirable”, “governance support”, “practicable”, and “better resource management”. Regarding improvements, participants express the need to be more specific, better exception handling, and minimize overhead. The more specific improvement in this context was attenuated once we explained more on how we apply our approach on specific projects. Improved exception handling means that participants would like to see more measures on how to deal with errors during assessment. We incorporate such measures in our evaluation phases although there is no specific error routine defined. The actual overhead minimization is considered as an ongoing effort and is a continuous objective for improvement.

From an EA practitioner’s perspective we still receive positive feedback with 65 percent expected stakeholder satisfaction, and 62.5 percent for both, utility and adoption. Accordingly, they find it moderately satisfying, useful, and likely to be adopted. For expected stakeholder satisfaction, the keyword most significant is once again “valuable” followed by “reduced complexity”. Other keywords included “detailed”, “effective”, and “reduced risks”. The following keywords decreased the rating in the opinion of some participants, namely “complex”, “unclear benefits for practitioners”, and “overhead”. Utility is most commonly associated with “practicability”. Again, our approach is considered “detailed”. Other keywords include “improved automation”, “improved planning”, and “applicable on many levels”. One participant deems the approach too difficult to use. For adoption, the EA practitioner’s perspective yields positively influencing keywords such as “clear instruction set”, “practical”, “success”, “governance”, and “reduced risk”. On the downside, it is considered as “difficult to implement”. Improvements expressed by EA practitioners are to provide clear implementation guidelines and more clarification on how practitioners benefit from the approach.

Finally, we have the group of EA customers which score 62.5 percent on expected stakeholder satisfaction, 57.5 percent on utility, and 55 percent on adoption. This translates into customer stakeholders being moderately satisfied, finding it moderately useful, and willing likely to adopt it. This result comes as expected as customers mostly benefit indirectly from such approach for it improves time and quality of service delivery. Directly, EA customers benefit in terms of their project planning, since they have an extended information base to make decisions regarding budget, staffing, and delivery time. When it comes to keywords that positively influence expected stakeholder satisfaction, customers name “useful” and “improved planning” most often. Additionally, “improved ROI” is perceived as being positively affected by our approach. A significant number achieves the keyword “irrelevant” as EA customers are not

clear about how they would be satisfied by this approach. Besides that, “overhead” is named as a factor that negatively influences the assigned rating. Utility yields positive keywords such as “relevance”, “flexibility”, and “simplicity”. Negative keywords are “complex” and “unclear improvement”. Enablers for adoption are “increased information”, “better feedback”, and most prominently “useful data” and “improved planning”. The main inhibitors for adoption are considered to be “overhead” and “time consumption”. When it comes to improvements, EA customers are mostly concerned about improving simplicity and clarifying EABV. In addition, it is suggested to better argue on how customers can benefit from this solution.

Influencing factors are perceived in some cases contradictory by participants within a particular stakeholder group. Hereby, the exception are EA managers which remark positive influencing factors consistently. The rating results are summarized in Table 6-4. It has to be noted, that scores for customers were slightly lower on all three criteria than the ones from practitioners.

Expected Stakeholder

Satisfaction Utility Adoption EA Managers Satisfied Useful Very likely adoption

EA Practitioner Moderately satisfied Moderately useful Likely adoption

EA Customer Moderately satisfied Moderately useful Likely adoption

Table 6-4: Concept evaluation: rating results

In the next Section, we outline the detailed evaluation with our chosen evaluation method within the organizational environment.