The Test of Change with faculty allowed me to return to an aspect of step 1 – Name and Frame the Problem – in the improvement journey by seeing the system at a micro or departmental level but simultaneously seeing the system from the macro or institutional level. Schein (2017) points out the value in studying dimensions of culture particularly “when we try to understand how
organizational cultures are nested in broader macro cultures” (p. 103). It is important to pay attention to this complexity in culture in order to spread and scale the Test of Change. My work with the “J” Department uncovered several overarching themes triangulated through the focus group interview, open-ended survey questions, field notes, and PDSA reflections including: 1) valuing teaching and learning; 2) combatting assessment deterrents; 3) welcoming collaboration; and 4) appreciating the PDSA/micro-change framework. These themes provide necessary perspective both at the micro (departmental) and macro (institutional) level to understand the barriers and the opportunities of implementing the spread of change. I will highlight the themes that emerged through analyzing the focus group and open-ended question survey data.
3.6.3.1 Valuing Teaching and Learning
This theme emerged most prominently appearing n = 48 times including sub-categories of discussing particular course development or enhancements to the curriculum. Faculty talked about focusing on what students are learning, how they are learning it, and how faculty can improve opportunities to enhance students’ learning. During the focus group, one faculty member perhaps summed it up best by saying, “We are a cohesive group so we’re very fortunate in that we’re very supportive of each other both in ways that we all want to be creative in what we teach and how we teach but then also how are we preparing our students in our department curriculum.” Another comment noted, “Knowing that we want students to graduate with certain skills and knowledge has led to us communicating more and more frequently about both the goals and the sorts of assessments we are doing.” This theme aligns with Kotter’s (2012) steps 6-8 – generating short- term wins, consolidating gains and producing more change, and anchoring new approaches in the culture.
3.6.3.2 Combatting Assessment Deterrents
Faculty mentioned deterrents and challenges of executing assessment n = 42 times. The sub-themes included: time constraint/workload issues, preconceived notions about what assessment is or is not, restrictions on academic freedom, and being in a compliance mindset. During the focus group one faculty commented, “In my opinion, assessment is primarily understood as a quantifiable objective not as a qualitative objective and it’s making entrenched faculty twitchy that they may a) lose academic freedom and b) have to scientize (sic) stuff they perceive as not assessable.” Another faculty member said, “If somebody came in tomorrow and said ‘guess what we don’t have to assess ever again,’ I don’t think anybody would say ‘but wait it’s so valuable;’ we’d be like – thank God.” In this theme I noticed Heifetz et al.’s (2009) adaptive challenge archetypes especially the gap between espoused values, speaking the unspeakable, and the gap between behaviors and competing commitments. These data tell me that faculty are still grappling with many of these challenges in regard to SLO assessment and that it is not yet embedded in the culture.
3.6.3.3 Welcoming Collaboration
Faculty expressed the value in collaborating with colleagues n = 25 times. This theme was resounding in the analysis of faculty PDSA notes and research field notes. Most faculty mentioned in one way or another how much they liked being able to have a discussion with a colleague about pedagogy, student learning, and assessment. In answering the survey question what are the opportunities to take advantage of in sustaining a good assessment program in this institution, one
faculty member wrote, “People LOVE to get together and talk and work with each other. People like collaborating. One of the super fun things about this project has been getting to meet with Julia about the small experiments, and talking with my colleagues in my department about theirs.”
In one-on-one meetings with faculty I frequently noted that faculty were talking more among each other about their micro-changes and pedagogy in general. Faculty would often mention a conversation with a fellow “J” colleague about their micro-change and appreciated the feedback they were able to give and receive.
3.6.3.4 Appreciating the PDSA/Micro-Change Framework
Faculty in the “J” Department were not familiar with PDSA cycles prior to our work together in October 2019. After the brief introduction to improvement science and the PDSA cycles, all faculty embraced the tool as a path forward. Faculty mentioned the PDSA framework/structure n = 24 times. One faculty member noted how to use PDSA cycles moving forward, “It was just like now I have almost a fall back for when something’s not working.” Another faculty member said, “…I like structure very much and so this [PDSA tool] has given me some structure and it’s been fairly transformative.” These comments told me that faculty felt rejuvenated by this new possibility of thinking about teaching, learning, and assessment, which I recognized as Kotter’s (2012) step 7 – consolidating gains and producing more change.
3.6.3.5 Researcher Insights During Study Phase
Though the department is small in size in regard to full-time faculty, it is representative of a department size at the institution. It was also interesting to notice that the “J” Department ultimately provided a typical bell-shaped normal distribution in regard to engagement. The charts at the end of each faculty member’s vignette document level of engagement as evidenced by observations, meetings, and other communications going from low engagement to mid- engagement to high engagement. “Understanding the sources of variation in outcomes, and
The variation that I observed in the “J” Department is what I expect to see across Small School at the macro level. There will be levels of excitement and high engagement in taking on this work, and there will be levels of low or no engagement; however, all data collected, no matter the faculty member’s level of engagement, pointed to finding ways for faculty to have the time and space to think more intentionally about student learning and teaching. The project enabled me to revisit step 1 in the improvement journey to see the system differently and think about this smaller system within the context of the larger system.