Análisis B/C

Timeline: Ongoing work of planning objectives, learning activities and small-scale testing should be completed before the late summer (2012) work prior to the preliminary field-test.

The second step of the R&D process involves planning objectives, learning activities, and small-scale testing. This includes stating specific learning objectives, defining the skills, developing a curriculum and timeline (Lorenz & Pichert, 1989).

To establish the components needed to successfully refine the Guidebook with its aim of impacting a larger institutional culture, I plan to ask, through online surveys, the intended audience what they need and want a Guidebook to do for them; I’ll interpret their feedback. The intended audience that I intend to ask is the faculty of CCC. To research the problems and solution sets, I plan to act as a participant observer. This unique perspective will allow me an insider’s view of the processes and personalities in place and afford me opportunities to enact solutions that serve this exact population’s needs. According to Nicol (2010), faculty buy-in will be key to the effort to reform assessment practice. I extracted information from this group about how it approaches the use of text-based instructional materials. Instructional design was informed by this data.

As stated earlier, the Guidebook supported aspects of a larger initiative, i.e., accreditation activities, taking place a CCC. A project taking place in the UK influences this project: Re-engineering Assessment Practices in Higher Education (Nicol, 2010) project (www.reap.ac.uk ). Indeed, the outline for the proposed dissertation project stated herein has been influenced by ideas articulated by REAP. I reviewed the REAP program and used it as model for some aspects for my project.

REAP’s principal author, David Nicol (2010), examined the improvement of assessment practices systemically and systematically; he acknowledges that in order for assessment practice to improve, efforts must be made at all levels of institutional

operations, i.e., from the top-down and from the bottom-up. Practitioners are central to the process described by Nicol; they must better understand and provide high quality summative and formative assessment and feedback as part of everyday practice with students in order to accelerate their learning, and students must participate in the dialogue enabled by feedback loops and change behavior based on assessments of their

performance (Nicol & Draper, 2009). It is assumed that other aspects will form part of the context around project development. As such, they are not part of the dissertation, but will inform the processes.

Objective: At this step, I hope to better understand the instructional needs related specifically to assessment literacy and leadership practice of CCC DCs and more broadly, to understand how this group interacts with other constituent groups, i.e., administration and faculty, in order to establish what these groups value and need / wish to know regarding assessment practice. In addition to surveying colleagues, I plan to gather data from outside the institution from knowledgeable others to examine whether the

experiences of CCC constituents regarding the processes of accreditation-based reform is unique or typical.

How will this be evaluated? This step will be evaluated by determining how well the prototype Guidebook meets the needs of DCs in terms of providing materials that strengthen their assessment literacy and capacity for leading assessment reform (i.e., increases confidence, builds knowledge of assessment theory and balanced practice within the classroom and department, provides practical examples for taking action with faculty). The success of this step will be revealed via analysis of participants feedback

data collected as part of the preliminary field test of the Guidebook and afterwards, once they have used the developed Guidebook . It should be noted that specific evaluation methods will change with the development of the program (Lorenz & Pichert, 1989).

What follows is a conceptual framework that will guide the R&D process, but actual procedures may be adjusted based on emergent needs and circumstances.

Preliminary Research Plan for Step 2 7. DC Surveys

Population: I plan to conduct online surveys of three DCs that are involved in or have expressed in interest in programmatic and/or online courseware assessment. The survey responses will be recorded and transcribed for the purpose of analyzing the answers to look for common themes and repeated elements.

The online survey will consist of the following questions (these were generated in conjunction with Buller (2011) and Chappius, Commodore, and Stiggins (2010).

To be answered before reviewing the Guidebook:

• Has your department as a group engaged in discussions about assessment and accreditation? If yes, describe the number of times this has happened and topics that have been discussed.

• Has your department ever conducted an assessment audit?

• Are there commonly accepted or agreed upon theories or principles of assessment that guide the work of the faculty in this department? If yes, what are the

principles?

• Does your department have a mission statement? If yes, what is it?

• Do you oversee programs that have articulated goals? If yes, which program(s) and what are the goals?

• What sorts of “indicators of quality” do you use to measure program or course quality?

• How do you assess whether the stated goals at the program-level are being met?

• What sorts of assessment activities do faculty in this department tend to favor and/ or commonly use?

• What evidence is currently collected and analyzed that informs the measurement of progress?

• Rate your department’s level of engagement with instructional technology? With Moodle? 1= none; 5= high

• Are you aware of assessment tools that are built into Moodle? If so, which ones?

To be answered after reviewing the Guidebook sections on balanced assessment, ensuring assessment quality and evidence, building communication systems, and strategies for providing professional development:

• Rate the level of assessment literacy within your department as a whole? 1= none;

5 = high.

• What skills do you need to know more about in order to build your confidence with assessment literacy?

• Rate the importance of providing professional development to ensure a foundation of assessment literacy throughout your department?

• Which statement best reflects your opinion of the role of the DC in providing this professional development within the department: The DC should:

1. Take a primary role in leading professional development in assessment literacy.

2. Take a secondary role in leading professional development in assessment literacy, and allow others to primarily lead this.

3. Not get involved and let faculty take the lead for themselves.

• Have you led departmental development in assessment literacy? If yes, what ideas and/or knowledge were important and useful to know?

• What gaps in assessment principles and practices do you perceive need to be addressed across the institution?

• What approaches for assessment reform do you believe could be successful with your faculty colleagues?

• How can the DL leadership help support your departmental efforts at assessment reform?

• What are the barriers to assessment reform at CCC?

To be answered after reviewing the Guidebook sections on Moodle assessment tools:

• Which Moodle assessment tools do think would be useful to you? By your faculty members?

• Does this set of training materials help you use Moodle more effectively? Would you use these materials for your own professional development? Do you think others in your department would find them useful?

2) CCC Administrative Survey

Population: I plan to survey at least three college administrators (The Dean and Associate of the Arts and Sciences division, and the Dean of the Curriculum, Planning, and Research division.)

The survey protocol will consist of these questions:

• Would your division (the college) benefit from increased assessment leadership from DCs?

• What sorts of skills and knowledge should DCs leading faculty training about assessment prioritize in their instruction?

• What are the primary barriers to assessment reform at CCC?

• What are beliefs about assessment that you have heard expressed by your faculty?

• What positive comments about assessment have you heard? What negative comments?

• What sorts of assessment activities does faculty in the division commonly use?

• What approaches for assessment reform do you believe will be successful with your division faculty?

3) Faculty Survey

Population: I plan to survey at least three faculty colleagues.

• Rate the level that SLOs as articulated in the course outline impact your course design and assessment? 1=none; 5= high

• What sorts of “indicators of quality” do you use to measure course quality?

• How do you assess whether the stated goals are being met? If yes, describe how this is done.

• What evidence is currently collected and analyzed that informs the measurement of student progress?

• What sorts of assessment activities do you commonly use in your classroom?

• What approaches for assessment reform do you believe cou;d be successful with your faculty colleagues?

• Rate your level of engagement with instructional technology? With Moodle? 1=

none; 5= high

To be answered after reviewing the Guidebook sections on Moodle assessment tools:

• Which Moodle assessment tools do think would be useful to you?

• Does this set of training materials help you use Moodle more effectively? Would you use these materials for your own professional development? Do you think others in your department would find them useful? Why or why not?

7. Research that engages knowledgeable people from outside the institution Population: I plan to survey at least two people involved in faculty

leadership/leading change processes or accreditation efforts at other colleges/universities (The Director of the Center for Online Learning and Director of Distance Education at Chemeketa CC.)

• What strategies have you found useful for engaging DCs in assessment literacy and leadership?

• What are the most important elements of leadership required for leading faculty to adopt new practices?

• What approaches have you found to be successful in getting faculty to adopt new practices?

• What approaches have you found to not lead to success in getting faculty to adopt new practices?

The project will be conducted over the course of four terms (three terms for R&D and project development and field testing, and one term to write up the results and to defend the completed dissertation.

Step 3: Develop Preliminary Form of the Product (Guidebook Program