Chapter 3 provides greater detail about the research methodology. The chapter includes a description of the research design, population, instrumentation, data collection and analysis, and protection of human rights.
Research Design
This study involved the measurement of faculty self-efficacy beliefs about their capability to facilitate discussion in small seminar classrooms. As mentioned in Chapter 1, the study used a pragmatic approach; hence, the research design reflected use of a mixed methods sequential explanatory design. The tools chosen for data collection included a Likert-type quantitative survey instrument and qualitative semi-structured interviews.
Rationale for Methodology
The mixed methods sequential explanatory (participant selection model) research design provided synergistic and holistic understanding about faculty self-efficacy beliefs in a higher education, small class seminar environment (Creswell & Plano Clark, 2007; Day et al., 2008; Morse, 1991a; Tashakkori & Teddlie, 2002). In addition, the research design achieved greater research sophistication (Johnson & Onwuegbuzie, 2004), answered questions that neither quantitative nor qualitative designs could answer singly, (Tashakkori & Teddlie, 2002), facilitated understanding via a common research language (Johnson & Onwuegbuzie, 2004), provided for a strong empirical support base for
research (Desimone, 2009), and avoided camping on a specific theory or method of research. It acknowledged the contributions from multiple perspectives across literature, theories, and research methods (Weis et al., 2009).
Renewed emphasis on research rigor and a call for increased sophistication of evidence in support of research conclusions led to qualitative data that provided stronger evidence for those conclusions (Creswell & Plano Clark, 2007; Johnson & Onwuegbuzie,
2004). One of the strengths of mixed methods research design was that it allowed the “research to develop as comprehensively and completely as possible. When compared with a single method, the domain of inquiry is less likely to be constrained by the method itself” (Morse, 2003). However, Morse also pointed out that the strengths of the mixed method design could also be its weaknesses in that methodological purists considered the quantitative data as thin and the mixed method design as less rigorous (Creswell, Plano Clark, Gatmann, & Hanson, 2003).
Tashakkori and Teddlie (2002) identified three challenges for researchers engaged in mixed methods research. First, the design used in this study required that the
questions, data analysis, and research sequence align with the mixed methods sequential explanatory research design. Second, conceptual consistency involved alignment of study inferences with social cognitive theory and teacher self-efficacy theory. Third, cross-reference consistency necessitated that answers to research questions be connected to one another, aligned with current theory, and consistent to other studies. Tashakkori and Teddlie (2002) stated, “Strategies for integrating these inferences are more critical than the dominance of one [quantitative or qualitative] approach over another” (p. 41). If the study lacked validity and consistency, the methodology that dominated the study was unimportant.
In addition to the considerations already mentioned, other factors affected selection of this research design. These factors consisted of timing, weighting, and mix of the data. Timing in the research design had less to do with the collection of data and more to do with analysis and interpretation of data, yet, according to Creswell and Plano Clark (2007), the data collection and analysis were interrelated. The mixed methods sequential explanatory design used quantitative data from phase one to develop questions plus the purposeful participant sample matrix essential for data collection in phase two.
Weighting in a mixed methods study indicated the importance of the data and reflected the worldview of the researcher. A positivistic researcher tended to weight quantitative data more heavily, while someone with a naturalistic worldview might have emphasized qualitative data. This study was pragmatic; consequently, the importance of the data emerged from the types of questions asked (Creswell & Plano Clark, 2007; Hanson, Creswell, Creswell, Plano Clark, & Petska, 2005) rather than from focus on the
method. However, the emphasis in the current research was on the qualitative data from participant interviews that provided context for how the independent variables affected faculty self-efficacy beliefs.
Research Questions
Quantitative
The primary quantitative research question was, “What were faculty self-efficacy beliefs about their capability to facilitate discussion in small group seminars?” The secondary questions appear as null hypotheses.
H10: There is no relationship between the dependent variable faculty self-
efficacy beliefs about facilitating discussion in small group seminars and the independent variable gender.
H20: There is no relationship between the dependent variable faculty self-
efficacy beliefs about facilitating discussion in small group seminars and the independent variable age.
H30: There is no relationship between the dependent variable faculty self-
efficacy beliefs about facilitating discussion in small group seminars and the independent variable ethnicity/race.
H40: There is no relationship between the dependent variable faculty self-
efficacy beliefs about facilitating discussion in small group seminars and the independent variable education level/degree.
H50: There is no relationship between the dependent variable faculty self-
efficacy beliefs about facilitating discussion in small group seminars and the independent variable teaching experience.
H60: There is no relationship between the dependent variable faculty self-
efficacy beliefs about facilitating discussion in small group seminars and the independent variable academic title.
H70: There is no relationship between the dependent variable faculty self-
efficacy beliefs about facilitating discussion in small group seminars and the independent variable leadership position.
Qualitative
The primary qualitative research question was, “How did faculty perceive themselves as facilitators of discussion in the classroom?” The interview protocol questions in Appendix F gathered the data to answer the research question.
Variables
The dependent variable used in this study was faculty self-efficacy beliefs. Independent variables were gender, age, ethnicity/race, education level, teaching experience, academic title, and leadership position.
Population and Sample
The quantitative phase one targeted the faculty of the U.S. Army Command and General Staff College via an Internet-based survey instrument. Phase two involved the researcher’s decisions about a deliberate sample of the participants who responded to the quantitative survey.
Quantitative
Faculty at the U.S. Army Command and General Staff College (CGSC) were the target population for the study. Faculty populations were located at the Fort
Leavenworth, Kansas main campus site (N = 369) and three satellite campuses at Fort Belvoir, Virginia (N = 24); Fort Lee, Virginia (N = 12); and Fort Gordon, Georgia (N = 12). Another satellite campus commenced teaching in January 2010 but was not included in this study because the CGSC had not completed hiring of faculty for that site. The CGSC Quality Assurance Office granted permission to survey and interview the faculty population at the Fort Leavenworth, Kansas, main campus and the three satellite
campuses (see Appendix A).
Chapter 2 provided an explanation of the CGSC structure and the Intermediate Level Education (ILE) program at the main and satellite campuses. Appendix G shows the faculty demographics for the main campus as reported in the September 2009 Officer Professional Military Education Program faculty report. This report identified 353 faculty members at the main campus, of which 240 (68%) were Department of the Army civilian employees (Title V and Title X), and the remaining 113 (32%) were military
officers in the rank of major or lieutenant colonel. Only 10 (3%) of the 353 faculty members were female and 22 (6.2%) self-identified as People of Color. Faculty members identified by the title of instructor were 154 (43.6%) and those titled assistant professor were 140 (39.7%). Seventy faculty members had two years or less of higher education teaching experience (19.8%) and 66 faculty (18.7%) had between two and five years of experience.
Qualitative
Faculty members who voluntarily responded to the quantitative survey in phase one of the study comprised the target for qualitative data collection. The purposeful sample matrix in Table 3.1 identified demographic characteristics that helped guide the selection of individuals for the interviews. Faculty members had an opportunity to volunteer for the interview process, and of those who provided contact information, 12 eventually followed through and completed interviews. One faculty member elected to complete the interview using an e-mail message. The interview sample resembled the faculty at CGSC as closely as possible, in terms of both demographics and locale.
Instrumentation
Quantitative
Theoretical foundations for the current study included social cognitive theory and the theory of self-efficacy (Bandura, 1997). Instruments used in the study aligned with theory to avoid confounding constructs, terminology, and developing measures that produced questionable data. The Faculty Self-Efficacy Beliefs Scale (FSEBS) used a 5- point Likert-type scale to measure faculty self-efficacy beliefs about their capability to facilitate discussion in small group seminar classrooms (see Appendix H).
Instrument Selection
Chapter 2 provided a detailed description of the TEBS-S four-phased
development process. That process resulted in a valid and reliable instrument consistent with Bandura’s (1977, 1997) theory of self-efficacy. The Dellinger (2008) TEBS-S short
form had little correlation to the concept of teacher efficacy as measured in other traditional instruments incorporating the RAND items.
Even though the TEBS-S was valid and reliable, its usefulness in higher education contexts was limited because Dellinger and associates intended it to measure teacher self- efficacy in grades K-12. Rather than the earlier 2001 instrument version, Dellinger recommended (see Appendix I) using the latest 2008 TEBS-S version (Appendix J). Consequently, the 31-question 2008 TEBS-S version became the basis for developing the FEBS instrument used to measure higher education faculty self-efficacy beliefs in the present study. Modifications to the instrument involved rewording questions to reflect the type of work at higher education institutions. The modifications to Dellinger’s TEBS-S (2008, p. 764) are in Appendix K
Internet Instrument Development Procedures
The CGSC QAO served as the access point and quality control for all survey instruments administered to staff, faculty, and students in the CGSS. This office approved the research survey on June 16, 2009 (Survey Control Number 09-080; see Appendix B). The tool chosen to develop the web-based survey instrument was Inquisite Web Survey Builder (9.0). The CGSC QAO reviewed and then published the instrument. Publishing the instrument referred to placing the survey on the Internet through Inquisite, with data storage occurring behind a U.S. Army controlled firewall that prevented
unauthorized access. Faculty members at the CGSS were familiar with the Inquisite type of web-based survey instruments. Computer technology within CGSS supported these instruments. Inquisite features included
• Survey review and editing functions;
• Automatic blinding of the researcher from the list of participant names; • Automatic confidentially (prevention of matching participants to responses); • Safeguards preventing multiple survey submissions by the same user,
forwarding the website access to unauthorized users, or changing the survey; • Tally functions of surveys sent out, undeliverable surveys, and non-
responses;
This researcher received training, provided by CGSC QAO, and developed the online survey instruments. The initial development required four hours and subsequent edits about one to two hours each. Inquisite software supported piloting the survey instrument, identification of Internet or survey functionality issues, and user
understanding of the instructions. Expert Panel Review
Developing an online survey was not an isolated event. Online survey development was part of a holistic fielding process that was important for achieving acceptable response rates and reducing survey errors (Schonlau, Fischer, & Elliot, 2002). Survey question development was essential for generating useful data. Poorly worded questions that were ambiguous and not representative of the content might have limited the usefulness of the data and jeopardized the validity and reliability of the study. A panel of six experts reviewed the initial adaptation of Dellinger’s (2008) survey instrument.
Three criteria were important for selecting expert panel members to review the FEBS. The criteria included (a) teaching experiences in a higher education institution, (b) a doctorate degree or engaged in a doctorate program, and (c) experience developing or administering an online survey. All six individuals selected to review the survey instrument had experience teaching in higher education, with four individuals having taught at multiple institutions. Three individuals had prior experience as CGSS faculty. All individuals possessed a doctorate degree or were engaged in a doctorate program. One individual had published portions of his dissertation research in professional
journals. Five of the individuals had experience in developing survey instruments. Two individuals had extensive experience in psychometrics. Two individuals had developed online survey instruments.
The expert panel received a web link to a copy of the survey instrument. Instructions to panel experts were to
1. Read the instrument and annotate grammatical problems, unclear instructions, and poorly worded questions.
2. Navigate through the instrument to ensure logical flow of the survey and ease of use.
3. Attempt multiple response sequences to ensure the instrument would direct individuals to the appropriate demographic questions, as well as to the survey itself.
The panel found some spelling and grammatical errors, poor word choices for some questions, and issues with the “finish” and “submit” selections at the end of the instrument. They recommended changes categorized under four headings: overall
instrument format and flow of questions, grammar, question clarity and choice of phrases or wording, and color and graphical layout of the survey. The expert panel members provided feedback that resulted in corrections made prior to the pilot.
Pilot
Following implementation of the recommended changes from the expert panel, the FSEBS underwent a pilot test at the U.S. Army Management Staff College (AMSC). The Dean of Academics authorized the pilot on June 30, 2010 (see Appendix E). The faculty (n =57) at AMSC resembled, as closely as possible, the faculty at the CGSS in terms of demographics. The pilot administration was online through the CGSC Inquisite software. Because the AMSC faculty population was small, the Dean of Academics had concerns regarding anonymity of the participants. Consequently, no demographic data collection took place in the pilot. Further details about the pilot appear later in this chapter.
Several individuals who participated in the pilot provided feedback by e-mail about several issues they noted with the instrument. There was one typographical error and at least two cases in which the “submit” option appeared to have failed. Because the “submit” option appeared to work on the survey test versions, it was not clear why a problem arose with the pilot survey. This problem could have affected the response rate, in that some participants might have exited without sending the survey to the database.
Follow-up with CGSC QAO determined that all submissions were successful, even though the submit option appeared unsuccessful. Following correction of the typographical error, the instrument was loaded into Inquisite, along with all the notification, follow-up, and survey closeout options.
Qualitative
The study used a semi-structured interview protocol consisting of nine open- ended questions with an average interview time estimated at 30 minutes. The mixed methods sequential explanatory design provided an opportunity to refine the interview questions.
Interview Development Procedures
The researcher developed an initial array of questions designed to solicit
information about participant experiences facilitating discussion in the classroom. Other initial questions asked about classroom environment, preparation for discussion,
facilitating discussion, drawing out student deployment or combat experiences, and thoughts about the institution’s role in discussion teaching. Reflection on the theoretical basis for the study ensured the questions aligned with self-efficacy (Bandura, 1977, 1997), discussion teaching (Brookfield & Preskill, 2005; Isaacs, 1999), and a motivational learning environment (Wlodkowski, 2008).
Expert Panel Review
In addition to reflection on the theoretical constructs, the researcher asked the panel of experts who had participated in the Internet survey development process to review the interview questions. These individuals provided feedback about sentence structure and clarity. The questions underwent revision, including interview protocol updating.
Pilot
Three of the interviews were used as a pilot and helped refine the interview process. The pilot interviews identified the following items, resulting in changes to the procedures. Two Faculty of Color and one Caucasian faculty member participated in the protocol pilot. The first interview lasted 41 minutes. Since the researcher originally told participants that interviews would not last longer than 30 minutes, a watch set next to the microphone served as a timer. The digital recorder and the watch were in clear view of the participants.
On the day of the interview, the researcher contacted the participant to confirm the time and location of the interview. This step became necessary when the individual participating in the second interview forgot about the meeting and had to leave before the interview began. Participants received interview questions prior to the interview.
Participants felt that having time to reflect on their experiences produced richer information and gave depth to the answers. One of the pilot participants felt restricted and slightly disoriented when he tried to think back on previous experiences. Sending the questions to the participants ahead of time gave them an opportunity to reflect on their practices in the classroom.
After the first interview, all the participants received an advance copy of the protocol. Participants seemed to be honest and candid when answering the protocol questions. The pilot process allowed testing of the transcription and member-check process. Additionally, the pilot provided an opportunity to use MAXQDA 10 to code data. After the interview, the researcher made field notes about the interview. Recorded initial thoughts about the interview proved to be an effective method for capturing the sense of the session and impression of the interview environment. Because the process was successful during the pilot, it followed all interviews. Downloading the interviews to a compact disk and removal from the digital recorder was a further step in ensuring security and confidentiality. A copy of the interview protocol is in Appendix F.
Sampling Procedures
Quantitative
The CGSC faculty (N =417) comprised the target population for quantitative data collection. The entire population was accessible via the Internet and e-mail.
Consequently, no sampling procedures were necessary for administering the FSEBS to the CGSC faculty.
Qualitative
Creswell and Plano Clark (2007) stated that researchers purposefully select “individuals and sites that can provide the necessary information . . . Researchers
key concept being explored” (p. 112). Morse (1991b) suggested that participants willing to participate in the interview process had to meet several selection criteria. He wrote that participants had to willingly participate, be able to provide information, and have experience with the phenomena.
The purposeful selection sampling method identified the individuals for the interviews. Those identified met the informational needs of the study (appropriateness) and had experience facilitating discussions in the small seminar classroom
(knowledgeable). The individuals agreed to talk about their experiences (primary and secondary selection criteria or control). Finally, the participants provided saturation of the topic (information adequacy). The number of interviews recommended by Creswell and Plano Clark (2007) was 4 to 10. The current study included 13 interviews and included one that was submitted by e-mail. The interviews provided an adequate number to address the research questions and achieve saturation of the data.
One of the reasons for selecting the mixed methods sequential explanatory design was that phase one quantitative data collection allowed the researcher to identify
participants for follow-up interviews in phase two. The data, along with the preset dimensions (independent variables and location), determined the purposeful sample matrix (Morse, 1991a). Procedures used for the purposeful selection of faculty members to participate in the Phase 2 are in Table 3.1. The matrix in Table 3.1 shows an ideal mix of interviews, accounting for demographic variables at all CGSS campus locations. The matrix required adjustment, depending on how many individuals agreed to the interviews. The bridge between the study phases one and two allowed
• Review of quantitative data analysis to determine unique instances that justified further investigation,
• Adequate coverage of all CGSC campus locations,