Once you incorporate a ScoL finding into class and design a study to test if it works, you then need to pay close attention to factors that can influence whether the innovation ‘worked.’ Social science
research nicely alerts us to the fact that the changes we observe in our students’ learning may be due to a whole host of factors. One of these factors indeed may be what we do as instructors or what the students have done (perhaps because of our instruction), but observable changes may also be due to other naturally occurring factors. People change over time (i.e., maturation). Factors outside our awareness influence results (i.e., history). It is possible that changes that we see in our student learning are due to these natural changes or factors external to our instructional interventions. Social science methodology alerts us to these two confounds, to many others, and most importantly, to ways to avoid them. Watching out for these confounds is critical when assessing the effectiveness of changes to instructional methods. Detailed descriptions of these confounds can be found in research methods books and are worth the perusal (Creswell & Piano Clark, 2012; Morling, 2012).
It is also important to ask if the changes are statistically significant. Social science methodology involves significance testing. Assessing change is one type of pedagogical endeavor that necessitates the
quantitative method, regardless of discipline. If you want to know whether students’ performance improved after a change you made (e.g., a new assignment, an innovative presentation, group work, flipping your class), you need to know if that change would have happened by chance or if any other factors could account for it. This requires quantification of the evidence (e.g., themes brought up in a close reading, concepts used in essays, levels of meaning). When social scientists ask if the change is
statistically significant, they want to ensure that the change is due to what was done, and not just due to chance. Stated in this way, it is seems hard to not care about statistical significance. If you have worked hard to change your instruction and improve student learning, it is important to know whether that change would have happened by chance and without your intervention in the first place. Before one spends more time and energy on changing instruction or even trying to get others to also change instruction based on the changes you have seen, you should be sure your changes are not random. Statistical testing does this for you.
The most common statistical tests used to assess the effectiveness of instructional changes are t-tests and analyses of variance (ANOVA). These statistical analyses test for differences between groups – the class or section that got an innovation and the class or section that did not. There are no gray areas when it comes to testing. The statistical program (e.g., SPSS) provides a test statistic (e.g., an F or t test) and a probability value (i.e., p value). If the probability value is less than .05, the difference between groups is considered statistically significant and you can assume your innovation is important. Note that a p value is influenced by the size of your sample (the number of students in your groups) so some tests fail to be significant because they lack sufficient statistical power due to a small sample size. That said, for starters you want to see a p value less that .05. For step by step directions on conducting statistical analyses for SoTL or an easy to read exposition of statistical testing, see Gurung and Schwartz (2012) and Field (2013).
Note that statistical significance need not be the ultimate and only criterion for SoTL but it is certainly something to be considered for appropriate and relevant research designs and questions. Furthermore, statistical significance should not be confused with or taken to be synonymous with ‘significant’ as used in everyday life (i.e., to mean important).
There are a number of additional research designs (see Morling, 2012). Each has pros and cons. Although there may be better and worse ways to design and conduct a study, there will never be one
clear right way (Bartsch, 2013). The best we can do is to control for as many factors as possible in a systematic and intentional way. Starting with disciplinary methodologies and approaches you are comfortable with (and know how to use) makes great sense, but why stop there? If your question is learning in general, it makes better sense to develop your question and then pick the best methodology for the question regardless of the discipline. This may involve collaboration and reviewing literature on how other disciplines conduct questions surrounding teaching and learning (Gurung, Chick, & Haynie, 2009).
The science of learning provides the teacher with a wide variety of findings ripe for modifying and using in the classroom. It is imperative to assess the effectiveness of any change in a robust way and be prepared for the possibility that what may work in a cognitive psychology lab may not necessarily work in a classroom. Even some science of learning results from other classroom studies may not necessarily translate to your classroom. Using basic research design methods well will at least ensure that you get a sense of what does work.
References
Bartsch, R. A. (2013). Designing studies to measure student success. In R.A.R. Gurung & J. H. Wilson (Eds.). Doing the Scholarship of Teaching and Learning: Measuring systematic changes to teaching and improvements in learning (pp. 00-00). San Francisco, CA: Jossey-Bass.
Bartsch, R. A., & Cobern, K. (2003). Effectiveness of PowerPoint presentations in lectures. Computers & Education, 41, 77-86.
Bartsch, R. A., Engelhardt Bittner, W. M., & Moreno, J. E. (2008). A design to improve internal validity of assessments of teaching demonstrations. Teaching of Psychology, 35(4), 357–359. doi:
10.1080/00986280802373809
Bernstein, D. & Bass, R. (2005, July-August). The scholarship of teaching and learning. Academe, 91, 37- 43.
Bishop-Clark, C., & Dietz-Uhler, B. (2012). Engaging in the scholarship of teaching and learning: A guide to the process and how to develop a project from start to finish. Sterling, VA: Stylus.
Boyer, E. L. (1990). Scholarship reconsidered: Priorities of the professoriate. San Francisco, CA: Jossey- Bass.
Creswell, J. W., & Piano Clark, V. L. (2012). Designing and conducting mixed methods research. Thousand Oaks, CA: Sage.
Dunlosky, J., Rawson, K. A., Marsh, E. J., Nathan, M. J., & Willingham, D. T. (2013). Improving students’ learning with effective learning techniques: Promising directions from cognitive and educational psychology. Psychological Science in the Public Interest, 14(1), 4-58.
doi:10.1177/1529100612453266
Felten, P. (2013). Principles of good practice in SoTL. Teaching and Learning Inquiry, 1, 121-126. Field, A. (2013). Discovering statistics using SPSS. Thousand Oaks, CA: Sage.
Glassick, C. E., Huber, M. T., & Maeroff, G. I. (1997). Scholarship assessed: Evaluation of the professoriate. San Francisco: Jossey-Bass.
Gurung, R. A. R. (2012). Consuming the Scholarship of Teaching and Learning: Using evidence-based pedagogy ethically. In R. E. Landrum & M. A. McCarthy (Eds.), Teaching ethically: Challenges and opportunities (pp. 67-76). Washington, DC: APA.
Gurung, R. A. R., & Schwartz, B. M. (2012). Optimizing teaching and learning: Practicing pedagogical research. Malden, MA: Wiley-Blackwell.
Gurung, R. A. R., Chick, N., & Haynie, A. (Eds.) (2009). Exploring signature pedagogies: Approaches to teaching disciplinary habits of mind. Sterling, VA: Stylus.
Hutchings, P., Huber, M. T., & Ciccone, A. (2011). The scholarship of teaching and learning reconsidered: Institutional impact. San Francisco, CA: Jossey-Bass.
Kang, S. K., & Pashler, H. (2012). Learning Painting Styles: Spacing is Advantageous when it Promotes Discriminative Contrast. Applied Cognitive Psychology, 26(1), 97-103. doi:10.1002/acp.1801
Karpicke, J. D., & Roediger, H. L. (2007). Repeated retrieval during learning is the key to long-term retention. Journal of Memory & Language, 57(2), 151-162. doi:10.1016/j.jml.2006.09.004 Karpicke, J. D., Butler, A. C., & Roediger III, H. L. (2009). Metacognitive strategies in student learning: Do
students practise retrieval when they study on their own? Memory, 17(4), 471-479. doi:10.1080/09658210802647009
Mayer, R. E. (2011). Applying the science of learning. Boston, MA: Pearson.
McDaniel, M. A., & Donnelly, C. M. (1996). Learning with analogy and elaborative interrogation. Journal of Educational Psychology, 88(3), 508.
Morling, B. (2012). Research methods in psychology: Evaluating a world of information. New York: Norton.
Noppe, I. (2007). Powerpoint presentation handouts and college student learning outcomes. International Journal for the Scholarship of Teaching and Learning, 1. Retrieved from www.georgiasouthern.edu/ijsotl
Pelham, B. W., & Harton, B. (2006). Conducting research in psychology: measuring the weight of smoke. (3 ed.). Belmont, CA: Wadsworth.
Potter, M. K., & Kustra, E. (2011). The relationship between scholarly teaching and SoTL: Models, distinctions, and clarifications. International Journal for the Scholarship of Teaching and Learning, 5. Retrieved from www.georgiasouthern.edu/ijsotl
Sansone, C., Morf, C., & Panter, A. (2003). The Sage handbook of methods in social psychology. Thousand Oaks, CA: Sage.
Schwartz, E., & Gurung, R.A. R., (2012). Evidence-based teaching in higher education. Washington, DC: American Psychological Association.
Stevens, D. D., & Levi, A. (2011). Introduction to rubrics: An assessment tool to save grading time, convey effective feedback, and promote student learning. Sterling, VA: Stylus.
Suskie, L. (2009). Assessing student learning: A common sense guide (2nd ed.). San Francisco, CA: Jossey- Bass/Wiley.
Wilson-Doenges, G., & Gurung, R. A. R. (2013). Benchmarks for scholarly investigations of teaching and learning. Australian Journal of Psychology, 65(1), 63-70. doi:10.1111/ajpy.12011