Previous self-efficacy research has employed a number of analytical methods to investigate the sources of self-efficacy, their causal influence, their multidimensionality, and the contextual factors that may moderate them (Usher, 2007). This dissertation utilizes a scale validation analytic procedure using quantitative analyses such as
exploratory factor analysis (EFA), principal components analysis (PCA), and descriptive exploratory analysis (means, standard deviations, and frequency distributions).
The Design Self-Efficacy measurement instrument developed for this research project contains fifty items. The goal of this large number of items was to capture the multidimensionality of self-efficacy in the design context. These elements align with the four realms (A, B, C, and D) of the 2014 National Architectural Accreditation Board (NAAB) Student Performance Criteria (SPC). The realms are defined as: (A) Critical Thinking and Representation, (B) Building Practices, Technical Skills, and Knowledge, (C) Integrated Architectural Solutions, and (D) Professional Practice (Appendix 10).
Whereas the intent of the accreditation process is to distribute these criteria across the entire curriculum, the DSE measure developed for this research focuses
specifically on those criteria that align within the context of the design studio. The centers used for the study were Texas A&M University (TAMU), the University of Kentucky (UKY), and the University of Kansas (KU). Since each accredited program has different methods for distributing SPC throughout their program, a content analysis of the studio syllabi was completed. Since self-efficacy requires contextualization within a situation that requires action (Bandura, 1997), SPC were translated into questions that relate to task specific activities.
III.4.2 Project Score
Design and creativity requires the experimentation and synthesis of a variety of variables in order to result in a creative and effective design solution. Comparing and assessing the products of creative design is often elusive. As noted by Lueth (2008), instructors should develop explicit rubrics and assess students on how they accomplish the goals of the studio (Lueth, 2008). The rubric developed for this research is intended to help design studio faculty, professionals, and community stakeholders evaluate and score student work samples (artifacts) resulting from an assigned project. The rubric can be used across architecture design studios to score the attainment of student learning outcomes and enable the assessment of creative designs across a broad range of studio contexts, project types, and scales. The rubric is intended to be neutral to the range of projects in architecture design courses, which may vary from small-scale objects to inhabitable buildings to urban design. An individual student or a group of students may complete these artifacts and unless described in the text, the method of developing the
The rubric has two primary categories—design and communication—that emerged from a content analysis of architecture design studio syllabi.
The Design categories and subsets are defined as:
1. Design Research and problem solving: The student raises the appropriate design questions in response to a given design challenge—site,
environmental condition, and material parameters and other design constraints included in a project brief
2. Design Iteration, data collection and analysis: The student effectively references and evaluates design precedents that relate to a given project 3. Design Evaluation and integrative learning: The student establishes and
compares evaluative performance criteria and then synthesizes them iteratively, to result in an architectural design solution.
The Communication categories and subsets are defined as:
4. Written presentation: The student communicates effectively with appropriate written materials that describe the project and design solution.
5. Graphic / visual presentation: The student conveys information accurately using the appropriate graphic and representational media.
Development of rubrics for assessing project documents are also presented and defended with respect to reliability and validity. As outlined by Moskal and Leydens, the assessment rubric used the three forms of evidence to ensure validity of the instrument: content, construct, and criterion (Moskal & Leydens, 2000). Content evidence focuses on sampling a student’s knowledge of a subject or domain (Moskal & Leydens, 2000).
For this research, the assessment of content evidence focuses on scoring criteria specific to design problem setting and problem solving, design research, and written and graphic communication. Construct evidence relates to processes that are internal to an individual and a student’s foundational methods of working (Moskal & Leydens, 2000). For this research, the assessment of construct evidence focuses on design iteration and design evaluation as the means to evaluate the designs prior to constructing the artifact for review. Criterion evidence focuses on the application of knowledge and its correlation to the current design and impact on future events (Moskal & Leydens, 2000). For this research, the rubric is designed to follow the design processes thus developing what Rafilson refers to as the transference of validity from one situation to another (Rafilson, 1991).
III.4.2.1 Justification
For the purpose of NAAB accreditation, curricular assessment is aligned with student learning outcomes and evidences of Student Performance Criteria (SPC). According to NAAB, all “SPC are treated as equal forms of evidence of student
achievement” (NAAB, 2014). Attainment of an SPC should be apparent in the products that a student produces in response to a design assignment. For this research, an
instrument was constructed to allow external evaluators to review, vet, and score the work produced in studio, indicating whether learning outcomes have been attained.
III.4.2.2 Population and Sample
The population for the study was all students enrolled in design studios at TAMU, UKY, and KU above the age of eighteen. The survey participants were self- selected.
III.4.2.3 Subject Recruitment and Privacy
Student recruitment consisted of visiting each design studio and presenting the research design. The recruitment was followed up with an e-mail distribution that provided the students with a unique link to a Qualtrics form that enabled the upload of the student artifacts (Appendix 12). There were no inclusion and exclusion criteria, recruitment methods, or possible issues that could lead to consistent biases in the anticipated sample.
III.4.2.4 Protecting Participants
All project scoring was de-identified by a data broker in the Offices of
Institutional Effectiveness (IE) at the respective institutions. The IE data broker coded and linked the student related content for analysis.
III.4.2.5 Data and Safety Monitoring
The call for student projects (Appendix 12) was administered to all students who had given an informed consent to have their information turned over to the researchers as de-identified data. The survey provided to students was de-identified by a data broker at the Office of Institutional Effectiveness (IE). The roles of the data broker were: email the panel survey, upload the panel data to Qualtrics Online Survey Software and Insight Platform (Qualtrics), then link it to institutional data such as demographics and project
score, then assign anonymous coding to data. This process ensured that all of the data that is sent to the researcher for analysis was de-identified.
III.4.2.6 Materials and Instruments
A project assessment and scoring rubric (Appendix 13) was initially developed in collaboration with assessment experts at the University of Kentucky to guide reviewers in their assessment of the artifacts.
This rubric was developed using three sources of data: the AAC&U VALUE Rubric, faculty focus groups and interviews with design studio faculty at the Texas A&M University (TAMU), The University of Kentucky (UKY), and the University of Kansas (KU); and the content analysis on architectural design studio syllabi at TAMU, UKY, and KU. This project scoring assessment rubric incorporates all four educational realms of the National Architecture Accrediting Board (NAAB) - Critical Thinking and Representation; Building Practices, Technical Skills, and Knowledge; Integrated
Architectural Solutions; and Professional Practice. (Source: 2014 NAAB Conditions for Accreditation; www.naab.org) and the corresponding twenty-six student performance criteria related to those realms.
This rubric is intended to help design studio faculty, professionals, and community stakeholders evaluate and score student work samples (artifacts) that demonstrate evidence of student learning outcomes in architecture design studios. The rubric may also be used as a foundation for the assessment of creative designs across a broad range of studio contexts, project types, and scales where experimentation and
synthesis of design constraints result in a creative and effective design solution. Studio projects may vary from small-scale objects to inhabitable buildings to urban design.
This rubric uses a 1 to 4 interval measure with 1 being the lowest value and 4 being the highest value for each of the five categories established by the research. These categories include design research and problem-solving, design iteration and data collection and analysis, design evaluation and integrative learning, written
communication, and graphic communication. The intent of the rubric was to assess the student’s knowledge in architecture for a given project and their ability to convey that knowledge in a graphic format. The instrument used during data collection of project scoring was the AQUA platform developed by Taskstream.
III.4.2.7 Procedure
At the end of the semester, artifacts were collected via Qualtrics. To ensure de- identification throughout the process, a workflow was developed in conjunction with the data brokers at each of the institutions. The artifacts were bulk uploaded into the AQUA assessment platform, aligned the work to a design evaluation-scoring rubric (Appendix 13), and evaluators randomly assigned to assess and score the artifacts. This evaluation was preceded with two sessions: a training session, and a norming session. The intent of the training session was to familiarize the evaluators with the rubric and the assessment tools. The norming session demonstrated how to identify evidences by using the rubric and assess effectively inter-rater reliability through the norming process. A cadre of external evaluators who were asked to assess the work and use the assessment rubric as a guide to score the work evaluated the final studio projects (Appendix 13). The scores
were reported electronically and coded by the data broker to associate them with the individual students who are participating in the study. The data brokers provided the linked data to analyze.
III.4.2.8 Quantitative Data Analysis Tools
Data relative to project scoring was collected and displayed in the Analytics Dashboard as part of the AQUA platform. Rater scores for each artifact were
incorporated into this scoring system to increase its validity and studied for inter-rater and intra-rater reliability. The scores from multiple reviewers was averaged and compared to determine the level of inter-rater agreement.