In the systems approach to instructional design, the output from one step is the in-put to the next. Therefore, it is important to stop periodically to determine whether products being created are consistent from step to step in the process.
At this point in the design process, the goal has been analyzed, subordinate skills have been identified, learners and contexts have been analyzed, objectives have been written, and assessments have been developed. It is imperative that the skills, objectives, and assessments all refer to the same skills, so careful review is required to ensure this congruence.
Materials for Evaluating the Design You must have all the materials produced thus far to complete the design evaluation, including the instructional analysis diagram, performance objectives, and summaries of learner characteristics, as well as performance and learning contexts, performance objectives, and assessments.
Recall that there are four main categories of criteria to be considered for evaluating your overall design to this point—goal, learner, context, and assessment criteria.
These criteria are woven through the evaluation of your design.
Procedures for Evaluating the Design Using these main criteria, there are five steps in evaluating the design:
1. Organize and present the materials to illuminate their relationships.
2. Judge the congruence between the information and skills in instructional goal analysis and the materials created.
3. Judge the congruence between the materials and the characteristics of the target learners.
4. Judge the congruence between the performance and learning contexts and the materials.
5. Judge the clarity of all the materials.
Concepts 155
Organization. How can you best organize and present your materials to evaluate them at this point in the instructional design process? Each component builds on the product from the previous one; therefore, the materials should be presented in a way that enables comparison among the various components of your design. The designer should be able to see at a glance whether the components are parallel, which can be achieved by organizing the materials such that related components are together. Consider the structure in Table 7.3 . The first column is a list of the subskills from the instructional goal analysis, the second column includes performance objectives for each skill, and the third column shows test items for each objective. The last line contains the instructional goal, the terminal objective, and the test item(s) for the terminal objective. Using such a table, the evaluator can determine at a glance whether the materials are congruent. Table 7.4 includes an example of the type of material listed in each section of Table 7.3 .
The sequence of subskills presented on your chart is important. If you place them in the order you believe they should be taught, then you can receive addi-tional feedback from a reviewer concerning the logic you have used for sequenc-ing skills and presentsequenc-ing instruction. This additional feedback may save steps in rewriting or reorganizing your materials at a later point. The topic of sequencing skills is addressed in greater detail in Chapter Nine .
You should have other documents available for evaluators to use with your design evaluation table, including the instructional analysis diagram, the table of target learner characteristics, and the table describing the performance and learn-ing contexts. All items in the design table should be keyed to the numberlearn-ing of the subskills in the analysis diagram. This complete set of materials represents your instructional design to this point.
Congruence. The second step requires using the goal-centered criteria to judge the congruence among your materials. The congruence among the subordi-nate skill in the goal framework, its intended performance objective (conditions, behavior, and content), and the intended test items is critically important to the quality of your materials.
Subskill Performance Objective Sample Assessment
1 Objective 1 Test item
2 Objective 2 Test item
3 Objective 3 Test item
Instructional Goal Terminal Objective Test item Structure of the Design Evaluation Chart
t a b l e 7 . 3
Skill Performance Objective Test Item(s) 1. Write the formula
for converting yards to meters.
1. From memory, correctly write the formula for con-verting yards to meters.
1. In the space provided below, write the formula used to convert yards to meters.
2. Convert measures in yards to comparable meters.
2. Given different lengths in yards, convert the yards to meters, correct to one decimal place.
2. 5 yds = _______ meters 7.5 yds = _______ meters 15 yds = _______ meters Example of a Design Evaluation Chart
t a b l e
7 . 4
A recommended procedure to follow for this part of the analysis is to compare the (1) subordinate skills in instructional goal analysis with the subordinate skills listed in the design evaluation table, (2) subordinate skills in the table with the performance objectives in the table, and (3) performance objectives (conditions, performance, and criteria) with the test items prescribed in the table.
The wording of the subordinate skills in the goal analysis diagram and the de-sign evaluation table should be the same. Once this congruence is established, the goal analysis can be set aside. Your subordinate skills and performance objectives should differ only in that conditions and perhaps criteria have been added. Finally, determine whether the performance objectives and test items match in conditions, performance, and criteria. Are students given the information and materials in the items as prescribed in the objective? Once the design is parallel in content, evalu-ators can turn to examining the congruence between the performance objectives and the characteristics of the learners.
Learner Characteristics. The third step is to compare the materials with the characteristics of the learners. For this analysis, evaluators should judge the con-gruence between the materials and learners’ ability, vocabulary, interests, experi-ences, and needs. The best materials in the world are ineffective if intended learners cannot use them effectively. Do reviewers believe the performance objectives and assessments are set at the right scope and complexity for the defined target group?
Are the objectives broken down finely enough, or too much? Are the test items at the appropriate level of complexity for the learners?
Contexts. The fourth step is to judge the congruence of the performance and learning contexts with the performance objectives and test items in the design evaluation table. Reviewers should judge the authenticity of the tasks prescribed for the performance context, because this authenticity helps ensure transfer of skills from the learning to the performance context. They should also examine the feasibility of the tasks for the learning context. Can the designer expect the resources required (e.g., costs, time, personnel, facilities, equipment) to implement the implied instruction and assessments to be available in the learning context?
Materials Clarity. With congruence among the materials and the goal, con-texts, and learners established, the final step is for reviewers to judge the clarity of the materials. Unfortunately, this step is sometimes where the evaluation begins, but without determining the alignment of the design documents, clarity may be a moot point. During this step, evaluators should be asked whether the structure and scope of the goal analysis make sense to them. Are the subordinate and entry skills identified correctly, and are they in the correct order? Are the performance objectives written clearly, and do they know what is meant by each? What is their perception of the quality of the test items, including language clarity; vocabulary level; gram-mar, spelling, and punctuation; assessment format; and professional appearance?
After you have received feedback concerning the adequacy of your design and made appropriate revisions in your framework, you have the input required to begin work on the next component of the model, namely developing an instruc-tional strategy. Having a good, carefully analyzed, and refined design at this point facilitates your work on the remaining steps in the process.
Examples
W
hen you examine test items and assessments in this section, you can use the four categories of criteria summarized by the rubric at the end of the chapter to help focus your attention on particular aspects of the item.Examples 157