performance required in the test should match the performance required in the LO. Remember, test items were developed after the LOs were written (see 7.7).
8.5.1 Purpose. The primary purpose of testing is to assess the student's attainment of the behavior specified in the LO. Tests can also serve several other purposes such as those noted in Table 31.
TABLE 31. Types of tests.
TEST CATEGORY
PURPOSE OF TEST
Criterion • Used to measure the student's attainment of the objective.
• Used to measure the effectiveness of the instruction.
Pretest • Used to measure the student's ability to attain each objective.
• Used after the instructional system becomes operational to determine how much instruction individual students need.
Diagnostic • Used to determine attainment of supporting knowledge and skills necessary to perform the TLO.
• Used during validation to predict success, to identify and correct weaknesses in the instruction.
Survey • Used to determine what prospective students already know and can perform before receiving instruction.
• Used during development of instruction to gather data for design of instruction.
8.5.2 Characteristics of tests. There are several characteristics to be considered when developing tests. These characteristics ensure that the tests measure what is intended each time they are administered. The characteristics are shown in Table 32.
TABLE 32. Characteristics of tests.
CHARACTERISTIC DEFINITION
Validity • Degree to which the test measures what it is intended to measure.
• Degree to which the test predicts performance.
Reliability • Degree to which the test yields the same results consistently.
• Consistency across two administrations to the same students.
• Consistency across two forms of the same test. Usability • Tests that are easy to administer, score, and interpret.
8.5.3 Assessment method. Most tests can be classified into two main groups: predictive and performance tests. A common question is: When should designers use performance tests and when should they use question-answer tests? These are the wrong categories for comparison. The comparison should really be between performance tests and predictive tests.
8.5.3.1 Performance test. A performance test is one in which the student actually performs the skill required by the TLO. Many concept and rule-using type performances are tested with paper-and-pencil tests. For instance, many problem-solving skills involving the use of scientific principles can be observed from written performance tests. Many types of tasks, especially
equipment operation tasks, involve many different capabilities that have to be performed in an integrated manner. For example, the task of bleeding a hydraulic brake system involves recall of a procedure (cognitive skill), physical performance of the steps (psychomotor performance), recognition of the parts and tools (knowledge), observation of the brake fluid conditions in the system (cognitive skill), and cleanliness and safety (attitudes). In these types of tasks, full task performance cannot be measured by a paper-and-pencil test. A performance test would require a real or operational mock-up of a brake system. Performance tests require the student to
demonstrate mastery of the task previously learned.
8.5.3.2 Predictive test. Performance tests of integrated tasks are generally time-consuming because they often have to be conducted one-on-one with real equipment or simulators. If the actual behavior cannot be tested in a performance test (because it is too costly, dangerous, or impractical), the next best option is a synthetic performance based test. The object of this type of performance test is to simulate as closely as possible the performance or portion of performance a student would be expected to perform on the job. For example, if a student can appropriately respond to visual and auditory cues in a computer based test there is a better probability that the student will be able to transfer the behavior to the actual task.
8.5.3.3 Knowledge based testing. Knowledge based testing is not normally considered predictive of job performance. It may however be useful when the object of the test is to determine that an individual has learned facts.
8.5.4 Test construction factors. Several key factors should be considered when constructing tests, such as:
a. Testing level. The level of testing (perform, comprehend, etc.) should correlate with the stated learning level for that portion of the instruction being tested - no higher and no lower.
b. Test length. Adequate coverage of the LO is the major factor in determining the length of test that is required. One or more test items may be needed to adequately measure each LO.
c. Arrangement of test items. Test items should cover the most essential and significant portions of the material. Test items should be clear, concise, and well written to minimize misunderstandings. Items of the same type should be grouped together in a test, if possible. Individual test items should also be arranged in approximate order of difficulty, which allows the students to progress as far as they can without spending excessive time on difficult items at the first part of the test.
8.5.4.1 Determine constraints at testing sites. Constraints are limiting or constraining conditions or factors. These constraints can affect all testing decisions. Identify specific constraints (by LO) which may include:
a. Time available. b. Manpower available.
c. Costs of alternatives.
d. Equipment, tools, and materiel required/available. e. Potential for equipment damage.
f. Facility/testing area required/available. g. Acceptable degree of realism.
h. Logistical support required/available. i. Supervisory requirements.
j. Adequate supervisors available.
k. Communications required/available. l. Ethical considerations.
m. Safety considerations.
n. Environmental damage considerations. o. Legal considerations.
8.5.4.1.1 Compare what is needed with what is available. Annotate constraints and identify potential resources for testing (by objective). Check resources such as space (i.e., facilities and ranges), equipment, tools, materials, and personnel. Determine if specific constraints apply to all LOs to be tested, a subset of LOs, or a single LO.
8.6 Develop instructional materials. During the design phase, the instructional method and