The development of the final framework employed a Delphi method of 3-round emailing to experts for comments and feedbacks. The experts were 50 lecturers from various public and private institutions who have at least five years of experience in teaching differential calculus at pre-university and university levels. 40 of them were from local institutions while the other 10 were from international institutions. The questionnaire that was emailed to the experts consisted of 10 items (Likert scale) and an open-ended question intending to seek clarity on the categories, their flow and logical sequence together with the grammatical and spelling errors, if any. The measurement on the Likert scale were defined as Unsatisfactory, Poor, Satisfactory, Good and Outstanding (Appendix H ).
The details of the numbers of experts responding to the questionnaire in the 3- round emailing are as shown in Table 4.8. In the first round, two locals and one international expert did not respond to the request, giving a 94% rate of return. After the analysis of the responses from 47 experts, another three of them were dismissed due to their responses being outliers, one of them was responding almost all Poor while the other two of them assigned almost all Outstanding. In Round 2 of the emails, 42 experts returned their feedbacks resulting in 95% rate of return. One of the experts was treated
Table 4.7: The refined framework for assessing visual reasoning Visual reasoning process Category Description Enc oding
Correct graph with correct solution
Produces correct graph to solve and represent the solutions and managed to arrive to the correct solution
Correct graph with incorrect solution
Produces correct graph to solve and represent the solutions but did not manage to arrive to the correct solution
Incorrect graph with correct solution
Produces incorrect graph to solve and represent the solutions and managed to arrive to the correct solution based on the wrong graphs.
Incorrect graph with incorrect solution
Produces incorrect graph to solve and represent the solutions and did not manage to arrive to the correct solution
No graph with correct solution
Produces no graph to solve and represent the solutions and managed to arrive to the correct solution
No graph with incorrect solution
Produces no graph at all to solve and represent the solutions and did not manage to arrive to the correct solution
No answer / Not attempted
Left the item un-attempted – no graphs or any algebraic solutions.
De
coding
Correct solution with valid reason
Produces correct solution based on the graph and managed to provide valid reason(s) to arrive to the correct solution
Correct solution with invalid reason
Produces correct solution based on the graph but did not manage to provide valid reason(s) to arrive to the correct solution
Correct solution with no reason
Produces correct solution based on the graph but did not manage to provide any reason(s) to arrive to the correct solution
Incorrect solution with invalid reason / no reason
Produces incorrect solution based on the graph and did not manage to provide valid reason(s) to arrive to the correct solution
No answer / Not attempted
Left the item un-attempted.
as outlier and being dismissed for assigning all items as Outstanding. In Round 3, three of the experts did not return the feedbacks resulting in an 86% rate of return. The details of the responses to each item in the questionnaire throughout all the three round of emailing are as displayed in Table 4.9, Table 4.10 and Table 4.11.
On the overall, as can be seen in Table 4.9, at least 90% of the experts agreed with Satisfactory and Good for all items describing the clarity on the framework. Minor adjustments were made based on the proposal from the open-ended questions :
1) The term solution in the Description column for the Encoding process was proposed to be replaced by explain or describe
2) The additional description of Solutions may differ from the original solutions set for the Incorrect graph with correct solution to enhance explanation.
3) The spelling of unattempted was re-spelled as un-attempted. 4) The inclusion of s for possible pluralism in the word reason.
The analysis for Round 2 is as shown in Table 4.10. All experts were at least satisfied with the refined framework although one of them answered all Outstanding. It was assumed that he/she did not really evaluate the framework thoroughly or he/she might had assumed that the framework was totally refined.
The analysis for Round 3 is as displayed in Table 4.11. The responses were fairly distributed between the Satisfactory and Good. All of them did not find any spelling or grammatical error in the framework.
Based on the final feedbacks from the experts and the final refinement process, the final framework that can be used to assess the visual reasoning ability of pre- university students is as shown in Table 4.12.
Table 4.12: The final framework for assessing visual reasoning Visual reasoning process Category Description Enc oding
Correct graph with correct solution
Produces correct graph to explain and represent the solutions and managed to arrive to the correct solution
Correct graph with incorrect solution
Produces correct graph to explain and represent the solutions but did not manage to arrive to the correct solution
Incorrect graph with correct solution
Produces incorrect graph to explain and represent the solutions and managed to arrive to the correct solution based on the wrong graphs. Solutions may differ from the original solutions set.
Incorrect graph with incorrect solution
Produces incorrect graph to explain and represent the solutions and did not manage to arrive to the correct solution
No graph with correct solution
Produces no graph to explain and represent the solutions and managed to arrive to the correct solution
No graph with incorrect solution
Produces no graph at all to explain and represent the solutions and did not manage to arrive to he correct solution
No answer / Not attempted
Left the item un-attempted – no graphs or any algebraic solutions.
De
coding
Correct solution with valid reason
Produces correct solution based on the graph and managed to provide valid reason(s) to arrive to the correct solution
Correct solution with invalid reason
Produces correct solution based on the graph but did not manage to provide valid reason(s) to arrive to the correct solution
Correct solution with no reason
Produces correct solution based on the graph but did not manage to provide any valid reason(s) to arrive to the correct solution
Incorrect solution with invalid reason / no reason
Produces incorrect solution based on the graph and did not manage to provide valid reason(s) to arrive to the correct solution
No answer / Not attempted
Left the item un-attempted.
The final framework was then sent to an expert, an international professor, for final validation (Appendix I). It is named Visual Reasoning over Graph (VR-G), an in- depth assessment of how graphs of functions and derivatives and their concepts must be
It is principally used in categorising students’ encoding and decoding ability. In this study, constructs for students’ visual reasoning ability when using graph to solve mathematical problems on functions and derivatives were extracted from the above- mentioned frameworks. The items in the instruments are conceptualized based on the content domain but enhanced according to the knowledge and scheme.
The term ‘correct graph’ refers to the students being able to produce a complete and effective graphs that are characterized by labelling of the axes, scales and function(s). The complete graphs that are constructed accurately and neatly reflects understanding on the conceptual knowledge and the relationship between symbolic or algebraic representations and their visual representations on the graphs. The term ‘incorrect graph’ refers to students demonstrating limited understanding of graphing and some understanding on the relationship between symbolic or algebraic representations and their visual representation on the graphs.
The term ‘valid reason’ refers to the students being able to infer on the relationships between the properties of functions or/and derivatives, being able to integrate contextual knowledge, and understand the purpose of the information displayed in the graphs. On the other hand, ‘no valid reason’ refers to students that has no or appropriate engagement with the context where he/she may understand the single or basic elements of graph reading, at the same time serves as an indicator of employing memorization techniques or procedural knowledge.
4.3 Usage levels of visual representations