El estado a nivel local y la participación regulada de los ciudadanos

Various definitions and methods have been used to ease the generic assessment skills process (Yusoff et al., 2012). Silva (2009) claimed that generic skills can be measured accurately and in a common and comparable way. The problem of generic skills assessment is whether the observations of the students’ progress and performance can be carried out in a credible and trustworthy manner (Jonsson et al., 2007). No matter how data is presented, there is always the issue of interpretation, although it is helpful to look at the statistical measures (Prince, 2004). Stemler (2004) stated that the accuracy and consistencies of the assessment can be determined through three main approaches: consensus estimates – determining the degree to which different assessors give the same score to the same performance; consistency estimates – determining the correlation of scores among assessors; and measurement estimates – determining, for instance, the degree to which scores can be attributed to common scoring. Current Engineering Education literature suggested generic skills are assessed through several methods as described below.

 Self- and Peer Evaluation

Yasin et al. (2009) suggested that courses or programmes within an AL environment should employ continuous and alternative assessment besides the traditional assessment; for example, process evaluation, peer evaluation and self-evaluation. It is advantageous for students’ learning to be involved in giving and receiving feedback (Jonsson et al., 2007). Kennedy (2005) described peer assessment as “any of a variety of approaches where group members are required to evaluate other members of the group on their relative contribution to a project” (p. 2). Peer assessments use typical peer rating systems, where student team members confidentially rate how well they and individual team members are doing in fulfilling their tasks or rate individual behaviours (Shuman et al., 2005). The approach contributes individual multiplier towards the team performance (Wellington et al., 2002). Peer assessment and self-assessment requires students to reflect and evaluate their own participation, learning progress, and the results of autonomous learning (Hart, 1994). Both self- and peer-assessment are expected to decrease the central role of the lecturer in assessment activities (De-Grez et al., 2012).

49

However, regarding both, peer and self-evaluation, on the one hand students can be very accurate in grading their own work; but, on the other hand, students tend to give more marks than teachers (Topping, 2003; Jonsson et al., 2007; De-Grez et al., 2012). Kennedy (2005) research pointed out the challenges when adopting peer assessment: students may be reluctant to judge peers; students may discriminate; dysfunctional effects of peer assessment; and students’ different perceptions of fairness (pp. 62-64). Other challenges in self- and peer assessment include students being concerned about their inexperience in marking and the amount of time consumed through the activity (De- Grez et al., 2012). Regardless, many researchers and educators have adopted peer and self-assessment in assessing students’ generic skills especially for teamwork skills (for example, see Brooks et al., 2003; Kennedy, 2005; Marin-Garcia et al., 2008; Kim, 2013; Mohd-Yusof et al., 2013)

 Formative and Summative Assessment

Several studies have shown that feedback can be an effective strategy that promotes active learning (Nicol et al., 2006; Biggs et al., 2007; Oliveira et al., 2013). According to Biggs et al. (2007), there are two types of assessment, summative and formative. Summative assessment is usually made after learning and informs students whether or not their learning corresponds to what is expected. Formative assessment is made during the learning process and informs students (and teachers) on how it is progressing and what actions should be taken to improve it. For this formative feedback to be effective, it is necessary that students are aware of what they have learnt, what they need still to learn and know what is expected of them (Biggs et al., 2007; Oliveira et al., 2013). Nicol et al. (2006) suggested that feedback should give the necessary information for students to minimise this difference. Knight (2001) summarised the key concepts related to summative and formative assessment as shown in Table 2-5.

Table 2-5Summative and formative assessment (Knight, 2001, p. 9) Dimension of difference Summative – assessment as measurement Formative – assessment as judgement Assumptions about achievement

Achievements are seen as transferable. Data can be used to predict achievement.

A limited transfer of

learning. Data cannot be used as a good predictor.

Products “Feedout” in the shape of

warrants to achievement

Feedback in the shape of improvement

Priorities  Reliable measures of

achievement.  Motivating learners.

 Providing feedback

50  Providing information to guide learning. opportunities to improve learning.  Motivating learners. How assessments are

communicated

Often in numerical form. Often in words - narrative.

Common assessment techniques

Fixed response test. High-inference judgements

of authentic achievements

on projects, work

placements, peer and self- assessment.

Suggestions for

improvement

 Use programme-wide

assessment plans to

identify what is going to be assessed and when.  Develop the assessment

criteria.

 Ensure that there are repeated observations of

ILO’s that are

summatively assessed.

 Use multiple

observers/assessors.

 Use learning indicators as points of reference

in assessment

conversation.

 Allow enough time - to design the assessment plan.

 Provide awareness of how to do formative assessment (peer and self-assessment).  Link formative assessment with employability requirement.  Rubric Assessment

A common practice in AL assessment is the rubric that is used in evaluating student generic skills performance in the Department of Aerospace at the United States Naval Academy (USNA) via the CDIO syllabus (Boden et al., 2007) and assessed ABET professional skills (Shuman et al., 2005; Kranov et al., 2008). Jonsson et al. (2007) listed several benefits of the rubric widely stated in the literature. These include: to increase or enhance consistency of judgement, to provide valid judgment, and to provide the desired validity in assessing complex generic skills. The rubric structurally tells both assessor and student what is considered important and what to look for when assessing (Perlman, 2003). A rubric articulates gradation of quality for each criterion it contains, from excellent to poor (Morreale et al., 2011).

When utilising the rubric, assessors use an analytic rating system where each component is scored individually or performance is rated holistically based on an overall

51

impression (Pomplun et al., 1998). Jonsson et al. (2007) briefly distinguished two main categories of rubrics, analytic scoring and holistic scoring, as summarised in Table 2-6. Table 2-6Analytic and holistic scoring in rubric assessment (Jonsson et al., 2007)

Analytic Scoring Holistic Scoring

Assessor assigns a score to each of the dimensions assessed in the task.

Assessor makes overall judgment on quality performance.

Useful in small-scale such as classroom. Used for large-scale assessment.

The problem in rubric assessment is that many lecturers have an instinctive grasp of what generic skills are, but struggle to provide a clear definition of them and to define rubrics for their assessment (Cajander et al., 2011). Feldt et al. (2009) claimed that the rubric form should help assessors in the approach to the definition, clarification and assessment of generic skills performance. However, Jonsson et al. (2007) argued rubrics do not facilitate a valid judgement for performance assessment unless a more comprehensive framework of validity is used. Many educators and researchers in the literature utilised a rubric assessment to assess generic skills (Williams, 2002; Dunbar et al., 2006; Boden et al., 2007; Feldt et al., 2009).

 Standardised Assessment

The literature shows general agreement among HE institutions, accreditation organisations and employers that it is important for students to develop generic skills in order to secure employment (DEST, 2006; Yorke, 2006; Zaharim et al., 2009; Yusoff et al., 2012). A number of researchers conducted a study to define a set of generic skills. The outcome of the study contributed to a number of national frameworks of generic skills proposed in Malaysia, Australia, Japan, the United States of America, the United Kingdom as well as in the European Union (DEST, 2006; Yusoff et al., 2012; ABET, 2014). In the same respect, Singh et al. (2014) listed several countries which have a centralised and standardised generic skills assessment tool in place, for example, Australia – Graduate Skills Assessment known as GSA, England – Cambridge Thinking Skills Assessment and America – Work Keys System. Similarly, there is a centralised generic skills assessment conducted in America called the Collegiate Learning Assessment (CLA) (Klein et al., 2007). These countries, identified in the literature, have developed frameworks and/or guidelines for HE institutions to define clearly, observe and articulate graduate attributes in their curricula. However, each approach of the generic skills assessment as listed above has its advantages and drawbacks.

52  Reflective Report/ Portfolios

Assessing the students’ generic skills in an AL environment demands a careful consideration of various assessment techniques, where both the content and the assessment should be authentic (Hosseinzadeh et al., 2012). Authentic assessments are categorised into performance assessment and portfolio assessment (Tai et al., 2007). Regarding portfolio assessments, the literature often refers to the following definition provided by the Northwest Evaluation Association (NWEA): “A portfolio is a purposeful collection of student work that exhibits the student’s efforts, progress, and achievements in one or more areas. The collection must include student’s participation in selecting contents, the criteria of selection, the criteria for judging merit, and evidence of student self-reflection.” (NWEA, 1990, cited from Barrett, 2000, p. 14; Williams, 2002, p. 201).

Gibbings et al. (2007) encouraged teachers to prepare guidelines to the students on the requirements of an individual portfolio. The portfolio is to create documentary evidence of what has been learnt and how skills and competence levels have increased. Reflective reports or portfolios are to be used during generic skills assessment and also encourage students to reflect on their learning during group work (Wellington et al., 2002). Portfolio assessment allows the assessor to assess students’ progress in developing intended skills over a period of time, sometimes across several years (Barrett, 2000). The approach has also been widely used by many engineering accreditation bodies as offering an acceptable measure of students’ attributes (S. Palmer et al., 2011). Portfolio assessment includes notes, commentaries and articles the students have read, and discussions of the evolution of their ideas to formulate and report their findings and conclusions (Tai et al., 2007; Hosseinzadeh et al., 2012). Reflective reports or portfolio practices provide an opportunity for students to reflect his/her own writing practice, provide evidence of his/her progress in writing over time, and becomes a showcase of his/her work (Williams, 2002, p. 201). However, Cajander et al. (2011) argued that students are not likely to reflect on the development of generic skills unless required to do so, and that students are not likely to incorporate reflection as part of their on-going professional practice. Dannels et al. (2003) identified another reason for students not to complete reflective reports: they thought the reflections were part of the writing and speaking assignment which not contribute to any marks.

53

2.4.2 Current Approaches in Assessing Problem-Solving Skills and their Challenges