RQ1. To what extent are the national standards related to the assessment sys- tem of the leadership development programme?
RQ2. To what extent does the assessment system relate to recent research findings on effective school leadership and best practice in assessment system design?
RQ3. What are the perspectives of those involved in the assessment process regarding its fitness for purpose and the extent to which it reflects those re- search findings and best practice?
RQ4. How can these findings inform the future development of SLDP assess- ment systems in Hong Kong, in England and globally?
For England, the documentary analysis for RQ 1 was carried out on the revised (January 2015) national standards of excellence for headteachers and on the NPQH assessment framework current at February 2015. For Hong Kong, the ‘Key Qualities of the Principalship in Hong Kong’ (Walker et al., 2000) constitute the official leadership framework; are the equivalent of the English national standards; and were used in the analysis. The assessment framework for the Hong Kong CFP was taken from the EDB Circular No. 1/2017 applications.ed- b.gov.hk/circular/upload/EDBC/EDBC17001E.pdf.
For RQ 2 the two assessment systems were compared with a summary docu- ment (Appendix 3) which encapsulated recent research findings on:
•
those instructional and transformational leadership practices most efficacious in improving pupil outcomes•
how SLDP assessment systems should be designed, organised and opera- tionalised.The comparative analysis conducted cast light on how well designed each as- sessment system was and how well it covered those instructional and transfor-
mative leadership practices and activities that impact most on student out- comes.
For RQ 3 the research method employed was semi-structured interview. This enabled an exploration of participant perspectives and attitudes that reflected the ontological position, and recognised the ideas, experiences and knowledge of the participants as an important aspect of the social reality of aspiring princi- pal assessment. Interviewing also allowed for interaction with the participants in a flexible, responsive, and sensitive manner. This enabled spontaneous follow up questions to be asked in response to comments or experiences (Bryman, 2006) and represented a dialogic interaction in the conversation style described by Kvale (1998, p.16). Structured interviews were not used as they may intro- duce a level of formality that does not facilitate conversation, nor allow access to contextual and situational knowledge in a way that enables association with other aspects of interviewee experience. This last consideration was particularly significant in the Hong Kong context. Providers, assessors and graduated were interviewed. Details are provided in the sampling section below.
Fully informed consent was obtained (Appendices 4-11). All interviewees were given the opportunity to select the location of the interview and whether it would be face to face or online. The major considerations when agreeing locations were noise, privacy and the certainty of not being interrupted. The interviews lasted approximately one hour and were recorded. Interviewees were asked for permission to record the interviews in advance and this was confirmed at the interview. All respondents were relaxed about this and agreed readily, so re- cording was not judged to have had a detrimental impact on the interviews, dur- ing which notes were made. The recordings were later transcribed, but not strictly verbatim. These transcripts formed the basis for the initial analysis and extracts are attached as Appendices 12-14.
With providers the TOA was deployed to provide the basis for a three stage dis- cussion. In stage one participants were invited to consider ‘their’ assessment system in the light of the TOA, the aim being to encourage them to critique both. The second stage focused on the circumstances surrounding the creation and development of the assessment system, including any challenges and
constraints such as the policy context and the influence of policy makers. The final stage invited participants to evaluate the assessment system in terms of its utility in measuring candidate readiness for principalship. Although these inter- views were structured in the sense that they followed a three stage process, wi- thin this framework the discussions themselves were unstructured, with the par- ticipants having free rein to develop their own thoughts and ideas. Given the intensive nature of these interviews, the protocol should be viewed more as an interview guide (Lofland and Lofland, 2006).
In the interviews with assessors and graduates, participants were asked to consider the summary document of IL and TL practices and design features to assess the extent to which both groups perceived these measures to be present in their respective assessment systems. The final section of the inter- view focused on each assessment system’s perceived performance in providing accurate and useful assessment information, including its ability to confirm rea- diness for school leadership. To stimulate discussion interviewees were asked to consider their assessment system in the light of Huba and Freed’s (2000) de- finition of assessment (Appendix 3).
For RQ 4 the findings from each set of data were subjected to a comparative analysis in order to identify the commonalities and differences between the two systems as well as their strengths and weaknesses. The data was used to cri- tique the assessment systems and to consider how they could be developed further. Whether and how competency-based assessment is a feasible and ap- propriate mechanism for assessing readiness for headship was then con- sidered, as was the future development of SLDP assessment systems gener- ally.
4.72 Sampling
Initially in both England and Hong Kong purposive sampling was used to identi- fy appropriate participants. Bryman’s approach was adopted as it was necessa- ry to interview individuals who had a particular expertise or held a particular of- fice:
Such sampling is essentially strategic and entails an attempt to establish a good correspondence between research questions and sampling. In other words, the researcher samples on the ba- sis of wanting to interview people who are relevant to the research questions.
Bryman (2008, p. 458)
However once in the field, peer esteem snowball sampling (Christoponlos, 2009) was used, as participants recommended various individuals who they thought would be of interest to me and be interested in the research. This situ- ation applied particularly in Hong Kong where I arrived having only one initial contact who I had researched before departure as an expert professional in the field of study. While this participant was selected purposely, because of the cir- cumstances, other participants in Hong Kong were obtained through peer es- teem snowball sampling as the initial contact acted as a gatekeeper, recom- mending other elite members of the expert group. An introductory email was sent to recommended participants inviting them to be involved, with an informa- tion sheet giving more detail on the study and its ethical considerations. In Hong Kong I also cold-called principals, contacting 40 secondary and 17 primary schools, writing a concise email that established my research purpose and my credentials as a research student (Appendix 15). This tactic secured some valuable CFP graduate participants. Overall the sampling strategy used was a mixture of purposive, peer esteem snowball and cold calling a random sample. I had initially planned to secure six participants in each of the three categories for each country.
Table 2: The final (achieved) sample
Hong Kong England
Providers 5 5
Assessors 2 9
Constraints of time and resource, imposed by my position as a solo part-time researcher, had to be recognised when coming to terms with the tension bet- ween what was ideal and what was actually possible in sampling terms. This was particularly acute in the Hong Kong context.
Although this method of sampling precluded achieving high confidence in the generalisability of the findings, the study - particularly with regard to providers and assessors - investigated the perspectives of a relatively small population of expert opinion. Clearly this did not apply to the graduate populations but within the field of educational leadership there are many examples of such small sample sizes being used in qualitative research projects, for example Shapira et al. (2011). The lack of balance in the sample with regard to assessors was caused by the difficulty the researcher experienced in securing access to the assessor cadre in Hong Kong. The limited sample was secured after a personal approach to the Principal Assistant Secretary of the Hong Kong Education Bur- eau (Appendix 16).