Análisis económico de la red

Since the findings of this research are too limited to make a definite statement on the effectiveness of the performance agreements in increasing quality of education at Dutch universities of applied sciences, the author will address instead several policy recommendations on how the latest design of the performance agreements can be improved in the future. The recommendations are of high actuality since new ‘Quality agreements’ (‘Kwaliteitsafspraken’) for the period 2019 to 2024 are currently being arranged (Interstedelijk Studenten Overleg et al., 2018). As such, the policy recommendations can offer valuable inputs to the Ministry of Education, Culture and Science, The Netherlands Association of Universities of Applied Sciences, the Association of Universities in the Netherlands, the Dutch Student Union and the Intercity Student Consultation7 _{on how to shape certain}

features in the design of the future ‘Quality agreements’ (Interstedelijk Studenten Overleg et al., 2018). The research findings underpin a certain relevance of student satisfaction for universities of applied sciences in retaining students at their institution. In the past, higher education institutions were accused of displaying strategic behavior to increase graduate numbers and enrolments for maximizing budgets, which went at the expense of students’ impressions of the quality of education (De Boer & Van Vught, n.d.; Reviewcommissie Hoger Onderwijs en Onderzoek, 2017). This highlights the relevance of holding Dutch universities of applied sciences accountable for satisfying the needs of students.

7 _{The policy recommendations do not address the Higher Education and Research Review Committee}

that was abolished in 2016. Monitoring of the Quality agreements will be done by the official accreditation agency NVAO (Interstedelijk Studenten Overleg et al., 2018)

41 Therefore, it is recommended to the contracting parties to include indicators with stakeholder judgments also in the future ‘Quality agreements’.

Since the research found no significant relationship between the quality measures included in the performance agreements and student satisfaction and retention rates, it is recommended to the contracting parties to reconsider the inclusion of these indicators in future performance agreements. In terms of the indicator ‘Indirect costs’, this recommendation matches own evaluations by the Review Committee itself who acknowledged operationalization issues of the indicator (Reviewcommissie Hoger Onderwijs en Onderzoek, 2017). Regarding the indicator ‘Teacher quality’, it is undisputed that students should acquire analytical and research competencies during their studies (Reviewcommissie Hoger Onderwijs en Onderzoek, 2017). Yet, there might be better suited indicators for ‘Teacher quality’ to include in future performance agreements. In the case of Dutch research universities, teacher quality was measured in terms of the share of teachers with a ‘University Teaching Qualification’ (In Dutch: BKO) out of the total number of teachers. It is an institution-overarching qualification for didactic competencies of teaching staff (VSNU, n.d.) that has become increasingly implemented at Dutch research universities (Reviewcommissie Hoger Onderwijs en Onderzoek, 2017). In the case of Dutch universities of applied sciences, the Netherlands Association of Universities of Applied Sciences agreed in 2013 to implement a similar ‘Basic Qualification of Didactical Competence’ (in Dutch: ‘Basiskwalificatie Didactische Bekwaamheid; ‘BDB’) at Dutch universities of applied sciences (Zestor, n.d.). Since critics have highlighted that teacher quality is not only ensured by the educational background of a teacher (Interstedelijk Studenten Overleg, 2016), policy-makers might consider including the share of teachers with a ‘BDB’ qualification as a better indicator for teacher quality in future performance agreements.

Given the negative relationship between student-staff-ratios and student retention rates, policy efforts should aim at decreasing student-staff ratios at Dutch universities of applied sciences. In the performance agreements from 2012 to 2015, the student-staff-ratio (teaching staff) was included as an optional indicator that universities of applied sciences could choose to include in the agreements (Reviewcommissie Hoger Onderwijs en Onderzoek, n.d.-c). Given its effect on student retention rates, policy makers should consider including the ratio as a prescribed component in the future ‘Quality agreements’.

Regarding the findings that larger shares of ‘MBO’ students were associated with lower retention rates, the contracting parties of the future ‘Quality agreements’ might consider adjusted budget allocations for higher education institutions. As Honig (2006) suggested, not every policy may work in every setting and implementation also depends on the people. Hence, the budget allocation might not suit every higher education institution due to different student compositions. Universities of applied sciences with higher shares of ‘MBO’ students might require proportionally more financial resources to implement adequate support interventions for these students. This recommendation is given as a new impulse for discussion rather than as a concrete recommendation. This is because the study could not

42 draw causal relationships between student characteristics and outcomes. Nonetheless, it can give an interesting input for the discussion on how to customize the design of the ‘Quality agreements’ to the individual needs of institutions.

Finally, the author awaits with eagerness further developments in the new round of the Dutch ‘Quality agreements’ starting 2019 to 2024. With certainty, the next round of quality agreements will offer valuable, new insights into the mechanisms of the policy tool ‘Performance agreements’. The author hopes that the suggestions made for further research and policy recommendations will find at least some attention on the path until then.

43

References

Bernal, J. L., Cummins, S., & Gasparrini, A. (2017). Interrupted time series regression for the evaluation of public health interventions: A tutorial. International Journal of Epidemiology, 46(1), 348–355. https://doi.org/10.1093/ije/dyw098

Bowerman, B. L., & O’Connell, R. T. (1990). Linear statistical models: An applied approach (2nd ed.). Belmont, CA: Duxbury.

Bradley, D., Noonan, P., Nugent, H., & Scales, B. (2008). Review of Australian Higher Education. Final Report.

Browne, B. A., Kaldenberg, D. O., Browne, W. G., & Brown, D. J. (1998). Student as Customer: Factors Affecting Satisfaction and Assessments of Institutional Quality. Journal of Marketing for Higher Education, 8(3), 1–14. https://doi.org/10.1300/J050v08n03_01

Crawford, L. M. (1999). Extended Opportunity Programs and Services For Community College Retention. In The Eighth Annual California Community Colleges Chancellor’s Office Statewide

Conference (pp. 1–32). Monterey, CA. Retrieved from https://files.eric.ed.gov/fulltext/ED429642.pdf

Crosby, B. C., Bryson, J. M., & Bloomberg, L. (2014). Public Value Governance: Moving Beyond Traditional Public Administration and the New Public Management. Public Administration Review, 74(4), 445–456. https://doi.org/10.1111/puar.12238

Dansereau, F., Cho, J., & Yammarino, F. J. (2006). Avoiding the “Fallacy of the Wrong Level.” Group & Organization Management, 31(5), 536–577. https://doi.org/10.1177/1059601106291131 De Boer, H., Jongbloed, B., Benneworth, P., Cremonini, L., Kolster, R., Kottmann, A., … Vossensteyn,

H. (2015). Performance-based funding and performance agreements in fourteen higher education systems: Report for the Ministry of Education, Culture and Science. Retrieved from www.utwente.nl/cheps

De Boer, H., & Van Vught, F. (n.d.). Higher Education Governance in the Netherlands: from a Janus- head to a Trimurti. Retrieved from https://ris.utwente.nl/ws/portalfiles/portal/5596904

de Lourdes Machado, M., Brites, R., Magalhães, A., & Sá, M. J. (2011). Satisfaction with higher education: Critical data for student development. European Journal of Education, 46(3), 415–432. https://doi.org/10.1111/j.1465-3435.2011.01489.x

de Oliveria Santini, F., Ladeira, W. J., Hoffmann Sampaio, C., & da Silva Costa, G. (2017). Student satisfaction in higher education: a meta-analytic study. Journal of Marketing for Higher Education,

27(1), 1–18. https://doi.org/10.1080/08841241.2017.1311980

Dougherty, K. J., Sosanya, M. J., Lahr, H., Natow, R. S., Pheatt, L., & Reddy, V. (2016). Looking Inside the Black Box of Performance Funding for Higher Education: Policy Instruments, Organizational Obstacles, and Intended and Unintended Impacts. RSF: The Russell Sage Foundation Journal of the Social Sciences, 2(1), 147–173. Retrieved from https://www.rsfjournal.org/doi/pdf/10.7758/RSF.2016.2.1.07

44 Elliott, K. M. (2002). Key determinants of student satisfaction. Journal of College Student Retention,

4(3), 271–279. Retrieved from

http://journals.sagepub.com.ezproxy2.utwente.nl/doi/pdf/10.2190/B2V7-R91M-6WXR-KCCR Elliott, K. M., & Healy, M. A. (2001). Key factors influencing student satisfaction related to recruitment

and retention. Journal of Marketing for Higher Education, 10(4), 1–11. Retrieved from https://www-tandfonline-

com.ezproxy2.utwente.nl/doi/pdf/10.1300/J050v10n04_01?needAccess=true

Elliott, K. M., & Shin, D. (2002). Student Satisfaction: an alternative approach to assessing this important concept. Journal of Higher Education Policy and Management, 24(2). https://doi.org/10.1080/1360080022000013518

Field, A. (2012). Repeated Measures ANOVA. Issues with Repeated Measures Designs. Retrieved from http://health.uottawa.ca/biomech/courses/apa6101/Repeated Measures ANOVA.pdf

Field, A. (2013). Discovering Statistics Using IBM SPSS Statistics. (M. Carmichael, Ed.) (4th ed.). SAGE Publications.

Forrester, G. (2011). Performance management in education: milestone or millstone? Management in Education, 25(1), 5–9. https://doi.org/10.1177/0892020610383902

Halachmi, A. (2005). Performance measurement: test the water before you dive in. International Review of Administrative Sciences, 71(2), 255–266. https://doi.org/10.1177/0020852305053884

Harvey, L., & Green, D. (1993). Defining Quality. Assessment & Evaluation in Higher Education, 18(1), 9–34.

Heinrich, C. J. (2002). Outcome-Based Performance Management in the Public Sector : Implications for Government Accountability and Efectiveness. Public Administration Review, 62(6), 712–725. https://doi.org/10.1111/1540-6210.00253

Higher Education and Research Review Committee. (2012). Work Method and Assessment Framework. Retrieved from http://www.rcho.nl/images/NL PerformAgr Assessment Framework 2012.pdf Hillman, N. W., Tandberg, D. A., & Gross, J. P. K. (2014). Performance Funding in Higher Education:

Do Financial Incentives Impact College Completions? The Journal of Higher Education, 85(6), 826–857. https://doi.org/10.1080/00221546.2014.11777349

Hladchenko, M. (2014). Performance agreements in Dutch higher education. Euromentor Journal, 5(3), 34–47. Retrieved from https://ris.utwente.nl/ws/portalfiles/portal/18868866

Honig, M. I. (2006). New Directions in Education Policy Implementation: Confronting Complexity. In M. I. Honig (Ed.), New Directions in Education Policy Implementation: Confronting Complexity

(pp. 1–23). Albany, New York: State University of New York Press. Retrieved from https://www.sunypress.edu/pdf/61303.pdf

Interstedelijk Studenten Overleg. (2016). Goed onderwijs begint bij de docent. Retrieved from https://www.iso.nl/website/wp-content/uploads/2016/06/Docentkwaliteit-Goed-onderwijs-begint- bij-de-docent.pdf

45 Interstedelijk Studenten Overleg, Landelijke studentenvakbond, Ministerie van Onderwijs Cultuur en Wetenschap, Vereniging Hogescholen, & Vereniging van universiteiten. (2018). Investeren in Onderwijskwaliteit. Kwaliteitsafspraken 2019-2024.

Kelchen, R. (2018). Do performance-based funding policies affect underrepresented student enrollment?

The Journal of Higher Education, 3, 1–26. https://doi.org/10.1080/00221546.2018.1434282 Kelchen, R., & Stedrak, L. J. (2016). Does Performance-Based Funding Affect Colleges’ Financial

Priorities? Journal of Education Finance, 41(3), 302–321. https://doi.org/10.1353/jef.2016.0006 Kickert, W. (1995). Steering at a Distance: A New Paradigm of Public Governance in Dutch Higher

Education. Governance, 8(1), 135–157. https://doi.org/10.1111/j.1468-0491.1995.tb00202.x Kotler, P., & Fox, K. F. A. (1995). Strategic marketing for educational institutions (2nd ed.). Englewood

Cliffs; Upper Saddle River, N.J.: Prentice-Hall.

Li, A. Y. (2018). Performance Funding Policy Impacts on STEM Degree Attainment. Educational Policy, 00(0), 1–38. https://doi.org/10.1177/0895904818755455

Linden, A. (2015). Conducting interrupted time series analysis for single and multiple group comparisons, 15(2), 480–500.

Lindsay, A. (1992). Concepts of Quality in Higher Education. Journal of Tertiary Education Administration, 14(2), 153–163. https://doi.org/10.1080/1036970920140203

Maassen, P. A. M., & Van Vught, F. A. (1988). An Intriguing Janus-Head: The Two Faces of the New Governmental Strategy for Higher Education in the Netherlands. Source: European Journal of Education European Jfournal of Education, 23(1/2), 65–76. Retrieved from http://www.jstor.org/stable/1502965

Mark, E. (2013). Student satisfaction and the customer focus in higher education. Journal of Higher Education Policy and Management, 35(1), 2–10. Retrieved from https://www-tandfonline- com.ezproxy2.utwente.nl/doi/pdf/10.1080/1360080X.2012.727703?needAccess=true

McDonald, G. (2013). Does size matter? The impact of student-staff ratios. Journal of Higher Education Policy and Management, 35(6), 652–667. https://doi.org/10.1080/1360080X.2013.844668 McLellan, D. L., Cabán-Martinez, A. J., Nelson, C. C., Pronk, N. P., Katz, J. N., Allen, J. D., …

Sorensen, G. (2015). Organizational characteristics influence implementation of worksite health protection and promotion programs: Evidence from smaller businesses HHS Public Access.

Journal of Occupational & Environmental Medicine, 57(9), 1009–1016. https://doi.org/10.1097/JOM.0000000000000517

OCED. (2014). OECD Reviews of Innovation Policy: Netherlands 2014. OECD Publishing. Retrieved from http://dx.doi.org/10.1787/9789264213159-en

OCW. (2011). Quality in diversity: Strategic Agenda for Higher Education, Research and Science. Den Haag.

OCW, & HBO-raad. (2011). Hoofdlijnenakkoord OSW - HBO-RAAD.

46 Institutional Spending Patterns. Journal of Public Administration Research and Theory, 22(4), 675–700. https://doi.org/10.1093/jopart/mur069

Reviewcommissie, A., & Onderwijs, H. (2014). Interventies uitval en rendement, (november), 1–25. Reviewcommissie Hoger Onderwijs en Onderzoek. (n.d.-a). Eindadviezen hogescholen. Retrieved June

30, 2018, from http://www.rcho.nl/eindadviezen/eindadviezen-hogescholen

Reviewcommissie Hoger Onderwijs en Onderzoek. (n.d.-b). Eindadviezen universiteiten. Retrieved June 30, 2018, from http://www.rcho.nl/eindadviezen/eindadviezen-universiteiten

Reviewcommissie Hoger Onderwijs en Onderzoek. (n.d.-c). Overzicht van indicatoren (verplicht en facultatief). Retrieved from http://www.rcho.nl/images/RC overzicht indicatoren.pdf

Reviewcommissie Hoger Onderwijs en Onderzoek. (2017). Prestatieafspraken: het vervolgproces na 2016. Advies en zelfevaluatie.

Sanford, T., & Hunter, J. M. (2011). Impact of performance-funding on retention and graduation rates.

Education Policy Analysis Archives, 19(33), 1–30.

Schertzer, C. B., & Schertzer, S. M. B. (2004). Student Satisfaction and Retention: A Conceptual Model.

Journal of Marketing for Higher Education, 14(1), 79–91. Retrieved from https://www.tandfonline.com/doi/pdf/10.1300/J050v14n01_05?needAccess=true

Shin, J. C. (2010). Impacts of performance-based accountability on institutional performance in the.

Higher Education, 60(1), 47–68. https://doi.org/10.1007/s10734-009-9285-y Taylor, A. (2014). Standardised Effect Size in a Mixed/Multilevel Model.

Umbricht, M. R., Fernandez, F., & Ortagus, J. C. (2017). An Examination of the (Un)Intended Consequences of Performance Funding in Higher Education. Educational Policy, 31(5), 643–673. https://doi.org/10.1177/0895904815614398

van den Broek, A., Wartenbergh, F., Bendig-Jacobs, J., Braam, C., & Nooij, J. (2016). Monitor Beleidsmaatregelen 2015. Nijmegen. Retrieved from https://www.mboraad.nl/sites/default/files/monitor_beleidsmaatregelen_2015.pdf

Vereniging Hogescholen. (n.d.-a). Definities personeel kengetallen. Retrieved from https://www.vereniginghogescholen.nl/system/knowledge_base/attachments/files/000/000/172/or iginal/Definities_personeel_kengetallen.pdf?1438936767

Vereniging Hogescholen. (n.d.-b). Feiten en Cifjers. Retrieved May 12, 2018, from http://www.vereniginghogescholen.nl/kennisbank/feiten-en-cijfers

Vereniging Hogescholen. (n.d.-c). Vereniging Hogescholen | Sectoren. Retrieved June 26, 2018, from https://www.vereniginghogescholen.nl/sectoren

Vereniging Hogescholen. (2018a). Bron en definities studentenaantallen Vereniging Hogescholen (Vol.

1). Retrieved from

http://www.vereniginghogescholen.nl/system/knowledge_base/attachments/files/000/000/171/ori ginal/Bron_en_definities_studentenaantallen.pdf

47 http://www.vereniginghogescholen.nl/system/knowledge_base/attachments/files/000/000/841/ori ginal/factsheet_studentenaantallen_2017_definitief.pdf?1518013848

VSNU. (n.d.). Basiskwalificatie Onderwijs. Retrieved June 14, 2018, from https://www.vsnu.nl/bko.html

VSNU. (2016). Prestaties in perspectief. Eindrapport Hoofdlijnenakkoord en prestatieafspraken. VSNU Prestaties in perspectief. Retrieved from https://vsnu.nl/files/documenten/Nieuwsberichten/VSNU_Prestaties_in_perspectief_%28web%2 9.pdf

VSNU. (2018). Aantal ingeschreven studenten. Retrieved May 1, 2018, from https://www.vsnu.nl/f_c_ingeschreven_studenten.html

Wild, L., & Ebbers, L. (2002). Rethinking student retention in community colleges. Community College Journal of Research and Practice, 26(6), 503–519. https://doi.org/10.1080/2776770290041864 Zestor. (n.d.). Inventarisatie trajecten Basiskwalificatie Didactische Bekwaamheid (BDB). Retrieved

June 25, 2018, from http://www.zestor.nl/inventarisatie-trajecten-basiskwalificatie-didactische- bekwaamheid-bdb

Zijlstra, W., Asper, H., Amrani, A., Tupan-Wenno, M., Van Stapele, M. C., & Van Stapele, N. (2013).

48

Figures and tables

Figure 1. Causal Logic of Performance Funding Policies (Rabovsky, 2012, p. 679) ... 14 Figure 2. Conceptual model of the hypothesized development of student satisfaction (H1) and retention (H2) over time and the suggested impact of student satisfaction on student retention (H3). . 18

Figure 3. Conceptual model of the hypothesized impact of student satisfaction on student retention (H3) and the assumed impact of the quality measures on student satisfaction and student retention (H4). ... 18

Figure 4. Mean Student satisfaction rates at Dutch universities of applied sciences from 2011 to 2015. N = 160. ... 25

Figure 5. Mean Retention rates at Dutch universities of applied sciences from 2003 to 2014. N = 420. ... 27

Figure 6. Conceptual model (Figure 3) revisited. Outcome of the analyses of the impact of student satisfaction on student retention (H3) and the impact of the quality measures on student satisfaction and student retention (H4). ... 33

Table 1. Summary Descriptive statistics table with all variables included in the analyses. Assignment of variables to statistical analyses according to time periods. ... 24

Table 2. Results of the repeated-measures ANOVA for the time trend of student satisfaction at Dutch universities of applied sciences in 2011 to 2015 (N = 160). Display of pairwise comparisons to show contrasts between groups (Differences in mean student satisfaction rates in universities of applied sciences at a certain time point x)... 26

Table 3. Results of the repeated-measures ANOVA for the time trend of student retention at Dutch universities of applied sciences in 2003 to 2014 (N = 420). Display of pairwise comparisons to show contrasts between groups (Differences between mean student retention rates in universities of applied sciences at a certain time point x). ... 28

Table 4. Results of random parameters in the Linear Mixed Model analysis of the impact of quality measures on student satisfaction (H4). ... 29

Table 5. Results of fixed effects in the Linear Mixed Model analysis of the impact of quality measures on student satisfaction (H4). ... 30

Table 6. Results of random parameters in the Linear Mixed Model analysis for the impact of quality measures on student retention (H4). ... 31

Table 7. Results of fixed effects in the Linear Mixed Model for the impact of student satisfaction on student retention (H3) and the impact of quality measures on student retention (H4). ... 32

Table 8. Frequency distribution of the Number of enrolled students in academic year 2011/2012 ... 1

49 Table 9. Frequency distribution of the created variable ‘Size’ of universities of applied sciences, N = 35 ... 2

Table 10. Frequency distribution of the created variable ‘Profile’ of universities of applied sciences, N = 35 ... 2

Table 11. Non-significant interaction effect between time and the size of universities of applied sciences ... 3

Table 12. Non-significant interaction effect between time and the profile of universities of applied sciences ... 4

1

Appendix. Extended analysis

The output of the repeated-measures ANOVA for the development in student satisfaction rates over time showed a significant contrast in student satisfaction scores between the year of policy implementation (T2) and 3 years after (T5). This means that student satisfaction scores have significantly increased after the introduction of the performance agreements. A spontaneous interest aroused in whether the trend in student satisfaction differed according to institutional factors. This extension of the analysis aimed to obtain more detailed information about the conditions under which the trend in satisfaction occurred.

The first step of this extended analysis was to define institutional factors according to which universities of applied sciences would be categorized. The author oriented herself toward previous research to define institutional factors (McLellan et al., 2015). McLellan et al. (2015) investigated whether organizational factors in terms of size, sector, leadership support and organizational capacity were associated with the implementation of worksite health protection and promotion programs in smaller businesses. The authors defined size as the number of employees in each company (operationalized as a dummy variable) and categorized businesses across four broad sectors on a nominal variable.

Since information on size and profile could be easily retrieved in the case of Dutch universities of applied sciences, the author chose to investigate whether the trend of student satisfaction differed across institutions regarding their size and profile.

The idea behind including the factor ‘Size’ was that universities of applied sciences with more students and thus more financial resources might have had more capacities to respond to the introduction of the performance agreements so that the trend of student satisfaction might have been stronger at these universities. Universities of applied sciences were rated from ‘very small’ (‘1’), ‘small’ (‘2’), ‘large’ (‘3’), and ‘very large’ (‘4’). Similar to the procedure in McLellan et al. (2015), the categories had been determined on basis of the max. and min. value and the median of the number of enrolled students in academic year 2011/2012, i.e. the first measurement moment included in the ANOVA. Table 8 provides the Frequency distribution for the variable ‘Number of enrolled students’ in AY 2011/2012. The median was chosen to create categories due to the uneven distribution of universities of applied sciences regarding their size. The boxplot can be found in the appendix. Universities of applied sciences were assigned categories on basis of the following intervals: Category 1 [473; 1797], Category 2 [1798; 4067],