As already mentioned, the analysis starts by distinguishing between evaluations that provide a (significantly) positive and a non-positive impact. Table 3 shows the summary of results when using the broad target measure. The first column in the table corresponds to results when only the policy characteristics are included. The probability of obtaining a positive impact (compared to a non-positive one) in an evaluation of a reform involving an employment incentive is significantly higher than that implemented through a decrease in the employers’ SSCs. In contrast, there seems to be no differences with respect to the other two types of reforms. As for the other characteristics of the policy intervention, neither the direction nor the duration and scope seem to affect the result of the evaluation. The only source of variability in the impact worth mentioning is that corresponding to policies that target a specific group of companies. With respect to non-targeted reforms, those targeted at specific companies seem to be less likely to provide a positive impact on employment. Interestingly, there are no difference between untargeted reforms and those directed at a particular group of workers.
The following columns show the results when subsequently adding the three groups of controls. The effect of the type of reform when controlling for differences in the design, publication and macroeconomic context remains largely unaltered. The only difference is in the effect associated with employment incentives, which in some specifications becomes insignificant while in others remains significantly positive. Its effect is only marginally significant when the dummies for the group of countries
are excluded in the specification that controls for the macroeconomic context variables (last column in Table 3). The inclusion of controls does not lead to major changes in the results of the effect of the other policy characteristics, with the interesting exception of that for the group of workers. The specification that includes the three set of controls suggests that the probability of obtaining a positive impact of the reform is higher if the policy was targeted to a particular group of workers.
Overall, the results suggest that when a reform affects only a specific group of the working population, it is more effective in increasing employment than when there is no target. In turn, the evidence derived from the evaluations indicates that interventions designed for particular groups of companies may be less effective even than untargeted reforms.
As for the influence of differences across evaluations in the controls, the results confirm the strong influence of the design of the study itself. In summary, it is more likely to estimate a positive employment impact of the reform when using data drawn from periods of less than a year, and when using administrative registers rather than data from a survey. On the other hand, using matching techniques may increase the chance of estimating a positive impact compared to applying D-i-D in isolation. In any case, the latter econometric method may be linked to a higher propensity to obtain positive evaluations when compared to alternative techniques (such as regression discontinuity).
Regarding the time horizon of the assessment, the results indicate that a positive impact is less frequent in the medium and long term compared to the short term. Less frequent positive impacts are also obtained in studies that combine the assessment on employment and wages versus those that only focus on the impact on employment. On the other hand, the results summarised in Table 3 reveal some differences across groups of countries in the propensity to obtain positive impacts. With respect to the CEE group, which is the reference category, evaluations for the other groups of countries tend to provide less frequent positive effects. The fact that part of the differences across countries vanish when the GDP growth and the unemployment rate are included as controls suggests that some of these differences are caused by disparities in the economic background of the economies in which the policy reform is evaluated.
It can be inferred that the likelihood of a positive impact increases in countries and periods of high pace of growth and decreases when the economy is relatively stagnant. As regards the influence of the unemployment rate, the evidence derived from the meta-regression of evaluations is not as robust that of GDP growth. If anything, the evidence points to a decline in the chances of observing a positive effect when unemployment is high.
The results of the probit meta-regressions obtained when including the detailed target group of workers instead of the variable for the general target are synthesised in Table 4 (overleaf). The inclusion of the detailed groups of targeted workers indicates that there may be some differences in the effectiveness of the policy intervention, depending on the particular group of workers targeted by the reform. Based on the specification that includes all the controls (fourth column of results in Table 4), the probability of obtaining a positive impact on employment is higher when the policy is targeted at female workers and, to a lesser extent, the low- skilled. The differential effect observed for the long-term unemployed in the specification with no controls and
only with controls for differences in the design vanishes when the country group dummies and the macro-context controls are added. The opposite holds true as regards the effect of women. When the dummies for the groups of countries are excluded (fifth column of results in Table 4), a positive effect is again observed for the long-term unemployed and, particularly, for workers with a fixed- term contract. This points to the concentration of the differentiated effect for these groups in specific countries or under specific macroeconomic circumstances. The results of a final set of specifications that add several controls for differences in the labour market institutional setting in which the evaluations were obtained are summarised in Table A5.3 of Annex 5.
Table 3: Probit models for positive versus non-positive impact – general target*
Policy intervention
Type of reform Employment increase ++ O O ++ +
Other O O O O O
Payroll cost O O O O O
Direction Increase O O O O O
Duration Permanent O O O – O
Scope Single reform O O + O O
Target Group of companies – – – – – – – – – – – – –
Group of workers O O O + + + + + +
Design (data, method and so on)
Data frequency Low frequency O O – – – – –
Undated – – – – – – – – – –
Data source Survey O – – – – – – – –
Econometric method Matching + + O O + + +
Other – – – – –
Regression discontinued – – – – – – – – – – – –
Outcomes Employment and wages – – – – – – – –
Number of years analysed – O O O
Time horizon assessed Long term – – – – – – – –
Medium term – – – – – – – – – –
Characteristics of the study
Group of countries Continental – – – –
Nordic – – – – –
Southern – – – – –
Type of publication Other O – – –
Report O O O Working paper O + + + Language English O O O Context – macroeconomic GDP growth + + + + + + Unemployment rate – – O Number of observations 207 207 207 207 207
Joint significance Yes Yes Yes Yes Yes
Pseudo-R2 0.11 0.26 0.30 0.35 0.33
Note: *Full statistical outputs for this table and other results of the meta-analysis are available in an online annex at http://bit.ly/nonwagelabour
Models are probits, fit to binary data with value 1 for significant positive estimates, and 0 for negative and non-significant estimates. + + + positive p < 0.01; + + positive p < 0.05; + positive p < 0.1; – – – negative p < 0.01; – – negative p < 0.05; – negative p < 0.1; O p ≥ 0.1. Based on standard errors clustered by article. Joint significance denotes the result of the Wald test of the joint significance of all the coefficients. Omitted categories are: SSCs, Decrease, Temporary, Comprehensive package, No target group, High frequency, Administrative data, D-i-D, Only employment, Short-term, CEE, Journal article, Other than English.
The main conclusion that can be derived from the results is that none of the labour market controls seem to affect the probability of a positive impact in the evaluation.