The assessment of effectiveness, for this analysis, involved first looking at key performance measures (rates of job entry and sustained jobs) and then adjusting these to take into account the different client profiles of the Job Brokers. Rates of job entry and sustained jobs were derived from the NDDP evaluation database. Differences in client profiles were assessed through a combination of the evaluation database and the qualitative research data on Job Broker selection practices.
7.2.1 Performance as measured by the evaluation database
The NDDP evaluation database provides systematic information about performance, based on Job Broker services’ claiming of job entry and sustained work payments. It was therefore used as a starting point in identifying different levels of effectiveness among the Job Brokers in the in-depth study.
Local level performance data were derived from the evaluation database. This involved defining geographic areas, based on the recorded home addresses of registered clients, which matched, as closely as possible, the service delivery area described by Job Broker managers and staff and reflected in the selection of clients and Jobcentre Plus staff. This was relatively straightforward where there was just one Job Broker team in the organisation covering either a prescribed area or operating at the national level, and where the discussion in the staff focus group reflected this coverage. Where the Job Broker organisation had a number of teams of which the in-depth study focused on only one, or a single national team which was not fully reflected in the composition of the staff focus group, the research team proceeded by selecting the local authority areas or Jobcentre Plus districts which matched the area of operation of the staff involved in the in-depth study.
The evaluation database does not distinguish between individual organisations which form part of a consortium or partnership to deliver Job Broker services. Data are available only at the level of the partnership as a whole. As a result, where the in- depth study had involved one partner only, it was not possible to match the performance data with that specific job organisation.
Data from the commencement of the Job Broker services (at various points in 2001) to November 2003 were used, partly because the more recent months’ data are incomplete and partly because there are unusual patterns in job entry and registration around Christmas and New Year, which might have distorted the picture. It is important to note here that the Job Brokers in the in-depth study began operating at different times in 2001 and in 2002. The contractual arrangements changed over this period with, on 1 October 2003, the reduction of the in-work period required for a sustainable job payment from 26 to 13 weeks and the increase in the registration fee. The database records a job as sustained at the point when a sustained payment is claimed. This produces a difficulty in comparability, since for newer Job Brokers, more of their operation will have been during a period when the shorter definition of sustained work was used.
The evaluation database was used to look at job entry rates (ie the proportion of registrations which result in a job entry payment) and sustainable job rates (ie the proportion of job entries which result in a sustained job payment) for all Job Brokers. It was not possible to take into account the interval between registration and job entry. Since there are differences between Job Brokers in the types of clients with whom they work, and some use other services prior to registering clients on NDDP, this would not be very meaningful without complex further analysis. The total set of Job Brokers were grouped into three bands, based on their job entry performance,
with equal numbers of Job Brokers in each band. The local performance of the Job Brokers studied in-depth was then mapped against these groups, that is, identifying whether each Job Broker matched the performance of the highest, middle or lowest third of performers nationwide. The same process was carried out, separately, with local level performance of the in-depth study Job Brokers in terms of sustainable job rates.
7.2.2 Adjustments to reflect differences in client profiles
It was also necessary to recognise, however, that there are differences in the client profile of the Job Brokers in the in-depth study. These reflect the types of clients with whom organisations have historically worked, differences in the routes by which clients come to the Job Broker services, and differences in organisations’ practices around registration, discussed in Chapter 2. It is clear that some Job Brokers work particularly with clients who might be expected to face more barriers to entering work, and it is important to take this into account in comparing performance. The main information provided by the evaluation database about client profile is the duration of the benefit claim which preceded registration. The evaluation database was, therefore, used to identify, for all Job Brokers, the proportion of their clients whose latest benefit claim was of 24 months’ duration or longer. (The analysis was repeated using a definition of five years-plus duration, but the eventual groupings of the in-depth study Job Brokers did not change.) Two equal-sized groups of national Job Brokers were then constructed, representing those with above and below average proportions of clients with 24 months’ duration of benefit claim. Again, the local level performance of the in-depth study Job Brokers was mapped against these two groups.
However, the qualitative research data provides richer information about client profiles, and the grouping of the in-depth study Job Brokers was, therefore, adjusted in the light of what had been learned from the fieldwork with Job Broker managers and staff. This refined the evaluation database information in several ways:
• first, duration on benefit is a fairly crude proxy for the type of service a client is likely to need to move into work, and sustained work. And because the data are based on the duration of the most recent claim, it will under-record durations for clients who moved from a non-eligible benefit such as Jobseeker’s Allowance to an incapacity benefit. It also does not reflect cumulative claim durations, where clients have a number of periods on incapacity benefits interspersed with intervals on another benefit or in work;
• second, the qualitative data allowed other indicators of the proximity of clients to work to be taken into account, such as whether the Job Broker specialised in working with people with more severe or enduring impairments, and whether clients came to the service through routes which might indicate more barriers to work (such as mental health service or learning disability services);
• third, it meant it was possible to look at registration practices, in particular whether other internal services are used to provide early work preparation prior to the client being registered on NDDP, and whether registrations are slanted, deliberately or not, towards clients who are perceived as being closer to work readiness; and
• finally, some Job Brokers begin working with clients but do not notify the Department of the client’s registration until they feel more confident that the client will take up work, conscious of the Department’s monitoring of the target of 25 per cent of registrations resulting in job entry.
The qualitative data were, therefore, used to adjust the initial categorisation of the in-depth study Job Brokers based on the proportion of clients with benefit claim durations of 24 months-plus. This produced two groups based on a broader set of indicators of clients’ proximity to work – a further-from-work group and a closer-to- work group. This resulted in four Job Brokers changing groups, all from the further- from-work to the closer to work group. The judgements made here were not always clear cut, particularly because some Job Brokers had begun to change their registration practices only quite recently at the time of the Wave Two fieldwork. It was not possible to reflect differences in local areas, in terms of labour markets or the provision of relevant services, in the analysis. It is also important to note that the number of sustained jobs, job entries and, less often, registered clients was below 100 for some Job Brokers, and sometimes below 50. This means that quite small changes in the number of clients, job entries or sustained jobs could change the groups to which an individual Job Broker was initially allocated on the basis of the database. The eventual allocation of Job Brokers between groups was, however, agreed across the research team.