Finally, surveys have been criticised for lacking policy relevance (Viotti and Gusmao, 2007; Colecchia et al., 2007). Arundel (2007) demonstrates this point via the lack of references to CIS indicators in a study featuring interviews with 67 members of the policy community across 19 countries, and by the noted dominance of traditional innovation indicators in policy rhetoric, related program targets, and academic studies, which also indicates an enduring influence of the linear model despite proclamations of its demise (Salazar and Holbrook, 2004; Arundel, 2007). Perhaps the best demonstration of this critique is via reference to the primary mandate of innovation surveys – to provide a mechanism to benchmark and compare inter-country innovation performance. This is hindered by the lack of comparability of data due to remaining cross-country methodological differences, but also due to inadequacies of the main innovation rate indicators based on the proportion of innovative firms. Arundel‟s (2007) study notes that less than 5% of European program expenditures on innovation were non-R&D based. Thus with policy directed at high R&D intensity firms and surveys geared towards non-R&D types of innovation, Arundel (2007) argues that the lack of policy interest comes as no surprise. Prevalent rate based indicators fail to distinguish between different levels of „innovativeness‟ for firms with both non-R&D and R&D modes of innovation, and better indicators could inform a wider understanding of innovation and discourse to inform policy.
The uptake of innovation survey indicators for policy related measurement exercises has some way to go. For example, in the 2011 European Innovation Scoreboard exercise, of 26 featured indicators used for calculating indices, only 6 are sourced from innovation surveys. Incidentally, 4 of these are rate based indicators. The OECD STI Outlook for
76
2010 (OECD, 2010) provides comparison of 13 innovation indicators for Australia with the OECD average, only 3 of which are sourced from innovation surveys and all are rate based indicators (rate of firms collaborating, rate of firms with new to market product innovation, rate of firms with non-technological innovation). Arundel et al. (2008) note that the cross sectional nature of many CIS based studies has limited many analyses, and that study of causal relationships provide analyses of greater policy relevance. This requires panel data, which is limited in availability, and impacted by methodological factors such as low sampling fractions and changes in methods and content over time.
What factors are important to the policy relevance of indicators? And what is meant by policy relevance? These questions are considered here to provide an appreciation of the need for new indicators, and the gaps that motivate this study. It must be made clear that it is not the intention to explore particular policy issues or policies, but rather to consider how innovation survey indicators might provide better information content to support policy, given the limited penetration so far in this respect.
As discussed in section 2.2.5, and noted in the literature (Arundel et al., 2008; Arundel and Mohnen, 2003; Smith, 1998, 2005; Archibugi and Pianta, 1996), a key advantage of innovation survey indicators is offered by wide coverage across large populations of firms, and for providing a descriptive, economy-wide picture of the distribution and patterns of innovation. In an early discussion on the policy relevance of survey indicators, Pianta and Sirilli (1998) note the need for balance between two approaches to policy, supporting larger, high technology based firms that are highly innovative, or assisting smaller firms to become more innovative. The challenges that different types of firms face in terms of innovating can be vastly different. For example, larger firms can often focus on incremental innovations that maintain their advantage in particular product markets, though face an „innovator‟s dilemma‟ when disruptive technologies emerge (Christensen, 1997). In contrast smaller firms could be highly innovative in dynamic sectors such as information and communications technologies. Innovation survey indicators will be inadequate for capturing all innovations of course, such as those outside of the business sector (such as in public sectors organisations) or in micro- businesses excluded from most surveys, however they can produce indicators that can
77
better reflect the diversity in innovation activities, and outputs across different types of firms.
Thus to tell a story of relevance to policy, indicators need to be able to capture the different patterns and modes of innovation across firms in different industries and of different sizes. They need to reflect differences in innovation performance across the economy, the different inputs to innovation processes, including technology, research, knowledge, and capabilities, and different innovation outputs, such as new products, or new marketing methods. They need to reflect varied degrees of novelty in outputs (e.g. new to world or new to business), and the systemic environment, consisting of different interactions and knowledge flows between firms and other organisations and institutions.
Indicators need to reflect these elements in order to „tell a story‟ that is relevant to the design, monitoring and evaluation of policies (Finnbjornsson, 2008). Arundel and Hollanders (2008), in their discussion of innovation scoreboards (which draw on numerous innovation indicators), note that indicators have policy relevance by acting as „early-warning‟ systems for potential problems at an economy-wide level, for tracking changes in strengths and weaknesses, and for helping to motivate reactions across government and businesses that may result in improved innovation capabilities. Veuglers (2007, p.35) discussion highlights the role that indicators play in assessing „innovative capacity‟, defined as „the ability of systems not only to produce new ideas but also to commercialise a flow of innovative technologies in the longer term‟. Much of the focus in the repeated series of European Innovation Scoreboard reports (EIS, 2001; 2003; 2004; 2005; 2006; 2008; 2009; IUS, 2012), which are designed to inform European innovation policy, is on tracking national strengths and weaknesses across different dimensions of innovation, in order to build a picture of innovation capability, and how it develops and changes over time. Thus drawing on relevant parts of the literature, „innovation capability‟ can be defined as the ability to successfully turn innovation inputs (activities and investments such as R&D and non-R&D activities) into innovation outputs (new products, processes, marketing or organisational methods) (Smith et al., 2012).
78
It should be noted here that the author has worked with local, state, and federal policy agencies in Australia in relation to production and interpretation of innovation indicators, and this experience partly shapes assumptions behind the concept of „policy relevance‟. Synthesising this experience, discussions in the literature above and in previous sections, it can be noted here then, that by policy relevance, indicators should meet any or all of the following objectives:
1. They reveal strengths or weaknesses in innovation performance or characteristics.
2. They provide a map of the patterns of innovation activities, inputs, outputs or impacts.
3. They provide some differentiation between different levels of innovation intensity, novelty or capability across firms or firm groupings.
4. They reveal changes or trends in innovation characteristics or performance over time.
5. They provide results that may inform policy directed at firms operating in different sectors, of different sizes, or in different regions.