Otras Técnicas de monitoreo y diagnóstico

Deviations from protocol

Table 10presents deviations both from the original funding proposal when finalising our review protocol

at the start of the project, as well as deviations from this protocol in the course of completing the review.

Other limitations

Assessing the quality of theory reports

It is rare for systematic reviews of theory to assess their quality; this is exemplified by those reviews reported by Campbellet al.,146_{Harriset al.}147_{and Kreiger.}148_{Our quality criteria were informed by previous}

work by Michieet al.64_{and Bonellet al.,}131_{and were accompanied by guidance which was applied by two}

researchers who then met to discuss their scores. However, as shown inAppendix 13, reviewers found it difficult to make decisions in applying the criteria and there was very little inter-reviewer agreement on the scores.

There were a number of challenges. Few studies presented a single theory of change; many instead drew on a range of existing theories. This meant that reviewers were uncertain about whether or not they should assess each contributing theory (which would be difficult without references to primary texts) or to give an overall score for each report (which would be limited in terms of transparency). Reviewers also found it difficult to assess criteria of testability and parsimony because, although guidance was provided, this did not provide a clear benchmark against which to make such judgements, and reviewers felt that their judgements often shifted in the course of reviewing the literature. Application of the criterion of generalisability was, to some extent, easier to assess and more likely to result in agreement between reviewers simply because this was a more absolute judgement; most reports had been included because they offered an abstracted and therefore potentially generalisable framework for understanding PYD rather than, for example, reporting local data. However, we were unsure of the extent to which the theories would actually be relevant to varying structural or cultural contexts in practice. Finally, our criteria were intended to assess the quality of causal theories of change and thus were not useful in assessing the quality of the normative elements of theories.

TABLE 10 Deviations from proposal and protocol

From original proposal or

published protocol Deviation Reason for deviation

Proposal Scope: language–any language. Our original proposal did not exclude studies published in other languages; we added this exclusion criterion at an early stage in the review, including this in our registered protocol51

This was decided because PYD interventions have been overwhelmingly developed in English-speaking countries

Proposal Scope: date. Our original proposal did not restrict studies by date. We decided early on the review process to search for and include studies published from 1985 onwards. This exclusion criterion is reported in our registered protocol51

PYD interventions were developed from 1985 onwards

TABLE 10 Deviations from proposal and protocol (continued)

From original proposal or

published protocol Deviation Reason for deviation

Proposal Search: databases and websites. The list of databases included in the proposal was reviewed at the protocol51_{stage. At this early stage in the} review, three databases were not included in the search (CAB Health, EMBASE, PAIS). The Health Technology Assessment Database was added and further efforts were placed in searching topic specific websites relevant to the intervention and health outcomes of interest. The revised list was published in our registered protocol51

On the advice of our search specialist

Proposal Search: journals. Our original proposal stated that we would hand-search the five journals that yield the highest numbers of studies that meet inclusion criteria. However, we amended this aspect of our search, including it in our registered protocol51

We decided very early in the review that the original approach was not a good use of resources, because it would not be specific to reports missed by other approaches

Proposal Pilots of screening at title and abstract. We piloted 100 rather than the original 50 references at title and abstract

This provided more opportunities to discuss potential variations in applying and to aid discuss of the exclusion criteria

Protocol Scope: not in school time. We included one programme (YPPD) in which PYD was in practice delivered in a few sites in school hours, deviating from the intended model of delivery

This delivery was an unintended deviation from YPDP theory of change and occurred only in a minority of sites

Protocol Synthesis of economic evaluations. We did not include any economic evaluations in our synthesis

Our searches yielded no relevant economic evaluations

Protocol Meta-analysis: multivariate meta-analysis. As indicated in the protocol,51

we intended to use multivariate meta-analysis or another method to synthesise effect size. Instead, we used multilevel meta-analysis with random effects at both the outcome and study level

It was not possible to use multivariate meta-analysis or another method to synthesise effect sizes because of the heterogeneity of outcomes and lack of availability of variance- covariance matrix for reported outcomes

Protocol Synthesis: metaregression and qualitative comparative analysis. We stated in the protocol51 that we would use a combination of

metaregression and qualitative comparative analysis to test hypotheses generated from the theory and process synthesis, as well as funnel plots to examine potential publication bias

We were unable to conduct metaregression models to examine subgroup effects because of inconsistent subgroup reporting. We were unable to test hypotheses on other moderators of effects because of insufficient heterogeneity. We were unable to conduct qualitative comparative analysis because of insufficient qualitative variation in effectiveness to examine conditions predicting effectiveness. We were unable to conduct funnel plots because of insufficient studies per outcome Protocol Risk of bias. In addition to allocating a score of

‘high risk’,‘low risk’or‘unclear risk’within each critical appraisal domain, we also applied the code‘not applicable’to studies where codes were not suitable (e.g. methods of sequence generation and allocation concealment for controlled trials and whether or not studies controlled for key confounders in RCTs)

This allowed more transparent reporting of risk of bias

Protocol Partner collaboration. We consulted with a slightly different array of policy stakeholders

This was due to people’s availability for consultation

PAIS, Public Affairs Information Services. DISCUSSION AND CONCLUSIONS

Limitations in included theory reports

The theoretical literature did not in general focus on descriptions of causal theory of change for how PYD interventions might reduce substance use or violence among young people. Much of the literature instead aimed to assert the normative value of PYD as an approach to youth provision. This normative theory was, however, useful in understanding the goals and assumptions of PYD programming. Causal theorising was a minor and generally unsystematic element of most theory reports, with a few exceptions. Although our synthesis of theories of change was hampered by a lack of clarity within included reports about how PYD might optimise young people’s capacity for‘intentional self-regulation’and by a lack of systematic consideration of how promoting positive assets might lead to reductions in risk behaviours, we nonetheless developed a synthesis of causal theory which described the mechanisms by which PYD interventions might reduce violence and substance use. In the case of theorising how PYD interventions affect intentional self-regulation, it went beyond synthesis to fill in some gaps. The synthesis was successful despite the lack of success of our quality assessment of the theoretical literature. We included reports in our synthesis regardless of their quality. Our synthesis involved bringing together theoretical fragments (which specified only certain parts of the pathway from PYD intervention to substance use or violence reduction) and would have been less comprehensive had we synthesised only theories of change that themselves set out a comprehensive and clear path from intervention to risk reduction. Because of this, quality criteria focused on, for example, parsimony would not have proven very useful even if they had been easier to apply.

However, the resulting synthesis was quite‘thin’, particularly in terms of how positive assets might enable reductions in risk behaviours. As well as not constituting a comprehensive theory of how assets reduce risk, the suggested pathways offer little that is distinctive to that provided in more traditional psychological theories used in prevention science such as the social learning model128_{and the social control theory,}135

both of which PYD theorists cite.

Limitations in process and outcome evaluations

Process evaluations overall were generally of low or medium quality. Sampling and analysis methods were poorly reported. Analyses were generally descriptive and did not develop clear, second-order interpretations. Few quotes were used to substantiate the analysis. Nonetheless, we were able to develop a synthesis that provided some useful answers to our RQ concerning the characteristics of contexts and participants that could influence the implementation and receipt of PYD interventions. Only one process evaluation from the UK was included, but this did include similar themes to those reporting from the USA and Australia. The lack of studies from the UK was also apparent in the case of outcome evaluations, as was the lack of studies of cost-effectiveness. There were also various methodological problems with outcome evaluations. Authors rarely presented data in a format that was readily analysable, which meant that our analyses required both extensive transformation of effect sizes and sensitivity analyses. We decided to exclude one study both because of the quality of the evaluation and because of the uninterpretable effect sizes it reported. Although our rationale was transparent, it is possible that another meta-analyst may have taken a slightly different approach or made somewhat different transformation decisions. Moreover, we

performed a sensitivity analysis in which we excluded one study79_{that did not report findings in the same}

standardised metric as other studies (i.e. as change from baseline rather than adjusted by baseline). The force of the conclusions did not change as a result of excluding this study.

The variable quality of evidence (including inadequate adjustment for clustering in several included studies) and the need for extensive data transformation and for sensitivity analysis for key statistical decisions must qualify the interpretation of our statistical results. Although, on balance, it was an appropriate decision to meta-analyse the included studies, challenges we faced with the data may suggest an interpretation of the pooled effect size that relies more on its general magnitude and precision than on statistical significance per se. We were also unable to test the key hypotheses derived from our consultations owing to the nature of the included evidence.

Moreover, although multilevel meta-analysis is perhaps a more robust method than those using one effect size per programme, it was not as robust as the multivariate meta-analysis originally proposed in the protocol. This is because multivariate meta-analysis uses the known variance–covariance matrix between included outcomes to account for dependencies between outcomes within interventions or studies, whereas multilevel meta-analysis achieves this by partitioning the variance between outcomes into that attributable to variation within interventions and variation across interventions.

Finally, there were problems associated with including evaluations of two interventions (PYDC79_{and Stay}

SMART111_{) in the meta-analysis, for different reasons discussed above. Although we ultimately sensitivity-}

analysed findings with Tebeset al.,79_{our findings still may not present the most complete picture of effects}

on substance use.

Finally, we did not perform funnel plots because these would not have been a good guide to publication bias given the small number of studies included.149_{Our very comprehensive search methods, although not}

precluding the possibility of publication bias, mean that we took all reasonable steps to prevent this arising from an insensitive search.