(JFAIP) has identified the need for a repeat firelighter risk screening measure
(McDonald, 2010; Kurt, 2014). As previously highlighted, a limited number of risk screening measures exist for use specifically by fire services (refer to Chapter 3). Some established tools are extensive, intrusive, and not suited to the particular skills of fire services practitioners. Furthermore, they are generally theoretically-based and/or have not been subject to tests of validity and reliability (Henderson, MacKay, & Peterson- Badali, 2010). Study 1 in this thesis attended to the latter by evaluating the validity of the pre-established Family Risk Survey (FRS) and Child Risk Survey (CRS; Moynihan & Flesher, 1998) using a Victorian JFAIP sample. In brief, the results showed that the FRS demonstrated predictive capability but limited specificity. It appeared to be an overly inclusive tool that was able to detect most of the repeat firelighters, but
incorrectly over-classified about half of the non-repeat firelighters as at-risk. In contrast, the CRS was found to not distinguish between repeat and non-repeat firelighters in the JFAIP sample. Additionally, the Behaviour Risk Tool (BRT) was empirically developed in study 1 to be tested for validity and reliability in study 2 of this project.
Investigating the validity and reliability of the BRT was the overarching aim of study 2. Primarily, this study sought to determine if the BRT demonstrated more utility for screening young firelighters in the JFAIP than the FRS. The first step (Study 2: Part A) was to assess the BRT’s validity with a new sample, and determine a numerical cut- off point that would distinguish cases as predicted repeat and non-repeat firelighters, depending on whether their score fell above, below, or on the cut-off value. The ability of the BRT to accurately detect a repeat firelighter (sensitivity) and the degree of
correctly identified non-repeat firelighters (specificity) was then compared to same parameters for the FRS. Firstly, it was hypothesised that one third of the sample would be repeat firelighters post-JFAIP-intervention (as was expected and found in study 1). It was also hypothesised that the screening tool would demonstrate predictive validity for determining a future repeat firelighter from a future non-repeat firelighter. The second step of study 2 (Part B) was to assess the reliability of the BRT. Firstly, an evaluation of the BRT’s test-retest reliability across time was performed, along with an exploration of the internal reliability/consistency of the BRT. It was hypothesised that the BRT would demonstrate both test-retest reliability and internal reliability/consistency.
Method
Each part of study 2 utilised data collected from different samples, depending on the requirements of the aims being addressed. Part A utilised a sample made up of parents/carers of JFAIP cases who completed the BRT and provided follow-up information about their child’s repeat/non-repeat firelighting behaviour. This
information was used to assess the predictive validity of the BRT (i.e., to separate repeat and non-repeat firelighters). In contrast, Part B utilised a community sample (whose repeated firelighting status was unexplored) to assess the test-retest reliability of the BRT over a (minimum) two-week interval. A community sample was deemed appropriate since questions surrounding the stability of the instrument are important regardless of repeat firelighter status. Both samples were used to assess the internal reliability/consistency of the BRT (reported under Part B). The method of data collection for both samples used in this study is outlined below.
Part A: Predictive Validity of the Behaviour Risk Tool Participants
Participants were recruited as a result of their contact with, and completion of, the JFAIP. The parents/carers of young participants who engaged with the JFAIP during the period December 2014 to June 2016 were asked to complete the screening tool developed in the previous chapter (the BRT). During this time, 270 cases were referred to the JFAIP; 204 were actioned and the remaining 66 resulted in no action from the JFAIP. Consent to contact the parents/carers for repeat firelighting follow-up was requested and demonstrated via a tick box option on the front of the paper version of the BRT. Eighty parents/carers (39%) of the actioned cases provided consent to be
contacted for follow-up. Of the 80 cases, 77.57% were able to be contacted for follow- up resulting in a total sample size of 62. Most of this JFAIP sample were male (96.8%). The age range was 5 to 17 years (m = 12.43, SD = 3.25), with the exception of one 30- year-old with an intellectual disability who was considered eligible for the study because his intellectual functioning age was 12 years. The only two females in the group were aged 11 and 15 years. As in the previous study, descriptive statistics were used to provide an overview of the younger and older firelighters separately (where the split was at > 12 years). The new sample consisted of an uneven proportion of younger (n = 24, m = 9.00, SD = 1.84, range 5 to 11 years) and older (n = 38, m = 14.65, SD = 1.58, range 12 to 17 years) firelighters.
Materials
The BRT, as developed by Victoria University and JFAIP, (Appendix T) was used in this study.
Procedure
Ethics approval. Appropriate approval for the collection and analysis of Part A data was sought and granted for this project (HRE14-203) in 2014 from Victoria University’s Human Research Ethics Committee (HREC).
Data collection. At the time of the pre-intervention interview conducted by a JFAIP practitioner, each parent/carer was asked to complete the 25-item BRT. In addition, they were asked to tick the appropriate box if they consented to being
contacted for follow-up by the author. Each parent/carer was provided with an envelope to ensure confidentiality when returning their completed BRT to the JFAIP practitioner before their departure. The JFAIP practitioner passed the sealed BRT envelope to the JFAIP coordinator, who subsequently forwarded the BRT and contact details of the consenting parents/carers to the author for follow-up. It is worth noting that the individual BRTs completed during this period were not used to screen cases for risk because the optimum cut-off value that would distinguish between predicted repeat and non-repeat firelighters had not yet been determined (one aim of this study). Hence, no JFAIP cases were predicted to be repeat firelighters and similarly no referrals to mental health services were made based on the outcome of this measure.
A follow-up phone call was conducted after a minimum 6-month period to determine the presence or absence of repeat firelighting (using the same criteria as study 1). While an attempt was made at 6 months to contact each parent/carer, some follow- up attempts were conducted after a longer period (up to 18 months) depending on the parent/carer’s availability at the time of contact. Each parent/carer was asked the following questions:
1. Has your child continued to light fires or play with fire since the JFAIP sessions were completed? Yes/ No?; and
2. Has your child used mental health services since the completion of the JFAIP sessions? Yes/ No? If yes, how many sessions have been attended? Question two above only sought basic information (e.g., yes/no and number of sessions). Some parents provided additional brief qualitative information relating to the types of mental health services utilised, but this was not required.
A number of parents/carers were difficult to contact for follow-up and therefore, in cases where a direct telephone call failed to reach them, a text message was sent to each of them seeking a Y or N to indicate Yes or No response respectively to the above questions. Six parents/carers responded via this method.
Part B: Test-retest Reliability and Internal Consistency of the Behaviour Risk Tool Participants
Participants were parents/carers of children from the general community aged between 5 and 17 years with no specific history of firelighting. In total, 115 responses were collected at Time 1. Thirty-nine parents/carers did not complete the Time 2 survey (2 weeks later), which meant that data was analysed for a total of 76 participants. The frequency of male children was 54.9%. The mean child age (n = 76) was 9.20 (SD = 3.17) for Time 1 and 9.24 (SD = 3.20) for Time 2. Four children had birthdays in the period between test and retest which accounts for the slight age increase in the mean. Over 90% of respondents were the mothers of the children reported on. Fathers and one step-mother made-up the rest of the respondents. All participants were from Victoria, Australia, and were recruited via snowball sampling from social media pages of the research team.
Materials
The 25-item BRT (Appendix T) was used to collect the data. Additionally, demographic information relating to the child’s age, sex, and their relationship to the participant, was recorded.
Procedure
Ethics approval. Appropriate approval was sought and granted for Part B of this project (HRE17-038) in 2017 from HREC.
Data collection. A Qualtrics online survey was developed to collect the data, specifically, responses to the demographic questions and the 25 items of the BRT, as well as an email address for the retest survey. The survey link was posted on social media pages with an advertisement seeking participation (Appendix U). Participants were provided with a description of the study and gave informed consent by agreeing to participate online. The survey was brief, taking approximately 5 minutes to complete for each child. Parents with multiple children were given the opportunity to complete the survey for each of their children in order from oldest to youngest. At Time 1, participants were asked to provide an email address so that they could be sent a link to the follow-up survey 2 weeks later.
An automated message (Appendix V) with information and contact details for the JFAIP was set-up to pop-up on the screen for any parent who responded positively to the first five questions relating to firelighting behaviour (e.g. items 1 to 5 on the BRT). While this study did not specifically seek the parents of firelighting young people as participants, it was certainly possible that a child with a history of firelighting
behaviour or interest would be identified based on their parent’s responses. In the interest of ethical practice, it was important to make any parents reporting firelighting behaviour or interest aware of the services provided by the JFAIP. At Time 1, 52% of
parents/carers responded positively to one or more of these items and received the pop- up information about the JFAIP. It is unknown how many (if any) of these families contacted the JFAIP seeking their services as a result of the pop-up message. These participants were still included in the sample assessing the test-retest reliability of the BRT.
Two weeks after the first survey was completed (Time 2), a link to the follow-up survey was sent to the corresponding email address provided by parents/carers. The survey was identical to the first one with the exception that a request for an email address was no longer included. At Time 2, 43.8% of parents/carers responded positively to the fire-specific questions resulting in the pop-up information about the JFAIP. An independent research assistant downloaded the completed surveys into a Statistical Package for Social Sciences, Version 22 (SPSS) dataset and matched the Time 1 and 2 survey responses by parent email and age of children to allow for the anonymity of participants to be maintained during statistical analyses (considering researcher social media accounts were used to recruit participants).
Data Analysis: Parts A and B
The raw data collected for Part A was stored securely at Victoria University. An electronic dataset with the corresponding raw data was developed for statistical analyses to be conducted. An electronic database of Sample B data also was downloaded for use. SPSS was used to analyse the data. For Part A, descriptive statistics were used to determine the rate of repeat firelighting and provide a basic overall description of the repeat and non-repeat firelighters. The receiver operating characteristic (ROC) area under the curve (AUC) analysis was conducted for the screening tool. If any tool is able to perfectly discriminate between repeat and non-repeat firelighters at a particular cut- off value, the value of the AUC of the ROC curve would be 1.0. If a tool’s ability to
distinguish between repeat and non-repeat firelighters was equal to chance, the AUC value would be 0.5. An AUC ranging from 0.5 to less than 1.0 is indicative of a test that
is better at distinguishing than chance, but is still flawed. The ROC curve plots the
sensitivity of the measure at each possible cut-off value against the parallel 1-
specificity. Sensitivity reflects the extent to which a screening measure can avoid false negatives (e.g., failing to detect a repeat firelighter). Specificity represents the
measure’s ability to avoid false positives (e.g., inaccurately classifying a non-repeat firelighter as “at-risk” of repeat firelighting). In any screening measure, there generally needs to be a trade-off between sensitivity and specificity. The optimal cut-off value is determined by the researcher based on the most suitable sensitivity and specificity (Safari, Baratloo, Elfil, & Negida, 2016). More specifically, the cut-off value is determined by the investigator depending on what is considered to be more important (e.g., over-classifying negative cases to ensure fewer positive cases are missed or under- classifying positive cases correctly in order to avoid classifying a negative incorrectly).
For Part B test-retest reliability was determined via a Pearson product-moment correlation coefficient measure of the association or consistency between the Time 1 and Time 2 data (Pallant, 2009). The assumptions of normality, linearity, and
homoscedasticity were assessed and found not to be violated, relevant to a Pearson’s correlation coefficient (r; Pallant, 2016). Cronbach’s alpha was used to measure the internal reliability/consistency of the BRT (for both samples), based on the average inter-item correlations and the number of items.
Results
The results that describe the sample of young repeat and non-repeat firelighters in the JFAIP sample are presented first. The results of the sensitivity and specificity analyses of the BRT are then reported, followed by the findings in relation to the
reliability analyses conducted with the JFAIP sample and the sample from the general population.
Part A: Predictive Validity of the Behaviour Risk Tool
In the total sample of 62 JFAIP cases, the repeat firelighting rate post-JFAIP intervention was 16.1%. The repeat and non-repeat firelighting children and adolescents ranged from 5 to 17 years of age (plus one 30-year-old with a mental age equivalent to 12 years) with no significant difference in the mean age between the groups, t = -1.16 (60), p = .52, CI = -4.26 to 1.14. A chi-square test of contingencies (α = .05) showed that age group (5 to 11 and 12 to 17 years [plus one 30-year-old]) was not significantly related to repeat firelighting status, Fisher’s Exact Test p = .166, two-tailed. In addition, the repeat firelighting rate of males alone was 15%, and for females, 50%. The latter figure should be interpreted cautiously as there were only two female participants in the entire sample, but is nonetheless important to note. Of the total sample, 27.4% had engaged with mental health services to some extent within the period between completion of the JFAIP and follow-up. Specifically, 50% of the repeat firelighters reported using mental health services, and 23.1% of the non-repeat group reported the same. There was a significant relationship between repeated firelighter status and mental health service use post-intervention, Fisher’s Exact Test p = .037, two-tailed. This indicates that significantly more repeat than non-repeat firelighters were linked in with mental health services to some extent. Information about the type and level of mental health services utilised by the young people in this sample is largely unknown. This data was not specifically sought, but some parents/carers provided some brief qualitative information in regards to the mental health services used in the follow-up period by the young person. This information indicated varying degrees of mental health intervention from very brief contact or early stages of assessment through to
more intensive, ongoing, and regular therapy. The limited and inconsistent information obtained pertaining to the type and degree of mental health service use by young people in this sample prevents further interpretation of the possible impact of mental health services on firelighting behaviour.
Sensitivity and specificity. Fifteen participants (24%) failed to respond to at least one question on the screening tool, either accidentally or by choice. However, of the total number of questions across the entire sample, 98.7% received responses. Table 13 displays information relating to the missed questions. This missing data needs to be considered and taken into account. The most ideal approach to managing the missing data was explored using a few methods described below (options one to three). Table 13
Number and Percentage of Each Item on the BRT with Missing Data
Item Number of missing
responses
Percentage of missing responses 2 How many other family members have a
fascination with fire?
5 8.1%
3 How many times has your child hidden matches, lighters or other fire-starting materials?
2 3.2%
4 How curious is your child about fire? 3 4.8%
5 How often do you worry about your child playing with fire when left unattended?
2 3.2%
10 Is your child very shy? 2 3.2%
18 Does your child eat more than he/she should?
1 1.6%
19 Does your child show depressed mood? 2 3.2%
20 Does your child physically fight with peers or siblings?
1 1.6%
Option one: Analyses using raw totals (missing values become zero). Firstly, an independent samples t-test analysis was conducted to compare the repeat and non- repeat firelighters based on the raw totals of the BRT. Therefore, missing values were inherently considered as a value of zero. The analysis demonstrated there was a significant difference between the mean total screening tool score for repeat (m = 68.10, SD = 11.89) and non-repeat firelighters (m = 52.21, SD = 19.27), t = 2.51(60), p
= .015, CI = 3.21 to 28.56.
The ROC AUC analysis was then used to determine how useful the screening tool was for predicting repeat firelighters, and what the most suitable cut-off total was to classify future repeat and non-repeat firelighters. The AUC value was .784 indicating that the screening tool was a significantly (p < .01) fair to good measure for predicting continued firelighting (Safari et al., 2016). The most optimal cut-off value was 57.5 demonstrating a sensitivity of .80 and a specificity of .31, indicating that the screening tool was 80% sensitive and 69% specific. This infers that the screening tool with that cut-off score can accurately detect a future repeat firelighter 80% of the time but will inaccurately classify a non-repeat firelighter as at-risk for approximately 30% of not at- risk cases.
Option two: Analyses using mean response for missing values. A second independent samples t-test was conducted on the total BRT score. However, this time, missing values were replaced with the mean response score across the number of questions that were responded to (e.g., if the participant responded to 24 questions, the raw total was divided by 24 and the resulting mean score was rounded to the closest whole number to replace the missing value, and a new total score was calculated). The