Manejo posparto en el bovino lechero para maximizar la fertilidad

The demographic questionnaire employed in stage two of the study was identical to that designed for stage one of the study (refer to section 2.5.1 for details).

89 5.5.4 Cloze procedure

The first measure used to compare the unrevised and revised reports was the cloze procedure test. As described in the first chapter, this test is thought to be a valid and reliable measure of reader comprehension, with the advantage that it is applicable to any document and is easy to score (Friedman et al., 2009). A cloze test was developed for both the unrevised and revised reports following the procedure outlined below.

Consistent with Doak et al., 1996’s recommendations, an extract of approximately 300-350 words was taken from the report. Specifically, the unrevised extract length was equal to 305 words, whereas the extract from the revised report was equal to 325 words. The first and last sentences of both passages remained intact, with every fifth word deleted between these sentences and replaced with an underlined blank space of uniform length. Applying this procedure to the unrevised report resulted in 50 cloze units, whereas 52 units were formed for the revised document. This number is consistent with the recommendation that the generated “cloze test should have about 50 blanks to fill in’ (Doak et al., 1996, p. 35) A copy of the cloze test produced for each version is provided in Appendix F.6 and F.7, respectively.

Scoring the cloze tests used an exact-match method, where the proportion of correct insertions made by the participant was calculated, and then multiplied by 100. The resulting percentage was then compared between the two groups, and in relation to traditional

comprehension tests.

5.5.5 Subjective questionnaire 5.5.5.1 Overview

As explained in section 1.3, an important step in the revision of health documents is the process of learner verification, where feedback from the target audience is sought to

90

verify the suitability of the document. Although it would have been preferable for this process to involve interviews with audience members (Doak et al., 1996), conducting

interviews was not feasible during this stage of the study due to time and resource constraints. Additionally, because the research questions for this stage of the study focus on comparing the two versions of the report, it was desirable to collect quantitative data.

Thus, a questionnaire was used in this study to satisfy these requirements and compare the two versions along measures of comprehension, self-efficacy and parental perception. The questionnaire served an important role in (1) reinforcing the results of the cloze test through an additional comprehension measure, (2) assessing self-efficacy which is known to be associated with health outcomes, and (3) understanding whether the report actually satisfied the needs of parents. As a literature review failed to yield any current questionnaires that address these three constructs, a new questionnaire was conceptualised.

5.5.5.2 Item development

The developed questionnaire consisted of 3 sub-scales: comprehension, self-efficacy and perception. The items included in each subscale were generated from a combination of literature review, expert opinion, and the interview data from part one of the study.

Six multiple-choice questions were included to evaluate reader comprehension. These were based on questions used during an ‘introduction to hearing loss’ lecture given to adult learners. Each question aimed to assess parental understanding of critical information in the report, including test purpose, the meaning of test results, the type of hearing loss and future recommendations. Each multiple-choice question used a 4-choice answer format, and only one of these answers was correct for each question.

Self-efficacy was measured by asking participants to note their level of confidence in understanding the terms, test procedures, test results and recommendations included in the

91

report. It was ensured that the items were task specific rather than addressing global beliefs, as this is thought to provide a better estimate of an individual’s self-efficacy (Desjardin, 2003). Participants were asked to mark their confidence on a 10-point scale, where 0

indicated participants felt ‘not confident at all’ and 10 signified that the participant felt ‘very confident’.

Participant perceptions of the report were gauged through eight questions, which were developed by revisiting the themes present in the interview data from stage one of the study. For example, the commonly reported issue of difficult language was evaluated by the

statement ‘I thought the report used too much jargon’, whereas the need for revision of the order of the report was assessed through the item ‘The order of information in the report was helpful’. Both positively and negatively worded items were included in this subscale, in an attempt to minimise acquiescence response bias, that is, the tendency for participants to agree with statements irrespective of their content (Rattray & Jones, 2007). Participants indicated their opinions of the report using a fixed choice, 5-point scale of agreement, where 1 indicated ‘not at all’, 3 indicated ‘moderately’ and 5 ‘very much’.

5.5.5.3 Scoring

All responses to the questionnaire were scored by hand and subsequently entered into a Microsoft Office Excel spreadsheet. Every correct answer to one of the comprehension multiple-choice questions was awarded a ‘1’, whereas incorrect answers were awarded ‘0’. The scores for each question were then tallied to provide the total comprehension score, where the minimum possible score was 0 and the maximum possible score was 6. The total self-efficacy score was calculated by adding the ratings given for each self-efficacy item. Possible scores on this sub-scale ranged from 0 to 40, with higher scores indicating greater self-efficacy. To score the opinion sub-scale of the questionnaire, the number circled by participants for the positively worded items were added. For negatively worded items, the

92

number circled was first reverse-scored and then combined to the score from the items above. The total opinion score range from 8 to 40, where higher scores indicate a more positive opinion of the report.

5.5.5.4 Pilot testing

Assessment of both the content and face validity of this questionnaire occurred prior to the questionnaire being used in this study. First, to evaluate content validity, an audiologist with extensive research experience was asked to comment whether she felt the items

accurately represented what the questionnaire was designed to measure. The face validity was assessed by asking three target audience members to read the questionnaire. Each individual was then asked about their view of the comprehensiveness, order and the clarity of the questionnaire.

Following these assessments, only minor amendments were made to the

questionnaire. No items were removed, but four items were added; including three fixed- response questions and one open-ended question. The open-ended question simply asked participants if they had any comments about the readability and comprehensibility of the report. This provided participants an opportunity to expand on their answers and express additional perceptions about the report.

5.5.6 Readability

Both the unrevised report and the revised report were subjected to the same readability analysis as was described in section 2.4. Specifically, the analysis was again performed using the Readability Studio (Windows) 2012.1 software (Oleander, 2014), with the same textual features selected. The four formulas utilised in stage one of the study (the F- K, FRE, SMOG and Fry) were again used, and the Bormuth Cloze mean was also added. This test was included for interest as a comparison between the predicted cloze score and the cloze

93

score actually obtained by participants for each version of the report. It should be noted that the unrevised report and the glossary were excluded from the readability analysis of the revised report as both were considered only supplementary to the revised report.

5.6 Statistical Analysis

The statistical analyses performed during this study used the Statistical Package for the Social Sciences (SPSS version 19). The statistical tests were chosen based on the characteristics of the data, but the significance level was always set at α = 0.05, and a clinically significant effect size was considered to be d = 1.0.

First, the readability estimates and the textual features of each report were compared. Second, a combination of descriptive statistics, chi-squared tests and Analysis of Variance (ANOVAs) were used to describe the participants in the study and test for any significant differences in the demographic characteristics of the two report groups. Next, a multivariate Analysis of Variance was conducted to determine whether a linear combination of cloze score, comprehension score, self-efficacy score and perception rating was significantly affected by the assigned version of the report each group was assigned. Follow-up univariate ANOVAs using a Bonferroni correction were then conducted to investigate the effect of report allocation on each respective outcome variable. Correlation analyses were also used to investigate whether cloze scores, comprehension scores, self-efficacy scores or perception ratings were significantly related to education level. Subsequently, an Analysis of Covariance (ANCOVA) was used to determine the effect of report allocation on the assessed outcome variables while education level was controlled for as a covariate.

94

CHAPTER SIX

RESULTS STAGE THREE: VERIFICATION

6.1 Overview

This chapter presents the results of the data collected in stage three of the study: verification. To begin, the results of the readability assessment are presented and compared between the unrevised and revised versions of the report. Next, the results of the experimental verification process are described. In total, 32 participants were recruited for this part of the study, with half of these individuals randomly assigned to read the unrevised report, while the remaining half were assigned the revised report. The two groups did not significantly differ on the compared demographic variables, but did significantly differ on the two

comprehension measures, in addition to their self-efficacy scores and perception ratings. These differences remained statistically significant with large observed effect sizes even when level of education was controlled for.