As with the elicitation of 𝜌0 the elicitation of γ by individual experts lasted 25 minutes in total (i.e. when using both approaches of the pre-specified doses available in the protocol and the method of allowing the experts to choose their own doses). The process started by the facilitator ascertaining that the relevant kisspeptin literature described in the SHELF framework was complete for the published literature. Again, it was agreed that the unpublished finding from the single volunteer was permissible to be used during the elicitation process. For the first method of individual elicitation, the experts confirmed the range of doses intended to be used in the clinical trial, namely 0.4, 0.8. 1.6, 3.2, 6.4 and 12.8 nmol/kg. For the second method, the experts agreed on the doses of 0.8, 1.6, 3.2, 4, 5, 6, 8, 12, 24 and 25 nmol/kg.
5.3.4.1 Forming Probability Distributions for γ (Pre-specified Doses)
The template form for placing the 30 counters was provided to each expert and the facilitator commenced the next stage of the elicitation process by asking them (without conferring) to place the first counter in the “bin” corresponding to the most likely value of γ. The experts appeared to be able to interpret this instruction
successfully. However, it became apparent during the next step that there was potential for confusion in their original interpretation of that first task.
In the next step, the experts were asked to place their second counter in the bin corresponding to the lowest possible value for γ. The experts expressed confusion so an alternative formulation of the question was posed (at short notice), namely “Please place your second counter in the bin corresponding to the least likely value for the highly likely dose”. This did not appear to increase clarity. A third formulation of the question was attempted which re-formulated the task for all counters, namely “By
- 107 -
assuming that a dose-response curve has a plateau at the 90% response rate then where does that plateau start?”. It is noted that this formulation is not completely satisfactory since there is no guarantee that a plateau exists and furthermore γ can still be defined even without a plateau. Further discussion of this issue appears later.
However, within the revised assumption that a plateau exists then the placing of counters continued in order to describe the beliefs of the experts for the parameter γ.
Expert 1 Expert 2
Figure 11 Results of Individual Elicitation for Parameter γ (Pre-specified Doses)
As shown in Figure 11 and consistent with the similarity of experts for the elicitation of 𝜌0, the range of possible values for γ was the same for the two experts (0.8 to 12.8 nmol/kg) with the distribution for Expert 2 having a slightly higher negative skew compared with Expert 1 (the mode was in the bin corresponding to 6.4 nmol/kg for Expert 1 compared with Expert 2 who thought γ was equally likely to be 6.4 or 12.8
- 108 -
nmol/kg). Also shown in Figure 11 are the hand-annotated smoothed distribution curves although no formal fitting of distributions was performed by the experts.
As was the case when the experts discussed eliciting 𝜌0 earlier, the reasons for the differences in beliefs between the experts was thought to be influenced by the dose at which a hyper-sensitivity effect might occur.
5.3.4.2 Fitting Individual Probability Distributions for Parameter γ (Pre-specified Doses)
The fitting of a formal probability distribution to each individual’s elicited beliefs using the R software that accompanies SHELF was not straightforward because the software enforces 10 (equally-spaced) bins to be used whereas the trial was designed to have 6 pre-specified doses.
Since the pre-specified doses in the planned trial were spaced according to a
doubling-dose rule then the linear scale of bin widths could be mis-leading if the true dosing scale was ignored. For future elicitation sessions I recommend developing software that provides a user-specified number of bins and also permits bin labels that acknowledge unequal spacing of values (e.g. dose levels). Other recommendations are provided in Section 5.5.2.
For the purposes of distribution fitting, an approximate non-uniform spacing was achieved by using only the virtual bins closest to the actual doses. The results of that fitting process are shown in Figure 12.
- 109 -
Expert 1 Expert 2
Scaled Beta(1.47, 2.33) No satisfactory fit of the 6 available distributions in SHELF
Figure 12 Fitted Individual Probability Distributions for Parameter γ (Pre-specified Doses)
The mean value of γ for Expert 1 was estimated to be 4.95 nmol/kg using the Scaled Beta (1.47, 2.33). The 10th and 90th percentiles of the distribution (i.e. 1.3 and 9 nmol/kg respectively) showed consistency with the counters for Expert 1 in Figure 11.
The pre-planned Scaled Beta distribution could not be made to fit the elicited
distribution for Expert 2 and this was due to the equal assignment of probabilities to the top two doses for a negatively skewed distribution. Similarly, none of the five other available distributions in SHELF (Normal, t, Beta, Log-Normal, Log-t and Gamma) could be made to fit the data for Expert 2. For future elicitations sessions, I recommend the development of software that permits fitting a mixture of distributions. For example, Gajewski and Mayo (2006) describe the process of eliciting a mixture of Beta
distributions by eliciting individual Beta distributions from each of two experts and then forming a mixture distribution by eliciting the relative expertise of each expert as an additional parameter which is then used to provide relative weights of the two individual Beta distributions.
- 110 -
Overall, the Scaled Beta distribution appears to be a reasonable representation of the beliefs of the individual elicitation for Expert 1 but not Expert 2 when only the
pre-specified doses from the protocol are considered.
5.3.4.3 Forming Probability Distributions for γ (Unrestricted Doses)
As described in Section 5.2, elicitation was repeated with a single change to the task so that before elicitation started, the experts were able to choose a range of doses that would span all potential values of γ (i.e. the dose highly likely to lead to a response) and not just those available in the protocol. Performing elicitation using two different approaches would permit comparison of the results.
For the second method, the experts agreed on the doses of 0.8, 1.6, 3.2, 4, 5, 6, 8, 12, 14 and 25 nmol/kg. The experts wrote these values under the 10 bins in a new template form and then, without conferring and under the guidance of the facilitator, placed 30 counters to form their probability distribution for γ in the same manner as that described previously for the case of pre-specified doses.
As was the case with pre-specified doses, the experts were most comfortable with the elicitation of γ using the question “By assuming that a dose-response curve has a plateau at the 90% response rate then where does that plateau start?”
- 111 -
Expert 1 Expert 2
Figure 13 Results of Individual Elicitation for Parameter γ (Unrestricted Doses)
The differences between experts in the shape of the probability distributions was more pronounced in this case of unrestricted doses (see Figure 13) compared with elicitation when only doses pre-specified in the protocol were considered (see Figure 11). For Expert 1, the mode was in the bin corresponding to 6.4 nmol/kg when considering the doses available in the protocol and 6 nmol/kg when doses were unrestricted. This contrasts with Expert 2 who thought γ was equally likely to be 6.4 or 12.8 when considering the doses available in the protocol but equally likely to be 14 and 25 nmol/kg when doses were unrestricted. Overall, the distribution for Expert 1 was nearly symmetric whereas for Expert 2 there was a high negative skew. Also shown in Figure 13 are the hand-annotated smoothed distribution curves although no formal fitting of distributions was performed by the experts.
- 112 -
As was the case when the experts discussed eliciting 𝜌0 and γ earlier, the reasons for the differences in beliefs between the experts was thought to be influenced by the dose at which a hyper-sensitivity effect might occur.
5.3.4.4 Fitting Individual Probability Distributions for Parameter γ (Unrestricted Doses)
As in Section 5.3.4.2, due to the non-uniform spacing of doses being considered, only approximate fitting of distributions could be produced using SHELF, though I
recommend future elicitation sessions use software that permits unequal spacing (see Section 5.5.2). The results of that fitting process is shown in Figure 14.
Expert 1 Expert 2
Scaled Beta(3.68, 9.24) Scaled Beta(1.82, 1.37)
Figure 14 Fitted Individual Probability Distributions for Parameter γ (Unrestricted Doses)
The mean value of γ for Expert 1 was estimated to be 7.12 nmol/kg using the Scaled Beta (3.68, 9.24). The 10th and 90th percentiles of the distribution (i.e. 3.4 nmol/kg and 11 nmol/kg respectively) showed consistency with the counters for Expert 1 in Figure 13. If one is prepared to assume that intermediate doses were available (i.e.
doses between the available doubling-doses) then the fitted distribution for Expert 2
- 113 -
was also a reasonable fit when compared with the counters in Figure 13. As such, the mean value of γ for Expert 2 was estimated to be 14.3 nmol/kg.
Overall, If one is prepared to assume that intermediate doses were available then the Scaled Beta distribution appears to be a reasonable representation of the beliefs of the individual elicitation for both experts when the choice of doses was unrestricted. This contrasts with the observations from the difficulty fitting probably distributions when doses were confined to those pre-specified in the protocol. In retrospect, it would have been informative to have conducted the elicitation by transforming the doubling-doses to a linear scale and then compared the resulting prior distributions.