Managing the risk of suicide in acute psychiatric inpatients: A clinical judgement analysis of staff predictions of imminent suicide risk

Abstract
Background: Predicting suicide risk in psychiatric in-patients in order to inform risk management decisions is compromised by the poor predictive validity of the available models.
Aims: This study explored the factors influencing judgements regarding suicide risk in psychiatrists and nurses working in acute psychiatric in-patient units in Scotland.
Method: Clinical judgement analysis. Information used by 12 psychiatrists and 52 nurses to make judgements about suicide risk were analysed over 130 hypothetical cases. Correlations and linear regression analysis were used to examine judgement consistency and information use.
Results: There was agreement between clinicians on the relative but not absolute degree of risk of each patient case. Consistency of judgments was low, particularly amongst nurses. All clinicians rated those with more previous suicide attempts, men, those with shorter admission times, and those who were less compliant and not improving clinically as at greater risk of suicide. Conclusions: Clinicians use cues that have been associated with suicide in traditional predictive models based on epidemiological studies and short term factors that may be particularly relevant to acute psychiatric settings. The inconsistencies observed can be interpreted to cast doubt on the validity of predictions of risk for imminent suicide and the role of such predictions in the assessment process.

Introduction
Psychiatric in-patients are at significantly increased risk of completing suicide in comparison
to the general population in all countries reflecting admission criteria (Eagles et al., 2001;
Powell et al., 2000). Preventing such suicides occurring is therefore a legitimate concern in
acute psychiatric care settings (National Confidential Inquiry, 2006) Despite criticisms of
the current practice in the area suicide risk assessments continue to be a mainstay of such
efforts and predictions of suicide risk are used to inform decisions that are potentially critical
to patient safety such as the level of observation/engagement ordered (National Confidential
Inquiry, 2006).
Clinicians working in acute psychiatric in-patient environment are tasked with assessing
who may be at increased risk of imminently completing suicide over periods of hours, rather
than months. However, the existing models of proven predictive validity are based on large
scale community samples and identify factors that are predictors for suicide over the ‘‘longterm’’
and at a group level. They are unable to accurately predict which individual patients
will commit suicide over the short term (Cassells et al., 2005; Hughes, 1995). In the absence
of tools with proven predictive validity how clinicians use the information available to them
in order to make suicide risk judgements and whether or not a consensus exists between
practitioners over who is at a higher risk are therefore important questions which, are
addressed in this study.
Clinicians are rarely able to accurately describe the clinical information they use
when making decisions (Denig et al., 2002; Harries et al., 2000). Clinical judgement
analysis is a research method that examines the relationship between an individual’s
judgement and the information they use to make that judgement (Harries et al., 1996).
Using linear regression techniques it identifies the relative weight or importance
individuals attach to different information items, without relying on potentially unreliable
self-reports. These ‘‘captured’’ judgement policies can then be compared between
individuals to identify areas of agreement or disagreement (Harries et al., 1996). The
approach has been used successfully within medicine to explore how clinicians use
information to inform their diagnosis of heart failure (Skane´r et al., 2000), and to explain
variation in practice for prescribing decisions (Backlund et al., 2000; Harries et al.,
1996).
This article present the results of a study which used clinical judgement analysis to
explore the factors influencing judgements regarding suicide risk of a group of clinicians
working in acute psychiatric in-patients in Scotland. The study examined the information
cues that clinicians used to inform their judgements of suicide risk, comparing them to
risk factors identified by a review of the literature (Cassells et al., 2005). The study also
considered the reliability of clinicians’ predictions. This study formed part of a larger
study examining the relationship between predictions of suicide risk and decisions
regarding observation/engagement whose results are not reported here and are the focus
of a further paper in preparation. Ethical approval for this study was granted by Lothian
MREC.
Method
The information used by clinicians to make risk judgements was analysed over a series of
hypothetical patient cases. The cases were presented in written form in a booklet.
Hypothetical cases are useful in this context for several reasons. First, it allows us to
study a complex judgement process, associated with a relatively rare outcome, easily.
Second, the correlation between cues across cases can be minimised in order to allow for
identification of the influence of individual cues on each person’s risk of suicide
judgements. Third, it allows for a subset of cases to be repeated within the booklet to
measure the consistency of judgements. Finally, it also allows for direct comparison
between clinicians, and clear measurement of agreement as all clinicians see the same set
of cases.
Participants
All psychiatrists and registered mental health nurses involved in the assessment of suicide
risk in acute psychiatric in-patient settings, in four Primary Care Trusts in Scotland were
invited to take part in the study.
Procedure
On agreeing to take part, participants were sent a data collection booklet, consisting of a
questionnaire and the judgement task. The questionnaire was designed to collect
demographic data on participants, together with details of the environment where they
worked.
The Judgment task
The judgement task consisted of a set of 130 hypothetical cases (scenarios), each defined in
terms of 13 pieces of information, plus 15 cases that had been picked at random and
repeated. An example of a case is shown in Figure 1. Each booklet consisted of the same set
of cases, presented in the same order.
For each case, participants were asked to judge how likely it was that the patient described
in the scenario would try to commit suicide within the next 24 hours. They indicated this on
a 10 cm bar (see bottom of Figure 1) anchored on the left with ‘‘no risk’’ and on the right
with ‘‘very high risk’’. Their judgement of likelihood on this bar was encoded as a rating
between 0 (no risk) and 100 (very high risk). They were also asked to state what observation
level they thought the patient should have as an intervention (on pass, general observation,
constant observation, special observation). These observation levels were those in use in the
majority of Scottish services at the time of the study (Table I) (Scottish Executive, 2002).
Participants returned the completed booklet to the researchers in a prepaid envelope. Only
data from fully completed and returned booklets was analysed.
Case profile
The 13 different potential predictors of successful suicide attempts in acute psychiatric inpatient
populations identified by (Cassells et al., 2005) were used as a basis for the cases.
These factors were a mixture of more traditional ‘‘long term’’ predictors (e.g., psychiatric
diagnosis, previous self harm) and more short term predictors (e.g., changes in clinical state,
comorbid drug or alcohol use).
For each of these predictors, a number of different potential levels of severity related to
suicide risk were developed (Table II). For each of the predictors, at each level, a number of
verbal descriptions were constructed and validated by a panel of experts. The descriptors
were developed following interviews with experienced clinicians. A computer program
(written in visual basic 6 by CH and adapted by DD, based on the cue generation program
used in (Evans et al., 1995), was then used to randomly generate a number of series of 130
scenarios. For each predictor, each level had an equal probability of being included in each
scenario. These sets of scenarios was sampled and re-sampled until, for each set that was
generated, the inter cue correlation was negligible.
To ensure that the final sample of 130 scenarios used in the study had face and content
validity they were selected from those scenarios that represented real in-patient cases with a
distribution designed to mimic the overall in-patient caseload. Scenarios were examined for
face validity by a panel of 4 experts (operationally defined as experienced psychiatrist/mental
health nurse practitioners). Participants were aware that no observation level had been
decided in these cases. Scenarios that did not represent a realistic acute in patient case were
discarded. Caseload distribution was determined via a local survey of the range and
prevalence of diagnoses within in-patient services together with an examination of national
statistics. The final number of scenarios used in the study was based on a ‘‘rule of thumb’’
suggesting that between 5 – 10 scenarios are necessary for every item of information used, to
ensure sufficient variety in the judgements that are made, and to provide stable statistical
estimates of cue weights (Cooksey, 1996; Harries & Harries, 2001).
Analysis
All data were analysed using SPSS (version 12.0). Clinicians’ judgements of likelihood that a
patient would attempt suicide within the next 24 hours, rated as a mark on a bar were
encoded as a rating between 0 (no risk) and 100 (very high risk). To examine the extent to
which clinicians agreed on the relative degree of risk for each case (i.e., if they identified the
same case as being at a higher or lower risk), Pearson’s correlations between risk judgements
by each pair of clinicians was calculated for each of the participants, and the mean
correlation was calculated via Fisher’s z transformation (Fisher, 1921). Agreement between
clinicians’ judgements’ of risk across all cases was measured using Kendall’s W measure of
concordance. Kendall’s W varies from 0 (no agreement) through to 1 (perfect agreement).
(Howell, 1992, pp. 280 – 282).
The reliability of clinician’s risk judgements were also examined by calculating
Spearman’s rho correlation on the two sets of judgments for the 15 cases that were
repeated within the vignette booklet. These 15 and their original equivalents form a test and
retest set of cases. The mean for nurses and for psychiatrists was calculated separately via
Fisher’s z transformation and an independent samples t-test of the mean difference was
used. Finally, an individual’s judgements of risk across the 130 cases were standardized and
regressed onto standardized cue values giving a set of standardized regression coefficients
(their judgement policy). These indicate how the participant used each item of information
to judge suicide risk. Mean differences in the beta-weights attached to information use were
analysed using independent sample t-tests.
Results
Participant characteristics
Twenty eight psychiatrists and 92 nurses consented to take part in the study (from a
potential pool of 88 psychiatrists/269 nurses) with 12 psychiatrists and 51 nurses returning a
completed booklet (53%). The mean age of participating psychiatrists was 39 years (SD 7.9;
range 25 – 53), 50% were male and 50% female. The mean age of participating nurses was
40 years (SD 8; range 20 – 54), 40% were male and 60% female.
Agreement between judgements of suicide risk
There was considerable variation in both psychiatrists’ and nurses’ absolute ratings of the
suicide risk for each individual vignette. On average the range between the lowest and highest
ratings for a vignette was 61.3 for psychiatrists and 78.4 for nurses. The range was over 75 in
15/130 (11.5%) cases judged by psychiatrists, and 79/130 (60.8%) cases judged by nurses.
The extent to which clinicians agreed in terms of the relative degree of risk was calculated by
examining the extent to which their judgements for the vignettes correlated with each other
(a significant correlation would indicate that they agreed on cases that were at greater/lesser
risk than others). The correlation of judgements for each clinician across all 130 cases was
compared to all other clinicians, giving a total of 3294 comparisons. Of these 3060 (92.9%)
were significantly positively correlated (p5.01). Agreement ranged from an r=0.018 to
r=0.733, with a mean correlation of 0.416. Agreement between psychiatrists was slightly
greater (98.5% of comparisons significantly positively correlated, mean correlation 0.486)
than agreement between nurses (92.9% of comparisons significantly positively correlated,
mean correlation 0.412).
Concordance between clinicians was assessed using Kendall’s W. The greatest agreement
was between the psychiatrists (W=.5, n=12), with agreement between the nurses slightly
lower (W=.41, n¼51). Concordance when comparing the judgements of the whole
clinician group was also lower (W=.41, n=63).
Reliability of risk judgements
Of the 12 psychiatrists, 7 (58%) had significant (p5.01) correlations between their risk
predictions on their test and retest cases (mean correlation 0.614, range .158 – .737). Of the
51 nurses 11 (22%) had significant correlations (p5.01) between their risk assessments
(mean correlation 0.479, range 0.024 – 0.844). An independent samples t-test of the mean
difference in Fishers z transformations of consistencies indicated that psychiatrists showed
greater reliability in their judgements than nurses did (t(61)=72.053, n=63, p5.05).
Factors influencing risk judgements
Individual judgement policies, which examined how each clinician used information to
reach their judgement of suicide risk, were calculated. The number of significant cues in a
judgement policy that predicted risk judgements varied between clinicians, ranging from 1
to 6. The average number of cues for both psychiatrists and nurses was 3.7, with a median
of 4. There was substantial variation in the fits of the linear regression models (R2)
between participants. The mean adjusted R2 for psychiatrists was 0.34 (median=0.34,
range .22 – .51) and for nurses 0.28 (median 0.29, range .04 – .51).
There was also some variation in the way clinicians were influenced by cues. Mean
standardized beta-weights for each of the cues for psychiatrists and nurses can be seen in
Figure 2.
Figure 3 indicates the percentage of clinicians who had a particular cue as a significant
predictor within their judgement policy, together with the direction of the weighting.
Both psychiatrists and nurses associated suicidal ideation with increased suicide risk,
although psychiatrists were significantly more influenced by this cue (t(61)=2.103,
n=63, p5.05). Psychiatrists were also significantly more influenced by the patient’s
diagnosis than nurses were (t(61)=5.387, n¼63, p=1.0). The number of previous
suicide attempts, being male, lack of clinical improvement, lack of compliance and
shorter admission times were also associated with higher risk of suicide judgments by
both groups of clinicians.
Three of the cues (insight, adverse events and protective factors) were not significant
predictors at all in psychiatrists’ judgement policies. Two cues (co-morbidity and insight)
were not significant predictors in nurses’ judgement policies. Suicidal ideation and previous
suicide attempts were important factors for the majority of psychiatrists and nurses. For 50%
of the psychiatrists, but only 8% of nurses, diagnosis was also an important predictor.
Clinical improvement, length of admission, gender, compliance and hopelessness were
important predictors for between approximately 20% and 40% of clinicians.
Discussion
Clinicians who took part in this study rated the patient cases represented in the scenarios at
varying degrees of risk, often with very large differences in ratings. A patient case therefore
could have likelihood ratings of completing suicide from one practitioner of 25 and from
another of 100. However, when comparing the relative degrees of risk, there appeared to be
consensus regarding cases that were of relatively higher risk, compared to others. Therefore,
although individual clinicians may have different anchor points on a scale related to what
they consider high or low risk to be, they did appear to agree on who was at higher risk
compared to another patient.
What is perhaps of more concern are the results of the analysis of the reliability of
clinicians risk judgements, across the 15 repeated patient cases. Overall psychiatrists were
more likely to provide roughly similar risk assessments for the same case at two different
time points, than nurses. However, for a significant proportion of psychiatrists (42%) and
the majority of nurses (78%), risk judgements across the same patient case at two different
time points were significantly different. This implies that the predicted risk of suicide in the
same patient, exhibiting the same symptoms and behaviour, seen at two different times by
the same clinician could vary substantially. However, low reliability, agreement and
accuracy is also associated with greater uncertainty in the decision task (Harvey, 1995).
These findings may reflect the inherent complexity associated with the prediction of suicide
in this population. An increased familiarity with the task is however, associated with
increased consistency and given the pre-eminent role ascribed to psychiatrists in the decision
making process in most settings this may explain the greater reliability observed in
psychiatrists’ judgements of risk (Shanteau et al., 2003).
Clinical judgement analysis is a method that examines the relationship between the
judgements’ that individuals make and the information they use to make them. The analysis
of judgement policies of clinicians that took part in this study highlighted variation in the
number of information cues clinicians use to inform their judgements of suicide risk, and
some variation in which cues are used according to professional group.
The psychiatrists in our sample were more likely to use the patients’ diagnosis as a
predictor of the likelihood of suicide than nurses, and appeared to place more significance
or weight on the presence of suicidal ideation as a predictor than nurses. However, what is
evident is the extent to which there is considerable agreement between the two groups on
the relative significance of other factors such as previous suicide attempts, gender, length
of admission, clinical improvement, compliance and hopelessness when assessing suicide
risk. Predictive models derived from epidemiological studies suggest that factors such as
suicidal ideation, previous suicide attempts, diagnosis, gender and length of admission are
long term predictors of successful suicide (Cassells et al., 2005) all of which appear as
factors used by clinicians in this study to inform their risk judgements. However,
clinicians’ judgement policies also included other factors, such as clinical improvement,
and compliance when making such risk judgements. These factors, along with others such
as the degree of insight a patient has into their condition, comorbid substance abuse and
social factors (such as the level of social support an individual receives) (Cassells et al.,
2005) have been identified as dynamic or short term factors linked to increased suicide
risk in psychiatric in-patients. It is still uncertain whether using these more short term
factors increases the accuracy of suicide risk predictions within acute psychiatric inpatients.
It is also unclear whether some short term factors are more significant than
others either generally of for individuals when trying to estimate suicide risk. Clinicians’
use of clinical improvement and compliance as predictors of risk within their judgement
policies imply that these factors may have more clinical utility than other short term
factors. However, whether they are more useful in terms of accurate prediction remains
uncertain.
One of the main methodological issues that needs to be considered when examining
the results of studies that use case vignettes is the transferability of subjects performance
from the judgement task simulated in the vignette to their performance in real task
situations. A number of studies have indicated that clinicians performance on ‘‘paper
cases’’ in clinical judgement analysis studies appears to be no different to that on ‘real
patients (Braspenning & Sergeant, 1994; Denig & Rethans, 1996; Kirwan et al., 1983).
In order to increase the transferability, the case vignettes constructed for this study
were based on both evidence from the research literature on key information that is
deemed to be associated with suicide risk in psychiatric in-patients, and expert validity.
However, it should be recognized that clinicians’ information use was limited to that
presented in the case vignettes. Individuals judgements in reality may be influenced by
elements of the task situation such as the context of the ward environment and resource
issues (such as staff availability), which, were not represented in the vignettes used within
this study.
One way to increase the validity of the judgement task that is carried out is to either base
the case vignettes on real patient cases (Skane´r et al., 1998), or to ask clinicians to carry out
judgements in the clinical setting and analyse the patient case retrospectively. Although
these approaches would overcome the limitations of presenting clinicians with paper-based
vignettes, they would also make it harder to control the information that clinicians receive,
or to directly compare clinicians’ judgments and policies. The combination of risk factors
identified and used in this study would probably rarely be recorded in totality for each
in-patient, and often a combination of a number of risk factors together in one
patient case is rare (Powell et al., 2000). If patients are used in clinical practice, then the
ability of comparing clinicians across the same cases for consistency is also lost.
A further issue that should also be acknowledged when considering the results of this
study is the nature of how risk assessments are made in clinical environments. This study
examined the judgements of individual practitioners when studies have highlighted that
this is often a process involving members of the multidisciplinary team (Bowers et al.,
2000).
The prediction of the likelihood of suicide forms one element of the broader process of
risk assessment and management of suicide risk that in turn forms part of the overall care of
the patient. Prediction of suicide risk is however in and of itself a complex task, evidenced by
the difficulties in producing models in those studies that have attempted to identify risk
factors. The importance of clinicians’ judgements is therefore paramount, as they need to
continually evaluate a variety of different information sources to reach a judgement about an
individual patient. Although there were some differences in how psychiatrists and nurses
used and weighted information regarding patients diagnosis and suicidal ideation to inform
their judgements in this study, overall there was remarkable agreement in the relative use
and weighting of information.
Within acute psychiatric in-patient areas, patients who are judged as being at higher
risk of suicide are often identified as requiring intensive support by means of higher
levels of observation/engagement (Bowers et al., 2000). The relationship between risk
assessments and decisions regarding interventions is therefore potentially critical. It has
been suggested that there is a relationship between risk assessments and the level of
observation a patient is placed upon (Kettles et al., 2004). However, research in other
areas has suggested that a clinicians judgements may not necessarily influence their
treatment decisions with the nature of the relationship between judgements reached and
decisions made varying between clinicians (Poses et al., 1995; Sorum et al., 2002).
Despite the relative agreement with which clinicians in this study identified patients they
considered to be at higher or lower risk, relative to other patients, this may not
necessarily lead to agreement regarding the interventions those patients receive. The
relationship between the risk judgements made by the clinicians in this study and their
subsequent decisions regarding observation level are however the subject of a further
paper currently in preparation.
Overall, psychiatrists and nurses appear to agree on the characteristics of patients who are
considered to be at higher or lower risk of suicide, within acute psychiatric in-patient
environments. They use a mixture of both long term predictive factors and more dynamic
short term factors, to inform these judgements. There are some discrepancies in the
importance that different professional groups attach to the patients diagnosis and level of
suicidal ideation, when making judgements. However, it has been possible from this study to
provide some transparency as to the way in which clinicians use different predictive factors
to inform their judgements regarding imminent suicide risk in acute psychiatric in-patients,
an area under explored to date. The inconsistency in risk judgments observed is significant
and must cast some doubt on the validity of using predictive models based on aggregated
risk factors.
Alternative approaches to risk management focused not on statistical models but rather
the lived experiences of service users may though offer scope for improved patient outcomes
in this area. Such improvements may lie however not in any improved accuracy in the
prediction of suicide risk but instead in terms of a decreased likelihood of suicide both
during and after the in-patient episode. A phenomenological perspective on suicide suggests
that we must seek to understand individuals’ unique reasons for suicide and not committing
suicide at a particular point in time. Central to this perspective is the belief that suicide is an
endpoint in a trajectory following high levels of societal, intra and interpersonal stress which
result in unendurable psychological pain described, compellingly by Shneidman (1993a) as
‘‘psychache’’. In the context of such unbearable distress suicide becomes a compelling and
even attractive means of escape. Risk factors such as suicidal attempts and suicidal thoughts
are expressions of distress which can exist independently or co-exist with underlying
pathology. An understanding of the sources of stress, particularly at an individual level is
therefore necessary because it is the individuals idiosyncratically understood psychological
anguish which is the driving force behind suicide and not simply an aggregated collection of
risk factors (Shneidman, 1993b). Adopting a phenomenological model of suicide prevention
seeks therefore to ascertain in partnership with the service user their reasons for living and
dying (Jobes, 2000, p. 11). These reasons, revisited regularly, form the basis of a care plan
which focuses not on the treatment of the disorder but on addressing explicitly these issues
which the patient gives for wishing to kill themselves as treatment priorities, whilst
maintaining and expanding on the reasons for the patient’s ambiguity about suicide (i.e.,
their reasons for not committing suicide). Practice is focused on changing the balance in
favour of living with suicide seen not as a symptom of a mental illness which can be
addressed via treatment of the supposedly underlying disorder but simply as a coping
strategy ‘‘albeit a limited and problematic one’’ (Jobes, 2000, p. 11). The challenge for
clinicians in such circumstances becomes not how to assess risk accurately but instead how
to engage constructively with the service user in order to enable them to want, and be able,
to ‘‘say yes to life’’ (Degan, 1996, cited by Barker, 2003, p. 97) not just in hospital but after
discharge.
The presence of a collection of risk factors whether static, e.g., diagnosis, history of
previous attempts or more dynamic such a recent history of substance misuse or the loss of a
significant relationship may tell us that the individual falls into a particular risk category. It
cannot reliably predict the choices that individual will make in the short term regarding
suicide. Engaging with the individual patient and their lived experience of that world on an
ongoing basis may however allow us to understand why for some patients at some times
suicide can come to seem their only option. Only by doing so can we then begin to explore
with the patient what they feel might need to change in order for them to decide to choose
life over death.
Conclusion
One interpretation of one of the findings of this study in terms of the inconsistent judgement
by clinicians might be that it lends weight to calls for further training of clinicians. However,
where as in this case there is no valid predictive model in which the relative importance that
should attached by clinicians is known then the question becomes what might such training
be expected to deliver in terms of improving the predictive accuracy of the judgement
reached. Practitioners should be under no illusions regarding the irreducible uncertainties
involved in making short term predictions of suicide risk in in-patient settings no matter the
approach used (Simon, 2006). This does not mean that the present practice of routinely
incorporating consideration of such risk factors into the risk assessment process should be
abandoned. The significant limitations involved in predicting risk based only on such
information must however be acknowledged and greater efforts to incorporate a
phenomenological perspective on suicide risk assessment and management into practice
should be made.