Leeds paediatric heart surgery: managing outliers

Childrens’ heart surgery in Leeds has been suspended. Concerns about an excess in mortality have been raised and denied and I have written about seemingly large variations in mortality (“twice the national average”) being explained by chance.

In June 1998, the then Secretary of State for Health announced the establishment of an inquiry into the management of the care of children receiving complex cardiac surgery at Bristol Royal Infirmary between 1984 and 1995. The inquiry identified failures that contributed to the death children undergoing heart surgery and the 529-page report was a blueprint for wider reform of the NHS.

Funnel plots are useful for comparing the results of surgery between hospitals. The funnel plots below are from here and are for open cardiac surgery in children under one year in the UK 1991-1995. The Cardiac Surgery Registry (CSR) and Hospital episode statistics (HES) data were used to compare institutions. The horizontal dotted line is the national average and curved dotted line the limit of variation which might be expected by chance (95% confidence interval). The “O” is Bristol Royal Infirmary and “*” the eleven other UK centres. Bristol, as became apparent, was a clear outlier.

Funnel plots for open congenital heart surgery in chilldren under 1 year in UK 1991-1995
Funnel plots for open congenital heart surgery in chilldren under 1 year in UK 1991-1995

How should we deal with outliers?

The question is pertinent given the recent suspension of Leeds Royal Infirmary from performing children’s cardiac surgery. The UK Department of Health has produced guidelines in 2011 on the recommended process should a unit hit the dotted line, summarised below.

Stage 1 | 10 days

Hospitals with a performance indicator ‘alert’ or ‘alarm’ require scrutiny of the data handling and analyses performed to determine whether there is:

‘No case to answer’

  • potential outlier status not confirmed;
  • data and results revised in clinical audit records;
  • details formally recorded.

‘Case to answer’

  • potential outlier status;
  • proceed to stage 2.

Stage 2 | 5 days

The Lead Clinician in the hospital is informed about the potential outlier status and requested to identify any data errors or justifiable explanations. All relevant data and analyses should be made available to the Lead Clinician.

A copy of the request should also be sent to the Clinical Governance Lead of the hospital.

Stage 3 | 25 days

Lead Clinician to provide written response to national clinical audit team.

Stage 4 | 30 days

Review of Lead Clinician’s response to determine:
‘No case to answer’

  • It is confirmed that the data originally supplied by the provider contained inaccuracies. Reanalysis of accurate data no longer indicate outlier status;
  • Data and results should be revised in clinical audit records. Details of the hospital’s response and the review result recorded;
  • Lead Clinician notified in writing.

‘Case to answer’

  • It is confirmed that although the data originally supplied by the provider were inaccurate, analysis still indicates outlier status; or
  • It is confirmed that the originally supplied data were accurate, thus confirming the initial designation of outlier status;
  • proceed to stage 5.

Stage 5 | 5 days

Contact Lead Clinician by telephone, prior to written confirmation of potential outlier status; copied to clinical governance lead, medical director and chief executive. All relevant data and statistical analyses, including previous response from the lead clinician, made available to the medical director and chief executive.

Chief executive advised to inform relevant bodies about the concerns: primary care trusts, Strategic Health Authority, professional society/association, and Care Quality Commission. Informed that the audit body will proceed to publishing information of comparative performance that will identify providers.

Stage 6 | 10 days

Chief executive acknowledgement of receipt of the letter.

Stage 7

Public disclosure of comparative information that identifies providers (eg annual report of NCA).

The Situation in Leeds

It appears that in Leeds the process is at stage 2 – the local doctors have just been informed. The guidance suggests the identity of the statistical outliers should be anonymous at this stage. It may be that concerns were so great that special circumstances dictated the dramatic public announcement. We should find out in the next few weeks.