Predicted and actual survival analysis

BCIS Patient Area

Dedicated to providing information for patients

Predicted and actual survival

Data quality
Assessment of predicted outcomes is dependent on the accuracy and completeness of certain key data submitted to NICOR for analysis. This section starts with a measure of the data completeness of these key fields, which are those in the equation used to predict survival (example below).

 

Thirty Day Survival post PCI (ONS Tracked)
Accurately reporting mortality following PCI can be difficult for some centres, particularly those that treat patients and then transfer to another hospital before they are discharged. The Office of National Statistics holds extremely accurate records of all deaths in England and Wales, and was therefore used to provide an independent source of information regarding life status.

 

A risk model in a contemporary cohort of patients was developed to measure risk adjusted survival at 30 days.

(reference: McAllister KSL, Ludman PF, Hulme W, de Belder MA, Stables R, Chowdhary S, Mamas MA, Sperrin M, Buchan IE. A contemporary risk model for predicting 30-day mortality following percutaneous coronary intervention in England and Wales. Int J Cardiol. 2016;210:125–132)

 

A risk calculator has been created and is  accessible at this link

 

Figure: Example of operator data. The horizontal axis shows survival, up to 100% on the far right. Two sets of data are shown (with values given in the table below the graphic). The actual survival to 30 days following the PCI is shown for the operator or centre, and the overall average in England and Wales (green dot and arrow). In addition the predicted survival is given for the operator (or centre) and England and Wales average (black dot and arrow).

 

In previous years we have been able to provide a statistical analysis to determine if the observed survival was significantly outside a range predicted by the risk model. The statistical methodology behind these analyses is complex and evolving. At NICOR, the statistical team have changed and are in the process of revamping the statistical tools and methods.

Using the previous methodology, no individual operator was identified as performing significantly less well than predicted in the years 2014 to 2016. However, until the new statistical team have completed their assessment and possible modification to those methods we decided not to include any confidence intervals.

 

Several observations can be made from the point estimates. In the plot above the National (England and Wales) actual and predicted are very similar which confirms that the risk model remains well calibrated. If an operator’s predicted and actual survival are very similar, this means that the operator is performing as the risk model would predict. If the predicted survival of an operator is lower than the National predicted survival (as in this example), that means that that operator is treating higher risk patients than the national average (and vice versa).

Plots of Risk Factor Prevalence and ‘Missingness’
The table at the top of this section gives overall data completeness, but to help identify potential data errors, and to place the data of any one operator or hospital in perspective, two sets of plots have been produced. In each set, the operator (or hospital) is plotted as a coloured dot, against the background of grey dots that represent all other operators (or hospitals). One set of plots shows data completeness, and the other risk factor prevalence. When the risk adjustment analyses are performed, if any risk factor data are missing, the value is assigned to the lowest risk category.

Figure. Example of 3 panels of the Risk Factor Prevalence Plots. Each dot represents a PCI hospital, and the prevalence of a particular feature is ploted against the total number of PCIs performed during the analysis period. In this example this PCI hospital (in yellow) can be seen to have performed about 5000 procedures over the 3 years and is treating patients that are older than those being treated by most other centres (other centres represented by the grey dots), but the sex ratio and prevalence of diabetes are similar.

 

Discussion

Survival to 30 days after a PCI is dependent on several interacting factors. These include the general well-being of the patient (both the severity of the patient’s presenting condition and their other co-existing medical problems), the risk of the medications given and of the PCI procedure used in their treatment and a variety of aspects of the quality of healthcare provided by the entire team looking after that patient.

Some adverse events may be related to the PCI procedure (such as the need for emergency surgery, or another complication caused by the PCI). Most however occur in spite of excellent care and are due to the serious nature of the condition with which that patient presents. That is, they occur in spite of the PCI, not because of it. In trying to dissect out the relative contributions of these factors to patient outcome, we try to predict what we would expect to be the outcomes in a well-run PCI centre on the basis of the patient’s clinical features and presentation. We compare this with what we observe, and try to see if there is a significant difference between the two. We do this using a technique called ‘risk adjustment’. This requires the use of complex mathematical and statistical techniques. While risk adjustment is an essential tool in trying to understand the quality of PCI treatment, there are many caveats to the methodology and so it can only ever be used to provide a rough guide. For example, some of the features that might predict a good or bad outcome for a patient are not collected (such as if patient is also suffering from a potentially fatal cancer when they have a PCI) or may be very hard to measure (such as ‘frailty’). There will always be random variation in any statistical measure, and so some fluctuation in the apparent risk adjusted outcomes will simply be due to the play of chance.

If we identify a PCI centre or a PCI consultant whose patients appear to have significantly worse outcomes than expected mathematically (called a ‘potential outlier’), we must try to find the correct balance between over reacting (and criticising what may turn out to be a very high standard of care), and missing genuinely poor practice. Other clues can help, such as if apparently poorer outcomes fall back within the expected range in subsequent years and if the operator concerned is treating an unusual case mix that may not be appropriately adjusted for by the mathematical model. The BCIS policy for looking into possible outliers is available from the BCIS web site.

It is also important that though the information here is presented according to the consultant who was responsible for the overall care of the patient during their PCI procedure, they represent part of a larger clinical team consisting of paramedics, emergency medicine doctors, nurses, radiographers and technicians, and the way this teams works together with the hospital facilities available to them may have an impact on the quality of care that can be provided.

One of the most important and worrying aspects of the public reporting of PCI outcomes is the potential for it to generate ‘risk-averse behaviour’. Very sick patients will always have a lower survival rate regardless of PCI or not, be at higher risk of not surviving whether they have a PCI or not, and yet they tend to are those who stand to gain the most from treatment by PCI. If it is felt that the risk model that is used to try to adjust for these patient factors does not work, then there is the potential for a PCI operator to be reluctant to treat the sickest patient who has the most to gain, in case it makes them look as if they have poorer outcomes in spite of them actually providing an excellent quality of PCI. Thus the models for risk adjustment are pivotal to optimising patient care, not only to allow assessment of quality of PCI practice but also to avoid the possibility of risk-averse behaviour.

List of abbreviations
ACS Acute Coronary Syndrome
BCIS British Cardiovascular Intervention Society
CABG Coronary Artery Bypass Grafting
CHD Coronary Heart Diseae
ECG Electrocardiogram
MACCE Major Adverse Cardiovascular and cerebrovascular Event
NSTEMI Non ST elevation Myocardial Infarction
PCI Percutaneous Coronary Intervention
STEMI ST elevation Myocardial Infarction
UA Unstable Angina

Individual operator information

If you would like PCI operators outcomes after performing PCI

Here