Journal of Health Services Research & Policy >
Volume 13, Number 4 >
Pp. 209-214
Original research |
Division of Public Health, University of Liverpool, Liverpool
Correspondence to: dctr{at}liv.ac.uk
Objectives: To explore attitudes to the use of models for coronary heart disease to support decision-making for policy and service planning.
Methods: Qualitative study using semi-structured interviews with 33 policy- and decision-makers purposively sampled from the UK National Health Service (NHS) (national, regional and local levels), academia and voluntary organizations. Interviews were transcribed, coded and emergent themes identified using framework analysis aided by NVivo software.
Results: Policy-makers and planners were generally enthusiastic about models to assist in decision-making through: predicting trends; assessing the effect of interventions on health inequalities; quantifying the impact of population level and targeted interventions, and facilitating economic evaluation. The perceived advantages of using models included: more rational commissioning; the facility for scenario testing; advocacy for population level interventions and off-the-shelf synthesis to aid real time decision-making. However, although participants were aware of models to support decision-making, these were not being used routinely. Some participants felt that models oversimplify complex situations and that there is a lack of shared understanding as to how models work. Factors that increase confidence in decision support models included: rigorous validation and peer review, the availability of user-support and increased transparency.
Conclusion: Policy-makers and planners were generally enthusiastic about the use of models to support decision-making, illustrating the potential uses for models and the factors that improve confidence in them. However, existing models are often not being used in practice. So new models that are fit for practice need to be developed.
 | Introduction |
|---|
 Top Introduction MethodsResultsDiscussionAcknowledgementsReferences |
|---|
A number of decision support models for CHD have been designed to assist policy-makers.4 A model can be defined as ‘an analytical methodology that accounts for events over time and across populations, based on data drawn from primary or secondary sources, whose purpose is to estimate the effects of an intervention on valued health consequences and costs’.5 Models are particularly suited to help deal with the complexity that faces decision-makers when considering cardiovascular disease. CHD is well studied and a plethora of epidemiological and economic information exits regarding risk factors, treatments and preventive interventions. The challenge for decision-makers is to bring this information to bear on specific populations in a timely manner.
This is particularly relevant at the local level in the UK, where there is a drive to improve the quality of commissioning of health services.6 Primary care trusts are expected to become world-class commissioning organizations but in order to do this they will need to base policy decisions on sound knowledge and evidence at a population level.7
A systematic review of CHD policy models identified several key issues. These include: how explicit authors are about assumptions, limitations and uncertainty around estimates; the population considered; the credibility of data; the scope of outcomes; and model quality, including assessment of validity.4 Models have the potential to improve the quality of decision-making, leading to more appropriate resource allocation,5,8 but they have limitations.4,9
Previous qualitative research has illustrated the conceptual gap between researchers and policy-makers.10 Whitehead et al. conducted focus groups with policy-makers and researchers in the context of health inequalities. They concluded that the availability of policy relevant evidence on health inequalities could be improved by closer enagagement between researchers and policy-makers. Issues raised included: the importance of timely, relevant and understandable evidence for policy; a lack of mutual understanding of the policy-making and research processes; the need to develop shared ways of dealing with and expressing uncertainty in evidence; the need to find more effective ways to share and disseminate evidence. It was suggested that more researchers and policy-makers should experience periods of working in each others' fields in order to bridge this gap.11,12
In the context of modelling and decision support tools, co-operation between researchers and policy-makers is advocated to optimize the use of decision-making tools.11,13
It is important to involve potential users in the development of decision support models11,13 and to close the gap between research and policy-making.10 Currently, it is unclear the extent to which decision-makers are using models, and how useful they find them. We therefore designed a qualitative study, using a framework approach, to explore decision-makers' attitudes to the use of models that support CHD policy-making. The study aimed to explore experiences both of CHD policy models and of modelling in health care more generally. We particularly wished to understand how policy models might potentially fit within existing decision-making processes. The IMPACT model is the only comprehensive CHD model validated in the UK population.14 This study was part of a wider project aimed at informing the development of an improved version called the IMPACT2 CHD model. Our intention was to maximize the value of the IMPACT2 model to planners and policy-makers by consulting with the potential end-users of a policy model.
| Methods |
|---|
|
TopIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
The sampling frame comprised 58 policy-makers from a range of organizations in England, Scotland and Wales. This reflected the variations in role found among UK CHD policy-makers. Two distinct strategies were used to generate this diverse pool of participants. A list of known CHD policy-makers was drawn up on the basis of existing professional and policy networks, and a purposive sampling strategy explicitly sought to include decision-makers from each UK region, and from any organizational types that were under-represented in the initial list of known policy-makers' organizations. These lists were then combined to generate the final sampling frame. A recruitment letter was sent to every person on the list – this gave background details and invited the recipient to participate in the study.
Data were collected by individual semi-structured interviews, conducted at a time and venue convenient for the participant. Prior to the interview, participants received an introductory letter which provided further information about the consultation process. The interviews opened with general questions about the participant's role, organization and professional background. Non-directive questions then explored key policy concerns in relation to CHD, current decision-making practices, existing experiences of policy models and aspirations for a model to support policy-making. Participants were also asked to consider a series of illustrative examples based on potential uses for a CHD policy model. The interviews were conducted by DTR and BM. Data were digitally recorded and then subsequently transcribed verbatim.
Data were analysed using NVivo software for qualitative data analysis (version 7). The analysis used techniques drawn from the framework method.15,16 DTR and BM carried out a familiarization analysis and identified a thematic framework. This thematic framework drew on both a priori issues and also on concepts that emerged from the data during the data collection and familiarization analysis stages. The a priori issues were developed by the IMPACT steering group. Each member of the group was asked to put together a list of key issues relating to decision support tools for CHD policy-making. In this respect we have taken a pragmatic approach and have not based the analysis on a specific theoretical perspective. This is in keeping with the framework approach to applied policy research.10 To our knowledge there is no published conceptual framework that explains the response of decision-makers to decision support models. The thematic framework that we developed was converted into a series of codes to be applied to the data. Data from all of the transcripts were systematically coded, charted and mapped. The analysis then sought to identify associations between themes and to carry out an in-depth exploration of the emergent findings.
Before the whole data-set was indexed by DTR and BM, a third researcher (FLW) independently coded a subsample of transcripts and then compared coding, as a check to ensure high levels of inter-researcher consistency in analysis.
| Results |
|---|
|
TopIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
As the data collection process progressed, it became clear that we also needed to speak to directors of finance and those involved directly in commissioning CHD services. Therefore, three additional participants were recruited, giving a total of 33 participants (Table 1). The final sample consisted of CHD policy-makers at all levels of the NHS (national, regional and local) in England, Scotland and Wales. In addition we interviewed influential academics and members of voluntary organisations relating to CHD. One potentially important group that was not sampled was data analysts.
|
Most participants were familiar with the concept of models. Very few participants had not encountered modelling, or had not used it to support their work. Some had carried out modelling within their own organizations and others were familiar with existing CHD models. What participants meant by a ‘model’ varied. Some described how their organizations had sought to model trends in disease prevalence or in health service use, whereas others had been involved in modelling of a more sophisticated kind. At a national level, some participants had experience of developing a model which they considered preferable to models developed elsewhere, because they had more confidence in using them and because existing models were seen as not being fit for purpose:
‘We started out looking at the gap in life expectancy so there was a very specific purpose for it and none of the existing models really did that... this is the thing about academia doing something which is very interesting academically but it needs to address the issues that PCTs have about how they hit the PSA (Public Service Agreement) targets.’ (National level decision-maker)
Who would use decision support models?
Box 1 summarises the groups of people identified as potential users of models. Participants generally described how models might be useful, particularly in terms of commissioning and service planning. Commissioners were identified as an important user group. Within organizations, participants suggested that it would be analysts or intelligence managers who might make practical use of models but that the results would subsequently be used by a number of policy-makers within the organization.
| Box 1 Who would use models? National Health Service staff
|
Although policy-makers seemed enthusiastic about the potential for using models to support decision-making, in general they were not being used routinely. The use of models was more common at national and regional levels than at local level.
What would decision-makers use models for?
Based on the illustrative examples, participants stated that it would be either important or very important for a CHD model to address questions about:
‘...PCTs and Trusts need to know what they need to do [around inequalities targets] ... particularly some quantitative ideas about what they need to do, how much effort they need to make.’ (National level decision-maker)
Advantages of using decision support models
Box 2 summarizes the perceived advantages of decision support models. Participants suggested that models could offer considerable advantages for health service planning and decision-making, specifically that these processes could potentially become more evidence-led. It was particularly important to participants that models could represent a rational and transparent basis for decisions:
‘I think it allows people to become much better at predicting and helping predict future need, service delivery, costs. So I think it's essential to do good planning and I think people have been very weak at forward prediction in the past.’ (Public Health Observatory director)
| Box 2 Perceived advantages of using models Factors relating to model functions
|
A number of participants felt that models could provide evidence to support public health interventions:
‘I will be going into conversations with Head of Health and Social Care and the Chief Exec... She'll definitely need convincing. Although we've got peer messages about not getting value for money out of the NHS, maximum health gains to be found by going upstream, health prevention and health promotion, she will need convincing of that... There will be a fight and a very clear and rational message that we will need to get across to make the argument... If we can actually show the graphs, do the figures and show the money saved at the end of the day, and the lives saved, that would be a very helpful tool to have.’ (Consultant in Public Health at local level)
A specific perceived advantage was the ability to simulate multiple policy scenarios and to ‘try these out’ before deciding which policy to adopt:
‘It would let you make your mistakes before you try them in real life! To me the key element has got be being able to run the "what if" scenario...’ (National level decision-maker)
Participants suggested that there was little available evidence to support policy-making and thus models represented an alternative in the absence of robust data on the effectiveness of CHD interventions:
‘I think we don't have an alternative, that's the stark reality. One has to start somewhere. I think it's a surrogate for good, cohort data.’ (Physician)
Disadvantages of using decision support models
Box 3 outlines the perceived disadvantages of models. Issues around accuracy were of more concern to academics than to those working in the NHS:
‘The validity and accuracy tend to be a small issue because you can put up with quite a degree of inaccuracy and quite a degree of invalidity because the broader policy decisions aren't quite as sensitive as some of the more academic differences between the models. You need to have some sensitivity and understanding as to whether the assumptions are really dependent on something we are a bit uncertain about, but by and large even a relatively poor model is better than the alternative which is nothing.’ (National level decision-maker)
| Box 3 Perceived disadvantages of using models Factors relating to modelling complexity
|
Some participants thought that a disadvantage of models was that the validity of model outputs is predicated on the quality of the data which have been used:
‘A policy model is only going to be as good as the model itself, so if it relies on or uses assumptions that are dubious or uses data which aren't reliable or valid then the model is going to be problematic.’ (Academic)
Complexity was seen both as a model strength and as a disadvantage. Some argued that real life is so complex, in terms of understanding and explaining the factors that are associated with CHD, that it would be almost impossible for a model to capture and thus enable users to manipulate all the variables that explain change:
‘The world is not like the model, which is my fundamental problem. The real world has got other factors and too many other directions to it.’ (Academic)
Other potential users, however, were concerned that a model that was too complex might be too difficult for decision-makers to understand. They were concerned that by attempting to explain the complexity underpinning policy models they risked losing credibility by introducing doubts into the minds of policy-makers:
‘I think the simpler they're kept the better because once you start using words like Markov... it all begins to leave us mere mortals blind.’ (National level decision-maker)
Factors associated with increased confidence in models
Participants cited the importance of information on validation, particularly evidence of validity in different national contexts (Box 4). They also discussed the importance of strong academic support, consensus around reliability and peer review. For some potential users, it was important that they were able to understand how model outputs were reached, especially when they encountered unexpected model findings:
‘That's a model that changed my mind completely about the way we should be managing head injuries... Because I was able to break it down, the result I didn't originally trust... I found out where it was coming from and how it was derived. I think that's a crucial thing.’ (Physician)
| Box 4 Factors associated with increased confidence in models Usability factors
|
Some participants were concerned that the model outputs inevitably relied heavily on the modellers' interpretations of the conclusions drawn from the scientific evidence. They were concerned that this meant that the model was built on a series of subjective judgements, and that the way model outputs were reached might be inaccessible to users. A few participants emphasized that it was important for the credibility of a model, that its creators were not seen to be promoting a particular agenda. They suggested that if this was the case, then user confidence might be undermined:
‘I think the model needs to be very objective in what it gives out and what I think the model must not be seen to be doing is pushing a particular agenda because that would undermine its credibility.’ (Regional level decision-maker)
| Discussion |
|---|
|
TopIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
This study increases our understanding of the utility of decision support models in policy-making, taking into account the views of decision-makers. Most participants were familiar with the concept of modelling and some had carried out – or had commissioned – some kind of modelling within their own organizations. However, it was apparent that the use of models to support decision-making was not common, especially at the local level. Despite this, participants welcomed the idea of a CHD model. They could offer clear suggestions as to how a model could be used to support policy-making within their organizations, similar to those previously suggested.8 Participants identified issues that a CHD policy model would ideally be able to answer, including questions about the burden of disease, prediction of trends, the impact of specific technologies, treatment versus preventative strategies, population level interventions, high risk versus whole population interventions and cost-effectiveness.
Participants suggested that this information would be used to support commissioning and service planning cycles. One of the benefits of models has been cited as their ability to synthesize complex information and to help answer questions not easily answered by traditional research questions.17 Models also facilitate timely generation of evidence to support decision-making – policy-makers have identified this as an important factor in a previous study.18 Respondents in this study suggested that a CHD model would be particularly helpful to synthesize information around the impact of interventions on social inequalities in CHD. Petticrew et al. identified the need for predictive models that can identify best buys in the context of health inequalities.18 Policy-makers are also keen to integrate cost-effectiveness information into decision-making, but this is often difficult. Grosse suggests that economic evaluation has had a relatively limited impact to date in public health decision-making.19 This study suggests it is important that future CHD models include economic considerations.
There are a number of CHD decision support models available. However, a systematic review has outlined the problems with generalizability, validity and transparency apparent with many of them.4 Factors associated with confidence in models include information on validation (especially across multiple national contexts), strong academic support, consensus around reliability and positive peer review. Potential reasons for the lack of uptake of policy models have been identified including concerns about the methodology, assumptions and poor quality data. Interestingly, health service participants were less concerned about data quality, as long as the model indicated the general direction of travel. Siebert suggests that models for resource allocation cannot take into account all of the values that interplay in decision-making.9 In line with this, some respondents suggested that models do not account for the complexity inherent in many decisions. There is a double-edged sword. On the one hand, researchers strive to produce models that take into account complexity, but are in danger of producing evidence that is clouded by methodological considerations.18 On the other hand simple, understandable models are criticized for being over-simplistic and unreal.12
Weinstein20 describes the potential for models to influence the allocation of resources, and individual decision-making, and identifies a range of potential model users in the US broadly similar in role to those identified in this study. Decision-making for CHD is complex and involves the input of several, heterogeneous people and organizations. A strength of this study is that we have included most of the key groups involved in the CHD decision-making process in the UK who are likely to be interested in models. Groups missing from our final sample that may have relevant views include general practitioners involved in practice based commissioning and data analysts. A limitation of the study is that there are small numbers within each subgroup. This means that generalisation of the results from a particular subgroup requires caution. In general we have therefore not attempted to explore differences in subgroups.
There are a number of important implications for policy and practice that can be drawn from this study. In order to increase the use of decision support models for CHD in the UK, researchers need to consider: the potential users and audience for models; the key functions that users expect of a model; the perceived advantages and disadvantages of models and the factors that increase confidence in models.
The results of this study are being used to inform the development of the IMPACT CHD model. We hope that by consulting with policy-makers and end-users we will be able to develop a model that is fit for purpose. Although the focus of this study is CHD, many of the results are generic, with much broader applications. Our findings may therefore be of interest to people involved in any form of analysis to support decisions in health care.
| Acknowledgements |
|---|
|
TopIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
| Footnotes |
|---|
| References |
|---|
|
TopIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
CiteULike 
Complore 
Connotea 
Del.icio.us 
Digg 
Reddit 
Technorati What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
