Hum. Reprod. Advance Access originally published online on July 24, 2008
Human Reproduction 2008 23(11):2493-2500; doi:10.1093/humrep/den281
Variation in subfertility care measured by guideline-based performance indicators
1 Department of Obstetrics and Gynaecology, Radboud University Nijmegen Medical Centre, PO Box 9101, 6500 HB Nijmegen, The Netherlands 2 Centre for Quality of Care Research (WOK), Radboud University Nijmegen Medical Centre, PO Box 9101, 6500 HB, Nijmegen, The Netherlands 3 Department of Obstetrics and Gynaecology, Rijnstate Hospital, PO Box 9555, 6800 TA Arnhem, The Netherlands 4 Department of Epidemiology and Biostatistics, Radboud University Nijmegen Medical Centre, PO Box 9101, 6500 HB Nijmegen, The Netherlands
5 Correspondence address. E-mail: j.kremer{at}obgyn.umcn.nl
 |
Abstract |
|---|
 Top Abstract IntroductionMaterials and MethodsResultsDiscussionSupplementary DataFundingAcknowledgementsReferences |
|---|
BACKGROUND: About 30–40% of patients do not receive care based on available scientific evidence. For subfertility, this may imply unnecessary and expensive diagnostic tests and treatments. It is therefore important to identify gaps in performance by monitoring current subfertility care. A set of 39 guideline-based performance indicators was previously developed for this purpose. This study aimed to assess several quality criteria of the indicator-set and to use the set to assess current subfertility care.
METHODS: A historic cohort study was performed in 16 Dutch subfertility clinics; 2698 couples were invited to participate. Indicator data were gathered by medical record extraction, and patient and professional questionnaires. Quality criteria for each indicator (measurability, reliability, applicability, improvement potential, discriminatory capacity, complexity and case-mix stability) were assessed. Current practice was measured as adherence to the separate indicators.
RESULTS: One thousand four-hundred and ninety-nine (56%) couples participated. All indicators were measurable, but the results for the other quality criteria varied. In total, 14 of the 39 indicators scored <50% adherence. Variation in performance between the clinics was up to 100%. The highest median adherence (86%) is found within the guideline ‘indications for IVF-treatment’. The lowest median adherence is found within the guideline ‘initial assessment of fertility’ (43%), followed closely by the guideline ‘anovulation’ (44%).
CONCLUSIONS: This study shows the quality of the developed indicator-set for monitoring clinical subfertility care. A first assessment in the Netherlands reveals large variation between clinics and ample room for improvement of care.
Key words: guidelines/performance indicator/monitoring/quality of care/subfertility
|
Introduction |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionSupplementary DataFundingAcknowledgementsReferences |
|---|
According to medical literature,
30–40% of patients do not receive care based on available scientific evidence (McGlynn et al., 2003
). On top of this, an estimated 20–25% of provided health care is unnecessary (McGlynn, 1995; Schuster et al., 1998; Grol, 2001). In the case of subfertility, this could mean the use of unnecessary and expensive diagnostic tests and assisted reproduction technologies (ART) or the realization of complicated high-order pregnancies. Subsequently, this may have substantial physical and psychological consequences for the patients involved (Greil, 1997; Kallen et al., 2005; Ombelet et al., 2006; Cousineau and Domar, 2007; Schieve et al., 2007). Patients should receive the best achievable care with a minimum of complications and at minimal costs. To improve both outcome and process of health care, clinical practice guidelines have been developed summarizing the best available evidence (Grimshaw et al., 2004). However, such guidelines do not implement themselves (Grol, 1997). Efforts to monitor actual health care and to identify gaps in performance show where improvement is needed. Guideline-based performance indicators are useful tools for monitoring health care (Grol et al., 2002). Consequently, they can be used to assess changes in clinical practice as well, which makes them indispensable instruments within clinical improvement programmes.
We reported previously on the development of a guideline-based set of 39 process and structure indicators for the entire spectrum of clinical subfertility care by means of a systematic six-step RAND-modified Delphi method (Mourad et al., 2007). The selected indicators focus on process and structure of subfertility care rather than outcome. Although current international debate focuses mainly on outcome measures (Dickey et al., 2004; Schieve and Reynolds, 2004), process indicators are valuable instruments within performance assessment and improvement programmes because they reveal exactly where changes in care might be necessary (Palmer, 1997; Eddy, 1998; Brook et al., 2000; Mant, 2001; Rubin et al., 2001b; Mourad et al., 2007; Nelen et al., 2007). This does not mean, however, that clinical outcomes are unimportant, but to change outcomes, it is necessary to first initiate changes in the process and structure of care. The primary aim of this study was to test the indicator-set for several quality criteria (Terwee et al., 2007). Such a practice test is necessary to demonstrate its value as an instrument for monitoring and improvement of clinical performance (Rubin et al., 2001a; Campbell et al., 2003; Schouten et al., 2005; Hermens et al., 2006; Ouwens et al., 2006). The second aim, conditional on sufficient quality of the instrument, was to assess the variation in current subfertility care in a large sample of clinics.
|
Materials and Methods |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionSupplementary DataFundingAcknowledgementsReferences |
|---|
Study design and study population
We conducted a retrospective cohort study in 16 Dutch clinics using medical record and questionnaire data. The study was approved for all clinics by ‘the Regional Review Board for Human Research (CMO) Arnhem-Nijmegen (CMO no. 2004/193)’. To assess current clinical subfertility care in the Netherlands, we aimed at including a broad patient cohort from the participating clinics. The clinics' characteristics varied to ensure that delivered care was representative for Dutch standards. There was one academic and one tertiary care clinic and seven clinics offered secondary care (IVF/ICSI-treatment). These clinics were all also teaching clinics and of large or intermediate size; the other seven clinics were smaller, non-teaching facilities. In total, 15 clinics were national health, and one of the smaller secondary care clinics was a private clinic.
To include a representative patient group, potential participating couples were randomly selected by means of each clinics’ financial DBC (Diagnosis/Treatment Combination code) registration database. In this national registration, patients undergoing diagnostics or treatment for subfertility are identified with a specific Fertility-code (F-code). Couples were apt for inclusion if they had an active F-code anytime between 1 April and 30 June 2005. The assessment focused on the care they received in the period 1 January–30 June 2005. In each clinic, a random sample of eligible couples (50–500 depending on the size of the clinic) was invited to participate in the study. They were sent an informed consent form and a questionnaire: the former included consent to use their medical data from the clinic. Couples who had insufficient knowledge of the Dutch language or did not visit the subfertility clinic in the study period were excluded. In total, 2698 couples were invited.
The clinical performance indicators
We used a set of 39 clinical performance indicators for subfertility care, which was developed using a rigorous and systematic six-step approach. The set was based on literature, existing international guidelines and indicators and 10 Dutch (evidence and consensus based) subfertility guidelines (Mourad et al., 2007). The latter includes the Dutch Embryo Act and nine subfertility guidelines of the Dutch Society of Obstetrics and Gynecology (NVOG): initial assessment of fertility, anovulation and child wish, male subfertility, tubal pathology, endometriosis, premature ovarian failure (POF), intrauterine insemination (IUI), indications for IVF treatment and ovarian hyperstimulation syndrome (OHSS).
The indicator-set encompasses 37 process indicators and 2 structural indicators, and comprises the entire spectrum of subfertility care: the initial assessment of fertility, diagnostics, treatments and possible complications. Indicators were operationalized by the construction of a numerator and denominator, each consisting of several variables. The numerator is formed by the proportion of patients in which there is adherence to a recommendation and the denominator by the proportion of patients in which the recommendation is applicable. The indicators within the guideline ‘initial assessment of fertility’ are shown in Table I. The remaining indicators, ranked per guideline, are shown in Supplementary Table Ia–e in the web-based version of this article.
|
Data collection to assess quality criteria and current practice
Data for indicators regarding indications for ART (n = 7), diagnostic procedures (n = 6) and treatment procedures (n = 20) were extracted from the individual medical records by three trained investigators. Data for indicators regarding information provision to patients (n = 4) were collected by means of a patient questionnaire and data for indicators regarding the structure of care (n = 2) by a short professional questionnaire. The patient questionnaire was piloted in a group of 30 subfertile couples and this led to minor adjustments in formulation of the questions before it was used in the total study population.
Definitions and analysis
The application of a systematic and rigorous consensus method for indicator development results in high face and content validity of this indicator-set. We measured the following quality criteria of all indicators in the set, in order to demonstrate its value as an instrument for assessment of subfertility care (Streiner and Norman, 1994; Rubin et al., 2001a; Grol et al., 2002; Campbell et al., 2003; Grol et al., 2005; Schouten et al., 2005; Ouwens et al., 2006; Wollersheim et al., 2007). Interested readers are referred to Supplementary Table II in the web-based version of this article which contains a list of suggested additional reading on the use and validity of (process) indicators in clinical quality and performance assessment.
- Measurability: an indicator is considered measurable if data to fill the numerator and denominator of the indicator can be made available through data collection (e.g. medical records, complication or treatment databases or a survey) (Rubin et al., 2001a).
- Reliability: if the measurement of indicator data by two different data-collectors is reliably comparable, there is high reliability (Rubin et al., 2001a). Two independent reviewers abstracted a random sample of 10% (n = 32) of medical records from two participating clinics. The extent of agreement between these data reviewers on the level of process indicator scores, corrected for chance, was calculated using Cohen's kappa coefficient of inter-rater reliability. Coefficients of 0.4–0.6 represent moderate inter-rater reliability, coefficients of 0.61 or higher are considered very good (Landis and Koch, 1977).
- Applicability: for accurate performance assessment, an indicator is preferably applicable to a substantial proportion of reviewed patients (>10 cases); this is referred to as applicability (Grol et al., 2002).
- Improvement potential: when an indicator is used to detect changes in clinical performance, it is a prerequisite that improvement is possible at all; if overall performance for a certain indicator is already very high, the indicator has no improvement potential. We defined improvement potential as an overall performance score of <90% (Ouwens et al., 2006).
- Discriminatory capacity: the discriminatory capacity indicates whether an indicator is able to discriminate practice performance between different hospitals. High discriminatory capacity is therefore present when the range in scores between the lowest- and highest-scoring clinics is >20%. (Grol et al., 2002).
- Complexity: the concept of complexity was used for the number of variables needed to fill the numerator and denominator of an indicator. Complexity is thus a measure that reflects the amount of investments needed to assess an indicator; the higher the complexity the more effort it takes to measure an indicator. The maximum value for complexity was beforehand set at five different variables; in the case of complexity scores of more than five variables, subdivision of the indicator was considered. Variables relating to the inclusion of patients were not counted because those variables can vary per clinic and depend on design of the medical records. For the indicators measured by medical record extraction, two researchers (W.L.D.M.N. and S.M.M.) counted independently the number of variables to fill the numerator and denominator of each indicator. For the indicators measured by means of a professional or patient questionnaire, we counted the number of answers needed to determine the indicator. If there was disagreement between the two researchers’ findings, discussion took place to reach an agreement.
- Case-mix stability: case-mix stability is an important asset that enables the application of an indicator for monitoring within a specific hospital over time or to compare hospitals of different sizes and settings (Rubin et al., 2001a). Therefore, the relationship between certain patient-characteristics that may vary considerably between clinics [‘woman's age’, ‘type of subfertility (primary or secondary)’ and ‘duration of subfertility’] and the indicator scores was analysed to decide whether correction for case mix is necessary.
Analysis
Collected data were entered in a database using the Statistical Package for the Social Sciences (SPSS 14.0 for Windows®, SPSS Inc., Chicago, IL, USA). Descriptive analyses were performed for each indicator, and current practice was expressed as percentage of adherence to an indicator. The median adherence of each separate clinic was calculated, as well as the adherence of all clinics together.
|
Results |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionSupplementary DataFundingAcknowledgementsReferences |
|---|
Response
The mean participation rate was 56% (1499 of 2698 invited couples), varying between 47% and 72% per clinic. Demographic characteristics of the participating couples are shown in Table II. Mean female age was 33 years and male age 35 years. Forty-seven couples (3.2%) were of non-Dutch origin. Of the participating couples, 73% suffered from primary subfertility. The median duration of subfertility was 38 months. A total of 53% had a high education level and 72% had a more than modal income (gross >
2750/month).
|
Assessment of quality criteria of the indicator-set
Table III shows the quality criteria of the indicators, within the guideline ‘initial assessment of fertility’. Supplementary Table IIIa–i, available in the web-based version of this article, shows these criteria for the remaining guidelines. All indicators were measurable and made available through either medical record extraction, or a patient or professional questionnaire. The reliability is high for questionnaire data, because it is first-hand information. For medical record data, reliability was substantial, reflected in an average Cohen's kappa coefficients of 0.86 (range 0.48–1.0). In total, 18 indicators were inapplicable (<10 patients). For these indicators, the other quality criteria could not be assessed. A total of eight indicators had adherence scores of 90% or more, and therefore possess too little room for improvement. Regarding discriminatory capacity, a total of 11 indicators scored <20%. This understandably includes the eight indicators that had no room for improvement. None of the indicators required more than one data source. Complexity values ranged mostly from 1 to 5. An exception was the indicator concerning ‘the items that should be discussed during an intake prior to IVF/ ICSI-treatment’. This indicator encompassed many topics and concerned 18 variables and was therefore subdivided into five different parts for accurate analysis. Assessment of case-mix stability revealed that the following three indicators are in need of correction for ‘duration of subfertility’: (i) ‘the initial fertility assessment should consist of three parts: semen-analysis, tubal patency assessment and cycle analysis’, (ii) ‘life-style advice concerning bodyweight, smoking and the alcohol and drug use should be part of the counselling regarding pregnancy-probabilities’ and (iii) ‘in the case of IUI in the stimulated cycle, ovarian response should be monitored by transvaginal ultrasound’. The indicator ‘in the case of unexplained subfertility in a woman <36 years, there is an indication for IVF after 3 years of subfertility’ needs correction for ‘duration of subfertility’, ‘woman's age’ and ‘type of subfertility’ when the analysis takes place. All other indicators showed equal distribution of these patient-characteristics.
|
The overall results of this quality criteria assessment are summarized in Table IV.
|
Current practice: adherence to indicators
Table I shows the median clinical performance and the range of performance of the clinics for each indicator within the guideline ‘initial assessment of fertility’. Supplementary Table Ia–e contains the results for the remaining indicators and is available in the web-based version of this article. The indicators regarding endometriosis, OHSS, tubal pathology and POF (Mourad et al., 2007
) were applicable to too few patients (n < 10) in this sample and are therefore not shown in Table I. From Table I and Supplementary Table Ia–e, it can be concluded that the median adherence of 14 indicators was <50%. Of those, six indicators even scored <25%. The median adherence to the separate indicators varied from 0% (e.g. ‘in the case of abnormal semen-analysis, a complete andrological history-taking, physical examination and one extra semen-analyses should be performed’) up to 100% (e.g. ‘woman's physical examination should include an assessment of the body mass index’). The variation between clinics differed widely and is reflected in the range of adherence within one indicator. We found, for example, on the one hand exactly equal adherence between all clinics (e.g. all clinics scored 100%) for the indicator ‘routine use of hCG for luteal support after IVF is not recommended’ and on the other hand a 100% difference between the clinics (e.g. two clinics scoring 0% and 100%, respectively) for the indicator ‘initial fertility assessment should result in a prognosis and a diagnosis’. Finally, the results show that the highest median adherence (86%) is found within the guideline ‘indications for IVF-treatment’. The lowest mean adherence is found within the guideline ‘initial assessment of fertility’ (43%), followed closely by the guideline ‘anovulation’ (44%).
|
Discussion |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionSupplementary DataFundingAcknowledgementsReferences |
|---|
This study demonstrates several quality criteria of a set of 39 guideline-based process and structure indicators for comprehensive clinical subfertility care. Moreover, a first assessment of current subfertility care in the Netherlands discloses a large variation in the performance between clinics.
During the original indicator-selection procedure, the expert panels were asked to take five criteria into account when selecting key recommendations from the guidelines (Mourad et al., 2007). In this practice test, we assessed additional quality criteria of the proposed indicators. We can conclude that most of them meet these criteria, with several remarkable results. Improvement potential and applicability were not always high. Apparently, a considerable discrepancy exists between the experts’ estimate of these criteria on the one hand and currently provided care on the other hand. This underlines that even a rigorous indicator-development procedure should be complemented with a practice test. On the basis of the results of our practice test, particularly applicability and improvement potential (Tables III and IV and Supplementary Table IIIa–i), we can distinguish three subsets within our original set of 39 performance indicators. Indicators within the ‘first subset’ score highly for applicability and improvement potential and can therefore be used for quality monitoring purposes and as a baseline measurement for improvement programmes in the involved clinics. A ‘second subset’ contains indicators with low improvement potential, which makes them useless for improvement programmes, but with high applicability, which makes them still suitable for monitoring purposes, namely to ensure that adherence continues to be high in the future. Within the ‘third subset’, we find indicators with low applicability. They concern relatively rare conditions or complications (e.g. POF and severe OHSS), so inclusion numbers within our random patient sample were too small for adequate indicator assessment. This does not mean, however, that this subset needs to be discarded immediately for any future use; but the inclusion of enough cases for such indicators requires a more specific patient sample, e.g. drawn from separate complication registrations or large (national) databases.
The question arises whether this set of indicators is also suitable for countries other than the Netherlands. Only 2 out of the 39 indicators (e.g. laboratory accreditation by a Dutch committee and the maximum number of two embryos transferred per cycle) may be more or less specific for Dutch practice, but both can be easily adapted for international use by judging not the content but the topic of the indicator. Generalizability is thus not a barrier for international adaptation of the presented indicator-set.
Adherence to the guideline recommendations
Our study shows that adherence to guideline-based indicators ranged widely per clinic, per guideline and per individual indicator. Adherence to the indicators is relatively poor; 14 indicators score below 50% adherence. This means there is ample room for improvement of clinical subfertility care. Professionals should be urged to improve the implementation of those guideline-recommendations that showed poor adherence in their clinic. A 100% adherence score may not always be a viable goal, because there can in specific cases be good reasons to divert from guideline-recommendations. However, professionals should at least aim to rival the best scoring clinic, which is a realistic benchmark. The lowest adherence (less than 50%) is found within the guidelines ‘initial assessment of fertility’ and ‘anovulation’; priority should therefore be given to implementation of these guidelines. It is furthermore important to realize that a low score on an indicator does not automatically mean that there is a problem in the quality of care; it is, however, a signal to further estimate the matter in order to try to understand underlying causes and processes.
There are some results that stand out in particular. Several of the critical indicators with low adherence relate to communication and patient information. Again, the broad variation in adherence to these indicators and the fact that some clinics do score highly, illustrate that these results cannot be simply disregarded or attributed to recall bias (Reading, 1981; Kessels, 2003). These findings are in line with previously conducted qualitative and quantitative research, identifying physicians' ‘lack of self-efficacy regarding communication’ and ‘low outcome expectancy’ as main barriers for adherence to a subfertility guideline (Haagen et al., 2005). However, this is plainly disturbing from the perspective of patient-centred care, as former research clearly discloses subfertile patients’ preferences for comprehensible and complete information (Schmidt, 1998; Souter et al., 1998; Malin et al., 2001; Schmidt et al., 2003). Increased awareness among care-providers of these patient preferences and feedback on low indicator scores should lead to an improvement in information provision and transparency of care. Another low-scoring indicator is the one recommending unstimulated over stimulated IUI, in the case of unexplained infertility, to prevent multiple pregnancies (15% adherence, range between clinics 0–48%). This perfectly reflects the situation where consensus on a specific topic is lacking. None of the clinics score 100% adherence, meaning this recommendation is far from consistently followed. This might be because both patients and professionals are apt to change their treatment policy and are willing to take more risks, e.g. after previous unsuccessful cycles of unstimulated IUI. This probably also explains the poor adherence to the recommendation ‘mono-ovulation should be the result of ovulation-induction’. However, it is the responsibility and moral duty of the professional to guard patient safety during any treatment, and 0% scores should therefore be taken very seriously. Fortunately, more reticence is seen in the high adherence (79%) of the recommendation ‘IVF-treatment in women <36 years is only indicated after 3 years of subfertility’. This reflects the professionals’ commitment to prevent over-treatment and strive for cost-effectiveness and efficacy of this expensive and intensive type of subfertility treatment, by respecting the possibility of less invasive approaches and treatment-independent pregnancies (Goverde et al., 2000; Evers, 2002).
This study also has some limitations. Indicator measurement turned out to be quite a laborious exercise, including widely distributed questionnaires and an extensive medical record search to complete indicator data. Poor availability of medical data and the lack of adequate data resources are common problems in performance assessment efforts (Gagliardi et al., 2005; Pronovost et al., 2007), but this should never be the sole reason to discard rigorously selected performance indicators. On the contrary, care-providers and policy-makers should undertake efforts to overcome these barriers. The ongoing introduction of electronic patient records offers great opportunities for collaboration with clinical improvement programmes. It is therefore important to conduct further research to uncover which relatively small investments or adjustments to existing databases are necessary to facilitate monitoring of current care.
In conclusion, we demonstrated the quality of a developed set of guideline-based indicators for comprehensive clinical subfertility care. A practice test in the Netherlands revealed large variation and ample room for improvement of subfertility care. Such objective assessment of care can help professionals to identify and subsequently target the domains of care in need of improvement. The next step will be the translation of indicator results into appropriate implementation strategies that aim to bridge the gaps between the best available evidence we present in our guidelines and the care we deliver in daily practice.
|
Supplementary Data |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionSupplementary DataFundingAcknowledgementsReferences |
|---|
Supplementary data are available at http://humrep.oxfordjournals.org/
|
Funding |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionSupplementary DataFundingAcknowledgementsReferences |
|---|
The Netherlands Organisation for Health Research and Development (ZonMw) funded this project (Grant No. 945-14-116).
|
Acknowledgements |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionSupplementary DataFundingAcknowledgementsReferences |
|---|
We like to acknowledge all patient couples for their participation and the 16 subfertility clinics for their cooperation in data sampling. Special thanks to the coordinating gynaecologists of these clinics: P.A. van Dop (St Catharina Ziekenhuis, Eindhoven), B.W. Mol (Maxima Medisch Centrum, Veldhoven), A.M.H.W. Franssen (Canisius Wilhelmina Ziekenhuis, Nijmegen), M. Van Rozendaal (Ziekenhuis Bernhoven, Veghel), J.H.A. Vollebergh (Ziekenhuis Bernhoven, Oss), A.G. Minkhorst (Maasziekenhuis Pantein, Boxmeer), C.J.C.M. Hamilton (Jeroen Bosch Ziekenhuis, den Bosch), H. Ruis (behandelcentrum stichting Geertgen, de Mortel), G.J.J.M. Muijsers (Rivierenland Ziekenhuis, Tiel), J.H. Schagen van Leeuwen (St Antonius Ziekenhuis, Nieuwegein), L.J. van Dam (Gelre Ziekenhuis, Apeldoorn), E. Scheenjes (Ziekenhuis Gelderse Vallei, Ede), A.P.E. Schmoutziguer and L.F.Bancsi (Ziekenhuis Rijnstate, Arnhem), V.M. Blom (Streekziekenhuis Zevenaar, Zevenaar), E.M. Tepe, (Slingeland Ziekenhuis, Doetinchem). We also thank, medical student Evelyn Verheijden and research assistant Janine Liefers for their help in data management and analysis.
|
References |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionSupplementary DataFundingAcknowledgementsReferences |
|---|
Brook RH, McGlynn EA, Shekelle PG. Defining and measuring quality of care: a perspective from US researchers. Int J Qual Health Care (2000) 12:281–295.
Campbell SM, Braspenning J, Hutchinson A, Marshall MN. Research methods used in developing and applying quality indicators in primary care. Br Med J (2003) 326:816–819.
Cousineau TM, Domar AD. Psychological impact of infertility. Best Pract Res Clin Obstet Gynaecol (2007) 21:293–308.[CrossRef][Medline]
Dickey RP, Sartor BM, Pyrzak R. What is the most relevant standard of success in assisted reproduction?: no single outcome measure is satisfactory when evaluating success in assisted reproduction; both twin births and singleton births should be counted as successes. Hum Reprod (2004) 19:783–787.
Eddy DM. Performance measurement: problems and solutions. Health Aff (Millwood) (1998) 17:7–25.[Abstract]
Evers JL. Female subfertility. Lancet (2002) 360:151–159.[CrossRef][Web of Science][Medline]
Gagliardi AR, Fung MF, Langer B, Stern H, Brown AD. Development of ovarian cancer surgery quality indicators using a modified Delphi approach. Gynecol Oncol (2005) 97:446–456.[CrossRef][Web of Science][Medline]
Goverde AJ, McDonnell J, Vermeiden JP, Schats R, Rutten FF, Schoemaker J. Intrauterine insemination or in-vitro fertilisation in idiopathic subfertility and male subfertility: a randomised trial and cost-effectiveness analysis. Lancet (2000) 355:13–18.[CrossRef][Web of Science][Medline]
Greil AL. Infertility and psychological distress: a critical review of the literature. Soc Sci Med (1997) 45:1679–1704.[CrossRef][Web of Science][Medline]
Grimshaw JM, Thomas RE, Maclennan G, Fraser C, Ramsay CR, Vale L, Whitty P, Eccles MP, Matowe L, Shirran L, et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess (2004) 8:iii-72.
Grol R. Personal paper. Beliefs and evidence in changing clinical practice. Br Med J (1997) 315:418–421.
Grol R. Successes and failures in the implementation of evidence-based guidelines for clinical practice. Med Care (2001) 39:II46–II54.[Web of Science][Medline]
Grol R, Baker R, Moss F. Quality improvement research: understanding the science of change in health care. Qual Saf Health Care (2002) 11:110–111.
Grol R, Wensing M, Eccles M. Improving Patient Care. The Implementation of Change in Clinical Practice (2005) 1st edn. London, UK: Elsevier Limited.
Haagen EC, Nelen WL, Hermens RP, Braat DD, Grol RP, Kremer JA. Barriers to physician adherence to a subfertility guideline. Hum Reprod (2005) 20:3301–3306.
Hermens RP, Ouwens MM, Vonk-Okhuijsen SY, van der WY, Tjan-Heijnen VC, van den Broek LD, Ho VK, Janssen-Heijnen ML, Groen HJ, Grol RP, et al. Development of quality indicators for diagnosis and treatment of patients with non-small cell lung cancer: a first step toward implementing a multidisciplinary, evidence-based guideline. Lung Cancer (2006) 54:117–124.[CrossRef][Web of Science][Medline]
Kallen B, Finnstrom O, Nygren KG, Otterblad OP, Wennerholm UB. In vitro fertilisation in Sweden: obstetric characteristics, maternal morbidity and mortality. BJOG (2005) 112:1529–1535.[Web of Science][Medline]
Kessels RP. Patients’ memory for medical information. J R Soc Med (2003) 96:219–222.
Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics (1977) 33:159–174.[CrossRef][Web of Science][Medline]
Malin M, Hemmink E, Raikkonen O, Sihvo S, Perala ML. What do women want? Women's experiences of infertility treatment. Soc Sci Med (2001) 53:123–133.[CrossRef][Web of Science][Medline]
Mant J. Process versus outcome indicators in the assessment of quality of health care. Int J Qual Health Care (2001) 13:475–480.
McGlynn EA. Quality assessment of reproductive health services. West J Med (1995) 163:19–27.[Web of Science][Medline]
McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J, DeCristofaro A, Kerr EA. The quality of health care delivered to adults in the United States. N Engl J Med (2003) 348:2635–2645.
Mourad SM, Hermens RP, Nelen WL, Braat DD, Grol RP, Kremer JA. Guideline-based development of quality indicators for subfertility care. Hum Reprod (2007) 22:2665–2672.
Nelen WL, Hermens RP, Mourad SM, Haagen EC, Grol RP, Kremer JA. Monitoring reproductive health in Europe: what are the best indicators of reproductive health? A need for evidence-based quality indicators of reproductive health care. Hum Reprod (2007) 22:916–918.
Ombelet W, Martens G, De SP, Gerris J, Bosmans E, Ruyssinck G, Defoort P, Molenberghs G, Gyselaers W. Perinatal outcome of 12,021 singleton and 3108 twin births after non-IVF-assisted reproduction: a cohort study. Hum Reprod (2006) 21:1025–1032.
Ouwens MM, Marres HA, Hermens RR, Hulscher MM, van den Hoogen FJ, Grol RP, Wollersheim HC. Quality of integrated care for patients with head and neck cancer: development and measurement of clinical indicators. Head Neck (2007) 29:378–386.[CrossRef][Web of Science][Medline]
Palmer RH. Process-based measures of quality: the need for detailed clinical data in large health care databases. Ann Intern Med (1997) 127:733–738.
Pronovost PJ, Miller M, Wachter RM. The GAAP in quality measurement and reporting. JAMA (2007) 298:1800–1802.
Reading AE. Psychological preparation for surgery: patient recall of information. J Psychosom Res (1981) 25:57–62.[CrossRef][Web of Science][Medline]
Rubin HR, Pronovost P, Diette GB. From a process of care to a measure: the development and testing of a quality indicator. Int J Qual Health Care (2001) a 13:489–496.
Rubin HR, Pronovost P, Diette GB. The advantages and disadvantages of process-based measures of health care quality. Int J Qual Health Care (2001) b 13:469–474.
Schieve LA, Reynolds MA. What is the most relevant standard of success in assisted reproduction?: challenges in measuring and reporting success rates for assisted reproductive technology treatments: what is optimal? Hum Reprod (2004) 19:778–782.
Schieve LA, Cohen B, Nannini A, Ferre C, Reynolds MA, Zhang Z, Jeng G, Macaluso M, Wright VC. A population-based study of maternal and perinatal outcomes associated with assisted reproductive technology in Massachusetts. Matern Child Health J (2007) 11:517–525.[CrossRef][Web of Science][Medline]
Schmidt L. Infertile couples’ assessment of infertility treatment. Acta Obstet Gynecol Scand (1998) 77:649–653.[CrossRef][Web of Science][Medline]
Schmidt L, Holstein BE, Boivin J, Tjornhoj-Thomsen T, Blaabjerg J, Hald F, Rasmussen PE, Nyboe AA. High ratings of satisfaction with fertility treatment are common: findings from the Copenhagen Multi-centre Psychosocial Infertility (COMPI) Research Programme. Hum Reprod (2003) 18:2638–2646.
Schouten JA, Hulscher ME, Wollersheim H, Braspennning J, Kullberg BJ, van der Meer JW, Grol RP. Quality of antibiotic use for lower respiratory tract infections at hospitals: (how) can we measure it? Clin Infect Dis (2005) 41:450–460.[CrossRef][Web of Science][Medline]
Schuster MA, McGlynn EA, Brook RH. How good is the quality of health care in the United States? Milbank Q (1998) 76:517–563. 509.[CrossRef][Web of Science][Medline]
Souter VL, Penney G, Hopton JL, Templeton AA. Patient satisfaction with the management of infertility. Hum Reprod (1998) 13:1831–1836.
Streiner D, Norman G. Health Measurement Scales (1994) 2nd edn. Oxford, UK: Oxford University Press.
Terwee CB, Bot SD, de Boer MR, van der Windt DA, Knol DL, Dekker J, Bouter LM, de Vet HC. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol (2007) 60:34–42.[CrossRef][Web of Science][Medline]
Wollersheim H, Hermens R, Hulscher M, Braspenning J, Ouwens M, Schouten J, Marres H, Dijkstra R, Grol R. Clinical indicators: development and applications. Neth J Med (2007) 65:15–22.[Web of Science][Medline]
Submitted on March 25, 2008; resubmitted on June 19, 2008; accepted on June 25, 2008.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  A.M.E. Lintsen, D.D.M. Braat, J.D.F. Habbema, J.A.M. Kremer, and M.J.C. Eijkemans Can differences in IVF success rates between centres be explained by patient characteristics and sample size? Hum. Reprod., October 16, 2009; (2009) dep358v1. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
S.M. Mourad, R.P.M.G. Hermens, T. Cox-Witbraad, R.P.T.M. Grol, W.L.D.M. Nelen, and J.A.M. Kremer Information provision in fertility care: a call for improvement Hum. Reprod., June 1, 2009; 24(6): 1420 - 1426. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||