Hum. Reprod. Advance Access originally published online on July 30, 2007
Human Reproduction 2007 22(10):2665-2672; doi:10.1093/humrep/dem215
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Guideline-based development of quality indicators for subfertility care
1 Department of Obstetrics and Gynaecology, Radboud University Nijmegen Medical Centre, PO Box 9101, 6500 HB Nijmegen, The Netherlands 2 Centre for Quality of Care Research (WOK), Radboud University Nijmegen Medical Centre, PO Box 9101, 6500 HB Nijmegen, The Netherlands
3 Correspondence address. Tel: +31-243617351; Fax: +31-243668597; E-mail: j.kremer{at}obgyn.umcn.nl
 |
Abstract |
|---|
 Top Abstract IntroductionMaterials and methodsResultsDiscussionAcknowledgementsReferences |
|---|
BACKGROUND: Internationally, several organizations have developed clinical guidelines for subfertility care to supply patients with the best possible care. However, to improve the implementation of such guidelines, we first need to gain insight into the application of clinical guidelines in daily practice. Valid quality indicators are necessary to estimate actual guideline adherence. However, none of the existing subfertility guideline programmes is accompanied by a satisfactory set of quality indicators. In this study, we develop a set of valid guideline-based quality indicators for subfertility care.
METHODS: A systematic RAND-modified Delphi method was used to develop a set of key recommendations based on 10 national Dutch subfertility guidelines, international literature and existing international indicators. Experts’ opinions were used to appraise recommendations regarding specific criteria such as efficacy, level of health gain, applicability and potential for care improvement.
RESULTS: A representative set of 39 key recommendations was selected from 303 initial recommendations. The recommendations covered two structural and 37 procedural aspects, the latter encompassing ‘indications for treatment’, ‘diagnostic procedures’, ‘treatment procedures’ and ‘patient information’.
CONCLUSIONS: This study describes the systematic, stepwise method used to develop 39 process and structure indicators that can be used to monitor subfertility care.
Key words: guidelines/indicator/monitoring/quality of care/subfertility
|
Introduction |
|---|
|
TopAbstractIntroductionMaterials and methodsResultsDiscussionAcknowledgementsReferences |
|---|
The worldwide prevalence of subfertility is estimated to range from 4 to 30%, affecting approximately 80 million couples around the globe (Vayena et al., 2002
; Nachtigall, 2006). Subfertility should therefore be considered one of the major health problems of the 21st century; it has great societal impact. Even if sufficient subfertility services are available (as is the case in most western countries), they are not necessarily accessible for all couples nor is the quality necessarily satisfactory. The available diagnostic tests and assisted reproduction technologies (ARTs) are expensive and can have substantial physical and psychological consequences for the patients involved (Greil, 1997; Cousineau and Domar, 2007). Pregnancies after ARTs are more often complicated, particularly due to higher multiple pregnancy rates (Kallen et al., 2005; Ombelet et al., 2006; Schieve et al., 2007). Therefore, optimal organization of care should be strived for in order to supply patients with the best possible care and a minimum of complications at minimal costs. However, the question arises whether the current organization of subfertility care serves this goal satisfactorily, and to what extent patients effectively receive such ‘best care’.
To help physicians standardize subfertility care and improve health outcomes, several large professional organizations have developed clinical guidelines for subfertility care in the past few years. These include the National Institute of Clinical Excellence (NICE), the American Society of Reproductive Medicine (ASRM), the European Society for Human Reproduction and Embryology (ESHRE), as well as several country-specific fertility societies. In the Netherlands, the Dutch Society for Obstetrics and Gynaecology (NVOG) has issued nine national guidelines for subfertility care. In general, clinical practice guidelines, preferably based on solid research and valuable clinical experience, are increasingly seen as one of the crucial tools for achieving high quality of care (Woolf et al., 1999). Guidelines can easily provide clinicians with information regarding optimal health care, and they aim at increasing the efficiency of care and reducing variation in performance between different professionals and hospitals. Regarding subfertility care, adherence to guidelines can decrease unnecessary diagnostics and treatments, which reduces the number of complications and expenditures. However, publication and dissemination of guidelines is not automatically followed by a change or improvement of health care (Bero et al., 1998); guidelines do not implement themselves (Grol, 1997). From our own experience in a multicentre quality study of intrauterine insemination (IUI) in the Netherlands, we have learned that current guideline adherence is not optimal (unpublished data). The first step towards improving adherence is to gain insight into the application of clinical guidelines in daily practice, i.e. the ‘actual’ care. To estimate such quality of actual care, valid quality indicators are necessary (Grol et al., 2002) because they are the missing link in connecting evidence from guidelines to practice. Three types of quality indicators can be distinguished, referring to the ‘process’ and ‘structure’ of medical care and the ‘outcome’ of delivered care. Such quality indicators should preferably be developed by means of a systematic method (Campbell et al., 2003). The increasing need for evidence-based quality indicators in reproductive health care is recognized internationally as well, but the search for the most suitable indicators is still ongoing (Min et al., 2004; Nelen et al., 2007; Jahn et al., 2006). Unfortunately, none of the existing subfertility guidelines is accompanied by a satisfactory set of systematically developed quality indicators. In our current study, we therefore aimed to develop a set of valid quality indicators covering all aspects of subfertility care.
|
Materials and methods |
|---|
|
TopAbstractIntroductionMaterials and methodsResultsDiscussionAcknowledgementsReferences |
|---|
Setting
A systematic RAND-modified Delphi method (Dalkey et al., 1969
; Fitch et al., 2001), including independent expert ratings and repetitive feedback, was used to develop a set of key recommendations suitable for transcription into quality indicators. We based our study on the international literature and the nine NVOG subfertility guidelines: initial assessment of fertility, anovulation and child wish, male subfertility, tubal pathology, endometriosis, premature ovarian failure (POF), IUI, indications for IVF treatment and ovarian hyperstimulation syndrome (OHSS) (http://www.nvog-documenten.nl, in Dutch). These guidelines are consensus or evidence-based and were systematically developed according to the NVOG standards and issued between 1998 and 2005. In addition, the Dutch Ministry of Health, Welfare and Sport published a model protocol of the Dutch Embryo Act (http://www.vsop.nl/pdf/embryowet.pdf, in Dutch), which incorporates a diversity of clinical recommendations concerning IVF treatment, and which is considered a clinical guideline as well. The invited expert panels consisted of a representative diversity of clinical experts and guideline users (Campbell et al., 2004), working at various types of fertility clinics ranging from small regional hospitals to tertiary university clinics. Among them were gynaecologists, fertility clinicians (medical doctors who specialize in assisted reproduction, but who do not have a postgraduate specialization in obstetrics and gynaecology), gynaecologists in training and members of the original guideline development workgroups.
Procedure for indicator development
The procedure for quality indicator development consisted of six steps, which were completed consecutively as shown in Fig. 1: (i) literature search, (ii) selection of recommendations, (iii) written questionnaire, (iv) consensus meeting, (v) critical evaluation and (vi) consultation with guideline developers:
|
Literature search
We searched Medline for existing indicators for reproductive health and specifically subfertility care. The key words used were ‘subfertility’, ‘infertility’ and ‘reproductive health care’ combined with ‘guidelines’ or ‘clinical guidelines’ and ‘quality indicators’. Furthermore, we searched the Internet for potential sources of quality indicators, e.g. governmental reports, reports from national and international fertility societies and commercial initiatives. We paid specific attention to existing guidelines concerning reproductive health or subfertility. For this purpose, we used the Websites of the National Guideline Clearinghouse (www.guidelines.gov) and the Geneva Foundation for Medical Education and Research (www.gfmer.ch).
Selection of recommendations
First, four authors (S.M.M., R.P.M.G.H., W.L.D.M.N. and J.A.M.K.) separately selected all the recommendations from the nine NVOG guidelines and the Dutch Embryo Act. To take international consensus and solid evidence into account as well, we added all the recommendations with A-level evidence from the NICE subfertility guideline that did not have an exact equivalent in the Dutch set of recommendations. The collected recommendations were edited in three different questionnaires for the consensus panel method in Step 3. Questionnaire 1 covered ‘endometriosis’, ‘anovulation’ and ‘IUI’ (131 items); Questionnaire 2, ‘initial assessment of fertility’, ‘male subfertility’ and ‘tubal pathology’ (110 items); and Questionnaire 3, ‘indications for IVF’, ‘OHSS’, ‘POF’ and the ‘Embryo Act’ (99 items).
Written questionnaire
Questionnaires were sent to 63 experts who were divided into three expert panels of 21 members each, and the panels appraised the questionnaires according to a RAND-modified Delphi method (Campbell et al., 2003). Each panel received the questionnaire corresponding to the area of interest of its members. Whenever available in the guidelines, international evidence levels were provided to facilitate the decision making. Experts were asked to score the key recommendations on a nine-point Likert scale ranging from 1 (hardly relevant) to 9 (extremely relevant), with respect to their impact on both ‘health gain’, i.e. health as ‘physical, mental, and social well-being’ (World Health Organization (WHO), 1946), and ‘overall efficacy’, i.e. prevention of unnecessary medical treatment and promotion of cost-effectiveness. Moreover, the experts were asked to provide for each guideline a top-five ranking of recommendations they considered ‘most important’ and ‘representative’ to assess the quality of clinical performance. They were also offered an opportunity to provide comments and suggest additional items. First, we analysed the data from the returned questionnaires and calculated median scores for ‘health gain’ and ‘overall efficacy’ for each recommendation. The selected recommendations were rated as valid if they matched the criterion described by Campbell (Campbell et al., 2000): an overall panel median score of 8 or 9. To rate a recommendation as valid, there also had to be ‘agreement’ between the ratings of the independent panellists. Agreement was defined as 75% or more of ratings within a panel being in the lowest tertile (1, 2, 3) or the highest tertile (7, 8, 9), so that items that were mainly in the unequivocal tertile (4, 5, 6) were excluded.
Second, we scored the recommendations rated valid by means of the Campbell criteria by awarding them points according to the expert's top-five ranking. For each number-one ranking by an expert, we awarded a recommendation 5 points, for each number-two ranking, we awarded 4 points and so on. This way, we created a list of scores reflecting the weight that experts assigned to each recommendation. For each guideline, the recommendations with a value of three points or more were listed top-item down. Next, a consensus meeting took place with these listings as feedback.
Consensus meeting
All participants in the questionnaire round were invited to a consensus meeting, where a discussion about the results of Step 3 was initiated. The participants were divided into three panels, each assessing separate guidelines. Each panel was asked to discuss and reconsider the previous rankings and aim at a new top five; selected recommendations had to be ‘applicable’ for indicator development and had to have ‘potential for improvement’ as well. The panels were allowed to suggest additional recommendations. The discussions lasted approximately 2 h and were chaired by four of the authors (S.M.M., R.P.M.G.H., W.L.D.M.N. and J.A.M.K.), who are experienced in performance measurement. This resulted in a final consensus-based top five for each guideline, ready for consideration in step 5.
Critical evaluation
Four authors (S.M.M., R.P.M.G.H., W.L.D.M.N. and J.A.M.K.) critically evaluated the top-five rankings from Step 4. The recommendations were once more appraised with regard to both ‘improvement potential’ and ‘applicability’ after operationalization for the clinical setting. Some items were excluded or combined due to overlap between guidelines or pragmatic reasons concerning operationalization.
Consultation with guideline developers
As a sixth and final step, we presented the final top-five rankings in a third feedback round to 12 gynaecologists who had participated in the development of these guidelines, and are therefore considered opinion leaders in the field.
|
Results |
|---|
|
TopAbstractIntroductionMaterials and methodsResultsDiscussionAcknowledgementsReferences |
|---|
Step 1
The online search of Medline and the additional Internet search revealed that several national and international organizations issued specific guideline programmes in the field of reproductive health or subfertility. Nevertheless, even widely distributed programmes, such as the ASRM and ESHRE programmes, are usually unaccompanied by quality indicators. However, programmes that are indeed accompanied by indicator sets (WHO, 1997
; Ford et al., 2003; ICSI, 2004; Temmerman et al., 2006) focus mainly on a public health perspective. They aim, e.g. at monitoring prevalences of contraceptive use, Chlamydia infections and induced abortions to describe current reproductive health care, but are clearly not specifically developed for the extensive domain of clinical outpatient subfertility care. In contrast, NICE issued their Clinical Guideline (CG) 11 concerning subfertility (NICE, 2004), and it was accompanied by seven ‘key priorities for implementation’. Because of the lack of other suitable and systematically developed subfertility indicators for clinical care, we decided to include all 28 level-A-evidence recommendations from the CG11 for appraisal in our selection procedure (Fig. 1).
Step 2
Altogether, 275 recommendations extracted from the NVOG guidelines were added. This resulted in a total of 303 recommendations for appraisal by the experts.
Step 3
Initially, 63 questionnaires were sent to the experts, of which 47 (75%) were returned fully completed. The response rates were 71% (n = 15) for Questionnaire 1, 76% (n = 16) for Questionnaire 2 and 76% (n = 16) for Questionnaire 3. The 303 preselected recommendations were reduced to 139 items in this selection round, while 36 new items were suggested. This resulted in 175 items for appraisal during the consensus round.
Step 4
Eleven experts attended the consensus meeting: three guideline developers, six gynaecologists, one fertility clinician and one gynaecologist in training. They discussed and prioritized the 175 items, resulting in a new selection of 53 recommendations.
Step 5
Critical evaluation of the 53 recommendations led to the exclusion of 6 items, and another16 were combined to 8 new items. A semifinal set of 39 key recommendations was offered to the guideline developers for approval.
Step 6
All the guideline developers approved of the set of 39 key recommendations, which was thus finalized.
Final set
These 39 key recommendations were transcribed into 39 quality indicators concerning either structural or procedural domains of subfertility care. Table 1 reflects for each guideline the selection of key recommendations within the various domains. Both structural (n = 2) and procedural (n = 37) aspects were selected. The latter encompass ‘indications for ART’ (n = 7), ‘diagnostic procedures’ (n = 6), ‘treatment procedures’ (n = 20) and ‘patient information’ (n = 4). In expert rounds 3 and 4, 8 of 16 patient information items (50%) were selected, as were 24 of 132 treatment procedure items (18%) and 11 of 91 diagnostic items (12%). Furthermore, two of five structure items (40%) concerning the annual evaluation of IUI and the accreditation of IVF laboratories were selected (Table 2). Only one of the 36 newly suggested items (regarding lifestyle advice) was included in the final set. Of the 28 level-A-evidence recommendations from the NICE guideline that were initially added in the first step, only two were included in the final set. Another nine (non-level-A evidence) recommendations were common to both the NICE and the NVOG guidelines (Table 2).
|
|
|
Discussion |
|---|
|
TopAbstractIntroductionMaterials and methodsResultsDiscussionAcknowledgementsReferences |
|---|
When we consider our final set of recommendations, it appears that 40% of the structure items and 50% of patient-information items were selected, which suggests that professionals acknowledge the importance of an accurate quality framework and complete patient information as conditions for ‘best possible care’. Furthermore, the panellists suggested 36 new items before Step 3, but only one of them was included in the final set (Table 2). Experts apparently consider the current guideline contents as possibly incomplete, but still sufficiently satisfactory regarding clinical relevance.
The fact that the quality indicators in our set refer mainly to the process of care, a few to structure and none to outcome reflects the ratios of these types of recommendations in guidelines. The lack of evidence for the best outcome measures in clinical care explains the absence of outcome-related recommendations in guidelines that are consensus and evidence-based. Nevertheless, outcome indicators (e.g. ‘live-birth rate per treatment cycle’) are the mean criterion used for judging professionals and hospitals in daily practice (Min et al., 2004). Outcome measures, on the one hand, have the disadvantage of being sensitive to case mix and the methods of data collection (Eddy, 1998; Mant, 2001). In addition, they may affect our judgement of the quality of care provided because good outcomes do not necessarily mean that care was delivered well. Process indicators, on the other hand, assess the process of care and reveal exactly where changes can be made, while structure indicators provide us with the framework to do so. Therefore, process and structure indicators are more valuable in quality improvement programmes because, unlike outcome measures, they offer a concrete starting point for improvement. This is especially true when process measures have been proved to relate to outcome measures.
In reproductive health care, choosing the best outcome measures to define the standards of success is currently being discussed internationally (Schieve and Reynolds, 2004; Dickey et al., 2004; Min et al., 2004; Jahn et al., 2006). A comparison of this ongoing discussion and activity about outcome measures with the almost complete lack of initiatives for developing and measuring process and structure indicators, highlights a striking discrepancy. Apparently, little effort has been made to bridge the gap between the evidence we make available in our guidelines and the supposedly evidence-based care we deliver in practice.
The literature search revealed one similar initiative of indicator development for clinical subfertility care, the NICE subfertility guideline (NICE, 2004). NICE used a systematic approach similar to ours to develop their auditable standards (http://www.nice.org.uk). However, a comparison of the contents of our final indicator set with NICE's seven key priorities shows that only one indicator is common to both sets: ‘no more than two embryos should be transferred per embryotransfer’ (Table 2). Aside from similarities in guideline contents and health systems in both countries, different stakeholders and slightly different selection criteria have probably influenced the composition of both final sets. In particular, the NICE criterion ‘make efficient use of National Health Service resources’ already curtails potential indicators because the lack of suitable resources would render indicators ineligible. In addition, NICE explicitly tries to include between 5 and 10 key priorities in their guidelines. This upper limit is in line with most quality measure projects currently being realized (McGory et al., 2005). Regarding subfertility care, having many indicators would, in our opinion, better suit such a wide guideline programme that encompasses several different domains. However, before cutting back our set of 39 indicators, we recommend performing a practice test.
There are some methodology limitations that could be considered with respect to the range of our final set. Comparable combined RAND-modified Delphi techniques have proved effective in several indicator-development initiatives (Campbell et al., 2000; Schouten et al., 2005; Gagliardi et al., 2005). However, the limitations and validity of such procedures have been questioned regularly (Jones and Hunter, 1995). In particular, the accuracy and validity is suggested to depend on several expert characteristics (Campbell et al., 1999). Questions may arise about whether the group of experts is heterogeneous enough and whether the individual expert contributions are of equal value. However, the group of experts in a small country like the Netherlands (which has only 13 licensed IVF clinics including eight university centres) is limited. We therefore think that the level of professional participation in our expert panels does reliably reflect the country's overall level of expertise. Moreover, a consensus-based-panel approach has the advantage of directly involving the target group of professionals in the procedure. By obtaining their consent during the selection process, we founded a substantial supporting base for our set of quality indicators, and consequently, for any national measurement programme or guideline-implementation strategy that may follow.
The systematic approach and panel composition underline how face and content validity are accounted for in this study. However, content validity alone is not enough to entitle indicators as valid. As already stated, improvement programmes in health care should preferably be tailor-made and therefore based on adequate measurements of performance. Although we have already considered selected criteria for operationalization (e.g. measurability and potential for improvement) during the selection rounds, the suggested quality indicators need to be submitted to a practice test. Then certain clinimetric characteristics, such as measurability, feasibility, reliability and improvement potential, can be properly assessed. Furthermore, efforts should be made to try to relate indicator scores to outcome measures in order to assess the clinical importance of the separate quality indicators. Ultimately, this practice test will show whether these indicators will hold up in the future. Independent of any future adjustments, the complete indicator set presented in this paper complements the existing guidelines and can still serve as a benchmarking instrument for educational or quality-improvement activities in fertility centres.
This study took international literature into account, but still took place within the framework of the Dutch subfertility guidelines. This implies that some of the selected indicators are particularly valuable in the Dutch situation and less suitable internationally. For example, the recommendation ‘no more than two embryos should be transferred per embryotransfer’ would probably not withstand the selection if the panellists were from the USA. Still, the topic of this indicator. e.g. ‘the maximum number of embryos transferred’ could be maintained for international application when adapted to specific national standards.
We strongly encourage the development of international subfertility guidelines based on both international evidence and consensus, in initiatives like the one ESHRE is currently undertaking (www.eshre.com). However, we suggest that all such initiatives be accompanied by the simultaneous development of quality indicators by an international panel. Such quality indicators should be tested in practice first, and should preferably match the health-care systems and ethics of the various countries involved. Suitable data resources should be appointed or adapted to match the demands of proper indicator measurement. The consensus method and the domains of subfertility care described in this paper could serve as a framework for further work in this field.
In conclusion, we have described a systematic procedure of developing a complete set of process and structure indicators based on an entire national subfertility guideline programme. These 39 quality indicators can be used to monitor subfertility care. However, their potential to do so accurately must first be proved in a proper practice test.
|
Acknowledgements |
|---|
|
TopAbstractIntroductionMaterials and methodsResultsDiscussionAcknowledgementsReferences |
|---|
We thank all the members of our expert panels and especially the experts attending the consensus meeting for their expertise and contributions to the procedure. We are grateful to Prof. J.M.M. van Lith, chairman of the subcommittee ‘Quality indicators’ of the NVOG, for his support and guidance. The Netherlands Organisation for Health Research and Development (ZonMw) funded this project (Grant no. 945-14-116).
|
References |
|---|
|
TopAbstractIntroductionMaterials and methodsResultsDiscussionAcknowledgementsReferences |
|---|
Bero LA, Grilli R, Grimshaw JM, Harvey E, Oxman AD, Thomson MA. Closing the gap between research and practice: an overview of systematic reviews of interventions to promote the implementation of research findings. The Cochrane Effective Practice and Organization of Care Review Group. BMJ (1998) 317:465–468.
Campbell SM, Braspenning J, Hutchinson A, Marshall MN. Research methods used in developing and applying quality indicators in primary care. BMJ (2003) 326:816–819.
Campbell SM, Cantrill JA, Roberts D. Prescribing indicators for UK general practice: Delphi consultation study. BMJ (2000) 321:425–428.
Campbell SM, Hann M, Roland MO, Quayle JA, Shekelle PG. The effect of panel membership and feedback on ratings in a two-round Delphi survey: results of a randomized controlled trial. Med Care (1999) 37:964–968.[CrossRef][Web of Science][Medline]
Campbell SM, Shield T, Rogers A, Gask L. How do stakeholder groups vary in a Delphi technique about primary mental health care and what factors influence their ratings? Qual Saf Health Care (2004) 13:428–434.
Cousineau TM, Domar AD. Psychological impact of infertility. Best Pract Res Clin Obstet Gynaecol (2007) 21:293–308.[CrossRef][Medline]
Dalkey N, Brown B, Cochran S. The Delphi Method III: Use of Self Ratings to improve Group Estimates (1969).
Dickey RP, Sartor BM, Pyrzak R. What is the most relevant standard of success in assisted reproduction?: no single outcome measure is satisfactory when evaluating success in assisted reproduction; both twin births and singleton births should be counted as successes. Hum Reprod (2004) 19:783–787.
Eddy DM. Performance measurement: problems and solutions. Health Aff (Millwood) (1998) 17:7–25.[Abstract]
Fitch K, Bernstein S, Aguilar M. The RAND/UCLA Appropriateness Method User's Manual (2001) Santa Monica, CA: RAND.
Ford J, Nassar N, Sullivan EA, Chambers G, Lancaster P. Reproductive health indicators. (2003) Australia.
Gagliardi AR, Fung MF, Langer B, Stern H, Brown AD. Development of ovarian cancer surgery quality indicators using a modified Delphi approach. Gynecol Oncol (2005) 97:446–456.[CrossRef][Web of Science][Medline]
Greil AL. Infertility and psychological distress: a critical review of the literature. Soc Sci Med (1997) 45:1679–1704.[CrossRef][Web of Science][Medline]
Grol R. Personal paper. Beliefs and evidence in changing clinical practice. BMJ (1997) 315:418–421.
Grol R, Baker R, Moss F. Quality improvement research: understanding the science of change in health care. Qual Saf Health Care (2002) 11:110–111.[CrossRef][Medline]
ICSI. The Institute for Clinical System Improvement:The diagnosis and management of infertility (2004) http://www.icsi.org/infertility/diagnosis_and_management_of_infertility_2301.html.
Jahn A, Bloemenkamp KW, Hannaford P, Olsen J, da Silva MO, Temmerman M. Monitoring reproductive health in Europe–what are the best indicators of reproductive health? Hum Reprod (2006) 21:2199–2200.
Jones J, Hunter D. Consensus methods for medical and health services research. BMJ (1995) 311:376–380.
Kallen B, Finnstrom O, Nygren KG, Otterblad OP, Wennerholm UB. In vitro fertilisation in Sweden: obstetric characteristics, maternal morbidity and mortality. BJOG (2005) 112:1529–1535.[Web of Science][Medline]
Mant J. Process versus outcome indicators in the assessment of quality of health care. Int J Qual Health Care (2001) 13:475–480.
McGory ML, Shekelle PG, Rubenstein LZ, Fink A, Ko CY. Developing quality indicators for elderly patients undergoing abdominal operations. J Am Coll Surg (2005) 201:870–883.[CrossRef][Web of Science][Medline]
Min JK, Breheny SA, MacLachlan V, Healy DL. What is the most relevant standard of success in assisted reproduction? The singleton, term gestation, live birth rate per cycle initiated: the BESST endpoint for assisted reproduction. Hum Reprod (2004) 19:3–7.
Nachtigall RD. International disparities in access to infertility services. Fertil Steril (2006) 85:871–875.[CrossRef][Web of Science][Medline]
Nelen WL, Hermens RP, Mourad SM, Haagen EC, Grol RP, Kremer JA. Monitoring reproductive health in Europe: what are the best indicators of reproductive health? A need for evidence-based quality indicators of reproductive health care. Hum Reprod (2007) 22:916–918.
NICE. Fertility: Assessment and Treatment for People with Fertility Problems (2004) http://www.nice.org.uk/pdf/CG011niceguideline.pdf.
Ombelet W, Martens G, De SP, Gerris J, Bosmans E, Ruyssinck G, Defoort P, Molenberghs G, Gyselaers W. Perinatal outcome of 12,021 singleton and 3108 twin births after non-IVF-assisted reproduction: a cohort study. Hum Reprod (2006) 21:1025–1032.
Schieve LA, Cohen B, Nannini A, Ferre C, Reynolds MA, Zhang Z, Jeng G, Macaluso M, Wright VC. A population-based study of maternal and perinatal outcomes associated with assisted reproductive technology in Massachusetts. Matern Child Health J (2007) Mar 8; Epub ahead of print.
Schieve LA, Reynolds MA. What is the most relevant standard of success in assisted reproduction?: challenges in measuring and reporting success rates for assisted reproductive technology treatments: what is optimal? Hum Reprod (2004) 19:778–782.
Schouten JA, Hulscher ME, Wollersheim H, Braspennning J, Kullberg BJ, van der Meer JW, Grol RP. Quality of antibiotic use for lower respiratory tract infections at hospitals: (how) can we measure it? Clin Infect Dis (2005) 41:450–460.[CrossRef][Web of Science][Medline]
Temmerman M, Foster LB, Hannaford P, Cattaneo A, Olsen J, Bloemenkamp KW, Jahn A, da Silva MO. Reproductive health indicators in the European Union: The REPROSTAT project. Eur J Obstet Gynecol Reprod Biol (2006) 126:3–10.[CrossRef][Web of Science][Medline]
Vayena E, Rowe PJ, Griffin P. Current practices and controversies in assisted reproduction. In: Report of a WHO Meeting (2002) Geneva: World Health Organisation.
WHO. The Constitution of the World Health Organization. (1946).
WHO. Monitoring reproductive health: selecting a short list of national and global indicators. (1997).
Woolf SH, Grol R, Hutchinson A, Eccles M, Grimshaw J. Clinical guidelines: potential benefits, limitations, and harms of clinical guidelines. BMJ (1999) 318:527–530.
Submitted on March 26, 2007; resubmitted on June 11, 2007; accepted on June 18, 2007.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  S.M. Mourad, W.L.D.M. Nelen, R.P.M.G. Hermens, L.F. Bancsi, D.D.M. Braat, G.A. Zielhuis, R.P.T.M. Grol, and J.A.M. Kremer Variation in subfertility care measured by guideline-based performance indicators Hum. Reprod., November 1, 2008; 23(11): 2493 - 2500. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 C. Huyser Affordable ART services in Africa: synthesis and adaptation of laboratory services ESHRE Monogr, July 1, 2008; 2008(1): 77 - 84. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||