Hum. Reprod. Advance Access originally published online on May 13, 2008
Human Reproduction 2008 23(8):1786-1792; doi:10.1093/humrep/den120
The methodological quality of clinical guidelines of the European Society of Human Reproduction and Embryology (ESHRE)
1 Department of Obstetrics and Gynaecology, Radboud University Nijmegen Medical Centre, PO Box 9101, 6500 HB Nijmegen, The Netherlands 2 Centre for Quality of Care Research (WOK), Radboud University Nijmegen Medical Centre, Nijmegen, The Netherlands 3 Department of Obstetrics and Gynaecology, Sahlgrenska University Hospital, Göteborg, Sweden 4 Infertility Centre, Ghent University Hospital, Ghent, Belgium 5 IVF Unit, Sophiahemmet, Stockholm, Sweden
6 Correspondence address. E-mail: w.nelen{at}obgyn.umcn.nl
 |
Abstract |
|---|
 Top Abstract IntroductionMaterials and MethodsResultsDiscussionFundingReferences |
|---|
BACKGROUND: Clinical practice guidelines bridge the gap between the evidence from literature and clinical practice, and they may provide guidance in ethical, legal and societal dilemmas. To explore the potentials for future international guideline development within the field of human reproduction and embryology, we assessed the quality of existing guidelines produced by the European Society of Human Reproduction and Embryology (ESHRE).
METHODS: We systematically searched for the ESHRE guidelines produced after 1996 in electronic databases and on the Internet. Subsequently, we assessed the methodological quality of these guidelines using the validated Appraisal of Guidelines for Research and Evaluation (AGREE) Instrument.
RESULTS: The overall methodological quality of most of the 11 selected ESHRE guidelines was poor. Most of the guidelines scored <30% in the domains of ‘stakeholder involvement’, ‘rigour of development’, ‘applicability’ and ‘editorial independence’. Only one guideline was rated ‘strongly recommended’.
CONCLUSIONS: The methodological quality of the guidelines produced under the auspices of ESHRE can be improved. We suggest a systematic, up-to-date methodology, investment in guideline development specialists, systematic quality control and the incorporation of indicator development. Furthermore, attention should be paid to the document nomenclature, and an ESHRE guidelines’ summary on a special part of the ESHRE website would be a good initiative.
Key words: guideline/quality of care/subfertility/ESHRE
|
Introduction |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionFundingReferences |
|---|
Although European borders are fading and all clinicians have easy access to the same scientific evidence, distinct differences in national health laws and healthcare services remain slightly preserved. The consequences of this are probably most obvious in health sectors struggling with societal, religious and ethical dilemmas. This is especially true within the field of reproductive medicine, where, for example, there are issues about the appropriate handling of gametes, embryos or gonadal tissues, the identity of donors and the risk of multiple births after assisted reproduction (The ESHRE Task Force on Ethics and Law, 2003
, 2004; Kaariainen et al., 2005; Fenton, 2006; Soini et al., 2006). The differences have already resulted in couples moving around to get access to subfertility care more rapidly, and more cheaply. They may move to places where they believe such care will be provided at a higher standard (labour market and reproductive tourism). This illustrates, now more than ever, the need for European Instruments to overcome practice variations and quality failures (Crosignani, 2005; Jones and Keith, 2006).
In general, clinical practice guidelines can play an important role in decreasing practice variations and in improving quality of care. They can bridge the gap between the growing stream of scientific evidence and actual clinical practice by facilitating the choices of healthcare professionals and patients for the most appropriate, safe and cost-effective care (Woolf et al., 1999; Grol et al., 2003; Kaariainen et al., 2005; Soini et al., 2006). Thus, European clinical practice guidelines in the field of human reproduction and embryology can improve the quality of European subfertility care.
To achieve these goals, these clinical guidelines’ conditions should be of high quality: valid, reliable, clinically applicable, clear and timely updated (Field and Lohr, 1990). Consequently, high quality of guidelines requires not only knowledge on the clinical topic, but also expertise in guideline development. Enthusiastic clinicians offering their spare time for guideline development may not always have the required expertise. Moreover, certain parts of guideline development, e.g. the selection and interpretation of the evidence, are costly and time consuming and should comply with certain criteria, such as transparency and independence of the developers (Fervers et al., 2003).
The Appraisal of Guidelines Research and Evaluation (AGREE) collaboration developed and validated an instrument for assessing the quality of clinical guideline development based on 23 key items that are categorized in six domains: scope and purpose, stakeholder involvement, rigour of development, clarity and presentation, applicability and editorial independence (www.agreecollaboration.org) (AGREE Collaboration, 2003). The methodological quality of clinical practice guidelines, as assessed with this AGREE Instrument, is reported to vary significantly (Harpole et al., 2003; Burgers et al., 2004; Christiaens et al., 2004; Gaebel et al., 2005; Appleyard et al., 2006; Arnau et al., 2006; Hurdowar et al., 2007). Haagen et al. (2006) analysed the quality of national European subfertility guidelines for intrauterine insemination previously and have reported important imperfections as well as differences. They suggested international collaboration at an European level, e.g. at the level of the European Society of Human Reproduction and Embryology (ESHRE), for selecting and interpreting evidence and guideline development. They believe that this would improve the scientific validity of clinical practice guidelines, promote international consensus for their clinical content and reduce the duplication of effort and inefficient use of resources (Haagen et al., 2006).
Recent attempts at such European collaboration have been made in the development of the ‘Guidelines for the diagnosis and treatment of endometriosis’ (Kennedy et al., 2005) and ‘Guidelines for recurrent miscarriage’ (Jauniaux et al., 2006), both initiated by ESHRE Special Interest Groups (SIGs). However, the quality of these ESHRE guidelines is unknown.
Insight into the quality of the existing ESHRE clinical practice guidelines is needed to explore ESHRE’s potential role for future international guideline development within the field of human reproduction and embryology. Therefore, we investigated the quality of clinical practice guidelines developed under the auspices of ESHRE using the AGREE Instrument.
|
Materials and Methods |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionFundingReferences |
|---|
Searching for clinical practice guidelines
To find clinical practice guidelines produced under the auspices of ESHRE, we used several systematic search methods (Fig. 1). First, we searched the ESHRE website (www.eshre.com) using the keywords ‘guideline(s)’, ‘practice guideline(s)’, ‘recommendation(s)’ or ‘consensus’.
|
Second, we used the search function of the ESHRE Monographs website (for the period 2002–2007) with the same keywords (eshremonographs.oxfordjournals.org).
Third, we searched Medline and Pubmed in January 2007 with the same keywords, but in combination with ‘ESHRE’.
Selection of clinical practice guidelines
Two authors (R.W.P. and W.L.D.M.N.) selected the documents. They consulted another author (R.P.M.G.H.) to reach consensus for any disagreement. To be selected for appraisal, a guideline had to fulfil all of the following criteria, namely it had to: (i) be produced in the name of ESHRE; (ii) focussed on topics within the field of reproductive medicine or embryology; (iii) relate directly to patient care; (iv) be published after 1996 and (v) be written in English. Furthermore, the title and/or abstract of the guideline had to contain the words ‘guideline(s)’, ‘practice guidelines’, ‘recommendations’ or ‘consensus’ or the document had to contain at least 10 clinical recommendations. We decided to search in the documents systematically for terms like "should; must; recommend*; advis*; consider*; requir*; need*; accept*; prefer*" in order to identify recommendations, because the forms in which they are presented differ widely. Technical manuals and systematic reviews such as the Capri workshop documents were excluded, unless they were specifically referred to as: ‘Guideline(s)’.
Appraisal of methodological quality of the selected clinical practice guidelines
We used the AGREE Instrument (http://www.agreecollaboration.org) to evaluate the methodological quality of the selected ESHRE guidelines. An international group of researchers from 13 countries developed and validated this Instrument in a multi-staged process of item generation, selection, scaling and field-testing for 100 selected guidelines (AGREE Collaboration, 2003). Cronbach’s alpha as a measure of internal consistency ranged from 0.64 to 0.88, whereas the intraclass correlation coefficient varied between 0.57 and 0.91, indicating an acceptable interrater reliability within the 194 appraisers (AGREE Collaboration, 2003). This reliability seemed to be insensitive for appraiser type (MacDermid et al., 2005). The AGREE Instrument assesses the methodological quality of clinical practice guidelines with 23 key items, categorized in six domains: scope and purpose (three items), stakeholder involvement (four items), rigour of development (seven items), clarity and presentation (four items), applicability (three items) and editorial independence (two items).
The guidelines selected for this study were divided between two groups of reviewers. All reviewers in each group appraised the selected guidelines independently by scoring each item of the AGREE Instrument on a four-point Likert scale (1, strongly disagree; 2, disagree; 3, agree; 4, strongly agree) (Table I).
|
The first group consisted of a Belgian gynaecologist, a Swedish gynaecologist (P.S. and K.G.N.), and a Dutch quality-of-care expert (W.L.D.M.N.). The second group consisted of a Swedish gynaecologist (C.B.), a Dutch embryologist (A.M.M.W.) and a quality of care expert (R.P.M.G.H.). A medical research student (R.W.P.) was a member of both groups to guarantee coherence in the appraisal between the two groups. Two appraisers in the first group (P.S. and K.G.N.) shared the total number of appraised guidelines. The appraisers based their scores on the instructions and examples for scoring given in the AGREE Instrument training manual (http://www.agreecollaboration.org) and the contents of the guideline itself and of any accompanying documents.
After the initial evaluation, differences in the scores of more than one point per item were discussed within the group in a 1 h telephone conference. The purpose of this conference was to avoid misinterpretation of either the documents or the AGREE Instrument, but not necessarily to achieve consensus. One of the authors (J.A.M.K.) was the moderator of both teleconferences. After this teleconference, we used the final scores to calculate six domain scores for each guideline. We added up the scores of the individual items in a domain given by the three or four appraisers and standardized the total as a percentage of the maximum possible score for that domain (obtained score – minimum possible score)/(maximum possible score – minimum possible score) with a range from 0% to 100%.
The final step in the appraisal involved an overall judgement of the guidelines as described by the AGREE Instrument’s instructions. A guideline was ‘strongly recommended’ if over half of the domains scored >60%. A guideline was ‘recommended’ if most of the domains scored >30%. A guideline was ‘not recommended’ if most of the domains scored <30%. The most positive category was chosen in the case of a tie.
Descriptive statistical analyses were performed with SPSS for Windows Release 12.0 Standard Version (Chicago, IL, USA).
|
Results |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionFundingReferences |
|---|
Search and selection
The use of the search functions on the ESHRE website and the ESHRE monographs website produced 138 and 72 ‘search hits’, respectively (Figure 1). A careful review of the titles and abstracts of these ‘search hits’ provided 22 potentially relevant documents. We also found ‘Appendix D’ of the June 2005 edition of the ESHRE monographs. This appendix gives a summary of 65 additional documents that were published by ESHRE before June 2005. The MEDLINE search retrieved 13 citations and the PUBMED search 160.
The next step, screening for duplicates and irrelevant documents, reduced the selection to 54 potentially relevant documents. Application of the selection criteria further reduced this number. Most of the documents that were eliminated at this stage were excluded because of the publication date or because of no the direct relationship with patient care. The final selection included 11 documents that were produced under the auspices of ESHRE (Table II). ESHRE SIGs produced eight of the selected guidelines. No SIG was involved in producing three of the guidelines. All documents were published between 1997 and 2006. Nine out of the 11 guidelines were published in ‘Human Reproduction’ or ‘Human Reproduction Update’. Besides publication in Human Reproduction, the ‘Revised 2003 consensus on diagnostic criteria and long-term health risks related to polycystic ovary syndrome’ was also published in Fertility and Sterility. This last version was used for the evaluation. Only an outline version of the ‘Guideline for counselling in infertility’ was published in Human Reproduction. The full version of this guideline was only available on the ESHRE website. The ‘Ovarian hyperstimulation (OHSS) guideline’ has not been published in a peer-reviewed journal; it is only available on the ESHRE website.
|
Appraisal of the methodological quality of clinical practice guidelines
The results of the appraisals of the methodological quality of the selected ESHRE guidelines are summarized in Table III. The median scores for the six domains varied between 7% and 56%. The ‘ESHRE guideline for the diagnosis and treatment of endometriosis’ scored highest in five of six domains.
|
For the first domain, ‘scope and purpose’, almost all guidelines (10 out of 11) scored >30% and four guidelines had a score of >60%.
For the ‘stakeholder involvement’ domain, only four guidelines scored >30%. This is partly due to low median ratings of 0% on both key items 5 (involvement of patients’ view) and 7 (piloting of the guideline).
For the ‘rigour of development’ domain, all but one guideline scored <30%. Only the key items 11 (consideration of side effects, etc.) and 12 (linking to evidence) had median scores >10%, namely 44% and 33%, respectively.
The guidelines scores were relatively high for the domain ‘clarity and presentation’, with a median domain score of 42% and 9 out of 11 guidelines scoring >30%. However, the median score for key item 18 (tools for application) was only 8%.
All guidelines scored <30% for the ‘applicability’ domain. The median scores for key item 19 (organizational barriers) was 11%, but 0% for the key items 20 (costs implications) and 21 (review criteria).
For the last domain, ‘editorial independence’ (median score of 22%), only two guidelines scored >30%. Low median scores of 25% and 0% were obtained for the key items 22 (funding body) and 23 (conflicts of interest).
The overall judgement resulted in six ‘not recommended’ guidelines, four ‘recommended’ guidelines and one ‘strongly recommended’ guideline.
Many low scores were due to the absence of statements about certain aspects of the guideline development procedure, e.g. specific information about the individuals involved, the methodology used for both searching and selecting evidence as well as the translation into recommendations, the procedure for updating the guidelines and information about the funding body.
The appraisers agreed completely in 4 of the 66 domains, and there were no variations in their scoring of these domains. For the remaining 62 domains, the variance in domain score is given by the 95% confidence intervals. Some 95% confidence intervals were small, such as those for the ‘stakeholder involvement’ domain in guidelines 4 and 6 and the ‘rigour of development’ domain in guidelines 8 and 9. Some confidence intervals were broader than 40 points, for instance, for the ‘scope and purpose’ domain in guidelines 3, 7 and 8. Especially, in those domains, a more exact estimation could give little shifts between the three categories (<30%, 30–60% and >60%).
Comparison of the domain scores of guidelines produced before and after the development of the AGREE Instrument (before or in 2003 versus 2004 or later) showed no relevant differences. Even two of the five recommended guidelines were produced in 2000 and 2001.
|
Discussion |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionFundingReferences |
|---|
This study provides insight into the methodological quality of clinical practice guidelines developed under the auspices of ESHRE. They could be greatly improved, since the overall methodological quality is poor. This is particularly important, because the actual labour market in subfertility care and the increase in reproductive tourism illustrate and stress the need for qualitatively good instruments to reduce practice variations and quality failures at an international level.
Our study shows that guideline developers were moderate to good at describing the objectives and the patient groups to which the guidelines should apply. The scores for recommendations’ presentation were moderate. The low median scores for the ‘stakeholder involvement’, ‘rigour of development’, ‘applicability’ and ‘editorial independence’ domains indicate a great improvement potential in these aspects of guideline development. This score distribution across the various domains is comparable to those found by others, but our median domain scores are remarkably low (AGREE Collaboration, 2003; Appleyard et al., 2006; Arnau et al., 2006; Hulshof and Hoenen, 2007; Hurdowar et al., 2007; Zhang et al., 2007). Many of our low key item scores arose from a lack of statements. These statements about ‘stakeholder involvement’, for instance, are important to provide transparency and verification and promote acceptance among stakeholders. Tools like the AGREE Instrument focus on essential aspects of guideline development and could serve as a checklist during this developmental procedure. One explanation for the low scores may be that half of the included documents were developed before 2003, the year the AGREE Instrument was published. However, the comparison of the guidelines produced before and after 2003 showed no relevant differences in methodological quality; suggesting non-general use of such instruments, even after 2003. Nevertheless, some low scoring aspects of guideline development can be explained by the fact that they have become more important in just the last few years, e.g. the integration of patient preferences, procedures for updating the guidelines and application tools’ provision.
Owing to the absence of a simple summary of the produced guidelines on the ESHRE website, we used a systematic and rigourous approach to identify ESHRE guidelines. Other strong points of our appraisal are the use of the internationally validated AGREE Instrument and multidisciplinary composition of our appraiser group. Moreover, we organized additional telephone conferences to avoid rating errors, which resulted in robust domain scores with usually narrow confidence intervals.
However, our study has also some limitations. First, although the AGREE Instrument appraises the methodological quality of guideline development, there is more to a clinical guideline than how it is created. To come to an overall judgement about the guideline’s usefulness, the content’s quality of the individual recommendations and their impact on patients’ outcomes should be taken into account. No validated instruments for such appraisals are currently available.
Second, we had to discriminate between guidelines and reviews, manuals and consensus statements. Because of the few clinical recommendations and their ethical character, we decided to exclude the ESHRE Taskforces on Ethics and Law. However, further qualification of these documents may be valuable. Furthermore, the Capri workshop documents first discovered by our search gave the impression of systematic reviews rather than clinical guidelines. For each clinical question to be answered in a clinical guideline, a systematic review is needed. Therefore, future collaboration of guideline developers and participants of the Capri workshops may be preferable and fruitful.
Third, most studied ESHRE guidelines were not classified in minimal standards or best-practice guidelines, so we pooled all the guidelines and judged them with the same instrument. However, since there are known domain score differences between guidelines within and without a guideline programme and domain score differences between guidelines with and without detailed technical documentation, it is possible that ignoring such guideline classification may have influenced our results (AGREE Collaboration, 2003).
It is beyond doubt that ESHRE has an obligation to develop European clinical guidelines, but future ESHRE guideline developers should be selected with care. The group should minimally include representatives from all the professional groups that are likely to use the guidelines. But ESHRE should also train enthusiastic ESHRE members for guideline development experts or ESHRE should call out the experiences of specialized organizations like the National Institute for Health and Clinical Excellence (NICE) or the Guidelines International Network (GIN) to give guidance in all phases of future guideline development (Ollenschlager et al., 2004). In addition, ESHRE should adapt an up-to-date methodology to incorporate patients’ preferences in future guidelines. Moreover, obstacles for guideline implementation should be identified and solved in the early developmental phase to facilitate final implementation of a guideline. In preliminary studies, the ‘GuideLine Implementability Appraisal’ (GLIA) has proved useful for such barrier identification, but further evaluation is needed to assess GLIA’s effectiveness (Shiffman et al., 2005).
Within this implementation scope, it can also be important to define quality indicators during guideline development. Currently, reproductive healthcare is monitored mainly by outcome indicators with disadvantages such as: a lack of insight into the way the outcomes were achieved, a probability that outcome differences may be due to case mix and differences in data collection, and the risk of treatment refusal of patients with a poor prognosis (Eddy, 1998; Mant, 2001; Min et al., 2004; Jahn et al., 2006). These problems can be overcome by monitoring reproductive healthcare by a combination of outcome, process and structure indicators (Nelen et al., 2007). The central development of process and structure indicators could be incorporated into the guideline development methodology to facilitate the evaluation of reproductive healthcare over time and in various countries (Nelen et al., 2007).
In conclusion, ESHRE has the potential to develop international guidelines within the field of human reproduction and embryology, but the quality of the existing guidelines produced under the auspices of ESHRE can be improved. We suggest a systematic and up-to-date methodology, consideration of patients’ preferences, investment in guideline development specialists, a systematic quality control by tools like the AGREE and GLIA Instruments, and the development and incorporation of process and structure indicators for monitoring reproductive healthcare quality. Furthermore, attention should be paid to the nomenclature of the ESHRE documents to distinguish guidelines from manuals, statements and reviews easily. A summary of the existing ESHRE guidelines on a special part of the ESHRE website would be welcomed and useful.
|
Funding |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionFundingReferences |
|---|
P.S. is holder of a fundamental clinical research mandate by the Flemish Fund for Scientific Research (FWO-Vlaanderen).
|
References |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionFundingReferences |
|---|
AGREE Collaboration. Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care (2003) 12:18–23.
Appleyard TL, Mann CH, Khan KS. Guidelines for the management of pelvic pain associated with endometriosis: a systematic appraisal of their quality. BJOG (2006) 113:749–757.[CrossRef][Web of Science][Medline]
Arnau JM, Vallano A, Lopez A, Pellise F, Delgado MJ, Prat N. A critical review of guidelines for low back pain treatment. Eur Spine J (2006) 15:543–553.[CrossRef][Web of Science][Medline]
Burgers JS, Fervers B, Haugh M, Brouwers M, Browman G, Philip T, Cluzeau FA. International assessment of the quality of clinical practice guidelines in oncology using the Appraisal of Guidelines and Research and Evaluation Instrument. J Clin Oncol (2004) 22:2000–2007.
Christiaens T, De Backer D, Burgers J, Baerheim A. Guidelines, evidence, and cultural factors. Scand J Prim Health Care (2004) 22:141–145.[CrossRef][Web of Science][Medline]
Crosignani PG. Looking to Europe: something’s changed in Italy. Hum Fertil (Camb) (2005) 8:71–73.[Medline]
Eddy DM. Performance measurement: problems and solutions. Health Aff (Millwood) (1998) 17:7–25.[Abstract]
Fenton RA. Catholic doctrine versus women’s rights: the new Italian law on assisted reproduction. Med Law Rev (2006) 14:73–107.
Fervers B, Philip T, Haugh M, Cluzeau F, Browman G. Clinical-practice guidelines in Europe: time for European co-operation for cancer guidelines. Lancet Oncol (2003) 4:139–140.[CrossRef][Web of Science][Medline]
Field MJ, Lohr KN. Clinical Practice Guidelines: Directions for a New Program. (1990) Washington, DC, USA: National Academy Press.
Gaebel W, Weinmann S, Sartorius N, Rutz W, McIntyre JS. Schizophrenia practice guidelines: international survey and comparison. Br J Psychiatry (2005) 187:248–255.
Grol R, Cluzeau FA, Burgers JS. Clinical practice guidelines: towards better quality guidelines and increased international collaboration. Br J Cancer (2003) 89:S4–S8.[CrossRef][Web of Science][Medline]
Haagen EC, Hermens RP, Nelen WL, Braat DD, Grol RP, Kremer JA. Subfertility guidelines in Europe: the quantity and quality of intrauterine insemination guidelines. Hum Reprod (2006) 21:2103–2109.
Harpole LH, Kelley MJ, Schreiber G, Toloza EM, Kolimaga J, McCrory DC. Assessment of the scope and quality of clinical practice guidelines in lung cancer. Chest (2003) 123:7S–20S.[CrossRef][Web of Science][Medline]
Hulshof C, Hoenen J. Evidence-based practice guidelines in OHS: are they agree-able? Ind Health (2007) 45:26–31.[CrossRef][Web of Science][Medline]
Hurdowar A, Graham ID, Bayley M, Harrison M, Wood-Dauphinee S, Bhogal S. Quality of stroke rehabilitation clinical practice guidelines. J Eval Clin Pract (2007) 13:657–664.[CrossRef][Web of Science][Medline]
Jahn A, Bloemenkamp KW, Hannaford P, Olsen J, da Silva MO, Temmerman M. Monitoring reproductive health in Europe–what are the best indicators of reproductive health? Hum Reprod (2006) 21:2199–2200.
Jauniaux E, Farquharson RG, Christiansen OB, Exalto N. Evidence-based guidelines for the investigation and medical treatment of recurrent miscarriage. Hum Reprod (2006) 21:2216–2222.
Jones CA, Keith LG. Medical tourism and reproductive outsourcing: the dawning of a new paradigm for healthcare. Int J Fertil Womens Med (2006) 51:251–255.[Web of Science][Medline]
Kaariainen H, Evers-Kiebooms G, Coviello D. Medically assisted reproduction and ethical challenges. Toxicol Appl Pharmacol (2005) 207:684–688.[CrossRef][Medline]
Kennedy S, Bergqvist A, Chapron C, D’Hooghe T, Dunselman G, Greb R, Hummelshoj L, Prentice A, Saridogan E. ESHRE guideline for the diagnosis and treatment of endometriosis. Hum Reprod (2005) 10:2698–2704.
MacDermid JC, Brooks D, Solway S, Switzer-McIntyre S, Brosseau L, Graham ID. Reliability and validity of the AGREE instrument used by physical therapists in assessment of clinical practice guidelines. BMC Health Serv Res (2005) 5:18.[CrossRef][Medline]
Mant J. Process versus outcome indicators in the assessment of quality of health care. Int J Qual Health Care (2001) 13:475–480.
Min JK, Breheny SA, MacLachlan V, Healy DL. What is the most relevant standard of success in assisted reproduction? The singleton, term gestation, live birth rate per cycle initiated: the BESST endpoint for assisted reproduction. Hum Reprod (2004) 19:3–7.
Nelen WL, Hermens RP, Mourad SM, Haagen EC, Grol RP, Kremer JA. Monitoring reproductive health in Europe: what are the best indicators of reproductive health? A need for evidence-based quality indicators of reproductive health care. Hum Reprod (2007) 22:916–918.
Ollenschlager G, Marshall C, Qureshi S, Rosenbrand K, Burgers J, Makela M, Slutsky J. Improving the quality of health care: using international collaboration to inform guideline programmes by founding the Guidelines International Network (G-I-N). Qual Saf Health Care (2004) 13:455–460.
Shiffman RN, Dixon J, Brandt C, Essaihi A, Hsiao A, Michel G, O’Connell R. The GuideLine Implementability Appraisal (GLIA): development of an instrument to identify obstacles to guideline implementation. BMC Med Inform Decis Mak (2005) 5:23.[CrossRef][Medline]
Soini S, Ibarreta D, Anastasiadou V, Ayme S, Braga S, Cornel M, Coviello DA, Evers-Kiebooms G, Geraedts J, Gianaroli L, et al. The interface between assisted reproductive technologies and genetics: technical, social, ethical and legal issues. Eur J Hum Genet (2006) 14:588–645.[CrossRef][Web of Science][Medline]
The ESHRE Task Force on Ethics and Law. Ethical issues related to multiple pregnancies in medically assisted procreation. Hum Reprod (2003) 18:1976–1979.
The ESHRE Task Force on Ethics and Law. Taskforce 7: Ethical considerations for the cryopreservation of gametes and reproductive tissues for self use. Hum Reprod (2004) 19:460–462.
Woolf SH, Grol R, Hutchinson A, Eccles M, Grimshaw J. Clinical guidelines: potential benefits, limitations, and harms of clinical guidelines. BMJ (1999) 318:527–530.
Zhang W, Moskowitz RW, Nuki G, Abramson S, Altman RD, Arden N, Bierma-Zeinstra S, Brandt KD, Croft P, Doherty M, et al. OARSI recommendations for the management of hip and knee osteoarthritis, Part I: Critical appraisal of existing treatment guidelines and systematic review of current research evidence. Osteoarthritis Cartilage (2007) 15:981–1000.[CrossRef][Web of Science][Medline]
Submitted on November 27, 2007; resubmitted on March 11, 2008; accepted on March 20, 2008.
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||