Published 20 May 2019
A recent study has found that most trials and meta-analyses in reproductive medicine are insufficiently powered to produce a meaningful indication of live birth rate. Is the base of evidence-based medicine robust enough?
'Evidence' in reproductive medicine has been a hot topic for more than a decade. Questions became concerns with the increasing introduction of adjuvant treatments, with controversies emerging over the validity of PGT-A as a means of improving live birth rate in IVF. A widely cited 2012 report lamented the introduction of new technologies 'without appropriate development and evidence-based medicine to show that the procedure is safe and beneficial to the patient'.(1) The research and development considered necessary by the authors culminated in 'well designed RCTs with a follow up of all children born from the procedure'. A follow-up report five years later continued to call for 'robust studies' to confirm the safety and efficacy of any adjunct treatment before routine introduction.(2)
The power of results
Now, a systematic review of power calculations in trials and meta-analyses in reproductive medicine has concluded that even the largest trials are unlikely to detect plausible improvements in live birth rates, and meta-analyses are not large enough to make up for this shortcoming.(3) No wonder so many Cochrane reviews in reproduction end with an equivocal conclusion and a discouraging explanation that the data are weak and inconsistent.
The investigators reaffirm that a tested treatment is considered effective when a statistical test performed on primary outcomes is significant - that is, yielding a P-value of <0.05. Thus, they write, 'if a trial has high statistical power to detect any clinically relevant effect, then failure to observe statistical significance suggests that the treatment does not provide a meaningful benefit'.
To address this, the study itself analysed systematic reviews published in the Gynecology and Fertility section of the Cochrane Library which had live birth as a primary endpoint. The investigators calculated the statistical power necessary to detect improvements in LBR for each meta-analysis and for the largest RCT in each of those meta-analyses. Additionally, the 95% confidence intervals of estimated treatment effects from each meta-analysis and RCT were recorded to indicate the precision of the result.
Results of this analysis showed that the median power to detect an improvement in LBR of 5 percentage points (for example from 25 to 30%) was only 13% for the RCTs and 16% for the meta-analyses. Moreover, no RCT and only 2% of the meta-analyses achieved 80% power to detect an improvement of 5 percentage points.
So what does this mean for reproductive medicine? Are the results of our trials and meta-analyses strong enough to meet the demand of 'robust evidence', or are we interminably consigned to the Cochrane conclusion of inconsistent results and weak evidence? Indeed, the authors themselves raise the suggestion 'that interventions in reproductive medicine are frequently not assessed for plausible and clinically worthwhile improvements in LBR', and add that treatment effects were found to be 'generally imprecise, such that there remains substantial uncertainty around the relative merits of interventions'.
Finding the solution
Such findings have now been taken up as the subject of two editorials asking where reproductive medicine can find a meaningful evidence base. In the first Sjoerd Repping from the AMC in Amsterdam, who will be known by many ESHRE members as an unwavering supporter of randomised trials, once again reports that 'data demonstrating benefit are lacking for the vast majority of fertility treatments, thus calling into question their routine clinical use'.(4) Behind the assertion lie greedy fertility clinics into which patients are 'lured' with the promise of a claimed but unsupported clinical benefit. So how, asks Repping, to proceed?
His first answer is more RCTs, even if so many recent examples (such as HABSelect for sperm selection or ESTEEM for PGT-A) reach a neutral conclusion. His second proposition is a call for governing bodies (HFEA or ESHRE itself) to 'force' clinics to present accurate information and protect patients from treatments of unknown or no benefit.
The second editorial takes a more measured view and, while recognising the place of RCTs and meta-analyses within the evidence hierarchy, accepts that they are now 'very hard to do'.(5) Patient numbers, regulatory scrutiny and topicality all present barriers which may be insurmountable. As a more viable option, therefore, Macklon and colleagues propose the collective assembly of individual patient data to detect the mean rather than individual effects of treatments. Such a move, they suggest, in the emerging age of bioinformatics and personalised medicine ' would enable the link to be made between the underlying diagnosis and the effectiveness of therapies for specific patient phenotypes'. Relying on RCTs as the sole arbiter of clinical truth, they conclude, ' risks leaving many urgent clinical questions unanswered'.
As Repping urges, Macklon et al recognise that RCTs 'will continue to be of great value' - but the potential value of alternative approaches is an important step in providing the evidence that 'research orthodoxy' has failed to deliver in reproductive medicine. With so many studies of limited statistical power, the temptation is to draw a first-line conclusion from a secondary endpoint or trawl the data for an unanticipated result.
1. Harper J, Magli MC, Lundin K, et al. When and how should new technology be introduced into the IVF laboratory? Hum Reprod 2012; 27: 303-313.
2. Harper J, Jackson E, Sermon K, et al. Adjuncts in the IVF laboratory: where is the evidence for 'add-on' interventions? Hum Reprod 2017; 32: 485-491.
3. Stocking K, Wilkinson J, Lensen S, et al. Are interventions in reproductive medicine assessed for plausible and clinically relevant effects? A systematic review of power and precision in trials and meta-analyses. Hum Reprod 2019; 34: 659-665.
4. Repping S. Evidence-based medicine and infertility treatment. Lancet 2019; 393: 380-382.
5. Macklon NS, Ahuja KK, Fauser BCJM. Building an evidence base for IVF 'add-ons'. Repro Biomed Online 2019;