Lessons on standards from Clinical Trials

This week, the UK published new guidelines on Clinical trials in the field of medicine. As a field, they are ahead in terms of oversight and monitoring processes, and there are lessons to be learnt from this new document.

Are most published academic findings really false?

The issue of publication bias reared its head around the turn of the century. In 2005, John Ioannidis published a controversial paper called ‘Why Most Published Research Findings are False’ available here. Pharmaceutical companies and many others had noted that many of the findings in published academic papers were impossible to re-create in their labs. Was it possible that academics, people we trust to objectively sift through input data, have been fudging their results for career advancement?

Ioannidis’ article is a mathematical explanation of why clinical trials were producing so many seemingly spurious results. Instead of blindly assuming that a p-value of less than 0.05 was evidence of confidence, he looked at the chances of any given study with a positive finding (in other words one that has reported a statistically significant correlation between variables) is actually valid. Put simply, in any set of 100 studies with positive outcomes, 5 of those will be false positives. Add to that, the problem that:

  1. A p-value of 0.05 is a minimum requirement to get published, so the many, many studies that produce results with higher p-values will be filtered into the trash can
  2. Publications want surprising content, not additions to known content.

The chances of those 5 false positives ending up getting published are very high. The 95 other studies that found counter evidence won’t leave the researchers office. It is a problem of publication bias and misplaced confidence in statistics.

How did the Federal Drug Agency in the US respond to the findings?

Following a major review in the USA, the Federal Drugs Agency looked into the problem in the use of anti-depressant drugs and found trusting the publication process alone was an insufficient and flawed process. It is now required that ALL researchers carrying out clinical trials on medicines report their findings. This provides pharmaceutical companies and those in charge of life-or-death decision making complete data on which to base their decisions.

How do the new guidelines help my field?

Fast forward to this week and new best practice guidelines have been released, and serve as a guideline for all of us. The tone of the document  available here, is that it is the duty of those carrying out data initiatives to prove that they are trustworthy, putting the burden of trust on data analysts to be transparent and responsive in their actions. The document lays out steps at each stage of the process, where transparency is needed. Lead National Data Guardian for health, Dr Nicola Burne on launching the new approach has argued that it is not enough today for organisations to state that they have met minimal legal obligations. Those seeking respect for their work should be more open in sharing their processes and ready to defend their practices.

Support in this from the IoA

We’ve been concerned about transparency and trustworthy analytics here at the IoA too! If you have questions about current reporting best practices for your own data processes, check our Governance and Professionalism courses here, for some guidelines.

Related Articles

Datacamp - Learning Tracks

All IoA members can use the installation-free Data Camp environments to build, practice and test your skills in Data Camp. We have two custom built tracks to allow you to ensure your training is on course to fulfil your career goals. We’ve recommended two tracks of knowledge and analytics study aligned to all of the 7 first years in the Data Competency Framework.

Which Track is for me?

Business analyst with R: This track will take you through spreadsheet skills and BI tools in the early years, and build up your coding skills to use R environments in the later years with more challenging data projects.

Python analyst: This track goes straight into Python coding and will take you all the way to working with unstructured data and deep learning techniques.
Look for the track name and year when you search for a course.
With our custom tracks, we’ve selected the skills that we know employers are looking for but remember that you can also take any of the 300 courses and assessments and projects any time you want and add that to your CPD records, too. You can find a post discussing the aims and structure of the tracks here.

Datacamp - Learning Tracks

All IoA members can use the installation-free Data Camp environments to build, practice and test your skills in Data Camp. We have two custom built tracks to allow you to ensure your training is on course to fulfil your career goals. We’ve recommended two tracks of knowledge and analytics study aligned to all of the 7 first years in the Data Competency Framework.

Which Track is for me?

Business analyst with R: This track will take you through spreadsheet skills and BI tools in the early years, and build up your coding skills to use R environments in the later years with more challenging data projects.

Python analyst: This track goes straight into Python coding and will take you all the way to working with unstructured data and deep learning techniques.

Look for the track name and year when you search for a course.

With our custom tracks, we’ve selected the skills that we know employers are looking for but remember that you can also take any of the 300 courses and assessments and projects any time you want and add that to your CPD records, too. You can find a post discussing the aims and structure of the tracks here


View Learning Tracks


Go To DataCamp