Academic Criticism

Falsification of data in academic research is a known problem. Many choose to ignore it.

A review of anonymous self-reports estimated that 2% of scientists admitted falsification of data. [...] More important are the conscious or unconscious set of minor decisions that might be made by the researcher depending on what the data seem to be showing. These ‘tweaks’ might include decisions about changes in the design of the experiment, when to stop collecting data, what data to exclude, what factors to adjust for, what groups to emphasize, what outcome measures to focus on, how to split continuous variables into groups, how to handle missing data, and so on. [Uri] Simonsohn [a psychologist at the University of Pennsylvania] calls these decisions ‘researcher degrees of freedom’, while [distinguished US statistician and blogger] Andrew Gelman refers more poetically to the ‘garden of forking paths’.

Source: David Spiegelhalter. The art of statistics: Learning from data. Penguin UK, 2019. [B042]

Some initiatives try to prevent data snooping, e.g. HARKing (which stands for coming up with the Hypotheses After the Results are Known)1.

When asked about a list of ten questionable research practices [...] a 2012 survey of 2,155 US academic psychologists [found that] 58% said they had carried on collecting more data after seeing whether the results were significant. [And] 67% said they had failed to report all of a study’s responses.

Source: David Spiegelhalter. The art of statistics: Learning from data. Penguin UK, 2019. [B042]