Avoiding the “Shut Down Effect” from Uncertain Research Results
Lurking behind much of the debate in this symposium is anxiety that negative findings about access to justice services will strengthen and facilitate attempts to reduce resources for access to justice services.
While it would be impossible to rebut the claim that this might happen, that can not be an argument against conducting or reporting research. On the contrary, it has to be an argument for more and better research. Having been involved in access to justice for decades, I am all too aware that it never seems to be the right time to make ourselves vulnerable — so the answer has to be always, because that is the only way to gain credibilit..
But this does all raise the question as to how research should be structured, analyzed, and particulalry, reported in order to minimize the risk of results inconsistent with the ultimate findings of the research, in all their complexity and subtlety.
- Generality of Reporting. The headings, abstract, etc, must be structured to accurately reflect the generlity of the research
- The Reporting of Context. Studies should be very careful about describing accurately the context in which treatment is provided.
- Randomization/Observation. Where studies are not randomized, that must be very clearly reported, and selection bias must be loudly proclaimed, not the subject of a footnote.
- Explanation of Statistical Significance. The issues of statistical significance must, to the extent possible, be explained as clearly as possible, in lay terms. The failure to do so, when it occurs, makes both overstatement an unfair critique easier.
- Lay Version. Research should be made available in a lay summary version, without complexities but with the detail and cautions. This will reduce the risk of the results being oversimplified by the media and/or others
- Vigilance as to Over/Under Generalization. The text should not only be accurate as to the level of generality of the research, but should be explicit as to the kinds of generalizations that might erroneously be drawn from the research. (This makes it easier to rebut overdrawn conclusions made by legal or political opponents.)
I would very much appreciate additions to such best practices. There are surely many more in the social sciences. Those suggested here, however, are less about avoiding error, and more about avoiding error in the reporting or use by others.