After several rounds of amendments, we reached a state where all of us were happy with the checklist. Then the time came to put it to the test!
A good checklist should be able to correctly ‘accept’ a good preprint and correctly ‘reject’ a bad preprint - but could the PRECHECK checklist in its current state do that? Simon and Nora worked to find out. For ‘bad’ preprints, they selected three retracted COVID-19 preprints. For the ‘good’ ones, they selected three from this list of milestone COVID-19 studies that later converted into publications: https://www.nature.com/articles/d41586-020-00502-w
The checklist passed the test, as bad preprints got many negative answers and good preprints had many more positive answers - when taking into account answers on the items and the Let’s Dig Deeper sections together. Interestingly though, the checklist also flagged up the parts that were not so good about ‘good’ preprints, for example, their lack of transparency regarding data and materials. These and other exciting findings lead us to rework the checklist some more.
Next up: identifying and inviting relevant experts to provide anonymous review of the checklist following the Delphi method.