Research question

Why is this important? A study cannot be done without a research question/aim. A clear and precise research question/aim is necessary for all later decisions on the design of the study. The research question/aim should ideally be part of the abstract and explained in more detail at the end of the introduction.

Study type

Why is this important? For a study to be done well and to provide credible results, it has to be planned properly from the start, which includes deciding on the type of study that is best suited to address the research question/aim. There are various types of study (e.g., observational studies, randomised experiments, case studies, etc.),and knowing what type a study was can help to evaluate whether the study was good or not.


What is the study type?

Some common examples include:

  • observational studies - studies where the experimental conditions are not manipulated by the researcher and the data are collected as they become available. For example, surveying a large group of people about their symptoms is observational. So is collecting nasal swabs from all patients in a ward, without having allocated them to different pre-designed treatment groups. Analysing data from registries or records is also observational. For more information on what to look for in a preprint on a study of this type, please consult the relevant reporting guidelines: STROBE.

  • randomised experiments - studies where participants are randomly allocated to different pre-designed experimental conditions (these include Randomised controlled trials [RCTs]). For example, to test the effectiveness of a drug, patients in a ward can be randomly allocated to a group that receives the drug in question, and a group that receives standard treatment, and then followed up for signs of improvement. For more information on what to look for in a preprint on a study of this type, please consult the relevant reporting guidelines: CONSORT.

  • case studies - studies that report data from a single patient or a single group of patients. For more information on what to look for in a preprint on a study of this type, please consult the relevant reporting guidelines: CARE.

  • systematic reviews and meta-analyses - summaries of the findings of already existing, independent studies. For more information on what tolook for in a preprint on a study of this type, please consult the relevant reporting guidelines: PRISMA.


Let’s dig deeper

If the study type is not explicitly stated, check whether you can identify the study type after reading the paper.

  YES / NO / MAYBE

Use the questions below for guidance:

  • Does the study pool the results from multiple previous studies?
    • If yes, it falls in the category systematic review/meta-analysis.
  • Does the study compare two or more experimenter-generated conditions or interventions in a randomised manner?
    • If yes, it is a randomised experiment.
  • Does the study explore the relationship between characteristics that were not experimenter-generated?
    • If yes, then it is an observational study.
  • Does the study document one or multiple clinical cases?
    • If yes, it is a case study.

Transparency and Integrity

  1. Is a protocol, study plan, or registration of the study at hand mentioned?

    YES / NO / MAYBE

  2. Is data sharing mentioned? Mentioning any reasons against sharing also counts. Mentioning only that data will be shared “upon request” does not count.

    YES / NO / MAYBE

  3. Is materials sharing mentioned? Mentioning any reasons against sharing also counts. Mentioning only that materials will be shared “upon request” does not count.

    YES / NO / MAYBE

  4. Does the article contain an ethics approval statement (e.g., approval granted by institution, or no approval required)?

    YES / NO / MAYBE

  5. Have conflicts of interest been declared? Declaring that there were none also counts.

    YES / NO / MAYBE


Why is this important?

Study protocols, plans, and registrations serve to define a study’s research question, sample, and data collection method. They are usually written before the study is conducted, thus preventing researchers from changing their hypotheses based on their results, which adds credibility. Some study types, like RCT’s, must be registered.

Sharing data and materials is good scientific practice which allows people to review what was done in the study, and to try to reproduce the results. Materials refer to the tools used to conduct the study, such as code, chemicals, tests, surveys, statistical software, etc. Sometimes, authors may state that data will be “available upon request”, or during review, but that does not guarantee that they will actually share the data when asked, or after the preprint is published.

Before studies are conducted, they must get approval from an ethical review board, which ensures that no harm will come to the study participants and that their rights will not be infringed. Studies that use previously collected data do not normally need ethical approval. Ethical approval statements are normally found in the methods section.

Researchers have to declare any conflicts of interest that may have biased the way they conducted their study. For example, the research was perhaps funded by a company that produces the treatment of interest, or the researcher has received payments from that company for consultancy work. If a conflict of interest has not been declared, or if a lack of conflict of interest was declared, but a researcher’s affiliation matches with an intervention used in the study (e.g., the company that produces the drug that is found to be the most effective), that could indicate a potential conflict of interest, and a possible bias in the results. A careful check of the affiliation of the researchers can help identify potential conflicts of interest or other inconsistencies. Conflicts of interests should be declared in a dedicated section along with the contributions of each author to the paper


Let’s dig deeper

  1. Can you access the protocol/study plan (e.g., via number or hyperlink)

    YES / NO / MAYBE

  2. Can you access at least part of the data (e.g., via hyperlink, or on the preprint server). Not applicable in case of a valid reason for not sharing.

    YES / NO / MAYBE

  3. Can you access at least part of the materials (e.g., via hyperlink, or on the preprint server). Not applicable in case of a valid reason for not sharing.

    YES / NO / MAYBE

  4. Can the ethical approval be verified (e.g., by number). Not applicable if it is clear that no approval was needed.

    YES / NO / MAYBE

By ‘access’, we mean whether you can look up and see the actual protocol, data, materials, and ethical approval. If you can, you can also look into whether it matches what is reported in the preprint.

Limitations


Why is this important? No research study is perfect, and it is important that researchers are transparent about the limitations of their own work. For example, many study designs cannot provide causal evidence, and some inadvertent biases in the design can skew results.Other studies are based on more or less plausible assumptions. Such issues should be discussed either in the Discussion, or even in a dedicated Limitations section.


Let’s dig deeper

Check for potential biases yourself. Here are some examples of potential sources of bias:

  1. Check the study’s sample (methods section). Do the participants represent the target population? Testing a drug only on white male British smokers over 50 is probably not going to yield useful results for everyone living in the UK, for example. How many participants were there? There is no one-size-fits-all number of participants that makes a study good, but in general, the more participants, the stronger the evidence.
  2. Was there a control group or control condition (e.g., placebo group or non-intervention condition)? If not, was there a reason? Having a control group helps to determine whether the treatment under investigation truly has an effect on an experimental group and reduces the possibility of making an erroneous conclusion. Not every study can have such controls though. Observational studies, for example, typically do not have a control group or condition, nor do case studies or reviews. If your preprint is on an observational study, case study, or review, this item may not apply.
  3. Was there randomisation? That is, was theallocation of participants or groups of participants to experimental conditions done in a random way? If not, was there a reason? Randomisation is an excellent way to ensure that differences between treatment groups are due to treatment and not confoundedby other factors. For example, if different treatments are given to patients based on their disease severity, and not at random, then the results could be due to either treatment effects or disease severity effects, or an interaction -we cannot know. However, some studies, like observational studies, case studies, or reviews, do not require randomisation.If your preprint is on an observational study, case study, or review, this item may not apply.
  4. Was there blinding? Blinding means that some or all people involved in the study did not know how participants were assigned to experimental conditions. For example, ifparticipants in a study do not know whether they are being administered a drug or a sham medication, the researchers can control for the placebo effect (people feeling better even after fake medication because of their expectation to get better). However, blinding is not always possible and cannot be applied in observational studies or reanalyses of existing non-blinded data, for example.If your preprint is on an observational study, case study, or review, this item may not apply).