Results masked review: peer review without publication bias

© istockphoto.com/xijian We know that research data isn’t neat and tidy. It’s messy, complex and often throws something unexpected at us. At the Journal of Vocational Behavior opens in new tab/window, as well as some of its fellow journals we’re ok with that, because we want to publish findings that tell the “truth” (or as close as we can get) and we can prove it to you.

Results masked review (RMR) is a new form of peer review which allows an article to be judged on the merits of its research question(s) and methodology, not the findings. This ensures that we publish important results, regardless of their statistical significance.

Watch this video to find out about RMR articles opens in new tab/window.

A new sort of peer review

RMR articles are sent for review without the results, discussion or conclusion (although data has already been collected) and reviewers are asked to evaluate the article on the research question and the methodology only. The review process is split into two stages. In stage 1, only the research question and methodology are sent for review, and reviewers are asked to provide a recommendation. If the paper is given an in-principle “accept” decision, the paper moves into stage 2 where the author submits the full paper for review. More detail on this process can be found here opens in new tab/window.

What’s in it for the author?

If you asked a compelling question, used rigorous methods and data analysis, but got non-significant or unusual results, what would you do with that study? The sad truth is that many researchers would decide to not submit to a journal because of concerns getting it published. Alternatively, they might use questionable research practices, including submitting research based on inadequate statistical power, selective reporting of results, P-Hacking (finding patterns in the data which may cause the data to appear significant) and HARKing (Hypothesising After Results Are Known) in order to make the findings more appealing to a journal. With RMR articles, you don’t need to worry about what editors or reviewers might think about your results. As long as you have asked an important question and performed a rigorous study, your paper will be treated the same as any other. You do not need to have null results to submit an RMR article; there are many reasons why it can be helpful to have the results blinded at initial review.

How should reviewers approach RMR articles?

Reviewing a paper without the results, conclusion or discussion sections may allow you to focus on the research question and methods in more detail and without any preconceptions based on the findings. It is quicker too, as you have less material to digest. Admittedly, you are likely to see the paper more times than with the regular review process, but each time in smaller chunks. Feedback we have received from current reviewers is that it takes them the same amount of time to review an RMR article overall as a regular article.

I enjoy reviewing RMR articles in JVB as there is an opportunity to provide actionable feedback to authors to help them bring out the best in their work. - Anonymous reviewer

How is it different to Registered Reports?

This model has been adapted from Registered Reports opens in new tab/window, which is a form of empirical article in which the introduction, methods, and proposed analyses are pre-registered and reviewed prior to research being conducted. The key difference with RMR articles is that the data has already been collected before submission to the journal, but only the introduction, and methodology are initially submitted for review.

Find out more

The Journal of Vocational Behavior is one of a number of journals in psychology, business and management considering RMR articles, including the Leadership Quarterly, Journal of Business and Psychology, BMC Psychology and Journal of Personnel Psychology.

To find out more about the concept, process or to submit or review an RMR article for JVB, visit: https://www.journals.elsevier.com/journal-of-vocational-behavior opens in new tab/window