Mdthbs

This question applies both to papers people write to attain a degree, as well as academic papers submitted to journals by researchers.

In theory, the way science works is by building your own original research on top of other people's work. The general idea is that if they did their work well, and you did your work well, then new insights are revealed and human knowledge as a whole is advanced.

But not everyone is honest and not everyone cares for advancing their field. Bad science is a thing and takes many shapes. So people check each others work, and that includes the references to other people's work. After all, if you base your own work on flawed premises, then no matter how correct you make your own part, the conclusions are still going to be nonsense.

But how does this work in practice? When a reviewer gets a paper, how can they check if the references are reliable? There are so many potential obstacles in the way - many authors and papers will be unknown (so now you need to review those articles as well); articles can be taken from obscure, out-of-print books and journals that are nowhere to be found; paywalls that the reviewer would have to pay out of their own pocket; etc.

It seems to me so impractical as to be almost impossible. How is it done in practice?

But how does this work in practice? When a reviewer gets a paper, how can they check if the references are reliable? There are so many potential obstacles in the way - many authors and papers will be unknown (so now you need to review those articles as well); articles can be taken from obscure, out-of-print books and journals that are nowhere to be found; paywalls that the reviewer would have to pay out of their own pocket; etc.

It seems to me so impractical as to be almost impossible. How is it done in practice?

A properly chosen referee should be familiar with some of the cited references, but probably won't be familiar with all of them. If you're not familiar with any of the references, that's a red flag that you either should decline the referee request, or that the manuscript authors have little knowledge of the field. Leaving that case aside, most referees are probably more trusting of already published papers - at least ones published in reputable avenues. This is mostly reasonable, as they passed some kind of review already, hopefully from a knowledgeable and diligent referee. However, you shouldn't rely on that. Even when true, mistakes can be overlooked.

When you need to read another paper to understand a point/claim made in a manuscript you are reviewing there are some "smell tests" worth using (valuable for any paper you read, really):

• Does the manuscript represent the cited paper fairly and without misunderstandings?


• Do the results of the cited paper seem plausible?


• Are they supported by good arguments?


• How has the paper been cited (outside the paper you're reviewing)? After all, someone building on certain results is the person most likely to scrutinize them, so you may find additional support, counterevidence, or limitations in the literature. (Sometimes you can also find a better or more understandable exposition of the results.)

Depending on the specifics of the situation and the standards of the field, it may take more or less time to convince yourself that this paper is probably correct. However, fully "re-reviewing" the cited paper is (usually) not required or expected. If it is central for the manuscript you're reviewing then you might have to, but in that case the manuscript likely already retells the main argument, so it might be less additional work than you envision. What is much more common is to ask/suggest that the authors of the manuscript clarify, support, justify, qualify or hedge their statements as required. That is, you don't have to do the authors' work for them - they're supposed to convince you that their results are valid given the starting points. If they can't explain why the sources they build on are correct, well, they should at a minimum state their limits of their understanding clearly.

So ask if the approximations are justified under the current conditions, or if some external factors were accounted for in an experiment. Point out that technique X has some shortcomings that could affect the result, or if the validity of theory Y hasn't been established in the current regime. Doing so not only helps you figuring out if you trust the results, but also improves the paper for a less expert audience.

As for paywalls, you can ask the editor of the journal to provide you with a copy of papers you need for the review. If that doesn't work, or if the reference is an out-of-print book, inter-library loans can help you out. Ultimately, if the source is nowhere to be found (e.g. a unique original manuscript), or if the referee is unqualified to judge some aspect of the work, they should state so in the report.

The above is largely a description of how a referee could check if the references are reliable, and how they ideally should behave. In practice, however, corners will be cut in a number of places. Some people just aren't very meticulous, others don't want to spend too much time on working for free - especially if the manuscript is incremental, or maybe not all that interesting. Some people might see a famous name in the author list and conclude that the results are probably fine. Any reader should be aware that there are flaws in the reviewing process, and not trust papers blindly - especially when building on them.