Dr. Plopper’s Guide To Critical Reading of Primary Literature

    Overall tips: 

    1. Just because a scientific study has been published, this does not mean it is perfect.

     And the imperfections are not so minute that a non-expert cannot find them. Any student of science can, and indeed should, read science critically. This means you can read a paper by a Nobel laureate and find fault with it. It is in fact expected that, as a practicing scientist, you can criticize any science. The level of your critical analysis will differ with your experience level, but you should be able to criticize science even as a relatively inexperienced undergraduate.

    2. Manuscripts are not read the same way newspapers or novels are read. 

    One does not simply read every word, from the front to the back, in a single pass. The reason for this is fairly obvious: manuscripts contain a high density of information, and not all of this information is of equal value to the reader. In fact, quite a lot of effort is made by the authors and editors to break the information into discrete units that can be read almost independently. Inexperienced readers of primary literature should pay close attention to how manuscripts are organized, and seek specific types of information from each section. You should read the manuscript many times, but each time read only a portion of it, and do so because you are looking for specific information.

    3. Have a good textbook handy to look up background information and definitions.

    4. Reading a manuscript takes a lot of time, especially if you are unused to it. To do all of the procedures outlined here will take you at least three hours.

Step-by-step method:

1. Read the Title.

    1a. Understand each word. 

    The title provides the most important information in the manuscript. What, really, does this mean? This means the authors have combed through their data and have decided that if you remember nothing else about the paper, this is the one thing you should remember. You may be surprised to learn that most authors spend quite a lot of effort on the title, because they are forced to distil many months (or even years) of work into a single sentence. Imagine crunching an entire year of college courses into a single sentence! As a result, they shuffle and reshuffle their data until the most important point comes out. They also shuffle and reshuffle the words in the title.

    The result is that each word in a title usually carries a lot of weight. Because a lot of effort is put into creating a title, it should be rich in information content. To the reader, this means you should be able to get a lot of ideas out of the title. I usually spend at least five (real) minutes thinking about the title before I read anything else; most beginning students spend about five seconds on a title, figuring, “If it’s important, I’ll learn about it later in the paper.” Bad idea! 

    1b. Create a list of experiments you expect to see in the paper. 

    In practical terms, this means you should consider the implications of each word in a title. You should be able to derive a list of what sort of experiments you think the authors performed, what the data might have looked like, and how the authors reached their conclusions, before you even read the paper. Write this list down. It may be completely wrong, that’s OK. But try to think ahead about what will be coming. If eventually you discover that your list looks nothing like what the authors did, that is very instructional: did you miss a critical point, or (this happens) did they? Did they mislead you with their title? Many papers have titles that border on being advertisements: they paint the data in the most positive light, but may not be entirely accurate. Your job is to see through that!

    Example: Although not usually written as a declarative sentence, a title can be translated as such.Here is an example from a cell biology paper: “A Splice-Isoform of Vesicle-associated Membrane Protein-1 (VAMP-1) Contains a Mitochondrial Targeting Signal.” This sentence can be deconstructed easily: “X (verb) Y.” The verb “contains” is not overly scientific, but is has implications you should consider. The “X” in this case is rich in terms: “A Splice-Isoform of Vesicle-associated Membrane Protein-1…” When you read a title, make sure you understand what every single word means, or at least ask yourself how you could find out. In this case, if you don’t know what “splice isoform” means, go find out before you delve any deeper into the paper, because it must be a critical component of the whole manuscript (or it wouldn’t have made it into the title, right?). Usually a good place to find definitions or explanations for especially obscure words is the Introduction section. Likewise the words “vesicle-associated,” “membrane,” and “protein.” These are all very common terms in cell biology, and you should have no trouble understanding them. After you get the meaning of each word down, string them together and determine what they mean collectively. In this case, it means, “A structurally altered form of a protein that binds to vesicles…” and that should conjure an p_w_picpath in your mind: what do vesicle associated proteins do? Don’t know? Go find out. What would make them interesting, especially if they were structurally altered? Any predictions? I’ll wager that you won’t predict that they have anything to do with mitochondria. Hmmm…. 

    Do the same deconstruction to the “Y” term: what is a “mitochondrial targeting signal”? Where is it found? What does it do? Look it up if you don’t know.

    Now combine the two ideas with the verb. The two apparently disparate ideas are actually related, according to this paper. How would you show that? Spend some time thinking hard about that. Any idea how someone might do it? If so, write it down. If not, put down the questions you’d need answered in order to come up with some experiments: “How do you show something is a splice variant?” for example.

    So, before proceeding, you should have a list: It either has a series of experiments you think might be done, or it has a set of questions that you need answers to in order to generate that list of experiments. Maybe it contains a few statements of each type. A big difference between an inexperienced and an experienced researcher is how easily they can draw up this list of experiments. You should be striving to do this as quickly and thoroughly as possible. And remember: having a list that varies from the list of actual experiments done by the authors is perfectly OK.

    If you need to look in the paper to find the answers to your list of questions, do that next. Look at the data/figures, and check the Materials and Methods for the “how to” portion of addressing your question. Read these sections only for that information! Do not get bogged down in minute details: know what you are looking for, and find it. Skip/ignore anything that doesn’t answer your questions. Then, generate your list of experiments. 

    Do not read any more of the paper until you get this list together. I usually just abbreviate it in the margin of the first page. At this point all you have read is the title (plus maybe you’ve skimmed the rest), yet you have a good handle on what the paper should be about, and even a set of expectations to bring with you as you delve into the meat of the paper. This will help considerably when trying to decide what else in the paper really matters: Sometimes the pH of a buffer is boring, sometimes it is crucial.

    Given the list, now you can test your ideas by reading the rest of the paper to find out what experiments they actually did. How and where do you find that? In the Results section.

    But before we go there, let’s get a preview by reading a summary of the results in the abstract.

2. Read the Abstract. 

    The abstract is a summary of the whole paper. It usually contains a condensed introduction (1-2 sentences) explaining the rationale for the work. Then a brief summary of the data, with slight reference to the experimental methods at best. Finally, a concluding sentence, perhaps with mention of the implications of the work.

    Like the title, the abstract is a dense piece of writing. You can get lots of info here, but this time you are looking only for hints as to how and why the experiments were performed. You can then tell how close your list is to theirs. Don’t rewrite your list! Just get it in your head that your experiments and theirs are either similar or not. You will also get a sense for the order in which experiments are going to be presented. Ignore everything else.

3. Read the Results section. 

    3a. Read the text of the results without critiquing the data.

    Go slowly and carefully here. Since this is your first pass through the data, do not try to take it all in at once. Again, look for specific information and ignore the rest. This time, you are looking for a good understanding of what the experiments were: what did they do? The goal here is to get into the minds of the authors: Let them tell you why they did the experiments, and what they think happened as a result. Assume everything they tell you is the absolute truth. This is where you compare your list with theirs. Write down what differences there are, and note why you think the lists are different: are the authors leaving things out you’d like to see, or are they on a different track than you? The key here is to fully understand their train of thought. If you can’t figure it out, write that down, too, specifying exactly where you fell off the train. Then move on. From here on, focus on the material you understand from the Results, and ignore what you don’t understand.

    3b. Look at the figures and carefully read the figure legends. 

    This is where you get into the nitty-gritty of the paper. Look at each figure, having already understood what the authors claim is happening. An important thing to keep in mind here is that even if you don’t recognize the data, you can make inferences based on how the data are presented: If a figure includes a picture of some sort (e.g., a micrograph), this tells you where something is. It doesn’t really tell you how much of it is there, or when it got there, or what it is doing there- it just tells you what it looks like. So, pictures = spatial information. Likewise, a graph shows how one thing varies when compared against another: look at the axes on the graph, and you might see “cell #” on the Y axis and “time” on the X axis-, so it simply tells you the cell # at an indicated time; therefore graphs tell you how much per condition. Finally, pictures of gels (e.g., western blots, RT-PCR reactions, immunoprecipitations, etc.) are telling you approximately how much of something is contained in a sample- often these pictures are accompanied by a quantitative measure to support the picture. Don’t let a complicated (busy) figure scare you- in the end, most figures in cell biology-style papers boil down to addressing these three types of questions.

    BE CRITICAL: Assume they are trying to pull a fast one on you. Make sure that when they say something, the data actually show it. You may be surprised to learn that many papers fail in this respect, sometimes rather badly. Scientists often give themselves the benefit of the doubt, and your job here is to see whether you’d give them the same breaks. It is perfectly OK to be a complete stickler for facts: what are the controls for that experiment? How do you know that this result isn’t due to something else?  [Note: This is the one and only time you ever need to read the Materials and Methods section, and you only read it when you need a specific fact about a particular item. Do not dwell in the M&M section.] Challenge yourself to find alternate explanations for the results. An important thing to realize is that you can do this for every single paper ever published. All authors omit things, because they can’t publish every experiment they ever did: what do you think they chose to omit? Why did they omit it?

    Make sure you understand what they did before moving on. I sometimes have a list, next to my “expected experiments” list.

4. Read the introduction.

    Now, you are looking for a clear justification for why the authors chose to do their experiments. What is the main question that they claim to be answering? Amazingly, some papers never mention this, and the authors assume that we’ll just “get it.” Bad idea! Are you convinced that their question is relevant/interesting?

5. Compare the authors’ main question, their data, and their conclusions.

    Simply put, did they answer their question? Did they do the right experiments to address their question? If your list of experiments differs from theirs, is their line of experimentation better? Sometimes it isn’t! If you had to answer this research question, knowing all that you know now, how would you do it? Maybe borrow some of their experiments? Or do exactly what they did? Great papers are those that do it exactly right, and they are rare.

6. Read the discussion.

    Having reached your own conclusions about the quality of the work, let them have the last say by reading their defense of their work. The discussion should tell you why their work is important, and how it advances the field. By the time you read this, you will have a clear understanding of what you think the weaknesses of the paper are: did they anticipate your questions, and actually answer them here? If so, consider their response. If not, you get the satisfaction of knowing that you been especially thorough in your reading and critical evaluation of the manuscript.

7. Project into the future.

    Having now gotten into the minds of the authors, imagine yourself working along side them. If you joined that lab tomorrow, what would you suggest be done next to advance this study? Are there gaping holes to be filled? Or can you move on to the next issue? To be a critical scientist, you must be able to answer these questions. You may find yourself in a lab one day, or maybe in an interview, where you will be asked, “What would you do next?” and reading manuscripts is where you begin thinking along these lines. Do not neglect this responsibility.

8. Wrap up.

    Give the whole paper a quick read from front-to-back, and see how the paper flows. Look for style, not content. What phrases do they use to introduce their ideas? How are the figures labeled? Is this a well-constructed paper? Is there anything in the paper you'd like to emulate?