When conducting a literature review how many sources should you have during an initial research?

The first question I posed in the title of my blog post is one that all of my students (undergraduate and graduate) and most of my research assistants ask me: how do I know when I’ve read enough for a literature review? The answer is never clear cut, unfortunately. I am someone who loves reading, and who needs to read broadly because his own work is interdisciplinary. I’m also quite systematic in how I read, and I prefer to err on the side of having TOO MANY sources rather than be accused of not knowing the field.

I also realize not everyone has the time to be on top of the literature, particularly with teaching, service, family and caring and research obligations. I am going to answer the top six questions I get asked on breadth and depth of literature reviews, and in doing so, I am going to suggest a few shortcuts that may help narrowing a literature review search and finding the “sweet spot” where you’ve read enough that you feel confident enough to start writing your paper, chapter or thesis.

1. How many sources should I read for my literature review?

This is an absurd question that is prompted by arbitrarily setting a random number of sources as “enough”. If you read the right five sources, you’ve probably covered a full field. But if you read 40 sources that all tend to pull in different directions, you’ll still be unable to cover all the sources.

Here’s my totally non-scientific take for coursework-related materials: a final research paper should at least use 13 additional sources to those in the syllabus (one additional paper per week) for an undergraduate class, and an in-depth literature review for a graduate course should be in the realm of 26 (2 additional papers per week) to 39 (3 additional papers per week for PhD students). If somebody writes a final paper for my courses that only use the readings we did during the semester, it shows they didn’t go any further and I’ll probably penalize them.

2. Where do I get sources for my literature review if I am starting up a new topic? Well, here are a couple of strategies:

There are 2 clear sources of citations for literature reviews: (1) papers that are systematic reviews or meta-reviews pic.twitter.com/sYCUGv3kg0

— Dr Raul Pacheco-Vega (@raulpacheco) June 18, 2017

• Read literature summaries and reviews published in journals.
  There’s plenty of journals now that provide reviews of the literature. Three I’m well aware of are WIRES (Wiley Interdisciplinary Reviews) and Geography Compass, as well as Progress in Human Geography. When reading Progress in Human Geography, you probably want to look for the “Progress Reports”.
• Search for “a review of the literature” or “a meta-review” type of articles (either Google Scholar or other online databases). These articles will usually provide a pretty extensive range of sources. Given their goal and scope, they’re also probably comprehensive enough. Meta-reviews tend to be more synthetic and provide a research agenda and specific research questions that need to be looked at in future research.
• Ask specialists (or look for their work) for key citations from where you can do citation tracing. For example, if I were to do a search on water ethics, I would ask Christiana Peppard, Jeremy Schmidt, Lucy Rodina for suggestions. Or I would look for articles citing them. If I were to do something on climate politics, I’d look for Kathryn Harrison, Sarah Burch, Max Boykoff, Mat Patterson.

(2) papers that are "a literature review of" (this should be obvious, but a lot of people don't seem to know this!) pic.twitter.com/T1xaK2yvv9

— Dr Raul Pacheco-Vega (@raulpacheco) June 18, 2017

Also the scoping review, a derivative of systematic reviews that is meant to be broader and more exploratory:https://t.co/WQjEpnuZR3

— AJ Wray (@WrayAJ) June 18, 2017

3. “When should I stop reading and start writing?”

My answer to that question is: you should be reading AND writing. Apparently, a lot of people feel like they need to Read All The Things before they can write a literature review. That’s why I always suggest that when you process a reading (a PDF or a printed source), you should generate at the very minimum a row entry in your Excel conceptual synthesis, and a synthetic note (or a rhetorical precis). Obviously, you gain a lot more if you write a full-fledged memo, but you may want to wait to write the memo up until you’ve read a few sources. But you should ALWAYS be writing as you read. You may not assemble the full literature review, but at least you can start with an annotated bibliography.

4. How do I know when to stop reading/researching/seeking more sources?

This is again a very tough question. Having mapped a very broad survey of the literature on informal waste picking, I can assure you that I felt I could not stop even after reading 50 articles. There is just simply too much published. But one of the reasons why I encourage my students to stop when they reach conceptual saturation (e.g. when they start seeing the same themes repeated over and over again) is because I don’t think you gain too much, marginally, from reading yet another paper on the same topic but using a different case study.

For example, I recently wrote a series of memorandums on the urban commons. I had basically mapped the entire body of works on urban commons using the first 10 citations I found on Google Scholar. However, I wanted to see how much more I could go in depth on the topic. What I found was that there were many case studies, but all using the same conceptual framework. So that’s when I stopped. When I saw that basically every other paper was a variation of the same central 10 ones, but using different case studies. I added those sources to my bibliography, but I didn’t need to incorporate them to my literature review.

Another way to respond to this question is: read enough to answer your questions properly.

The two biggest questions that probably would encompass the previous ones are related to breadth and depth.

5. How far reaching should your literature review be?

Scoping a literature review, as Dr. Pat Thomson shows here, is not an easy task. It requires us to search through many months or even years of published literature. I always do Google Scholar searches at least 7 years into the past (e.g. 2010 articles would totally be welcome, as would be books published in 2010) because of the very long lag-time that exists between submission, acceptance and publication. My citation tracing process also looks at the last 7 years of scholarship of key authors.

Then you have the other associated question – what about the “seminal” (I prefer the word fundamental) articles or books?

For me, this is the most challenging component. When I know a field very well (for example, agenda-setting theory in public policy), I can easily decide which authors I will be seeking (Stuart Soroka, Michael Howlett, Baumgartner and Jones, Kingdon). If I am doing policy design, I’ll go with Helen Ingram, Ann Schneider, etc. And then based on doing a citation tracing exercise, I will go to those younger scholars who are citing these key authors. But again, this requires you to know the field already.

This is where a supervisor, a coauthor, a colleague or a trusted scholar on Twitter may be helpful with narrowing the search scope. You can ask “who are the key authors I should be reading on Topic A” or “which are the key citations I should be looking at to get a grasp of Field B“. And then use those authors to create a map of the literature.

And the last question, which just about everyone asks me:

6. Do I need to do an in-depth reading of All The Things?

This is completely a question that has arbitrary answers and a broad range of parameters to work around. It also depends on what type of literature review you are writing. If you are, for example, preparing your doctoral comprehensive examinations, you DO want to read EVERYTHING and do so IN DEPTH. You need to demonstrate that you know your field of study, broadly and deeply.

However, if you are writing a literature review, for example, of agenda-setting theory and its applications to health policy, you may want to read in depth 5-10 articles on health policy, 5-7 articles on agenda-setting theory, and then start writing from there. Again, in-depth reading is correlated with the extent and degree to which you need to demonstrate that you know a field.

My method, as most people may have noticed, is usually as follows:

• I read 5-10 citations that I find key. This reading is usually in-depth.
• I create the set of questions I want to answer. I choose 3-5 citations around each question. All the reading associated with these questions I do using the AIC method, or skimming and scribbling unless I find key ones that need to be read in depth.
• I write a memorandum for each one of the questions I’m trying to answer. In this memorandum, I assemble a mini-literature review that answers the question.
• I fuse all the memorandums into a larger document where I have mapped out how each question (and answer) relates to the overall topic.
• I read my entire literature review and restart with citation tracing until I reach concept saturation.

Hopefully this post will help those who are struggling with literature reviews, as the summer approaches! You may also want to revisit my Literature Review posts.