Research is to see what everybody else has seen, and to think what nobody else has thought.
– Albert Szent-Gyorgyi (Hungarian Biochemist, 1937 Nobel Prize for Medicine, 1893-1986)
I got the idea to document the way I read papers from a discussion with Lode (he uses mostly the same approach as I do). I’m pretty certain this is obvious for many people, but I hope it may still be useful for others.
In this post I focus on processing lots of (related) papers in the least amount of time. I will not discuss how you find interesting papers, since this is fairly easy. Reading prestigious conference proceedings or journals in your field usually allows you to end up with a couple of seed papers that are representative for your specific area of interest. If you do research in ubiquitous computing, reading the proceedings of Ubicomp, Pervasive or Percom, and journals such as Personal and Ubiquitous Computing will get you along quite nicely. If you focus on the human-computer interaction aspects of ubiquitous computing, throw in the proceedings of CHI and UIST, together with the TOCHI and Human-Computer Interaction journals. The papers you find here will be stepping stones to related papers (and conferences) using the simple technique I will discuss in this post.
I estimate that of all papers I read, I only process 1 out of 5 in full detail. Most of the time when I find a paper that seems interesting, I start with skimming the paper briefly. I focus mostly on the abstract, the last part of the introduction (which should list the contributions), relevant pictures in the main matter, and the conclusion/discussion. I do several passes over the paper to validate my early understanding of what the authors did. I also go through the related work and references. This allows me to see if the authors refer to work that I don’t know but which could be interesting (which I mark for later processing). This will usually take little more than 5 minutes.
Now I have a pretty good idea whether the paper is interesting to me. If it’s not, I just go through the interesting references (if any) using the same process and stop there. If it is interesting, I go one step further, treating the paper as a seed paper that allows me to find other relevant work.
If the paper is not too recent, there will probably be (lots of) other articles that refer to this paper. Even if the paper is fairly new, it is still worth looking up papers that already refer to it. Looking up these articles allows me to discover relevant related work and to see how others critically describe this paper. A critical review could contradict of the results of the paper (e.g. through replication, such as Shumin Zhai’s work on target expansion), or might offer new insights (such as suggesting a new direction for research related to the original paper, as Yvonne Roger’s Ubicomp’06 paper did for Mark Weiser’s original Ubicomp paper). Both types of critical reviews are very useful for your own understanding of the paper.
After entering a title in Google Scholar, it presents you with a list of matching papers. Below each item, there is a link with the number of papers that cite it (indicated in orange in the figure below). In this case there are 118 citations:
After clicking on this link, Google Scholar shows all articles that cite the previous one, sorted by their relevance (in other words, the number of times these articles are themselves cited):
This approach to reading papers has one disadvantage: your reading list will quickly grow. You can end up with 30 more papers to read starting from just 1 interesting seed paper (which is what happened to me when I looked for papers that cited Intelligibility and Accountability: Human Considerations in Context-Aware Systems by Victoria Bellotti and Keith Edwards). In my opinion, this is actually an advantage as it allows you to quickly find many related (and hopefully interesting) papers. Furthermore, because you’re skimming papers, you never spend more than 5 minutes on papers that are not relevant or interesting.
After I have found a significant number of papers that I want to read, I of course will go through them in more detail. Papers that are very relevant for my work or give me the feeling they deserve a more thorough reading (e.g. because they provide many new insights), will be read from beginning to end. A few examples that I also discussed in my blog are: Range: Exploring Implicit Interaction through Electronic Whiteboard Design, Evaluating User Interface Systems Research and two chapters from Beyond the Desktop Metaphor: Designing Integrated Digital Work Environments.