Learning to do systematic reviews

Row of books

Within the past few years, two different faculty colleagues from one of the subject disciplines I support asked me to help them conduct a systematic review. Naively, I said yes, of course I’d help. This led to a steep learning curve as I had never done this type of work before. I realize that for many librarians, this is routine, but not for me, and I made many mistakes in the process of getting the hang of how to do it.

I drastically underestimated the hard work and the countless hours it takes to do one of these things! I learned several new terms (e.g., PRISMA diagram) and also stretched my subject knowledge on aspects of this field of study for which I have no academic background. But at the same time, this is what librarians do and do well as generalists. That point was reinforced for me throughout the process of conducting the reviews.

The current review I’m working on has been particularly challenging. Normally, the construction of an appropriate search strategy takes a great deal of time and experimentation. But in this case, my faculty colleague did most of the heavy lifting rather than me (I think usually, this works the other way around, i.e., the librarian’s expertise is most useful at this stage). Even so, the resulting search in various databases netted over 27,000 citations.

Why so many? Was the search strategy faulty? I don’t think so, and the reason for so many is that the study topic is very broad and multi-faceted with few standard limitations such as the articles must be published within x years. But having so many initial citations presented an enormous challenge as each of us was required to manually review every single one of them. For me, this stage was not only challenging due to the number of citations to review but also because doing so meant I had to have a really firm grasp on the topic. Again, remember that this area is not one for which I have any academic background.

It took me weeks and weeks of work. I thought it’d never end. Toward the final stretch, though, I began whipping through citation after citation lightning quick, only spending a few seconds at a time and concentrating primarily on keywords in the title and only rarely perusing the abstract. One small and unexpected triumph I stumbled onto was that the online, web-based tool I prefer to use for this work (Covidence) has a mobile-optimized view that is lightning fast, much faster than using the desktop view. That is what finally helped me to complete this initial review. Even so, just imagine clicking through citation after citation for many thousands of times. Very tedious work.

Out of over 27,000 citations, we ended up with about 700 for which one of us made a different decision (yes or no) in terms of fit for the study topic, and we are in the middle of reviewing those to come to a definitive conclusion. That isn’t bad, but it is, again, quite daunting for me to do given my lack of subject knowledge and expertise.

About 900 citations remain that we both agree belong in the study, and we are both repeating the process of separately reviewing each of those with the difference in this round being that the review is focused on the full text of the included citations, not just a review of the title and abstract. Before finalizing this round, we will once again have to collaboratively review and resolve any decision conflicts between us.

Before we can fully begin on this round of review though, we obviously need to obtain full text for each citation. This is perhaps the only part (aside from developing and refining a search strategy) where I as a librarian feel completely comfortable, like I am certain of the value of my work. I have spent most of the past week doing this. The work involves multiple strategies for identifying full text, including at times, laboriously and manually searching each citation, one by one.

Worth mentioning here is that I have faculty appointments at two different institutions, one full-time and one part-time. And the task of identifying and retrieving full text for many hundreds of citations has relied heavily on the access to content I gain through both. My search for full text always begins with the holdings of institution A; if I can’t succeed in obtaining full text there, I then turn to look for it in institution B. Fortunately, I succeeded in obtaining full text for the vast majority of citations between these two institutions.

Thank goodness for key aspects like DOIs, tools like Zotero and Covidence, and library search interfaces that by and large work amazingly well (with some caveats). It may seem to many to be common sense but a comprehensive search process like what I’ve needed to use throughout the whole project would have been utterly impossible in the days of print indices and print journals as well as previous search interfaces. I am definitely old enough to remember when DOIs first came on the scene in the late 1990s. It is truly revolutionary to be able to rely on this unique identifier because at this stage it is nearly ubiquitous in the sciences and social sciences. Because of that, it is by far the most efficient and reliable tool for identifying and obtaining the desired full text.

Although I have spent many hours and days working on identifying and downloading full text, I have also heavily relied on Zotero’s find available PDF feature. I am able to export citation details for citations lacking full text from Covidence, import that detail into Zotero, and then let Zotero do its batch search and download functionality. This succeeded for perhaps about 60% of the citations needing full text, and the remainder was found by manually copying/pasting DOIs into library search interfaces, navigating to the full text if it was available, downloading it, and uploading it into Zotero as an attachment to the citation information. After I finished that step, I then exported the data from Zotero (including the PDFs) and batch uploaded information into Covidence. This resulted in about 90-95% of the full text being found and loaded into the main interface (Covidence) used for the review process.

Of course, not all citations have DOIs, and also of course, not all of them have full text readily available, so in some cases (about 30-40 out of 900), I submitted interlibrary loan requests. But overall, I am pretty happy with the result and have learned quite a few tips and tricks along the way to ensuring a good job in the review and also to utilize the next time I’m ever asked to help with a systematic review.

Aside from the valuable learning experience (which to me is the main benefit), what else do I get out of this? Well, I will get co-author status on whatever gets published, which I appreciate. Having now been involved in multiple systematic reviews, I feel a lot better prepared and a lot less naive about what it takes to do them. There are many other, less well-defined but no less important lessons learned as well, such as the value of setting clear expectations and timelines for stage completion when collaborating with someone else on such a major project.

To my knowledge, this level of librarian involvement has never been done before at my primary institution, and it makes me wonder whether this kind of involvement should be openly advertised there. I think it’s an example of embedding oneself as a library faculty member into the vital research work of other faculty colleagues and thereby contributing valuable expertise to the educational mission of the institution as a whole.

1 thought on “Learning to do systematic reviews”

  1. Pingback: Chinese for breakfast – The OW Factor

Leave a Comment

Your email address will not be published. Required fields are marked *

Follow by Email
Copy link
error: Content is protected !!
Scroll to Top