Journal of Research Practice

Volume 12, Issue 1, Article R1, 2016

Review:
Realist Review: Current Practice and Future Prospects

Rigmor C. Berg
Norwegian Institute of Public Health, Oslo, NORWAY;
Faculty of Health Sciences, University of Tromso, Tromso, NORWAY
rigmor.berg@fhi.no

Julie Nanavati
John Hopkins School of Medicine, Welch Medical Library, Baltimore, UNITED STATES
jenanavati@gmail.com

Abstract

Realist review has emerged as a specific literature review approach that is concerned with explaining the outcomes of complex intervention programs. We undertook a systematic scoping review to examine the current practice of realist reviews. A systematic scoping review is a process of mapping the existing evidence base on a particular topic. We identified a growing body of literature using the realist review approach. We selected 54 reviews for our study. These reviews covered a range of topics, including health care, education, management, and public safety. We found that the initial process of exploratory scoping of the literature was described in only 58 per cent of the reviews. The approaches regarding appraisal, analysis, and synthesis of the selected studies were poorly described in most reviews. Overall, there was little uniformity and transparency regarding many methodological issues. Specific methodological guidance may need to be developed if realist reviews are to have a more uniform and transparent approach.

Index Terms: systematic review; systematic scoping review; mapping review; social intervention; realist review; realist synthesis; RAMESES publication standards

Suggested Citation: Berg, R. C., & Nanavati, J. (2016). Realist review: Current practice and future prospects. Journal of Research Practice, 12(1), Article R1. Retrieved from http://jrp.icaap.org/index.php/jrp/article/view/538/449

1. Introduction to Realist Review

Increasingly, practitioners, policy-makers, and others call for reviews that accommodate complexity and context in drawing lessons from complex intervention programs (Bravata et al., 2005). In the last 10 years, realist review—also called realist synthesis (the terms are synonymous; Wong, Greenhalgh, Westhorp, Buckingham, & Pawson, 2013)—has emerged as a specific approach to review literature in policy-relevant areas. The approach is explicitly concerned with explaining the interplay between context, mechanisms, and outcomes of intervention programs (Pawson, 2006; Pawson, Greenhalgh, Harvey, & Walshe, 2004; Pawson, Greenhalgh, Harvey, & Walshe, 2005; Wong, Greenhalgh, Westhorp, & Pawson, 2012). In one of the early publications describing realist review, Pawson and colleagues characterize it in these words:

Realist synthesis is an approach to reviewing research evidence on complex social interventions, which provides an explanatory analysis of how and why they work (or don’t work) in particular contexts or settings. It complements more established approaches to systematic review, which have been developed and used mainly for simpler interventions like clinical treatments or therapies. (Pawson et al., 2004, p. iv)

Realist review is a theory-driven and interpretive type of literature review. This is in contrast to a systematic review, which “attempts to collate all empirical evidence that fits pre-specified eligibility criteria in order to answer a specific research question” (Higgins & Green, 2011). Realist review aims to answer “what works for whom in what circumstances, in what respects and how?” with respect to social intervention programs (Pawson et al., 2004, p. 3). It is based on realism as a philosophy of science, which involves identifying underlying causal mechanisms and exploring how they work under different conditions (Pawson & Tilley, 1997). The approach generates hypotheses and leads to recommendations to influence the design of new intervention programs (Pawson, 2006; Pawson et al., 2005). In fact, the aim of a realist review is “to articulate underlying programme theories and then to interrogate the existing evidence to find out whether and where these theories are pertinent and productive” (Pawson, 2006, p. 74). Thus, it does not look for a summative judgment about complex social intervention programs, only a plausible explanation or theory, which will be refined, or possibly rejected, as new evidence emerges.

Realist reviews follow most of the same standard steps as systematic reviews (Pawson et al., 2004):

1. Identify the question
2. Clarify the purpose(s) of the review
3. Find and articulate the program theories
4. Search for the evidence
5. Appraise the evidence
6. Extract the results
7. Synthesize findings
8. Draw conclusions and make recommendations

Pawson and colleagues (2004) stipulate that the process should be equally rigorous and transparent (i.e., auditable), where every judgment is written down. However, compared to a traditional systematic review, a realist review is more iterative and possibly more challenging from a methodological standpoint (Pawson et al., 2004). With respect to realist reviews, “experts still differ on detailed conceptual methodological issues” (Wong et al., 2013, p. 3).

In order to make the process of realist reviews rigorous and transparent, a set of 19 publication standards has been developed by a team of scholars (i.e., the RAMESES [Realist and Meta-narrative Evidence Syntheses: Evolving Standards] publication standards, Wong et al., 2013). The authors of the standards anticipate that these will prompt “further developments in theory and methodology” of realist review (Wong et al., 2013, p. 3). It is modeled after the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) statement (Moher, Liberati, Tetzlaff, Altman, & The PRISMA Group, 2009). The RAMESES publication standards indicate what should be reported in the write-up of a realist review.

Although the number of publications reporting a realist review appears to be increasing rapidly, there is a lack of clarity about its methods and applications. Therefore, we conducted a review of the published examples of realist review. Our approach and findings are presented below.

2. Methods Used in Our Study

2.1. Method: Systematic Scoping Review

We conducted a systematic scoping review of published and unpublished examples of realist review. Systematic scoping reviews (also known as systematic mapping reviews) have been described as a process of mapping and describing the existing literature or evidence base on a particular topic (Arksey & O’Malley, 2005). Such reviews take stock of the research available in a field of study, such that one can decide how to develop the field further (Arksey & O’Malley, 2005; Armstrong, Hall, Doyle, & Waters, 2011; Bates, Clapton, & Coren, 2007; Clapton, Rutter, D., & Sharif, 2009; Oakley, Gough, Oliver, & James, 2005; Shepherd et al., 2006).

We adhered to the framework proposed by Arksey and O’Malley (2005) for conducting a systematic scoping review. We also followed Levac and colleagues’ recent recommendations on clarifying and enhancing each stage of the review (Levac, Colquhoun, & O’Brien, 2010). Therefore, the steps were:

1. Identify the research question
2. Identify relevant studies
3. Select studies
4. Chart the data
5. Collate, summarize, and report the results

With respect to the final optional step recommended by Arksey and O’Malley (i.e., Consultation), we embedded it throughout the research process. During the review process, we communicated with experts in the field and invited comments on our preliminary results.

2.2. Inclusion Criteria

To be included in our systematic scoping review, a review had to meet all the following criteria:

(a) Applies a method that is consistent with the underlying principles of a realist review, specifically focusing on how and why an intervention worked, or did not work, in particular contexts (Pawson, 2002, 2006; Pawson et al., 2004, 2005; Wong et al., 2012)

(b) Refers to at least one of the five main publications describing realist review (i.e., those listed above, under item [a])

(c) Has been published between 2004 (the date of publication of Pawson and colleagues’ key text on realist review [Pawson et al., 2004]) and the date of our search

(d) Contains the description and the results of a completed review

We included reviews with a methodological focus, for example, the presentation of the realist review approach, if the authors reported on the findings of a particular review. However, we excluded papers commenting on methodological issues without including details of the results of the review. Books, book chapters, unpublished reports, brief reports, and preliminary reports were all considered for inclusion on the same basis as journal articles. Abstracts and protocols (a protocol is a document that describes the plan for conducting a study, including purpose and methods) of planned or ongoing reviews that appeared to meet the inclusion criteria were included on a separate list, but data from these were not extracted.

2.3. Search to Identify Relevant Reviews

We conducted a comprehensive and systematic search up to January 2015 in 11 international databases: CINAHL, Cochrane Library (CDSR, DARE), EMBASE, ERIC, MEDLINE In-Process & Other Non-Indexed Citations, ProQuest Dissertations & Theses, PsycINFO, Social Services Abstract, Sociological Abstracts, and Web of Science. A Cited Reference Search was also conducted with the Social Science Citation Index through the Web of Science database on key realist review methodological texts (Pawson, 2002, 2006; Pawson et al., 2004, 2005; Wong et al., 2012).

The search strategy was prepared by an information specialist (coauthor Julie Nanavati) in cooperation with the first reviewer (coauthor Rigmor C. Berg). Search terms were identified from known realist reviews. The strategy was piloted to ensure that it captured known realist reviews in the published literature. The final strategy used for MEDLINE was:

(realist systematic review* or realist review* or realist synthes*) AND (“2004/01/01”[PDAT]: “2014/12/31”[PDAT])

To ensure that the search was as comprehensive as possible, we applied neither methodology search filters nor language delimiters. We supplemented the electronic database searches with searches in reference lists of relevant reviews, contacted realist review experts, and requested suggestions for reviews on the RAMESES listserv.

2.4. Selection of Relevant Reviews

We (the two coauthors), working independently, read all the titles and abstracts resulting from the search process. We compared our assessments and obtained full text copies of the reviews deemed relevant. Acting independently and jointly, we then assessed whether the reviews read in full text met all inclusion criteria. Where disagreements occurred, we re-examined the review and resolved the question by discussion. For both of these stages, we used pre-designed inclusion forms. When necessary, we contacted the original authors to obtain further information.

2.5. Extraction of Data and Analysis

Coauthor Rigmor C. Berg extracted data from the reviews included in the study, using a pre-tested data extraction form (with 53 variables in total). Coauthor Julie Nanavati subsequently verified, confirmed, or disconfirmed the data. Disagreements were resolved by re-examination of the review and subsequent discussion. We extracted data on publication details, review topic, review methods (identification, selection, appraisal, extraction, analysis), and the number and types of documents included.

Although there is no generally agreed upon method for assessing the quality of realist reviews, a reporting guideline has been developed by Wong and colleagues, with 19 suggested items of information (“List of items to be included when reporting a realist synthesis,” Wong et al., 2013, Table 1). We extracted data on these 19 items. This was done on the recommendation of experts in the field and also because it aligns with the expected methodological practice in realist review (Greenhalgh, Wong, Westhorp, & Pawson, 2011).

We recorded the data under these items as either Yes or No, but also used the third option of Vaguely or Unclear for some that could not be dichotomized easily. We used Vaguely for issues that were addressed to some extent, but not fully. We used Unclear for issues that could not be answered based on the information available in the publication. The charting of review data allowed us to proceed to collating the data. The data were compiled in a single spreadsheet. We grouped reviews according to their chief characteristics and carried out descriptive analyses by using frequencies and cross-tabulations.

3. Results of the Systematic Scoping Review

We considered 159 reviews in full text (Figure 1). We included 71 publications representing 54 realist reviews. These reviews were published between 2004 and 2015 (listed in Appendix A). The 14 abstracts and 24 protocols of what appears to be realist reviews were not included in our review; nonetheless, we have listed those under Appendices B and C, for the sake of transparency and future reference.

Figure 1. Flow diagram of the search and selection process.

3.1. Characteristics of the Realist Reviews Included in the Study

Most of the 54 reviews were published in peer-reviewed journals (n=42). There were nine reports and three dissertations/theses. The reviews were published in 40 different journals, with most journals publishing only one realist review each (only two journals had two realist reviews each). Similarly, only two first-authors made repeated (two) contributions to the dataset, while the rest contributed only once. In contrast, often the country of origin of the first-author was the United Kingdom (n=23), although first-authors were also from Canada (n=11), Netherlands (n=6), Australia (n=5), Belgium (n=2), Sweden (n=2), the United States (n=2), Hong Kong (n=1), New Zealand (n=1), and Norway (n=1).

The reviews covered a range of topics: health care system and technology, mental and somatic (ill) health, management, education, safety (e.g. joint health and safety committees), health research, and advance directives (i.e., a living will that gives durable power of attorney to a surrogate decision-maker). Four of the realist reviews were re-reviews of completed systematic reviews, which used all or a subset of the studies originally included in the systematic review. Three of the realist reviews were done in parallel with a systematic review. Five reviews described themselves as a systematic review in their title (although three of these did not follow the basic procedures required for systematic reviews).

3.2. Reporting in Accordance With RAMESES Publication Standards

Only seven of the reviews included in our study, all published in 2014 or 2015, cited and stated that they used the RAMESES publication standards. A good example is one that addressed intercultural doctor-patient communication (Paternotte, Van Dulmen, Van Der Lee, Scherpbier, & Scheele, 2015). Yet, no review reported on all of the 19 items of the RAMESES Publication Standards. For example, two of the seven reviews stating that they followed the standards did not identify the review as a realist review in the title (RAMESES Item 1) and most did not report on the items regarding strengths, limitations, and future research directions (Item 16) or funding (Item 19). Table 1 shows the descriptive results for each reporting item to be included when reporting a realist review

Table 1. Reporting of the RAMESES Publication Standards Items in the 54 Realist Reviews

Notes.
[a] Among the reviews that were presented in more than one publication, two reviews did not meet this requirement in any of the publications associated with those reviews
[b] One could not be assessed with regard to this item because there was no abstract nor executive summary
[c] A lack of reference to changes may not indicate that no changes were made; therefore, we only noted when such changes were explicitly stated
[d] We did not assess rationale/justification
[e] Four reviews could not be assessed with regard to this item because these were re-reviews of completed systematic reviews and used all or a subset of the studies included in the completed systematic review
[f] We did not assess reasons for exclusion at each stage or source of origin
[g] Two of the four reviews that re-reviewed a completed systematic review did not include a diagram
[h] Ten reviews could not be assessed with regard to this item because the reviews were not funded

The first four items of the RAMESES Publication Standards deal with the information expected for the title, abstract, and introduction of a realist review (Items 1-4). Most of the reviews reported in accordance with these standards. For example, three quarters (74 per cent) of the 54 reviews identified the document as a realist review or realist synthesis in the title.

The Method section of the RAMESES Publication Standards includes seven items (Items 5-11). Concerning the literature search processes, while less than half (42 per cent) of the reviews described the initial process of exploratory scoping of the literature, almost all (94 per cent) gave information about how the literature search was done. On the other hand, less than half of the reviews described the selection and appraisal of documents, the data extracted from the documents, and the analysis and synthesis processes in detail. A typical description of the analysis and synthesis processes that we marked as inadequate is quoted below:

Data synthesis was undertaken by [the main author], and synthesis results were regularly shared and discussed with the research team to ensure validity and consistency. The research team discussed all the extracted data to find overarching categories in the context-mechanism-outcome model.

With regard to the Results section of the RAMESES Publication Standards (Items 12-14), about half of the reviews provided information about the documents included in the review, either fully (39 per cent) or incompletely (11 per cent). The incompleteness arose from the absence of information concerning study design, setting, intervention, and/or participants.

The final section of the RAMESES Publication Standards is labeled Discussion and includes five items (Items 15-19). While only 39 per cent discussed the strengths of the review, 67 per cent discussed the limitations. The limitations included, in particular: (a) the possibility of selection and publication bias, (b) the studies included in the review rarely detailed the mechanisms by which an intervention was expected to work, and (c) the diversity of contexts covered in the studies hampered generalization.

These results indicate a relatively low degree of similarity among those reviews meeting our broad criteria for realist review, and in particular great variability in reporting of methods in realist reviews.

3.3. Reporting of Review Methods

In accordance with our aim, we examined the description of methods and processes in the reviews, with respect to quality assurance and transparency. The results of this examination are summarized in Table 2. Information that could not be dichotomized is not included in the table, but is described in the following text.

As seen in Table 2 (Row 2), 40 per cent of the reviews clearly specified document characteristics used as criteria for eligibility while 46 per cent did so vaguely. An illustrative example of a vague specification of eligibility criteria is quoted below:

Studies were included if they were relevant to the synthesis question (i.e., examined some part of the relationship between [intervention] and [outcome]) and contributed to the refinement of a program theory identified during the first stage.

As an additional example, many reviews included vague inclusion questions such as:

Does the article/document focus on [topic]? . . . Does the article/document report on qualitative or quantitative data? Does the article/document make conclusion regarding the effectiveness of [intervention]? Is it published in a peer-reviewed scientific journal or been the subject of peer-review by an external body?

Table 2. Reporting of Quality Assurance and Transparency Measures in the 54 Realist Reviews

Notes.
[a] The studies were re-reviews of completed systematic reviews and used all or a subset of the studies included in the systematic review
[b] Typically, eligibility criteria in evidence-based practice is specified by the acronym, PICO, which stands for Population, Intervention, Comparison, and Outcome. This process almost always begins with a patient question or problem. A well-built question should include all the four PICO elements (Sackett, Richardson, Rosenberg, & Haynes, 1997).

Additionally, we found that while most reviews provided a general set of inclusion criteria, some reviews specified inclusion criteria for documents that addressed context, mechanisms, or outcomes, but not all three. (Recall that a characteristic of realist review is the focus on the context-mechanisms-outcomes [CMO] configurations that underlie interventions.) We found that 14 reviews specified eligibility for studies that addressed the outcome of an intervention, one review highlighted criteria for mechanisms (views) papers, and one did so for context papers. Inclusion criteria for studies that addressed outcomes were: studies had to be controlled studies (n=5), evaluation studies (n=3), any type of quantitative studies, quantitative and qualitative studies that evaluated any of the outcomes, or studies that report on one or more of the outcomes of interest. One review specified that studies that addressed outcome had to be randomized controlled trials.

Related to the RAMESES Publication Standard Item 8 (pertaining to the searching process), we extracted data about the methods used for searching for literature (i.e., the methods used to identify the documents to be considered for inclusion in the review). Three reviews made no reference to how the documents were identified. Five reviews used documents from a related systematic review. The remaining 46 reviews searched in electronic databases, and some of these including other sources as well. Seven reviews (15 per cent of the 46 reviews that actually searched in databases) searched only in electronic databases, while 18 (39 per cent) used one other search strategy in addition to electronic databases, typically reference lists and expert consultations. Nine reviews (20 per cent) explained how they searched for grey literature, such as performing searches in Google and examining government reports. Some authors referred to an interface (e.g., OVID, EBSCO), rather than a particular database, which made it difficult to evaluate in which database they had searched. Of the 46 reviews that searched for eligible documents in electronic databases, 8 provided the full search, 7 provided the search used in one database (typically MEDLINE), 24 stated some or all keywords used in the search, and the remaining 7 reviews provided no keywords. With respect to languages searched, 7 had no limits (15 per cent), 21 limited the search to English, and 4 limited the search to English and one more language, either French or German. The remaining 4 reviews did not mention the language. Related to this, 10 reviews had no information about the years searched, while 8 had no limits for the year of publication (9 per cent), and the remaining 28 reviews searched literature published after a specified year. The number of electronic databases searched varied greatly, from 1 to 25 (mean=15, median=6). Similarly, the specific databases searched varied, although the most commonly searched were MEDLINE (n=30), PsycINFO (n=25), CINAHL (n=23), EMBASE (n=22), and the Cochrane Library (n=18).

As shown in Table 2 (Row 3), less than a quarter (22 per cent) of the reviews stated that the process of selecting documents was done by two or more persons independently. For 21 (42 per cent) reviews, it was unclear whether the selection process was done in this manner, and for 14 (28 per cent) reviews, part of the process was done in this manner—a subset of the abstracts or the full-texts was read by a second reviewer, “for quality control purposes” as one review explained.

Moving on to examine transparency with respect to the RAMESES Publication Standards Items 9-13, three quarters (74 per cent) of the reviews stated the number of documents screened, assessed for eligibility, and included in the review (Table 2, Row 4). The number of documents finally included in the review varied from 6 to 276 (mean=66, median=35). We note that the number of documents included was unclear in four reviews. One review did not state how many documents were included. We also examined the types of studies or documents that were included in the reviews. Close to half (n=25) did not provide information about the types of documents included, while others included quantitative, qualitative, and mixed-method studies in different combinations. This included: only quantitative studies (n=9); quantitative and qualitative studies (n=7); and quantitative, qualitative, and mixed-methods studies (n=5). The remaining eight reviews included primarily quantitative studies in addition to other types of documents, such as documents titled research papers or conceptual papers. Of note, in the 29 reviews that specified the types of studies included, there were almost six times as many quantitative studies (n=951) compared to qualitative studies (n=165).

We also extracted details about the appraisal of the studies included by the 54 realist reviews (Table 2, Row 5). In 13 per cent of reviews, appraisal was done by two or more people. Additionally, we found that 30 per cent (n=16) of the reviews made no reference to appraisal, 17 per cent (n=9) used one or more checklist, 24 per cent (n=13) assessed the documents’ relevance to theory (and rigor), and one review explained, “no formal quality scoring was employed.” Among the remaining 15 reviews, there was a mix of self-created quality assessment tools and other vaguely described assessments, including “descriptive quality assessment was used,” “the usual criteria for high quality evidence was used,” and “we based our appraisal on whether the studies identified mechanisms.” We found only a few reviews that explained how the outcome of the appraisal was taken into account in the synthesis.

Lastly, we note that only a third of the reviews mentioned involving stakeholders at any point in the review process. The remaining 67 per cent made no mention of linkage with stakeholders, policy makers, or other decision makers.

4. Discussion

4.1. Scope of Realist Reviews

Our systematic scoping review has found that in the course of 10 years, a substantial amount of reviews are being conducted using the realist review approach. Our results suggest it is not only expanding into a wide range of topics, but a large number of journals and authors appear to be active in the field. Interestingly, while realist reviews are easily identified as such through either the title or the abstract, three of the reviews included in our study failed to use the term realist in both instances, identifying instead as “explanatory review,” “theory-led narrative review,” or simply as “review.” Similarly, given realist review is a theory-driven approach, it was perplexing that most reviews lacked a rationale as to why the realist review was a suitable approach to follow.

4.2. Reporting of Realist Reviews

We found that no review reported on all of the 19 items of the RAMESES Publication Standards, including the reviews that claimed to report in accordance with the standards. It is possible that more development or clarification of the standards is needed. For example, we speculate that the finding that more than nine out of ten realist reviews failed to report all nine elements of information required in the abstract (as per item 2) was because of the word limits applicable to abstracts in most scientific journals. On the other hand, that more than half of the reviews we studied made no reference to scoping of the literature cannot be explained by imprecise reporting standards, and seems a major oversight as exploratory background searching is highlighted as an important first step in the key realist review texts (Pawson, 2006; Pawson et al., 2004, 2005).

The results suggest consensus in the choice of electronic databases as the main method for identifying literature for a realist review. However, there was a divergence among reviews with respect to other search strategies, reporting of the actual search, and sophistication in database searching. Of all the 19 publication reporting items, the one about the searching process is the most detailed (Item 8, see Wong et al., 2013). Complying with the standard should overcome idiosyncratic reporting of how the documents are identified.

4.3. Methods Used in Realist Reviews

It has become a common practice within systematic reviews that document selection, appraisal, and data extraction are performed by two independent reviewers to minimize bias and errors (Littell, 2013). Experimental evidence shows that more errors occur with a single reviewer than with two reviewers (Buscemi, Hartling, Vandermeer, Tjosvold, & Klassen, 2005). Thus it is disquieting that only about one in five of the reviews in our study had selection, appraisal, or data extraction done independently by two persons. Similar results have also been found in systematic reviews (e.g., Braga, Pemberton, DeMaria, & Lorenzo, 2011; Dixon, Hameed, Sutherland, Cook, & Doig, 2005: Lang & Teich, 2014; Weir, Mayhew, Worswick, Fergusson, & Grimshaw, 2009). For example, an assessment of systematic reviews and meta-analyses in pediatric urology concluded that independent selection of studies by two collaborating reviewers was done only in 53 per cent of the reviews, and only 33 per cent described some form of quality assessment (Braga et al., 2011).

Reflecting a similar pattern, less than half of the reviews included in our study reported on how judgments were made about including data and which data were extracted. Only about half stated that a data extraction form was used. We are not arguing that realist reviews should adopt the processes that are standard for systematic reviews. However, as previous results indicate (e.g., Florence, Schulz, & Pearson, 2005), with a single reviewer and no standard forms, the likelihood of the results offering a partial and potentially misleading account of the evidence increases. Furthermore, one third of the reviews in our study made no reference to appraisal of the documents included, and those that did, generally ignored accounting for appraisal results in the synthesis. This is not unique to realist reviews, but also identified in systematic mapping reviews of qualitative evidence syntheses in general (Dixon-Woods, Booth, & Sutton, 2007), as well as in meta-ethnographies in particular (France et al., 2014). Analyses have documented the problem of inconsistency among reviewers in appraisal judgments (Lensen, Farquhar, & Jordon, 2014). The appraisal criteria used to assess the quality of documents differed substantially among reviews, including whether the document aims at identifying explanatory mechanisms, its actual relevance to theory, and the use of explicit checklists. Key realist review texts (Pawson, 2006; Pawson et al., 2004, 2005; Wong et al., 2013) explain that appraisal of the data should be made on relevance (whether it can contribute to theory building and/or testing) and rigor (whether it is credible and trustworthy), which was the case for only a quarter of the reviews included in our study.

It is also a common practice within systematic reviews, and also recommended by the realist review developers (Pawson, 2006; Wong et al., 2013), that information is provided on the characteristics of the documents included in the review. Yet, half of the reviews in our study lacked information about the documents used. For many, describing—not to mention analyzing and synthesizing—the documents would be an enormous undertaking (and may exceed journals’ word limits), as nine reviews included over a hundred documents and the average number of documents included was 66. The high number of documents suggests that realist reviews may assume a broader scope than systematic reviews. For a review type that accepts all forms of evidence—qualitative, quantitative, comparative, mixed-methods, administrative records, annual reports, legislative materials, conceptual critique, personal testimony (Pawson et al., 2004)—it was surprising that there was about six times as many quantitative studies as qualitative studies included, and that no reviews mentioned including letters to editors, newspaper articles, opinion surveys, oral histories, or the like. In contrast to some effectiveness reviews, one review only accepted randomized controlled trials (RCT). This may be related to Pawson and colleagues’ (2004) doubt about the utility of RCT in explaining the outcomes of complex interventions, as they state, “such trials are meaningless because the RCT design is explicitly constructed to wash out the vital explanatory ingredients” (Pawson et al., 2004, p. 22).

The description of the analysis and synthesis approaches was rather weak in most of the reviews included in our study. This is noteworthy particularly because it is the analysis of the context and mechanisms as well as the theory development and tracking that set realist reviews apart from other types of literature reviews. Similar to what Dixon-Woods and colleagues (2007) found for qualitative evidence syntheses, there is some intransparency about how the data are analyzed and synthesized in realist reviews. We found that there is a lack of clarity both surrounding the process of building up and testing of theories and the way in which the reviewers assess the effect of interventions. There was a consistent lack of measures of strengths of effects across the studies. Instead, the reviews appeared to rely largely on a qualitative assessment of the likelihood of certain outcomes.

Pawson and colleagues (2004) state that a realist review, as compared to a systematic review, “probably demands greater methodological expertise on the part of the reviewer” (Pawson et al., 2004, p. v). In our study, a high proportion of reviewers discussed the limitations of their review rather than the strengths, oftentimes highlighting the demanding methodological process. A recurring element was a call for additional research, usually without specification of research directions. Counter to the publication standards and the review developers’ expectation that realist reviews are particularly oriented towards practitioners and policy makers (Pawson, 2006; Pawson et al., 2004, 2005), 83 per cent of the reviews did not specify the main implications of their findings and 67 per cent did not state whether they had any policy linkage. In a realist review, stakeholder involvement from very early stages of the review process is a requirement, which is intended to ensure practical relevance of the end product.

Finally, we note that four of the realist reviews included in our study were re-reviews of previously completed systematic (effectiveness) reviews. While two of these reviews made no mention of the approaches’ relative merits, the other two noted the greater understanding gained with the realist review, concerning the ways in which impacts were produced. Specifically, one stated, the “re-review offers a fuller understanding of the impacts of these interventions and how they are produced . . . It suggests that some conclusions of the systematic review and earlier reviews should be reconsidered” and the other noted, “a greater understanding is gained of the contextual factors and the main mechanisms that triggered the effects of [intervention].” We believe that systematic comparative research of the two approaches’ strengths and limitations is needed before conclusions about their relative merits can be drawn.

4.4. Strengths and Limitations of Our Review

Our systematic scoping review comes with limitations. One of the limitations is that new realist reviews are being published quite regularly in a variety of areas (sometimes without specifying its realist nature), and some may have been missed in this review. Moreover, different researchers may interpret what counts as a realist review slightly differently. We aimed for an acceptable degree of methodological congruence between the realist review approach and what had been conducted by the review authors. Our understanding of realist review and our inclusion criteria were based on Pawson and colleagues’ description of realist review (Pawson, 2006; Pawson et al., 2004, 2005). It is likely that less stringent inclusion criteria would reveal greater variability in realist review practice. It was also challenging to designate some of the data points. As with all ratings, there is some degree of subjectivity and other researchers may classify the information differently. Lastly, as all reviewers assessing published reports, we were limited by what is reported.

On the other hand, strengths of our review include our systematic methods, including search, selection, and data extraction. The use of a standard data extraction form enabled consistency, and the charting of data enabled us to identify commonalities and trends in realist review practice. In accordance with recommendations by Levac and colleagues (Levac, Colquhoun, & O’Brien, 2010), we strove to analyze the data being mindful of the relevance of our work to the debates about developing review approaches.

5. Conclusions

The objective of this systematic scoping review was to examine the current practice of realist review. We found that realist review is still in development. Despite the rapid expansion of realist reviews, there is little uniformity in practice. There is inadequate transparency about methods for selection, appraisal, analysis, and synthesis of studies. The results show that realist reviews are by and large not systematic reviews. There is scope for improvement toward the goal of being explicit and transparent at each stage of the review, such that other researchers can understand and replicate the review. This would also allow controversies and agreements to be deliberated. When methods are not sufficiently justified or are poorly undertaken or reported, and the approach subsequently appears to exist in many variants, it becomes harder to understand what is meant when the term realist review is used. Similarly, without theoretical or methodological justifications, realist reviews may also be suspected of offering a partial and potentially misleading account of the evidence. Specific methodological guidance may need to be developed if realist reviews are to have a more uniform and transparent approach. It is likely that the existing reporting standards will contribute both to greater consensus on methods for realist review and better research practice.

Acknowledgements

We gratefully acknowledge the generous contribution by realist review experts and members of the RAMESES listserv in suggesting realist reviews and we thank methods experts in offering comments on our preliminary results.

References

Arksey, H., & O’Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8(1), 19-32.

Armstrong, R., Hall, B. J., Doyle, J., & Waters, E. (2011). Cochrane update: ‘Scoping the scope’ of a Cochrane review. Journal of Public Health 33(1), 147-150.

Bates, S., Clapton, J., & Coren, E. (2007). Systematic maps to support the evidence base in social care. Evidence & Policy: A Journal of Research, Debate and Practice, 3(4), 539-551.

Braga, L. H., Pemberton, L., DeMaria, J., & Lorenzo, A. J. (2011). Methodological concerns and quality appraisal of contemporary systematic reviews and meta-analyses in pediatric urology. The Journal of Urology, 186(1), 266-272.

Bravata, D. M., McDonald, K. M., Shojania, K. G., Sundaram. V., & Owens, D. K. (2005). Challenges in systematic reviews: Synthesis of topics related to the delivery, organization, and financing of health care. Annals of Internal Medicine, 142(12, Part 2), 1056-1065.

Buscemi, N., Hartling, L., Vandermeer, B., Tjosvold, L., & Klassen, T. P. (2005). Single data extraction generated more errors than double data extraction in systematic reviews. Journal of Clinical Epidemiology, 59, 697-703.

Clapton, J., Rutter, D., & Sharif, N. (2009). SCIE Systematic mapping guidance. London, UK: Social Care Institute for Excellence.

Dixon, E., Hameed, M., Sutherland, F., Cook, D. J., & Doig, C. (2005). Evaluating meta-analyses in the general surgical literature: A critical appraisal. Annals of Surgery, 241(3), 450-459.

Dixon-Woods, M., Booth, A., & Sutton, A. J. (2007). Synthesizing qualitative research: A review of published reports. Qualitative Research, 7, 375-422.

Florence, Z., Schulz, T., & Pearson, A. (2005, October). Inter-reviewer agreement: An analysis of the degree to which agreement occurs when using tools for the appraisal, extraction and meta-synthesis of qualitative research findings. Oral presentation at Cochrane Colloquium, Melbourne, Australia. Retrieved from http://abstracts.cochrane.org/2005-melbourne/inter-reviewer-agreement-analysis-degree-which-agreement-occurs-when-using-tools

France, E. F., Ring, N., Thomas, R., Noyes, J., Maxwell, M., & Jepson, R. (2014). A methodological systematic review of what’s wrong with meta-ethnography reporting. BMC Medical Research Methodology, 14, 119.

Greenhalgh, T., Wong, G., Westhorp, G., & Pawson, R. (2011). Protocol—realist and meta-narrative evidence synthesis: Evolving standards (RAMESES). BMC Medical Research Methodology, 11, 115.

Higgins, J. P. T., & Green, S. (2011, March). Cochrane handbook for systematic reviews of interventions (Version 5.1.0). Retrieved from http://handbook.cochrane.org/index.htm

Lang, L. A., & Teich, S. T. (2014). A critical appraisal of the systematic review process: Systematic reviews of zirconia single crowns. The Journal of Prosthetic Dentistry, 111(6), 476-484.

Lensen, S., Farquhar, C., & Jordon, V. (2014, September). Risk of bias: Are judgments consistent between reviews? Oral presentation at Cochrane Colloquium, Hyderabad, India. Retrieved from http://abstracts.cochrane.org/2014-hyderabad/risk-bias-are-judgements-consistent-between-reviews

Levac, D., Colquhoun, H., & O’Brien, K. K. (2010). Scoping studies: Advancing the methodology. Implementation Science, 5, 69.

Littell, J. (2013). The science and practice of research synthesis. Journal of the Society for Social Work and Research, 4(4), 292-299.

Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Med, 6(7), e1000097.

Oakley, A., Gough, D., Oliver, S., & James, T. (2005). The politics of evidence and methodology: Lessons from the EPPI-Centre. Evidence & Policy, 1(1), 5-31.

Paternotte, E., Van Dulmen, S., Van Der Lee, N., Scherpbier, A. J. J. A., & Scheele, F. (2015). Factors influencing intercultural doctor-patient communication: A realist review. Patient Education and Counselling, 98(4), 420-445.

Pawson, R. (2002). Evidence-based policy: The promise of ‘realist synthesis’. Evaluation, 8(3), 340-358.

Pawson, R. (2006). Evidence-based policy: A realist perspective. London, UK: SAGE.

Pawson, R., Greenhalgh, T., Harvey, G., & Walshe, K. (2004). Realist synthesis: An introduction (RMP Methods Paper 2/2004). ESRC Research Methods Programme, University of Manchester, UK. Retrieved June 16, 2016, from https://www.researchgate.net/profile/Gill_Harvey/publication/228855827_Realist_synthesis_an_introduction/links/0fcfd507f0b7cbb2ce000000.pdf

Pawson, R., Greenhalgh, T., Harvey, G., & Walshe, K. (2005). Realist review: A new method of systematic review designed for complex policy interventions. Journal of Health Services and Research Policy, 10(1, Suppl. 1), 21-34.

Pawson, R., & Tilley, N. (1997). Realistic evaluation. London, UK: SAGE.

Sackett, D. L., Richardson, W. S., Rosenberg, W., & Haynes, R. B. (1997). Evidence-based medicine: How to practice and teach EBM. New York, NY: Churchill Livingston.

Shepherd, J., Harden, A., Rees, R., Brunton, G., Garcia, J., Oliver, S., & Oakley, A. (2006). Young people and healthy eating: A systematic review of research on barriers and facilitators. Health Education Research, 21(2), 239-257.

Weir, M., Mayhew, A., Worswick, J., Fergusson, D., & Grimshaw, J. (2009, October). The epidemiology and quality of systematic reviews of health professional behaviour change interventions. Poster presentation at Cochrane Colloquium, Singapore. http://abstracts.cochrane.org/2009-singapore/epidemiology-and-quality-systematic-reviews-health-professional-behaviour-change

Wong, G., Greenhalgh, T., Westhorp, G., & Pawson, R. (2012). Realist methods in medical education research: What are they and what can they contribute? Medical Education, 46, 89-96.

Wong, G., Greenhalgh, T., Westhorp, G., Buckingham, J., & Pawson, R. (2013). RAMESES publication standards: Realist syntheses. BMC Medicine, 11(21), 1-14.

Appendix A:
Details of 71 Publications Included in This Study

Appendix B:
Details of 14 Abstracts (Not Included in This Study)

Note. We have not extracted data from these abstracts. Still, we present the list here for two reasons: We believe it is important to share this for the sake of transparency of the review process; besides, this may be a useful resource for readers interested in the realist review approach.

Appendix C:
Details of 24 Protocols (Not Included in This Study)

Note. We have not extracted data from these protocols. Still, we present the list here for two reasons: We believe it is important to share this for the sake of transparency of the review process; besides, this may be a useful resource for readers interested in the realist review approach.

Received 29 April 2016 | Accepted 11 June 2016 | Published 8 July 2016

Copyright © 2016 Journal of Research Practice and the authors