Editorial:
Introducing New Editorial Roles and Measures: Making the Journal of Research Practice Relevant to Researchers
Following a detailed review of the accomplishments and aspirations of the Journal of Research Practice, we have undertaken a restructuring of the editorial board, with inputs from people associated with this journal. In designing the new structure, we have taken into account the need for building the journal’s profile in the six focus areas recently clarified: (1) Research Applications, (2) Research Spaces, (3) Research Education, (4) Research Experiences, (5) Research Philosophy, and (6) Research on Research. Focus Editors will ensure that the journal remains well engaged with the developments in these focus areas. The new structure allows us to involve all contributors to the journal in playing a role to enhance the journal’s relevance to researchers and reflective professionals. This restructuring exercise has presented us with an opportunity to build on the strengths of the journal and address areas of concern so as to strengthen the journal’s quality, relevance, and impact. A review of different notions of impact has led us to a set of proposed measures for enhancing the relevance and utilisation of the journal in future.
Index Terms: bibliometrics; citation analysis; editorial roles; Impact Factor; professional development; research assessment; research education
Suggested Citation: Dash, D. P., & Ulrich, W. (2012). Introducing new editorial roles and measures: Making the Journal of Research Practice relevant to researchers. Journal of Research Practice, 8(1), Article E1. Retrieved from http://jrp.icaap.org/index.php/jrp/article/view/314/249
In this editorial, we review the origin and growth of the Journal of Research Practice (JRP) and suggest key initiatives to enhance the journal’s relevance and utilisation in future (Section 1). We introduce a new structure for the journal’s editorial board and outline the key responsibility areas for all editorial members (Section 2). For the new category of editorial role labelled Focus Editors, we present specific nomination criteria followed by brief introductions to our new Focus Editors (Section 3). Next, we introduce a discussion on the methods of assessing a journal’s relevance and utilisation. We first examine some of the prevailing bibliometric methods of assessment, which are based on citation analysis (Section 4). Going beyond bibliometrics, we examine other practical notions of relevance and utilisation which are more aligned with the character and aspirations of this journal (Section 5). In conclusion, we summarise the key measures proposed for enhancing the relevance and utilisation of the journal (Section 6).
The Journal of Research Practice (JRP) was introduced in 2005 in order to address what was perceived then to be an important gap in the literature of research. To the best of our knowledge, there was no other journal at that time seeking to involve practising researchers in recounting their research experiences in a critically reflective mode, in order to establish a learning conversation across disciplinary, professional, and institutional boundaries. This was considered to be an important unmet need because, without such a conversation becoming part of researchers’ education and experience, how else could we establish an ongoing review of the practical and conceptual repertoire of research? A new journal was considered necessary to create “conditions for an ongoing review” of what it means to do research and how the goals and methods of research may evolve through innovations by those who engage in it (Dash & Ponce, 2005). JRP was floated in the open-access format to facilitate access, participation, and utilisation.
At that stage, although it was clear that such a practically-focused journal of research that is not aligned with any specific discipline or field would be a good thing to have, there were still a number of doubts regarding such a journal’s chances of success. Whether a new electronic journal would be able to publish good quality articles was a major concern. Online journals were still a novelty then. Although JRP was first published in 2005, preparatory work had started 2 years earlier with the creation of the “Research_Practice” electronic forum on October 7, 2003. At that time, open-access (OA) journals were not widely perceived as an effective channel for publishing high quality research and scholarship. Systematic directories of OA journals did not yet exist; in fact, the Directory of Open Access Journals (DOAJ) was to be launched in the very same year, 2003 (Johansson & Wahlgren, 2008). The concept of peer-reviewed OA journals was relatively new then and there were considerable doubts as to whether OA publishing could be of high quality and if OA communities could implement reliable peer-review procedures successfully.
Another related concern more specific to JRP was how to ensure an effective peer-review process given the journal’s broad, transdisciplinary scope. With submissions expected from a vast range of research contexts, how could we possibly establish a peer-review process that would not only be efficient but also produce reviews of consistently high quality?
With regard to both these concerns, JRP seems to have fared rather well. The specific mechanism we put in place included three elements. The first of these elements was an electronic forum of interested volunteers from different specialist domains, implemented as the “Research_Practice” electronic group, which served as a support network for the journal. It proved to be remarkably successful in that finding suitable peer reviewers for JRP submissions has never been difficult. This forum has grown to 675 members (in April 2012) and continues to serve as the primary source of peer reviewers for JRP, although help from outside the forum is sought in specific cases. Another part of the mechanism was the “JRP Best Reviewer Award,” which allowed us to formally recognise the high levels of quality and voluntarism of our forum members. A third element was the practice of sharing the review comments as well as the editorial decision with all the peer reviewers of a submission (without revealing the author’s identity). This practice has been welcomed by the peer reviewers as it contributed to significant peer learning. These three elements (i.e., the electronic forum, the best reviewer award, and the sharing of reviews) constitute the mechanism which has ensured an effective peer-review process as well as the final quality of articles we have published.
Since 2003, the DOAJ initiative has been followed by other such initiatives and many of the new directories have found JRP worthy of being included in their data banks. Even so, a new concern has arisen in the recent years: it is the concern about JRP’s utilisation and impact. With our editorial attention focused largely on ensuring the quality and originality of what we publish, and with a number of online directories helping researchers to become aware of the journal, we might have assumed that impact would take care of itself, so to speak. We must have thought free access to the journal’s contents ought to draw potential users who would use the articles in their educational, professional, scholarly, or research contexts. But gradually it has become clear that open-access publishing does not obviate the need for taking measures to stimulate greater visibility and use of the journal. In an environment where citation figures rather than quality of content weigh heavily in a journal’s perceived significance, it has become essential to give JRP a clearer identity in terms of its areas of specialisation, international profile of editorial team members, and linkages with potential user communities. Hence, to achieve the level of utilisation and impact we think our journal deserves, we have launched the current initiative of which this Editorial is a part.
After a good year of intensive analysis and discussion, four specific strategies were formulated in 2011, as stated in a previous editorial article (Ulrich & Dash, 2011, Section 1, “Introducing New Initiatives to Develop JRP”):
Strategy #1. Strengthening JRP’s Profile and Visibility
Strategy #2. Restructuring the Editorial Team
Strategy #3. Lowering the High Rejection Rate
Strategy #4. Making Full Use of the Online Open-Access Format
To implement strategies #1 and #2 in a coordinated manner, the journal’s publication focus was sharpened in terms of six broad focus areas, each focus area being defined by an aim, two core questions, and a set of subject areas. Each subject area was further denoted by a set of keywords. This three-level hierarchy of focus areas, subject areas, and keywords was presented in the form of a “JRP concept hierarchy,” which is meant to express the journal’s identity clearly and provide useful guidance to all contributors and users regarding the journal’s contents.
In the limited time period of a few months since its introduction, the JRP concept hierarchy is already proving its merit as a very useful tool in the editorial management process. JRP editors and reviewers are using it to guide potential contributors, helping them orient their contributions more clearly towards the journal’s publication focus. A number of special issue proposals have been assessed for their alignment with the journal’s focus areas. We have received more submissions as well as inquiries from potential authors, who have used the new tool to specify the content of their planned contributions and its alignment with one or several of the journal’s focus areas. Clearly, the JRP concept hierarchy is living up to its promise as a useful tool for reflecting on, and writing about, research practice.
Realising the full potential of this tool in enhancing the journal’s profile and visibility will require us to take the next important step anticipated earlier: restructuring the editorial team so as to reflect a congruence with the journal’s emerging profile. This is to be done by inviting outstanding research practitioners and thinkers to provide intellectual leadership and serve as a vital link between the journal and the specific professional and scholarly communities relevant to the six focus areas which have been identified. A set of Focus Editors will play this role. Table 1 presents the new structure of the editorial team. Being introduced in the 8th year of the journal’s operation, the new structure is expected to be valid for the next 8 years, till 2020.
Table 1. Editorial Board Structure, Journal of Research Practice, 2012-2020
Role |
Team Sizea |
Key Responsibility Areas |
Tenure |
Editors |
2 to 3 |
|
3 years |
Focus Editors |
6 to 12 |
|
3 years |
Associate Editors |
10 to 20 |
|
3 years |
Distinguished Reviewers |
30 to 90 |
|
3 years |
Members, JRP Forumc |
Not specified |
|
Not specified |
Notes.
(a) The size is based on the journal’s projected growth till 2020.
(b) These discussions are held within the JRP Executive Group, which consists of Editors, Focus Editors, Associate Editors, and Distinguished Reviewers. Inputs are also taken from the wider JRP Forum, when necessary.
(c) This is implemented through the “Research_Practice” electronic group.
In the following section, we describe the Focus Editor role in greater detail, as this will be a new role in the JRP editorial team. We present our expectations from Focus Editors and introduce the individuals nominated into this role.
Although the Associate Editor role continues as before, our expectations from this role have become much clearer in the restructuring process. As stated in Table 1, we expect Associate Editors to play a supporting role to the Editors and Focus Editors. This implies they should have a clear potential for, and interest in, assuming Editor or Focus Editor roles in future. Besides, we expect Associate Editors to undertake an annual review of the journal’s quality, especially its relevance and utilisation in different contexts. This is expected to generate valuable information for the whole editorial team to guide the journal’s development.
The readers will notice that we have dropped the earlier role category of Editorial Advisors. As expressed in the previous editorial, we would like to move on from the nominal representative function played by them so far and specify their expected contributions according to their particular expertise (Ulrich & Dash, 2011, subsection 1.2, “Restructuring the Editorial Team”). In the new structure, this is achieved by replacing Editorial Advisors with the two new role categories: Distinguished Reviewers and Forum Members.
The Editors, Focus Editors, Associate Editors, and Distinguished Reviewers will constitute the JRP Executive Group. This would provide a mechanism to initiate and maintain strategic discussions about the journal. Of course, whenever necessary in such discussions, inputs from the wider JRP Forum would be taken.
As stated in Table 1, the Focus Editors are expected to support the journal through their area-specific expertise and their extensive institutional and professional networks. Their role requires them to enhance the relevance and utilisation of what is published in the journal, and contribute towards the journal’s future development. This is an important role and a key part of our strategy for consolidating the journal’s achievements and guiding the next phase of its development. We have arrived at a clear set of criteria based on which specific individuals are nominated to the Focus Editor role.
For each of the six defined focus areas, one or two Focus Editors are nominated. In the latter case, Focus Editors may share responsibilities for a focus area according to their specific individual expertise. Since each of the focus areas covers a very broad range of issues, not every qualified Focus Editor may feel at home with all these issues; the option of “job sharing” thus offers more flexibility in finding highly qualified Focus Editors.
As the role of Focus Editor is similar across the six focus areas, there is a set of common criteria applicable to all Focus Editors. Besides these common criteria, each focus area will necessitate some specific expertise from its Focus Editors.
3.1.1. Criteria Common to all Focus Areas
Four common criteria have been identified; accordingly, all Focus Editors would need to have the following type of background and preparedness to play the role effectively:
(a) Theoretical and practical interest in the actual work of researchers, as demonstrated through past didactic or research engagements
(b) Broad outlook on the philosophical, political, and methodological problems of research practice, as demonstrated through research, teaching, and publications
(c) International engagement in research practice and related scholarship, as demonstrated though prominent position in relevant professional and scholarly communities, editorial boards, or councils
(d) Willingness to play an active role in making the journal a distinctive and influential publication on research practice, as demonstrated through contributions to, or contacts with, JRP
3.1.2. Criteria Specific to Each Focus Area
In addition to these four common criteria, each focus area demands a specific profile of expertise from it Focus Editor(s). Based on our prior demarcation of the focus areas (in terms of their aims and core questions, see Ulrich & Dash, 2011), we describe the particular interest and expertise each Focus Editor would ideally bring to the journal.
Focus Area 1. Research Applications
Aim. To develop conventional or innovative forms of applied research with a view to meeting contemporary challenges
Core Questions. What happens when the procedures or results of research are applied to practical affairs? What constitutes competent practice in applied research?
Expectation. The Focus Editor(s) would be competent in addressing issues of applied research in technical, human, and complex overlapping domains; should have some personal experience in applied research.
Focus Area 2. Research Spaces
Aim. To examine the institutional, cultural, and historical factors that shape research practice so as to help researchers open up new spaces for innovative research
Core Questions. What societal or historical forces shape research practice and obstruct or promote open inquiry? Are there new spaces for meaningful inquiry to be discovered and cultivated?
Expectation. The Focus Editor(s) would be knowledgeable on the history of research practice in different disciplines and fields and have practical experience in opening up new spaces for inquiry; should be familiar with the perspective of practising researchers as well as that of research policy makers.
Focus Area 3. Research Education
Aim. To promote new directions in research education so as to prepare researchers for their role in society
Core Questions. How can research education be more effective? Are there ways to better prepare researchers for their roles and responsibilities in society?
Expectation. The Focus Editor(s) would have significant experience and expertise in researcher or professional development, demonstrated through previous didactic engagement in academic, professional, and community contexts.
Focus Area 4. Research Experiences
Aim. To offer researchers a platform for sharing research experiences, appreciating the experiences of other researchers, and developing their own understanding of good practice
Core Questions. As humans, how do researchers deal with the demands of research practice? What can be learnt from the experiences of individual researchers or research teams?
Expectation. The Focus Editor(s) would bring a sustained interest in the living dimensions of research, with demonstrated expertise in initiating and supporting reflective conversations on research; a mainly didactic focus on developing research practice would be welcome.
Focus Area 5. Research Philosophy
Aim. To encourage reflection on the philosophical underpinnings of research, the specific research frameworks they inform, and corresponding notions of what constitutes valid and relevant research
Core Questions. What kinds of thought patterns, beliefs, values, and conventions underlie research practice? Can philosophical reflection support new forms of good practice?
Expectation. The Focus Editor(s) would specialise on the philosophical underpinnings of research practice, especially on the difficult but essential connection between its logical/theoretical and ethical/practical dimensions; should combine an interest in research theory with an orientation to improving research and professional practice.
Focus Area 6. Research on Research
Aim. To review and innovate conventional thinking about research as it is contained in notions such as scientific method, objective attitude, and logic of inquiry, with a view to expanding their range of application and exploring new forms of research
Core Questions. What can be learnt from successful and unsuccessful cases of research, what constitutes its specific quality? How can the process of research be extended to cover new domains?
Expectation. The Focus Editor(s) would have special knowledge of the conventions of research as they have evolved over time in different domains; a focus on reviewing and adapting these conventions so as to help develop new forms of research is desirable.
Table 2 presents a summary of the nomination criteria for Focus Editors, highlighting the key elements of our expectations from Focus Editors.
Table 2. Overview of Expectations From Focus Editors: Key Elements
Common Expectations |
Area-Specific Expectations From Focus Editors |
||
Focus Area Aims |
Focus Area Questions |
Area-Specific Competencies |
|
|
INTEREST IN THE ACTUAL WORK OF RESEARCHERS
BROAD OUTLOOK ON THE PROBLEMS OF RESEARCH PRACTICE
INTERNATIONAL ENGAGEMENT IN RESEARCH AND SCHOLARSHIP
ACTIVE ROLE IN THE JOURNAL |
1. Research Applications |
Research applied to practical affairs |
Competence in addressing issues of applied research |
2. Research Spaces |
Forces that shape research practice |
Knowledge on the history of research practice |
|
3. Research Education |
Effective research education |
Expertise in researcher or professional development |
|
4. Research Experiences |
Dealing with the demands of research practice |
Sustained interest in the living dimensions of research |
|
5. Research Philosophy |
Thought patterns underlying research practice |
Knowledge of the logical and ethical underpinnings of research practice |
|
6. Research on Research |
Successful and unsuccessful cases of research |
Knowledge of how the conventions of research have evolved over time |
|
Focus Area 1. Research Applications
Richard J. Ormerod
Male, b. 1945, British citizen; raised in rural Oxfordshire, UK; studied civil engineering (BSc, London, external degree at the Royal Military College of Science) and operational research and management science (MSc, Warwick); professional experience as consulting engineer, operational research scientist, corporate strategist and management consultant; academic teaching and research experience primarily in the UK; emeritus professor of management at Warwick University; currently living in Plymouth, UK.
Ormerod’s research interests include application of theories of intervention in professional practice, publication of the learning experience through case studies, and use of such experience to generate theory. He is interested in the pragmatic nature of intervention and its possible underpinning by philosophy and sociology. In pursuit of these aims he has written case studies on consulting interventions in the retail, mining, and energy sectors and also in the UK Parliament. As a result he has developed the transformation competence perspective (TCP) to guide the design of participative interventions. A continuing theme has been reflection on the nature of professional practice in general and operational research in particular.
Major Aim as Focus Editor. To encourage both the publication of reflective case material describing applications and their critical use to develop practical theories and methods.
Focus Area 2. Research Spaces
Wiebe E. Bijker (home page: http://www.fdcw.unimaas.nl/staff/bijker)
Male, b. 1951 in the Netherlands; trained as an engineer in applied physics (Technical University of Delft), studied philosophy (University of Groningen), and holds a PhD in the sociology and history of technology (University of Twente). Bijker is Director of Studies of the research master MPhil-degree programme Cultures of Arts, Science, and Technology (CAST). He is founding co-editor of the book series “Inside Technology” of MIT Press and “Science and Democracy in South Asia” of Orient Blackswan. Bijker is member of the Health Council of the Netherlands and the Board of the Rathenau Institute (the Dutch office of technology assessment). Currently, Bijker is professor of Technology & Society at the Maastricht University.
Bijker’s interests are in studying the relations between science, technology, and society. His work primarily draws on a combination of sociology and history of technology, and often has an ethnographic angle. He developed, together with Trevor Pinch, the social construction of technology (SCOT) approach. Recently his research agenda has broadened to include questions about the role of science and technology for development. He is also interested in the interplay between understanding and intervening, between research and practice. Empirical domains of his research include nanotechnologies, flood management and coastal engineering, handloom weaving, sustainable agriculture, water sanitation and resource management, knowledge brokerage, sustainable energies, and biotechnologies.
Major Aim as Focus Editor. To help researchers open up new spaces for innovative research, especially by combining different disciplines, by crossing boundaries between research and practice, and by raising normative and political questions related to research and intervention.
Focus Area 3. Research Education
Lynn C. McAlpine
Female, b. 1948 in Canada; studied social sciences, education, adult and professional learning (McGill University, Concordia University, University of Toronto); 20 years’ experience in professional development before entering academia--first at McGill University (now Emerita), currently at University of Oxford (Professor of Higher Education Development).
The focus of her research has varied over the years but a constant thread has been: (a) how individuals engage in and learn from the work they are doing, e.g., as teachers or nurses, and (b) how to better support that learning. Her present research interests are directed at understanding how doctoral students, post-PhD researchers, and newly-hired lecturers/pre-tenure professors engage in and learn to do academic work--further, the career trajectories they experience including whether they remain in academia or choose to leave. This research, conducted in both Canada and the UK, is represented in the following Web sites: Doc Work (http://doc-work.mcgill.ca/)--a summary of the Canadian work with resources for early career researchers, and Research Supervision (http://www.learning.ox.ac.uk/supervision/)--a resource for new supervisors, doctoral students, and others involved in doctoral education which incorporates some of the UK research.
Major Aim as Focus Editor. To promote the view that the conduct of research is a form of workplace learning in which informal learning is constantly taking place; from this it follows that: (a) there are profound pedagogical implications in how we conduct research, and (b) the conduct of research is a significant site of research education as much as are workshops, modules, courses, and other formal provisions.
Focus Area 4. Research Experiences
D. P. Dash
Male, b. 1966, Indian citizen; raised mainly in Bhubaneswar, India; studied technology and management (BTech Honours, Indian Institute of Technology; PostGradDip, Xavier Institute of Management, India) and systems thinking and action research (PhD, Lincoln, UK); professional experience as engineer and management consultant in private companies; academic teaching and research experience in India, UK, and Malaysia; currently living in Kuching, Malaysia, working in an international branch campus setting.
Dash’s work reflects a curiosity about how persons from different backgrounds could converse and connect with each other, what might impede or facilitate such conversation and the range of outcomes possible. He has pursued this theme in a variety of contexts, including sensitivity training, participatory planning and designing, professional development, community and organisational development, leadership, and research. His key ideas are related to enhancing the capacity of a group or system to develop its own distinctive quality by drawing upon the capacities of its constituents and their interactions. Dash has been a key initiator of the Journal of Research Practice and the managing editor since its beginning.
Major Aim as Focus Editor. To encourage researchers and professionals to interrogate the dilemmatic aspects of their practice and consider upgrading the practice through innovation.
Focus Area 5. Research Philosophy
Werner Ulrich (home page: http://wulrich.com/)
Male, b. 1948, Swiss citizen; raised mainly in St. Gallen, Switzerland; studied social sciences and economics (Doctor of Economics and Social Sciences, Fribourg) and systems theory and practical philosophy (PhD, Berkeley); professional experience as chief evaluation researcher and policy analyst in government; academic teaching and research experience in Switzerland, England, and New Zealand; currently living near Bern, Switzerland.
Ulrich is a social scientist and practical philosopher with a particular interest in the philosophical underpinnings of research and professional practice in a variety of applied fields that include private and public sector management, operational research/management science, social policy and planning, environmental design, evaluation research, and information systems design. His work on critical systems heuristics (CSH) and its methodological core principle of “boundary critique” has been influential for theoretical developments in these and other fields, particularly in operational research/management science and applied systems thinking/systems methodologies. Ulrich is developing this work into a broader framework for professional practice under the names “critical pragmatism” and “philosophy for professionals.”
Major Aim as Focus Editor. To raise researchers’ interest in philosophical questioning as a way to improve their research practice.
Focus Area 6. Research on Research
Gerard de Zeeuw
Male, b. 1936 in Banjuwangi, Java, Indonesia (former Dutch Indies); studied mathematics and physics (Leiden University), continued with statistics and econometrics (Erasmus University) and mathematical psychology (Stanford University), and received his PhD in philosophy (University of Amsterdam) on a thesis entitled “Model Thinking in Psychology.” In the 1980s the Dutch minister of Education awarded him a very large personal grant for a 10 year research program on social support systems, titled “Support, Survival, and Culture” (in Dutch, OOC) at the Centre for Innovation and Cooperative Technology, University of Amsterdam. Besides being Emeritus Professor of Complex Social Systems at the University of Amsterdam, De Zeeuw has been formally associated with Sint-Lucas School of Architecture (RC University Leuven, Belgium), Faculty of Business and Law (University of Lincoln, United Kingdom), Institute of Social Psychology (London School of Economics), and Agricultural University of Wageningen, Netherlands. He was elected twice as Fellow of the Netherlands Institute for Advanced Study (NIAS).
The journal, Systems Research and Behavioral Science published a Festschrift issue on him in 2002 (Volume 19, Number 2), in which the editor, Ranulph Glanville wrote, “De Zeeuw has a lifetime of major work in areas central to cybernetics and systems--in both published form and in his activities as a teacher (and other forms of social activism)--which has been influential and important, probably more than is realized.” Gerard has substantially brought forward our thinking on the design of support systems for individuals and groups confronted with various kinds of predicament, introducing methods of research that engage individual contributions to a collective activity. Some of the topics he has written about are: non-observational research, soft knowledge accumulation, third-phase science, complete collectives, and high-quality experience.
Major Aim as Focus Editor. To help researchers develop a critical appreciation of what constitutes quality in research and why it is difficult to achieve in many domains, encouraging them to think in innovative ways to extend the process of research to such nontraditional domains where we aim to improve upon our intentions, emotions, and other experiences, but do not find a ready-made research approach.
One of the major tasks awaiting the new editorial team will be to enhance the journal’s profile, visibility, and impact. We initiate a discussion on different methods of assessing a journal’s impact in order to develop a shared understanding of this task and identify our priorities more clearly.
Research councils, universities, and other institutions connected with research commonly assess the quality and impact of journals by using bibliometric measures. Such measures have gained currency because of their relatively easy availability and their basis in verifiable data.
In bibliometric analysis, an article’s worth is measured in terms of the number of times other articles cite it. This logic is extended to compute an entire journal’s worth in terms of the average number of citations to the articles published in it. In practice, to keep this measurement process manageable, a number of operational simplifications are introduced:
(a) citations are collected for items published in a journal (e.g., research articles, editorials, letters, etc.) and the total citation count is averaged over the number of peer-reviewed items published (typically, editorials and letters are excluded; see, e.g., Hubbard & McVeigh, 2011),
(b) a specific set of journals is considered for sourcing citation data; citations made in any other journals or other types of publication (such as books, reports, theses, policy documents, professional outlets, course material, etc.) are excluded, and
(c) only citations made in a specific year are counted, which have cited items published within a specific preceding time range (typically, the previous 2-5 years).
A prominent measure for assessing the worth of scholarly journals in this way is the Journal Impact Factor (or simply Impact Factor, developed by The Institute for Scientific Information, which is known as Thomson Reuters now); it uses citation data from a set of journals listed in the Web of ScienceSM database--the data are analysed and presented periodically in the Journal Citation Reports®. The Impact Factor for a journal is computed annually. It is calculated by dividing the number of current year citations by the number of peer-reviewed items published in that journal during the previous 2 years.
Another citation-based measure is the SCImago Journal Rank (developed by Scimago Lab); it uses citation data from the publications listed in the Scopus® database. There are conceptual and operational differences between Impact Factor and SCImago Journal Rank: (a) SCImago Journal Rank weighs the citation data from different sources differently, depending on the “prestige” of the source, as measured by the number of citations the source title receives from other titles (Scimago Research Group, 2007); (b) moreover, the citation data sources used by the two are somewhat different, both quantitatively and qualitatively--the Web of ScienceSM database included 12,000 journals in February 2012 (Thomson Reuters, 2012), whereas the Scopus® database included more than 19,500 titles in July 2011, which included “18,500 peer-reviewed journals (including 1,800 open-access journals), 425 trade publications, 325 book series, and 250 conference proceedings” (Elsevier, 2012).
Yet another citation-based measure is the Eigenfactor (developed by Carl Bergstrom and colleagues, see Bergstrom, 2007), which uses citation data from the Web of ScienceSM database. The method for computing Eigenfactors is similar to the one used by Google for ranking the importance of Web sites in a search--weighing citations more heavily if they come from heavily-cited journals (Fersht, 2009). In this regard, it appears comparable to SCImago Journal Rank described above. In a study conducted by Philip M. Davis (2008), a set of medical journals were compared for their Impact Factors (based on raw citation counts) and Eigenfactors (based on source-specific weighting of the citation counts, with a view to reflect the “prestige” of the citations). The two measures were found to be correlated:
At least for medical journals, it does not appear that iterative weighting of journals based on citation counts results in rankings that are significantly different from raw citation counts. Or stated another way, the concepts of popularity (as measured by total citation counts) and prestige (as measured by a weighting mechanism) appear to provide very similar information. (Davis, 2008, concluding para.)
By virtue of being included in the Scopus® database, JRP receives a SCImago Journal Rank (SJR). The SJR indicator (0.027 in 2011) and the related citation analysis for JRP can be found on the SCImago Web site.
A Note on Terminology. In what follows, we will speak of “journal impact factors” when we generally refer to bibliometric measures such as the three measures just introduced; we will, by contrast, speak of the “Journal Impact Factor” (capitalised) when we refer to the specific journal impact factor proposed by Thomson Reuters. Further, since it is conceivable to calculate impact factors (total number of citations in a year divided by the number of citable items published in a preceding time range) not only for journals but also for a research institution or for an individual researcher, we speak of journal impact factors rather than just impact factors.
Journal impact factors are used for research assessment by various institutions and agencies associated with research. Based as they are on citation counts, such measures reflect to a certain degree the level of peer recognition of a publication within the global research community. However, in practice, certain types of institutional use of journal-level measures have raised alarms among researchers, particularly when such measures are used to assess the quality of individual articles and worse, the achievement of individual researchers. There are two sets of issues with such use of bibliometric measures: (a) issues concerning the misuse of journal-level measures at the article-level and (b) issues pertaining to the danger of a one-sided and uncritical use of bibliometric measures in general.
Re: (a). Journal impact factors are aggregate ratios, a single number for all articles published in a journal during a year. By its nature, such a number does not reveal the variability of citation rates across the journal’s articles. That is, it tells us little about an article’s contribution to the journal’s impact factor. Likewise, it tells us nothing about how high the hurdles are for an author to get published in the journal, an aspect that is relevant for assessing the author’s contribution; what it means to get published in a journal depends not on its impact factor but on its editorial priorities, its publication backlog, and the peer-review process adopted for the submission in question (see, e.g., Colquhoun, 2003, 2007; Oh & Lim, 2009; Singaravelan, 2009).
Re: (b). A publication’s impact need not be assessed solely on the basis of scholarly citations. Impact can be understood as actual influence in educational, professional, and everyday research contexts as well, albeit difficult to assess accurately. Sometimes an article is cited for the “wrong reasons,” so to speak, for example, when a flaw is pointed out. Citation rates often have to do more with current academic fads than with the intrinsic merit of what is cited. There are phenomena such as “citation cartels” in which groups of authors mainly cite one another while ignoring other relevant work published. Citation rates may accordingly turn out to be short-lived and a poor measure of the “impact” that an article makes in its field (i.e., the difference it makes for the field’s development). Quite generally, the merit of a research publication, say, in terms of quality, originality/innovativeness, and relevance, to mention just a few essential aspects, need not and frequently does not correspond to its popularity or prestige as measured by raw citation count, much less to average citation rates of the journal volume in which it was published.
Naturally, the validity of such bibliometric research assessment has been a hot topic of debate. An opinion survey of a broad sample of scientists from all over the world showed that the opinion is “neither positive nor negative,” although respondents with larger number of publications tended to have a lower opinion of the Journal Impact Factor (Buela-Casal & Zych, 2012). Despite its shortcomings, there are practical benefits of having a bibliometric measure of quality for a journal. For instance, in institutional environments where the expertise to assess the worth of an article or a journal is in short supply, a bibliometric measure does provide a reasonable indication that is objective enough for institutional use (Mishra & Patel, 2009, commenting on Indian universities).
As with all measurement tools, the merit of journal impact factors is ultimately a question of whether we make good use of them. In this respect, we may be well advised to consider what those involved in the development of journal impact factors, have to say about their proper use and their limitations. As for example Eugene Garfield, the founder and former chairman of The Institute for Scientific Information, the organisation that introduced the Journal Impact Factor, cautioned us:
Thomson Reuters does not depend on the impact factor alone in assessing the usefulness of a journal, and neither should anyone else. The impact factor should not be used without careful attention to the many phenomena that influence citation rates, as for example the average number of references cited in the average article. The impact factor should be used with informed peer review. In the case of academic evaluation for tenure it is sometimes inappropriate to use the impact of the source journal to estimate the expected frequency of a recently published article. Again, the impact factor should be used with informed peer review. Citation frequencies for individual articles are quite varied. (Garfield, 1994, Section “Using the Impact Factor Wisely,” cited in Thomson Reuters Web site)
Notwithstanding the usefulness or otherwise of bibliometric measures, we are deeply concerned to note the pressure such measures put on journal editors to put citation potential before scholarly merit--aspects that may, and often do, conflict. Moreover, there are temptations involved that we need to resist, for instance, the editorial behaviour known as “coercive citation.” Some journal editors have been reported to have requested authors to cite articles from their journals without giving any reasonable argument as to why the authors should do so (Wilhite & Fong, 2012). We lament this, together with all other forms of “Impact Factor engineering” (the expression is from David Tempest; see Thomson Reuters, 2008).
Bibliometric measures have proven to be handy tools in research assessment exercises conducted regularly by research managers and government agencies in various countries, despite the known shortcomings of such measures (see “Not-so-deep impact,” 2005; on their unintended consequences in some countries, see López-Cózar, Ruiz-Pérez, & Jiménez, 2007). One may wonder, for example, whether the periodic research assessment exercises (RAEs) conducted in the United Kingdom over the past 20 years have really helped to improve the country’s higher education and research landscape. It is clear that whatever may be the merit and adverse impacts of the use of bibliographic measures for assessing and financing entire research institutions, the consequences will affect individual authors, whose research careers depend on a positive assessment of both the institutions in which they work and the journals in which they publish. Accordingly it is important for researchers to understand how such measures work and for journal editors to pay attention to them as well, without thereby losing sight of broader notions of quality, relevance, and utilisation. We consider it as one of our tasks as editors to ensure that JRP improves its bibliometric ranking; but we equally--and primarily--consider it to be our task to maintain and improve the substantive quality, relevance, and utilisation of what we publish.
Besides scholarly citations, the impact of a research article or journal can also be seen in various other contexts, such as education, professional development, and research practice. A journal is likely to have impact in all these contexts if it offers material that is considered significant and if it is really accessed and used by the relevant actors in these contexts. Particularly in application-oriented fields of research, good use of publications is not identical with attracting citations but is to be assessed also in terms of practical application by others, whether in the form of inspiration, guidance, or direct application.
Given the breadth of focus of JRP, we are likely to publish a mixture of scholarly and more application-oriented articles. Accordingly, citation-based measures are not likely to be equally relevant for everything we publish. Ideally, a good journal assessment tool ought to be multidimensional, capturing the significance of the material published, ease of access, and proof of use in both scholarly and application contexts. We are not aware of any single assessment tool that captures these.
To be sure, we do not mean to rely on quantitative assessment tools alone. Qualitative assessment (e.g., independent peer review) is certainly conceivable. In fact, the JRP Forum has served us in the past to get informal feedback from among our readers and contributors, and we consider making more systematic use of this opportunity in the future. In the new editorial structure, we expect our Associate Editors to review the journal’s relevance and utilisation periodically and make their assessment available to the editorial team. Finally, we may occasionally ask research theorists or practitioners not associated with JRP to give us their independent assessment of the journal.
In what follows, we first discuss current developments in quantitative assessment tools. Subsequently, we focus on mainly qualitative approaches to enhancing the journal’s quality. These are approaches that we can pursue as part of JRP’s editorial policy without depending on the availability of satisfactory quantitative measures.
The purely bibliometric measurements described in Section 4 are based on citation analysis. As we have seen, such measures say nothing about a host of other quality dimensions a journal ought to have (e.g., editorial policies, peer-review process, ease of access, educational value, and so forth). It will be a complex task to capture several such dimensions into a composite quality index for a journal. But, it is encouraging to note that such multidimensional measures are gradually becoming available.
One such multidimensional measure is the Index Copernicus Value (IC Value or ICV, developed by Index Copernicus International), which is computed for the journals listed in their Index Copernicus Journal Master List. A journal may be registered in this master list free of cost. The ICV is a multidimensional index that combines the following five quality dimensions: (a) scientific quality (58.0%), (b) editorial quality (20.0%), (c) international availability (13.5%), (d) frequency-regularity-stability (5.0%), and (e) technical quality (3.5%) (for the definition and measurement of each dimension, see Index Copernicus, n.d.).
JRP is included in the Index Copernicus Journal Master List (see the JRP page in Index Copernicus; the ICV of the journal is not available freely on this page though, but we can access the ICV indirectly through the publishers’ profiles maintained by Index Copernicus: see Athabasca University Press in the Publishers’ profiles in Index Copernicus). JRP has an Index Copernicus 2010 value of 5.02.
Such an index is not entirely based on hard data; it involves a degree of judgement. That can become contentious because no single standard of judgement can be applied easily across the vast range of publications such an index is expected to cover. It will be interesting to watch the future development of multidimensional quality indices such as ICV.
While there can be different ways of measuring a journal’s quality, our primary interest as editors of JRP is in enhancing it genuinely. We are interested specifically in enhancing the relevance and utilisation of the journal’s contents in the broad arenas mentioned earlier: education, professional development, and research practice. We are sharing our preliminary ideas here for further consideration and development by the editorial team, which now includes Editors, Focus Editors, Associate Editors, Distinguished Reviewers, and Forum Members.
5.2.1. Relevance and Utilisation in Education
JRP is expected to be a source of learning materials for everyone associated with research education, including research students, early-career researchers, and research teachers, supervisors, and facilitators. This aim is linked to the original motivation behind this journal, as stated in the inaugural editorial:
One of the ideals of JRP is to develop itself as an important companion to every research-inclined individual and every research student in the world. There are obvious hurdles in achieving this. However, as a research-like project itself, JRP will remain open to inputs and initiatives that help it move closer to this ideal. (Dash & Ponce, 2005, last paragraph)
Virtually every article published in JRP has a potential to be used for research education and researcher development. We need to take steps to ensure that this potential is realised. “Research education” happens to be one of the six focus areas for JRP. Looking at the state of research education around the world today, much needs to be done in preparing researchers for their role in society. Although all the six focus areas have a link with research education and researcher development, we need to publish more articles addressing specific issues relevant to research education. The special issue on research assistantships recently proposed by Michelle K. McGinn and Ewelina K. Niemczyk (to be published in 2013) is a step in the right direction. We also need to build cooperative links with important centres of research education around the world.
5.2.2. Relevance and Utilisation in Professional Development
Of the six focus areas of this journal, “research applications” links us to professional fields which use research procedures in their work. Reflective professionals from such fields are likely to be interested in this journal. In fact, we do have indications of such interest from the following fields: (a) agriculture, (b) accounting, (c) architecture and built environment, (d) art and design, (e) chaplaincy, (f) community development, (g) community health, (h) disaster management, (i) education, (j) game design, (k) information technology, (l) innovation management, (m) knowledge management, (n) leadership development, (o) management, (p) marketing, (q) natural resource management, (r) nursing, (s) occupational health, (t) programme evaluation, (u) public relations, (v) rural development, (w) social work, (x) sports and exercise, (y) tourism, and (z) urban development. These fields were gleaned from a Google Scholar search on “Journal of Research Practice,” which yielded not only the articles published in JRP but also around 450 other titles (including journal articles, books, book chapters, theses, course outlines, etc., which contain the phrase “Journal of Research Practice”).
This hints at the virtually untapped potential of JRP in contributing towards professional development in a wide range of professional fields. These are fields which involve some forms of research in their everyday performances as well as in their attempt at long-term enhancement of professional competencies. We need to establish vigorous links with these and other professional communities around the world, aiming to raise their interest in “research practice” as a subject of reflection and exchange that can enhance their profile of competencies or help them in opening up new areas of professional development. This is a task for all our editorial members connected with professional fields. Again, all six of our focus areas have a potential link with professional development. But viewing professions as potential spaces for research application, we would expect a degree of creative guidance from our Focus Editors assigned to two focus areas related directly to the professions: “research applications” and “research spaces.”
5.2.3. Relevance and Utilisation in Research Practice
The very raison d’être of JRP is to facilitate peer learning among researchers, irrespective of their disciplinary backgrounds and institutional settings, and enhance research practice in various ways. There are few other journals with a comparable focus and scope, as it appears from a study by Kueffer and colleagues (Kueffer, Hirsch Hadorn, Bammer, van Kerkhoff, & Pohl, 2007). These authors support the view that “adequately addressing issues in the knowledge society requires a major transformation of research, away from disengaged disciplines toward socially engaged, transdisciplinary approaches” (Kueffer et al., 2007, p. 22). However, they also warn us: “previous attempts to establish broad-scope practice-oriented transdisciplinary-focus journals . . . do not seem to have been particularly successful” (p. 26). We believe insufficient engagement with the actual experiences of researchers and professionals could be a key reason behind such lack of success reported by these authors. In this regard, JRP has established the right benchmark by privileging the voice of the researcher who is on the ground that is fraught with practical challenges as well as opportunities for experimentation, rethinking, and innovation. Purely abstract contemplations are seldom considered for publication in JRP, unless a potential connection can be shown with the practical challenges experienced by researchers or professionals.
Accordingly, JRP articles have a potential relevance to all those researchers and professionals who are reflecting on their experiences and challenges, and looking for alternative ways of engaging with their contexts, going beyond the conventions of their fields, improving their results/outcomes, or even starting new lines of work. They are our potential readers and among them can we find our future contributors. It then becomes a task for the editorial team to establish a connection with reflective researchers and professionals. Using the Web-based publication infrastructure of our journal, we can explore various forms of interaction with this target group.
Given the cross-cutting connections JRP is able to establish among research areas and professional fields, this journal can be a vehicle for building unusual collaborative networks, and therefore, potentially open up new spaces for innovative research. This potential can be realised if our readers, authors, and other contributors take interest in each other’s work and develop collaborative projects, reflective accounts of which would ultimately appear on JRP pages.
We now present a summary of our proposed measures to enhance the relevance and utilisation of JRP.
(a) As Editors, we cannot disregard bibliometric methods of journal assessment (e.g., Impact Factor, SCImago Journal Rank), despite their limitations. However, we must avoid the temptation of “Impact Factor engineering.” A higher citation count should arise from a genuine increase in the utilisation of the journal in various communities and contexts.
(b) We ought to look out for new multidimensional measures of quality, which go beyond citation analysis. As and when such measures become available, we ought to review our performance on different dimensions and determine measures to upgrade the journal, if necessary.
(c) We need to take steps to ensure that the educational potential of JRP is realised. We should publish more articles and special issues addressing topics relevant to research education.
(d) We also need to build cooperative links with important centres of research education around the world.
(e) We need to establish links with various professional communities around the world, aiming to raise their interest in “research practice” as a subject of reflection and exchange, towards securing professional development.
(f) We should establish a direct connection with reflective researchers and professionals. Using our Web-based publication infrastructure, we can develop various forms of interaction with this target group, among whom we can find our potential users and contributors.
(g) JRP can be a vehicle for building collaborative networks among individuals who “meet” each other through the journal. This could open up new spaces for innovative research.
We invite you, our readers and contributors along with the entire editorial team, to assist us in making sure the above list will not remain a mere wish list.
We express our gratitude to the following members of the JRP Forum: (a) Amer Saleem Khan, who drew our attention to “coerced citation” and (b) Sushil Kumar Pare, who drew our attention to “Eigenfactor,” through their recent postings in the JRP Forum.
Bergstrom, C. (2007, May). Eigenfactor: Measuring the value and prestige of scholarly journals. College & Research Libraries News, 68(5), 314-316. Retrieved from http://crln.acrl.org/content/68/5.toc
Buela-Casal, G., & Zych, I. (2012). What do the scientists think about the impact factor? Scientometrics, Online First.
Colquhoun, D. (2003, May 29). Challenging the tyranny of impact factors [Correspondence]. Nature, 423(6939), 479. Retrieved from http://www.nature.com/nature/journal/v423/n6939/
Colquhoun, D. (2007). How to get good science. Physiology News, 69, 12-14. Retrieved from http://www.dcscience.net/colquhoun-goodscience-jp-version-2007.pdf
Dash, D. P., & Ponce, H. R. (2005). Journey of research practice. Journal of Research Practice, 1(1), Article E1. Retrieved from http://jrp.icaap.org/index.php/jrp/article/view/10/21
Davis, P. M. (2008). Eigenfactor: Does the principle of repeated improvement result in better journal impact estimates than raw citation counts? Journal of the American Society for Information Science and Technology, 59(13), 2186-2188. Retrieved from http://arxiv.org/pdf/0807.2678
Elsevier. (2012). Scopus in details: What does it cover? Retrieved from http://www.info.sciverse.com/scopus/scopus-in-detail/facts
Fersht, A. (2009). The most influential journals: Impact factor and eigenfactor. Proceedings of the National Academy of Sciences USA (PNAS), 106(17), 6883-6884. Retrieved from http://www.pnas.org/content/106/17/6883.full
Garfield, E. (1994, June 20). The impact factor. Current Contents, 25, 3-7. Retrieved from http://thomsonreuters.com/products_services/science/free/essays/impact_factor/
Hubbard, S. C., & McVeigh, M. E. (2011). Casting a wide net: The Journal Impact Factor numerator. Learned Publishing, 24, 133-137. Retrieved from http://dx.doi.org/10.1087/20110208
Index Copernicus. (n.d.). IndexCopernicus evaluation methodology 2012. Retrieved from http://journals.indexcopernicus.com/info.php
Johansson, A.-L., & Wahlgren, I. (2008). The one stop shop to open access journals--DOAJ. ScieCom Info, 4(4). Retrieved from http://www.sciecom.org/ojs/index.php/sciecominfo/article/viewFile/1449/1234
Kueffer, C., Hirsch Hadorn, G., Bammer, G., van Kerkhoff, L., & Pohl, C. (2007). Towards a publication culture in transdisciplinary research. GAIA, 16(1), 22-26. Retrieved from http://www.geobot.umnw.ethz.ch/publications/PDF_publications/804.pdf
López-Cózar, E. D., Ruiz-Pérez, R., & Jiménez, E. (2007, April 20). Impact of the impact factor in Spain [Letter to the editor]. BMJ, 334(7593). Retrieved from http://www.bmj.com/rapid-response/2011/11/01/impact-impact-factor-spain
Mishra, B. K., & Patel, S. (2009, February). On the controversy of impact factor [Correspondence]. Current Science, 96(3), 324. Retrieved from http://cs-test.ias.ac.in/cs/php/toc.php?vol=096&issue=03
Not-so-deep impact [Editorial]. (2005, June 23). Nature, 435, 1003-1004. Retrieved from http://www.nature.com/nature/journal/v435/n7045/full/4351003b.html
Oh, H. C., & Lim, J. F. Y. (2009). Is the journal impact factor a valid indicator of scientific value? Singapore Medical Journal, 50(8), 749-751. Retrieved from http://smj.sma.org.sg/smjmainpages/5008main.html
Scimago Research Group. (2007). Description of SCImago Journal Rank indicator. Retrieved from http://www.scimagojr.com/SCImagoJournalRank.pdf
Singaravelan, N. (2009, February). Looking beyond number of publications and journal impact factor [Correspondence]. Current Science, 96(3), 327-328. Retrieved from http://cs-test.ias.ac.in/cs/php/toc.php?vol=096&issue=03
Thomson Reuters. (2008, June 6). Thomson Reuters speaks with David Tempest, Elsevier. Retrieved from http://community.thomsonreuters.com/t5/Citation-Impact-Center/Thomson-Reuters-Speaks-with-David-Tempest-Elsevier/ba-p/718
Thomson Reuters. (2012). The Thomson Reuters journal selection process. Retrieved from http://thomsonreuters.com/products_services/science/free/essays/journal_selection_process/
Ulrich, W., & Dash, D. P. (2011). Introducing a concept hierarchy for the Journal of Research Practice. Journal of Research Practice, 7(2), Article E2. Retrieved from http://jrp.icaap.org/index.php/jrp/article/view/279/239
Wilhite, A. W., & Fong, E. A. (2012, February 3). Coercive citation in academic publishing. Science, 335(6068), 542-543.
Published 1 May 2012
Copyright © 2012 Journal of Research Practice and the authors