Main Article:
Forms and Levels of Integration: Evaluation of an Interdisciplinary Team-Building Project
Team science models are frequently promoted as the best way to study complex societal and environmental problems. Despite increasing popularity, there is relatively little research on the processes and mechanisms that facilitate the emergence of integration of interdisciplinary teams. This article evaluates a suite of recent team-building and grant-writing activities designed to address water management in the Western U.S. We use qualitative methods to document the emergence of integrative capacity at the individual, group, and institutional levels, with particular attention to the role of graduate students and non-academic practitioners in a team science planning project. Our findings highlight the importance of social integration as a basis for conceptual integration and an ability to relate these concepts to real-world problems. The findings also demonstrate the value of qualitative evaluation measures of team readiness, capacity, and intellectual outputs to complement conventional evaluation indicators that rely on quantitative scientific outputs, particularly for team science projects still in the planning stages.
Index Terms: science of team science; interdisciplinary teams; collaborative research; social integration; conceptual integration; research evaluation
Suggested Citation: Armstrong, A., & Jackson-Smith, D. (2013). Forms and levels of integration: Evaluation of an interdisciplinary team-building project. Journal of Research Practice, 9(1), Article M1. Retrieved from http://jrp.icaap.org/index.php/jrp/article/view/335/297
Social, environmental, and health problems are not effectively solved within single-discipline approaches. Major funding institutions are increasingly supporting integrative, cross-disciplinary research collaborations (National Academy of Sciences [NAS], 2005). Collaborative and transdisciplinary “team science” models are frequently promoted as the best way to tackle complex problems. Team science collaborations typically involve groups of researchers from diverse disciplines and engaged non-academic stakeholders. A “Science of Team Science” literature has emerged to explore the factors that influence the success of collaborative science projects (Stokols, Hall, Taylor, & Moser, 2008).
This article presents an evaluation of an interdisciplinary team-building project at Utah State University, United States that was designed to expand disciplinary scientists’ capacity to work collaboratively in formulating research questions and methods that address water management challenges in the Western U.S. Our in-depth, qualitative case study provides insights into the processes that support the emergence of different types of science team integration, with particular attention to the role of graduate students and non-academic practitioners within team science efforts.
Scholars in the Science of Team Science (SciTS) field have developed a conceptual model that highlights the different phases involved in building successful team science collaborations, specifying the antecedent, process, and outcome factors (Stokols et al., 2003). Antecedent factors recognize the importance of training, prior experience, and institutional contexts to the ability of team members to be able to work in a team setting. These factors can influence the “readiness” of a team to succeed (Hall et al., 2008). Process factors reflect the structured activities used to facilitate or improve interpersonal relationships among team science members who are expected to collaborate (Stokols et al. 2008). Outcomes of team science processes can be cognitive and emotional (including mutual understanding, feelings of trust) or include tangible indicators of scientific productivity, such as publications and successful external granting.
To facilitate innovation and the emergence of new interdisciplinary ideas, effective team science processes are designed to overcome barriers or deficiencies in antecedent conditions, to build the capacity of individuals and the group, and to encourage greater integration among team members. Most academic scientific institutions are organized to reward disciplinary research and have incentive systems that discourage participation in team science projects (NAS, 2005). Language barriers associated with differences in disciplinary epistemologies and research methodologies can also frustrate team integration (Eigenbrode et al., 2007), though group processes that familiarize team members with alternative term uses can be a creative force for innovation (Repko, 2008, p. 272). Team members’ expertise should “fit” the nature of the research topic and complement the expertise represented within the team (Klein, 2008). Scientists most engaged with team science research tend not to be risk averse, are comfortable with ambiguity and complexity, are receptive and tolerant to new or diverse ideas, have a passion for learning, and are capable of mastering basic knowledge in many disciplines (Repko, 2008). Teams are deemed most successful when “scientific areas are merging and/or ripe for collaborative study,” when team members have overlapping study subjects (e.g., water), and when teams are able to agree on common concepts and research questions (Hays, 2008, p. S193).
Given what we know about the factors linked to successful team science, approaches to evaluating or assessing team science projects remain diverse and inconsistent. Conventional assessments of science programs rely on measures of outputs such as the impact and number of publications, patents, or citations (Stokols et al., 2008). While these metrics are not irrelevant to assessing team science projects, they tend to reinforce disciplinary perspectives and goals, and are not adequate for comparing transdisciplinary teams with traditional research programs (Hall, et al., 2012). The mechanisms through which team research enhances scientific quality are also not well understood (Falk-Kresinski et al., 2011), and therefore criteria for evaluating this process are not fully developed.
A tailored approach to evaluation considers the unique qualities and goals of each collaborative research effort (Frodeman, Klein, & Mitcham, 2010, p. 317; Repko, 2008). One of the most well developed evaluation efforts has been the National Cancer Institute’s Evaluation of Large Initiatives project (Trochim, Marcus, Masse, Moser, & Weld, 2008), which used logic models to link program objectives to anticipated research outcomes, and then compared these to actual outcomes. Others recognized the need to use different evaluation metrics and methods at each stage of the team science process in order to capture distinctive project outcomes, allowing evaluators to place more or less emphasis across project sub-components (Falk-Kresinski et al., 2011; Stokols et al., 2008). The Research on Academic Research (RoAR) model developed to evaluate a health science collaborative research project in British Columbia is noteworthy for recognizing that distinctive team science outcomes necessarily occur at different levels of analysis—among individual participants, at the organizational scale, and within broader institutional or cultural contexts (Best et al., 2009; Norman, Best, Mortimer, Huerta, & Buchan, 2011).
We build on these evaluation approaches in this study. Specifically, we utilize evaluation metrics that reflect (a) the specific objectives for this project, (b) the unique outcomes one would expect from the early planning stages of a team science research project, and (c) outcomes that emerge at different levels of analysis. As outlined in more detail below, the project’s main objective was to improve the capacity of an interdisciplinary science team to collaborate on the writing of a scientific research plan. The outcomes of this type of effort include achieving high levels of team integration and development of a synthetic scientific research plan that would not have occurred without the collaborative process. These outcomes are assessed at the individual scale, at the level of the team, and in the larger university and water management context.
Best et al. (2009) recognized that research team integration often occurs at the interfaces among these three scales. For example, social processes and interactions within the team or organization may contribute to, or limit, individual interests or capacities to do team-oriented projects (Beyerlein, Freedman, McGee, Moran, & Beyerlein, 2003). Individual researchers are also positioned within the larger academic system, which structures promotion processes and can incentivize or discourage interdisciplinarity at certain career stages (NAS, 2005). Lastly, the interface between the science team and the institutional context is illustrated in the ways that research groups may or may not be formalized within universities, or how networks of researchers can grow to become a community of practice (Wenger & Snyder, 2000).
Team science projects often aspire to accomplish more than just better collaboration among existing university scientists. In most projects, as in our case study, a parallel goal is to provide opportunities for graduate students to participate in team science projects—both to gain from their perspectives and to help train a “next generation” of scientists better equipped to participate in collaborative research. To date, most evaluations of interdisciplinary training of graduate students have focused on highly structured programs, such as the National Science Foundation’s (NSF) Integrative Graduate Education and Research Traineeship (IGERT) program (NSF, 2012). However, many graduate students engaged in interdisciplinary projects are not affiliated with an IGERT program. Without the dedicated funding and supportive infrastructure of programs such as IGERT, graduate students seeking interdisciplinary science experience struggle to balance their collaborative work against institutional and departmental advising systems that hold students accountable to traditional disciplinary degree requirements (Ryser, Halseth, & Thien, 2009; for an exception, see Neuhauser, Richardson, Mackenzie, & Minkler, 2007). Regardless of form, the small published literature and anecdotal stories suggest that graduate student experience in team science research often does not provide opportunities for students to learn how to formulate tractable interdisciplinary research questions. Graduate students are often engaged in the research process only after the core focus for the project is formulated, funded, and collaborators are identified. Without including problem formulation in graduate student training, integrative research opportunities may resemble routine research endeavors (NAS, 2005). In the analysis below, we explore the degree to which the case study project accomplished its parallel goal of training graduate students to be future team science leaders.
Similarly, a growing number of interdisciplinary projects are becoming “transdisciplinary” through high levels of collaboration with non-academic stakeholders and decision-makers (Klein, 2008). The core idea is that engaging the end-users of scientific information in the design and implementation of science can ensure that research questions are relevant to real world problems. As with graduate students, a range of methods and approaches are used to incorporate non-academics into team science projects (Hadorn et al., 2008). The evaluation approach we use below considers whether or not the project was successful in achieving integration between scientists and practitioners, and the degree to which input from water managers shaped the collaborative research plan.
Our case study focuses on a Utah State University (USU) group that received a Category 1 Water Sustainability and Climate (C1-WSC) program grant from the U.S. National Science Foundation (NSF) during 2010-2012. The core objectives of the C1-WSC project were “to conduct a set of research and team-building activities leading to the development of an applied scientific research plan to study complex water systems in the transitioning, irrigated landscapes of the Intermountain West.” The Intermountain West is a region of the Western U.S. where human settlement and built water infrastructure dramatically alter the hydrology and related ecology within irrigated valleys. Current biophysical and social science approaches to understanding water systems in this landscape are disconnected from one another and from on-the-ground water management behaviors and policies. The WSC effort was designed to build the capacity of participating scientists to work in a truly interdisciplinary science team and to engage stakeholders and water managers in the framing of research problems and development of methodological approaches. The team that wrote the original grant was led by a sociologist, but included economists, policy scientists, water engineers, hydrologists, and ecologists.
The leaders of the C1-WSC anticipated three major outcomes from the process: (1) improved individual understanding of scientific knowledge and methodologies in other disciplines, (2) improved capacity of the team members to function as an integrated group (including meaningful involvement of water managers), and (3) development of a new interdisciplinary research plan that would lead to improved understanding of sustainable water management in this complex, coupled system.
USU is a land-grant research institution, with disciplinary research on water resources conducted in many departments. In the decade preceding our case study, funds from central administration also supported a “Water Initiative,” to promote increased collaboration of water researchers. Relationships built through the Water Initiative kindled interest among diverse scientists in seeking more sustained funding for large-scale interdisciplinary water research efforts. One such group wrote the successful NSF C1-WSC grant that supported the team science effort evaluated here.
The C1-WSC grant was awarded and work began in the fall of 2010 and extended through August 2012, with the majority of collaboration activities concentrated in the first year. The writing of the original grant, team membership, and all activities and project work were open to any interested USU faculty, graduate students, and non-academic water managers and stakeholders. The core project activities consisted of two types of interactive events—weekly research workshops and methodological retreats. The weekly workshops consisted of seminar-style research presentations related to recent or ongoing water research. Attendees included all of the project partners—typically including 12 to 18 faculty, four to eight graduate students, and four to eight stakeholders and water managers. Team members presented material for 30 minutes each, followed by at least 30 minutes of questions and discussion. In total, 15 faculty across a range of disciplines presented over 5 months. Two non-academic partners also presented on their work to manage water in agricultural and municipal settings. While several graduate students were regular attendees, none presented. A few workshop sessions were also devoted to brainstorming research site options and focusing the collaborative research questions. Food and free parking was provided. Presentations and discussions were videotaped and archived on a project website to allow people to see sessions that they may have missed in person (see http://wsc.usu.edu/).
The methodological retreats of C1-WSC activities were held on four weekends throughout the winter and spring of 2010-2011. The first was a half-day interdisciplinary “toolbox” workshop facilitated by University of Idaho experts that encouraged team members to identify linguistic and philosophical differences between their various disciplinary approaches (Eigenbrode et al., 2007). The next three retreats involved all-day, off-campus events designed for in-depth presentations and discussion on methodological topics. The first focused on social science research methodology and was designed to inform non-social science collaborators about the theoretical, methodological, and substantive questions that are employed by social scientists in their water resources research. The second retreat focused on potential research locations for integrated water research. The third retreat involved small-group and full-group brainstorming sessions designed to identify the key social, ecological, and hydrologic components and their linkages within of coupled water systems. Retreats were mainly attended by faculty and graduate students, although all were invited. In sum, the WSC project team-building activities provided over 40 hours of structured time for interactions between team scientist and practitioners.
The second stage of WSC team collaboration involved writing a formal proposal for a much larger NSF Category 2 WSC (C2-WSC) grant. The previous year’s work served as the basis for the new grant proposal, and most of the writing was concentrated between August and October, 2011. The new proposal sought 5 million dollars over 5 years to address integrative scientific research questions related to coupled water system processes with a major focus on developing and implementing an integrated socio-hydro-ecological computational model. Within this model, researchers proposed to integrate individual and institutional social processes with complex hydrologic and ecological models to better understand how human-altered hydrologic flows affect aquatic biota, habitats, and ecosystem services, and in turn how water infrastructure investments can be designed to meet long-term and diverse user goals. With the exception of one new junior faculty member, the C2-WSC grant writing team consisted of researchers and non-academic participants who were engaged in the C1-WSC activities. Proposal writing efforts were organized around four subteams: human, ecology, hydrology, and integration. The first three subteams each elaborated research questions and proposed methods to capture core subcomponents of the coupled water system, while the integration subteam was tasked with developing a plan to systematically link subcomponents in an integrated model. The integration subteam also worked to ensure that the proposed project was responsive to the needs of water managers and stakeholders. The proposal writing process provided researchers with many hours of structured interaction.
While team members reported high degrees of satisfaction with the group process and the proposed scientific research plan, the C2-WSC proposal was not recommended for NSF funding. Meanwhile, key elements of the proposal did provide a foundation for a different, successful large team science grant proposal. The relationships built among team scientists and water stakeholders continue to shape applied science and water management efforts in the region.
We conducted an evaluation of the WSC effort for two reasons. First, there are few case studies of team science formation within the environmental sciences, even though team science is an increasingly common approach within this field. Second, we were also interested in understanding the impact of our group activities and processes on the emergence of different types of integration among science team members. As such, results can provide guidance to future efforts to build team science efforts.
We developed criteria and standards that aligned with project objectives (Yarbrough, Shulha, Hopson, & Caruthers, 2011). Evaluation criteria were also specified at three levels of analysis, following an approach similar to Best et al.’s (2009) RoAR logic model: individual level, research team level, and university/systemic level. For each combination of project objective and level of analysis, we identified indicators of success, based on the literature presented above (see Table 1 for a summary). The matrix of evaluation criteria in Table 1 also highlights the barriers and opportunities for team integration at each of these scales. At the individual scale, social integration with other team members is indicated by the presence of new social and intellectual interactions (Klein, 2006). Integration across disciplines and the production of knowledge based on this integration are inherently social because scientists need to interact with one another to draw upon each other’s expertise. As these interactions foster an emergence of common understandings, integration is attained at the group level (Klein, 2008). When scientific approaches are integrated, similar understandings of the relationships and dynamics of a holistic system should theoretically occur among participants (conceptual integration), and this should be reflected in the creation of integrated products (in this case, a new scientific research plan), in which “the interdisciplinary whole is more than the sum of its disciplinary parts” (Boix Mansilla, Feller, & Gardner, 2006, p.16). We also developed criteria that were tailored to our case study context, such as “team collaborations are encouraged beyond the WSC project.”
Table 1. Evaluation Criteria for Three Project Objectives, at Three Levels of Analysis
We used semi-structured interview data to evaluate these team science activities. This rich, qualitative information included insights on participants’ involvement, impressions, criticisms, and potential use of WSC experiences in the future. Qualitative data can be critical for team science evaluations when the project is new and when seeking to understand patterns of researcher behavior (Norman et al., 2011; Trochim et al., 2008). As the USU team had only worked together for 18 months at the time of evaluation, interviews were more appropriate than quantitative metrics. Qualitative data are also well suited to identify causal processes and mechanisms and capture other hard-to-measure outcomes, such as changes in research capacity (NAS, 2005), effectiveness of project management (Hall et al., 2012), or changes in research philosophies (Stokols, Harvey, Gress, Fuqua, & Phillips, 2005). Our qualitative approach emphasizes the participants’ voices, and adds validity to our observations and analysis.
A downside to in-depth, qualitative understanding is that causal mechanisms of research experiences are more difficult to pinpoint than when using quantitative methods. Therefore, our study is limited to an in-depth exploration of how team members’ impressions are linked to institutional and group processes, and results should serve as the basis for more systematic quantitative assessments of underlying causal relationships. Interviews were conducted in April and May 2012. In total, 24 WSC team members participated in the research evaluation. Study participants consisted of 11 of the 14 academic researchers who co-authored the Category 2 proposal. Those who were not interviewed included the WSC principal investigator (this paper’s second author) and two who were constrained by research-related travel. The few participants in Category 1 activities who did not continue with Category 2 grant writing also were not interviewed. In addition to faculty, six interviews were completed with non-academic WSC team participants representing two environmental organizations, one federal agency, one irrigation company, one municipal representative, and one agricultural organization representative. Seven graduate students who attended C1-WSC workshops or retreats were also interviewed. All but three interviews were conducted prior to receiving notification that the C2-WSC proposal was not funded. The three remaining interviewees were not asked to reflect on this outcome, although they mentioned the decision without prompting.
Interviews ranged between 30 and 60 minutes in duration, with most lasting approximately 40 minutes. The interview instrument was designed to obtain feedback from participants on the WSC collaborative process, with specific questions about their motivations to participate, the influence of group interactions and communication on their levels of involvement, and the influence of institutional and language barriers. For example, participants were asked, “What are things that you’ve learned from participating in this team?” and, “Do you think the WSC group process allowed us to identify scientifically important research questions?” Interviewees reflected on the C2-WSC proposal, including the quality of the scientific questions and the perceived level of team integration. Lastly, interviewees discussed the participation of non-academics and graduate students. Transcripts were systematically coded for the presence of key themes, as well as insights that emerged from the responses upon review.
To encourage thoughtful and open responses, the authors assured anonymity in the analysis and dissemination of the results. Accordingly, interview quotations presented below are anonymous and do not provide information on respondent characteristics unless these are important to the conclusions.
The lead author of this paper, who conducted all of the interviews and was the only person to read the interview transcripts for confidentiality purposes, was not affiliated with USU at the time of the Category 1 activities and only attended one WSC event prior to her involvement with Category 2 writing efforts. Additionally, three C1-WSC members reviewed the manuscript to validate our interpretations of the interview data.
In the sections that follow, we present the evaluation findings. We organize these findings in accordance with the levels of analysis outlined in Table 1: individual, research team, and university/systemic.
The C1-WSC process provided numerous structured opportunities for team members to become more familiar with scientific work from other disciplines. Indeed, the bulk of the first 5 months of team-building activities involved attendance at workshops where individuals presented their past water-related research. Team members’ interview comments suggested that this exercise was successful in improving their understanding and appreciation for the theoretical, methodological, and empirical contributions of other disciplines.
Most participating scientists noted that the formal C1-WSC activities expanded their personal and professional linkages with other team members. The weekly workshops and methodological retreats were designed to be casual and to promote a more relaxed form of interaction that contrasted formal modes of academic presentation. One faculty member described it as, “people are eating jelly beans and M&Ms on a Friday afternoon; it was kind of loose.” Interviewees emphasized that this did not mean the material presented at workshops was not rigorous, or that discussion did not challenge participants to think in new dimensions, but that the C1-WSC workshops were different from their typical modes of interacting. As one faculty member said, “WSC made me aware of the research that was already going on here at USU, and the potential areas of research that already existed . . .” These social or professional interactions were a direct outgrowth of new interactions among team members.
Importantly, the C1-WSC effort was organized using an open-invitation format. This format influenced the antecedent skills and experiences that individuals brought to the team project. Efforts were made to ensure a broad cross-section of disciplinary expertise, though team members who participated in WSC activities self-selected over time based on their personal interest, availability, and perceived synergies with their own research. A few of the original C1-WSC presenters were more comfortable working in traditional disciplinary roles and eventually left the team. This meant that individuals who remained with the group committed to working as a team. While many of the team members reported having interests and experience in interdisciplinary research before joining the C1-WSC project, they appreciated the structured interaction opportunities provided by the project.
In contrast, the C2-WSC proposal-writing team was comprised of the scientists who were most “willing to play,” not necessarily those whose areas of research specialties were most appropriate for the proposed project. Late in the process the team identified some gaps in technical expertise that were required to address key aspects of the emerging scientific research plan (e.g., groundwater modeling). The team recognized this gap and invited four non-USU scientists to join the C2-WSC proposal-writing team. However, because they had missed the yearlong collaborative team building exercises, these scientists were much less socially or conceptually integrated into team discussions.
While generally perceived as valuable, most faculty members who attended weekly workshops indicated that the duration of this activity (nearly 5 months) exceeded their comfort level. They provided a variety of reasons as to why they grew impatient with the workshops:
I am somebody who is very methodical in what I do and always about being time efficient, particularly in this phase of my life. So I found a lot of those meetings, while they were beneficial, the cost benefit ratio wasn’t low enough.
The only thing that I could see doing [differently] is trying to do less hand holding and more doing—hand holding in the sense that we all need to feel comfortable together in the same room and that’s important; I think I certainly didn’t need the extent of it that was done.
The researchers who expressed the highest level of frustration with the duration of weekly workshops tended to be junior faculty who were facing the greatest pressures related to promotion and tenure. Many team members in addition to junior faculty believed that a downside the structure of C1-WSC activities was that there were not enough immediate written products of the activities.
It appears that the C1-WSC project built on a pre-existing level of comfort with interdisciplinary research for most faculty. When asked what they learned from workshops and retreats, researchers pointed to new methodologies or interesting approaches from other disciplines; however, they rarely indicated that interdisciplinary teamwork was an entirely new endeavor. Junior faculty in particular expressed a greater familiarity with interdisciplinary research and teamwork, which they attributed to their graduate training, but it made them less focused on cross-disciplinary presentation and discussion, and more anxious to “get on with the writing.”
On the other hand, the C1 and C2-WSC activities were completely new for many graduate students and most of the non-academic participants. Graduate students attended the workshops because they were interested in the substance of the research and because they were generally interested in interdisciplinary research: “I was curious to see what it [interdisciplinarity] involved, because you always come across people who are saying they want to be interdisciplinary but then it always seems like it’s kind of hard to actually get there.” The weekly workshop discussions were intimidating for some, particularly those working within fields with more senior faculty participants. Graduate students found more opportunities to contribute in the small group discussions on the weekend retreats. One graduate student noted, “whenever we actually break up into smaller groups you get more insight because some people would not talk unless they are asked, ‘What do you think?’”
Graduate students, especially doctoral students, were cognizant of the increasing importance of interdisciplinary collaborations, particularly in the environmental and water sciences. One student described his intellectual transformation, in which the WSC discussions played a role, from an engineer who “loves science and theories and solving equations” to a student who thinks about water resource problems with more complexity—something that he believed would further his career. Graduate student involvement also inspired two masters students to apply to formal interdisciplinary doctoral programs at other universities. Where faculty expanded their professional networks, graduate students expanded their interests in interdisciplinarity.
At the research team scale, we evaluated the degree to which team members developed a strong commitment to the group effort and a shared conceptual understanding of integrated water research. Initially, the interviews suggested that the C1-WSC activities were effective at generating strong social ties and group identity. This social integration was a direct outgrowth of many hours spent sharing their research backgrounds and discussing ideas for new areas of research. Within these shared hours, familiarity with one another grew, and it was appreciated:
It felt like we had really grown and we had really come together and I think [it was] as important as anything. We trusted each other and we knew each other and we could laugh at each other and we could criticize each other and so whether or not we had a coherent research theme is one thing—the thing that’s probably really important is just the level of comfort.
We also looked for evidence of how social integration contributed to eventual conceptual integration around an integrated scientific research plan. Here, the evidence was mixed. Overall, participants felt that social bonds within the research team were essential preconditions for progress towards conceptual cohesion, and eventual integration. In the challenging weeks of C2-WSC proposal writing, there were many structured opportunities for faculty to debate and formulate an approach to studying complex water systems. Within these discussions and written development of research approaches, many ideas were offered, critiqued, or outright rejected among group members. One researcher described an example of an attempted contribution:
There were many times that I would go to meetings and think that I had an idea for a direction to go with the WSC proposal, or something that I found, something that I thought was very interesting and I would pitch it in a meeting or mention it and it would just fall flat. And then I would come back and just try to figure out what I said that was just not quite sparking the interest of others . . . I never looked at it as though someone was flat out rejecting the [my] idea, it was more just of a “alright that might be good but you’ve got to come back and tweak it and sell it to me.” I mean that may be a salable point to someone in the [my] field, but it’s not outside of the field.
This researcher wasn’t dejected from this experience, nor did he take criticism of his ideas personally: “it’s not that disheartening . . .” Rather, he accepted critiques with the understanding that team members valued him, but perhaps not the merit of his suggestion. In this and other instances of coordinated team efforts, familiarity and collegiality among researchers were essential for pushing one another intellectually, and thus formulating new, integrated ideas. Mutual personal understandings reduced the sting of criticism or the sense that one’s work is misunderstood, thus improving overall social freedom to question and innovate.
Underlying the processes leading up to conceptual integration were the ways in which team members used certain terms or language. As the C2-WSC writing effort intensified, several team members reported frustration at the continued disagreements about what people meant by the term “modeling.” This was evidenced by quantitative modelers on the team who, when attempting to bring together social, ecological, and hydrologic models in a computational platform, critically examined the empirical approaches to the idea of a “model” among their social science and ecological science colleagues. Two researchers independently reflected on the same interaction:
I was working with [a subteam], but then [a different subteam] called me and said, “All of the [modeling] work in our subteam is going to you and we don’t understand what you’re doing.” They said that—just straight up. We had a couple different meetings and kind of talked it out. I guess being upfront about questions instead of assuming that they would go away was refreshing.
I cornered [the previous researcher] and I said, “How are you guys planning on [modeling] this?” Because everything that they had written, what initially was hypothesis driven, it was all science oriented. It didn’t fit into modeling anything. So there was a huge lack of integration . . .
This interaction exemplifies that team members respected each other’s abilities and trusted that criticism from their counterparts was aimed at the merit of their ideas, and not their overall intellect. The clear and honest conversation between the modelers allowed researchers to recognize and work through this linguistic and conceptual difference. The “huge lack of integration” felt by one researcher could have lingered into computational modeling stages, when much more time and resources would be needed to clarify this gap. The opportunity to communicate in both professional and personal settings fostered approachability and more in-depth integration of research ideas. These products, though not necessarily tangible, are the essence of team integration.
Decisions about how to structure team efforts also influenced communication flows and the ability to generate an integrated plan. As previously noted, the research team initially organized into social, ecological, hydrologic, and integration subteams. The last subteam shared members with the three topical subteams to improve connections across the smaller groups. However, many team members reported that the greatest intellectual cohesion occurred within, not across, the subteams. For example, ecology subteam members spoke of a small group meeting early in the C2-WSC writing process (but after C1-WSC activities were complete) in which they were illustrating the eco-hydrologic system in great detail on a blackboard and were sharing in-depth, crosscutting ideas with one another. At the end of this meeting, the group felt like they had made strides in understanding their component of the system, but when they shared this with the larger team, it was clear that the larger team didn’t quite know what to do with this breakthrough, or how to incorporate these fresh ideas into the greater, integrated vision.
We suspect that in this and similar cases, the problem was not communication or language barriers within the group, but rather lingering disciplinary differences in preferred epistemologies and methods that limited understanding across the subteams. In hindsight, one researcher believed that some of these disciplinary barriers could have been overcome with more shared field experiences—something that could have been pursued in the C1-WSC activities, but was not due to time and budget constraints.
Overall, respondents believed that the scientific research plan that emerged had a greater coherence than would have been possible without the C1-WSC activities. One comment captured this view:
I mean that whole process, I know it seemed slow and daunting and difficult but I don’t think any of that stuff would have come up otherwise. I think if we had tried to write that Category 2 proposal without that whole process, it would have just looked like six different proposals just loosely tied together and it would have just been six much less novel proposals. I think the working together part allowed each of us to come up with some interesting new things that we wouldn’t have come up with otherwise. Even after a year, there was still a little difficulty blending it all together and wrapping it up with a nice little bow; I think after a year we were at least able to come up with some interesting things that wouldn’t have come up unless [if] people from different disciplines hadn’t been talking.
While it appears that social and conceptual integration among scientists on the team were significant, the integration of non-academic practitioners and graduate students was less successful. Approximately five to seven non-academic water managers or stakeholders attended the majority of the weekly workshops. Most attended in person, while others followed a live interactive video broadcast. Students and faculty alike appreciated the “real world knowledge and experience” that non-academics offered in the discussions. One graduate student noted that: “Unfortunately, many of our professors do not have experience in the field . . . so [hearing from practitioners was] one of the advantages of these workshops.”
That said, the WSC non-academic participants were generally more engaged as consultants rather than as full collaborators in the framing of research problems. In retrospect, some team members saw this as a downfall: “They sort of were reactive participants rather than being given the opportunity to thoughtfully contribute and actually say what’s important about this process . . . or how can this work actually meaningfully inform what their agency does.” Practitioners for the most part agreed. While they liked the topic of the project, they often felt as though “conversation was too theoretical,” which limited their participation as full partners.
Additionally, while graduate students attended many of the activities, most were not very involved in the culminating effort to write the C2-WSC proposal. This can be attributed to the fact that graduate students had coursework and dissertation research obligations that competed for their scarce time, and the inability of the group process to fully integrate graduate students as co-equals in the intellectual discussions. It also reflected that many faculty on the team cautioned their own graduate students from spending too much time on the WSC project. One professor noted that, “I didn’t involve my graduate students in the workshops because students are so busy already, and I wasn’t sure that this was the right time for them to be involved because we [the faculty] were still figuring things out ourselves.” Untenured junior faculty, in particular, worried that time spent by their graduate students on WSC work would take time away from research and writing output on their other funded research projects. In hindsight, several team members noted that a more clearly defined role for graduate students within the team, including funding support, would have helped encourage and expand graduate student contributions and acquisition of team building and problem formulation skills.
The C1-WSC project provided an infusion of money that helped facilitate project administration and coordination tasks, paid for the group workshops and retreats, reduced transaction costs associated with collaboration activities, and subsidized the involvement of several key faculty and graduate students. Most interviewees recognized that they would never been able to devote this much time and energy to a team science project without the NSF C1-WSC award. These resources also built on the legacy of less intensive investments previously made by the USU administration in their campus “Water Initiative,” which had already helped create an institutional climate open to more collaborative and interdisciplinary work. University administrators viewed the C1-WSC award as a major accomplishment, and allocated central funds to the team to pay for a part-time grant writing advisor when they were writing the C2-WSC proposal.
Despite these institutional resources and investments, it was clear that fundamental reward structure of departmental graduate programs and university faculty promotion and tenure systems remained significant barriers to the team’s collaborative work. As noted above, there was prevailing concern among untenured faculty that time spent on the WSC project was detracting from their ability to secure disciplinary grants, get work done on their other research projects, and write papers for publication. Graduate students who spent time on WSC efforts were not able to make this part of their thesis or dissertation projects, and were not relieved of outside research and teaching responsibilities. In addition, since this was a team building and planning grant and did not support collection of primary data, the team’s work did not lend itself to traditional academic publications. The fact that the C2-WSC proposal was not funded made the time investments of some of the younger faculty look like a risky choice.
While the Category 2 proposal was not funded, the core ideas discussed and co-created within the WSC transferred into a suite of “spin-off” projects conducted by smaller sets of WSC team members. The impact of the collaborative team-building work was most immediately visible in the contributions of the WSC team to a separate major NSF Track 1 Experimental Program to Stimulate Competitive Research (EPSCoR) proposal submitted in November 2011 by a larger group of interdisciplinary scientists from multiple Utah institutions. This project, named innovative Urban Transitions and Aridregion Hydro-sustainability (iUTAH), which did receive funding, was co-led by several WSC team members and utilized concepts, methods, and computational models that were a direct product of the WSC collaborations (see http://iutahepscor.org/ for more information). It is too early to evaluate how else the WSC activities will impact the EPSCoR effort, particularly because the number of researchers involved in the EPSCoR has nearly tripled. That said, we expect that familiarity, comfort, and experiences among the WSC team members will be sustained, as long as the EPSCoR collaborative environment encourages these connections and incentivizes cross-disciplinary collaboration.
A similar set of structural imperatives prevented the non-academic team members from investing more time in the group effort. Initially, none of the water managers and stakeholders received any payments or other tangible rewards for the investment of their time and energy. As with most of the faculty on the team, their decision to participate was not a core part of their regular job, but reflected a mix of personal curiosity and a calculation that being part of a successful C2-WSC grant promised much greater resources for water resources research.
One indicator of success at the systemic level was that the group process and final scientific research plan was meaningful to the non-academic partners. Without prompting, all of the practitioners interviewed were eager to know the status of the grant proposal and the next steps, even knowing that the proposal was not funded. We interpret the managers’ enthusiasm for the proposed research ideas as an indicator of success at the systemic level in the present and future research. Practitioner support for the collaborative effort suggests that the topics and research frames considered by the team fit current information needs. In addressing these needs, the research team legitimated their scientific interests in that these ideas may translate into meaningful and applicable knowledge. In forthcoming opportunities for stakeholder engagement, managers involved in the WSC activities may have a closer sense of how the collaboration could unfold. We see the connections among researchers and practitioners as having potential to bring about systemic change in how information is produced and used among team scientists.
The research collaboration on Water Sustainability and Climate (WSC) at Utah State University (USU) is an example of a team building project that set out to encourage interdisciplinary understanding and collaborative capacity of team members, and to develop a comprehensive, interdisciplinary scientific research plan. Our evaluation at the individual, research team, and university/systemic scales revealed that these objectives were largely met, but not without encountering some obstacles. Additionally, we learned that social interactions encouraged interpersonal relationships that helped build team cohesion, which provided the foundation for constructive work toward conceptual integration.
At the individual and research team levels, the weekly workshops and weekend retreats helped create stronger interdisciplinary understanding and social ties among team members, and the development of a strong team identity, thus satisfying the first two project objectives. To develop a holistic plan of study within complex systems, team science requires high levels of integration among the participants. Social integration was particularly important at the interface between individual and research team levels. As team members became more familiar with each other’s research and thinking, they brought about greater comfort with one another’s epistemological approaches, research foci, and garnered familiarity among team members. In turn, their familiarity created the conditions for team members to create a whole greater than the sum of the parts in the shape of a scientific research plan that no standalone team member would have produced without engaging in these group exercises.
Nonetheless, conceptual integration remained a struggle for some on the team, and the translation of a common vision into concrete computational modeling plans highlighted areas of remaining conceptual differences. This suggests that social integration, though key to the overall team formation process, may be limited by disciplinary tendencies that are engrained within scientific problem solving. We also suspect that lingering concerns about incomplete conceptual integration reflected the absence of a final product or output (in terms of a funded research project) that would have implemented the scientific research plan. As noted by Stokols et al. (2005), attitudes towards project outputs can reshape team members’ perceptions of concepts and feelings of satisfaction about the process. Coordinated research activities were beyond the scope of the C1-WSC objectives, but were not beyond the sights of team members.
At the university/systemic level, the broader institutional context provided opportunities for all types of team members to engage in this collaborative exercise, but few direct rewards for their participation. In fact, all of the team members had to ensure that their “real jobs” were covered before they could justify spending time on this team science project. For many, this work was a calculated gamble that the investment of their time and energy would yield rewards in the form of a much larger integrated research grant down the road. Although the team’s C2-WSC grant was not funded, several members of the team, including some of the non-academic partners, have already begun to implement many of the conceptual and methodological elements of the proposal under the auspices of the funded iUTAH EPSCoR project (described in Subsection 4.3).
The WSC activities had mixed results in terms of involving water managers into the design of a scientific research project. The practitioners who participated in the team-building work generally felt more like advisors rather than full team members. However, the examples of real-world water management challenges that were shared by non-academics were highly valued by researchers. Graduate students who participated in the WSC activities enjoyed learning about the complexities of interdisciplinary project work; however, few made serious contributions to the formulation of concrete research questions during C1-WSC discussions and none of them (other than the lead author of this paper) was directly involved in the writing of the C2-WSC proposal. We sense that this is largely because graduate students faced systemic constraints and received mixed signals from their advisors as to the merits of taking time away from their core studies and research. We believe that more formal incentive structures such as course credits or opportunities to co-author research articles must be established to overcome barriers in interdisciplinary graduate student training.
Our results are based on a single case study, and may reflect unique features of this team science effort or institutional context. As such, it is difficult to know how well they can be generalized to team science projects in general. We were able to complete detailed interviews from almost all of the team’s members, and our findings reflect both the general areas of agreement and diversity of opinion among our informants. We believe that the insights and experiences of the C1-WSC team are most useful for groups at the team formation and planning stages. In that sense, our evaluation highlights the value of structured formal and informal opportunities for interdisciplinary conversations as a basis for both social and conceptual integration. It also reveals the difficulty of engaging junior faculty and graduate students in team science work without changing institutional reward mechanisms that focus on traditional indicators of scientific achievement.
This work was supported by the National Science Foundation (grant number EAR-1038973). The authors would like to thank the funding agency and the Utah State University researchers who participated in this evaluation study.
Best, A., Terpstra, J. L., Moor, G., Riley, B., Norman, C. D., & Glasgow, R. E. (2009). Building knowledge integration systems for evidence-informed decisions. Journal of health organization and management, 23(6), 627-641.
Beyerlein, M. M., Freedman, S., McGee, C., Moran, L., & Beyerlein, M. (2003). Beyond teams: Building the collaborative organization. San Francisco: Jossey-Bass/Pfeiffer.
Boix Mansilla, V., Feller, I., & Gardner, H. (2006). Quality assessment in interdisciplinary research and education. Research Evaluation, 15(1), 69-74.
Eigenbrode, S. D., O’Rourke, M., Wulfhorst, J. D., Althoff, D. M., Goldberg, C. S., Merrill, K., et al. (2007). Employing philosophical dialogue in collaborative science. Bioscience, 57(1), 55-64.
Falk-Krzesinski, H. J., Contractor, N., Fiore, S. M., Hall, K. L., Kane, C., Keyton, J., et al. (2011). Mapping a research agenda for the science of team science. Research Evaluation, 20(2), 145-158.
Frodeman, R., Klein, J. T., & Mitcham, C. (2010). The Oxford handbook of interdisciplinarity. Oxford, UK: Oxford University Press.
Hadorn, H. H., Hoffmann-Reim, H., Biber-Klemm, S., Grossenbacher-Mansuy, W., Joye, D., Pohl, C., et al. (Eds.). (2008). Handbook of transdisciplinary research. Berne, Switzerland: Springer.
Hall, K. L., Stokols, D., Moser, R. P., Taylor, B. K., Thornquist, M. D., Nebeling, L. C., et al. (2008). The collaboration readiness of transdisciplinary research teams and centers: findings from the National Cancer Institute’s TREC year-one evaluation study. American Journal of Preventive Medicine, 35(2 Suppl.), S161.
Hall, K. L., Stokols, D., Stipelman, B. A., Vogel, A. L., Feng, A., Masimore, B., et al. (2012). Assessing the value of team science: A study comparing center- and investigator-initiated grants. American Journal of Preventive Medicine, 42(2), 157-163.
Hays, T. C. (2008). Commentary on measurements of scientific readiness. American Journal of Preventive Medicine, 25(2S), S193-S195.
Klein, J. T. (2006). Afterword: The emergent literature on interdisciplinary and transdisciplinary research evaluation. Research Evaluation, 15(1), 75-80.
Klein, J. T. (2008). Evaluation of interdisciplinary and transdisciplinary research: A literature review. American Journal of Preventive Medicine, 35(2S), S116-S123.
National Academy of Sciences. (2005). Facilitating interdisciplinary research. Washington, D.C.: National Academies Press.
National Science Foundation. (2012). Integrative graduate education and traineeship program (IGERT). Retrieved August 2, 2012, from http://www.nsf.gov/funding/pgm_summ.jsp?pims_id=12759
Neuhauser, L., Richardson, D., Mackenzie, S., & Minkler, M. (2007). Advancing transdisciplinary and translational research practice: Issues and models of doctoral education in public health. Journal of Research Practice, 3(2), M19. Retrieved from http://jrp.icaap.org/index.php/jrp/article/view/103/97
Norman, C. D., Best, A., Mortimer, S., Huerta, T., & Buchan, A. (2011). Evaluating the science of discovery in complex health systems. American Journal of Evaluation, 32(1), 70-84.
Repko, A. F. (2008). Interdisciplinary research: Process and theory. Los Angeles: Sage.
Ryser, L., Halseth, G., & Thein, D. (2009). Strategies and intervening factors influencing student social interaction and experiental learning in an interdisciplinary research team. Research on Higher Education, 50, 248-267.
Stokols, D., Fuqua, J., Gress, J., Harvey, R., Phillips, K., Baezconde-Garbanati, L., et al. (2003). Evaluating transdisciplinary science. Nicotine & Tobacco Research, 5(1S), S21-S39.
Stokols, D., Hall, K. L., Taylor, B. K., & Moser, R. P. (2008). The science of team science: Overview of the field and introduction to the supplement. American Journal of Preventive Medicine, 35(2), S77-S89.
Stokols, D., Harvey, R., Gress, J., Fuqua, J., & Phillips, K. (2005). In vivo studies of transdisciplinary scientific collaboration: Lessons learned and implications for active living research. American Journal of Preventive Medicine, 28(2), 202-213.
Trochim, W. M., Marcus, S. E., Masse, L. C., Moser, R. P., & Weld, P. C. (2008). The evaluation of large research initiatives: A participatory integrative mixed-methods approach. American Journal of Evaluation, 29(1), 8-28.
Wenger, E. C., & Snyder, W. M. (2000). Communities of practice: The organizational frontier. Harvard Business Review, 78(1), 139-146.
Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Thousand Oaks, CA: Sage.
Received 31 October 2012 | Accepted 29 January 2013 | Published 4 February 2013
Copyright © 2013 Journal of Research Practice and the authors