Journal of Research Practice

Volume 7, Issue 2, Article D2, 2011

Research Design:
A Comparison of Internet-Based Participant Recruitment Methods: Engaging the Hidden Population of Cannabis Users in Research

Elizabeth C. Temple
School of Health Sciences, University of Ballarat
PO Box 663, Ballarat VIC 3353, AUSTRALIA
e.temple@ballarat.edu.au

Rhonda F. Brown
Department of Psychology, The Australian National University
Canberra ACT 0200, AUSTRALIA
rhonda.brown@anu.edu.au

Abstract

While a growing number of researchers are embracing Internet-based data collection methods, the adoption of Internet-based recruitment methods has been relatively slow. This may be because little is known regarding the relative strengths and weaknesses of different methods of Internet-based participant recruitment, nor how these different recruitment strategies impact on the data collected. These issues are addressed in this article with reference to a study comparing the effectiveness of three Internet-based strategies in recruiting cannabis users for an online study. Consideration of the recruitment data leads us to recommend that researchers use multipronged Internet-based recruitment campaigns with appropriately detailed recruitment messages tailored to the population of interest and located carefully to ensure they reach the intended audience. Further, we suggest that building rapport directly with potential participants, or utilising derived rapport and implicit endorsements, is an important aspect of successful Internet-based participant recruitment strategies.

Index Terms: Internet-based data collection; Internet-based research; research methods; public health; cannabis

Suggested Citation: Temple, E. C., & Brown, R. F. (2011). A comparison of Internet-based participant recruitment methods: Engaging the hidden population of cannabis users in research. Journal of Research Practice, 7(2), Article D2. Retrieved [date of access], from http://jrp.icaap.org/index.php/jrp/article/view/288/247

1. Introduction

While recruiting participants in research studies can be unbelievably difficult and frustrating at times, for many studies, no participants means no data, which means no study. Moreover, not all people are equally desirable as participants for every study, with researchers typically having a specific target group of interest. How successfully this segment of the general population is accessed and engaged in the study will depend on the recruitment strategy employed. Therefore, the eventual success of the study is often contingent on the recruitment strategy working as intended.

So, how do we plan and execute a successful recruitment strategy? Well, obviously, the answer to this question will depend on a number of factors specific to the study, including the nature of the intended participant group and the type of data to be collected, not to mention other factors such as the geographic location of the target population and of the researchers. The decision to employ an Internet-based recruitment strategy for the study discussed in the present article was similarly based on a number of factors. A primary consideration, with the researchers based in an Australian rural setting, was access to potential participants. Additionally, as the study involved collecting information on illegal activities (i.e., cannabis use) and other sensitive information (e.g., history of sexual abuse), the benefits afforded by the Internet in terms of anonymity were important. Moreover, examination of Internet access statistics indicated that it was an appropriate medium for accessing potential participants in the target population (i.e., English-speaking, younger age groups--the peak age for cannabis use is 20-29 years [Australian Institute of Health and Welfare, 2008]).

Having decided to recruit participants online, it became apparent that that was the easy part; the real hurdle was determining how to actually do it. A review of the available literature turned up little in the way of adequately detailed information on how to (or even how not to) recruit participants online and indicated that there was a scarcity of evidence regarding the relative advantages and disadvantages of different approaches. This article has been written, therefore, to assist other researchers who would like to recruit participants online but are unsure how to do so. We start with a review of the literature, followed by a detailed description of our experiences employing three overarching online recruitment strategies for a study involving cannabis users. The varying levels of success of these strategies are discussed, with recommendations made regarding what we would have done differently if we knew then what we know now.

2. Internet-Based Research and Participant Recruitment

The growing popularity of Internet-based research has been attributed to a number of factors, including the possible efficiencies in time and money, the capacity to reach a geographically diverse sample, and the ability to ensure the anonymity of participants. These factors have also contributed to the ability of Internet-based studies to provide relatively easy access to hard to reach or hidden populations, such as people engaging in illegal activities or behaviour that is socially atypical (popularity: Alessi & Martin, 2010; Hallett, Maycock, Kypri, Howat, & McManus, 2009; Miller & Sonderlund, 2010; Whitehead, 2007; efficiencies: Eysenbach & Wyatt, 2002; Reips, 2002; Riva, Teruzzi, & Anolli, 2003; reach and anonymity: Bowen, Williams, & Horvath, 2004; Nicholson, White, & Duncan, 1999; Rhodes, Bowie, & Hergenrather, 2003; access: Duncan, White, & Nicholson, 2003; Lieberman & Huang, 2008; Miller & Sonderlund, 2010; Rhodes, Bowie, & Hergenrather, 2003).

Surprisingly, while these same features are inherent to Internet-based recruitment strategies, many Internet-based studies continue to utilise traditional, or offline, recruitment methodologies, such as flyers/posters, direct mail outs, newspaper and magazine advertisements or articles, personal referrals, or in-class recruitment of college students (e.g., Alexander et al., 2008; Bowen, Williams, & Horvath, 2004; Burke, Wallen, Vail-Smith, & Knox, 2011; Hedman et al., 2010; Tokunaga, 2011; Warmerdam, van Straten, & Cuijpers., 2007). It is possible that the relatively slow adoption of Internet-based recruitment strategies is related to the lack of information in the literature on how to best utilise the Internet for recruitment purposes. For example, articles rarely include detailed descriptions of the recruitment strategies employed and the processes involved (Hallett et al., 2009; Whitehead, 2007), although Benfield and Szlemko’s (2006) and Alessi and Martin’s (2010) articles are welcome exceptions.

Nevertheless, a broad range of Internet-based recruitment strategies have been utilised to attract participants to Internet-based studies, including: direct emails to personal contacts, organisation members, or student populations, email snowballing, website membership newsletters/emailing lists, Internet research panel emailing lists, Internet advertisements, postings on online forums and message boards, website announcements/notices, and through social media tools such as Facebook or Internet chat rooms (e.g., direct emails: Dobrow et al., 2008; Leece et al., 2004; Joinson, Pain, Buchanan, & Reips, 2008; Moloney, Aycock, Cotsonis, Myerburg, Farino, & Lentz, 2009; email snowballing: Barnwell, Earleywine, & Gordis, 2005; Koo & Skinner, 2005; newsletters: Denson & Earleywine, 2006; Im, Chee, Tsai, Bender, & Lim, 2007; Looby & Earlywine, 2007; Ritter, Lorig, Laurent, & Matthews, 2004; Walden & Earleywine, 2008; research panel: Novak, Kroutil, Williams, & Van, 2007; West et al., 2006; Internet advertisements: Donker, van Straten, Marks, & Cuijpers, 2009; online forums: Baggott, Erowid, Erowid, Galloway, & Mendelson, 2010; Im, Shin, & Chee, 2008; Katz, Fernandez, Chang, Benoit, & Butler, 2008; Nicholson, White, & Duncan, 1999; Ritter et al., 2004; website announcements: Baggott et al., 2010; Im, Shin, & Chee, 2008; Katz et al., 2008; Nicholson, White, & Duncan, 1999; Ritter et al., 2004; social media: Fernandez, Warren, Varga, Prado, Hernandez, & Bowen, 2007; Lin & Lu, 2011).

Researchers are increasingly employing multiple recruitment strategies for Internet-based studies, with a multipronged approach suggested to result in successful recruitment. However, despite a number of studies comparing the effectiveness of Internet-based recruitment to traditional recruitment methods, very few studies have compared different modes of Internet-based recruitment, and little is known about factors that may contribute to the successful Internet-based recruitment of participants (multipronged approaches: Alessi & Martin, 2010; Benfield & Szlemko, 2006; Hidaka et al., 2006; Miller, Johnston, McElwee, & Noble, 2007; Moloney et al., 2009; Rodgers, Buchanan, Scholey, Heffernan, Ling, & Parrott, 2003; successful recruitment: Benfield & Szlemko, 2006; Miller & Sonderlund, 2010; Internet-based vs. traditional: Bowen, Williams, & Horvath, 2004; Elford, Bolding, Davis, Sherr, & Hart, 2004; Fernandez et al., 2007; Leece et al., 2004; Link & Mokdad, 2005; Ritter et al., 2004).

3. Factors Associated With Successful Internet-Based Participant Recruitment

Moloney et al. (2009) compared several Internet-based recruitment methods including direct emails to personal contacts, Internet advertising, posts on discussion boards, and website notices. Of these, discussion board posts led to the highest number of contacts, however the direct email strategy had the highest rate of conversion from contacts to study participants, and Internet advertising was the least effective strategy for converting contacts into participants. In addition, networking and referrals, such as online and offline word-of-mouth (e.g., participants posting study details on forums) were central to the success of the recruitment campaign, suggesting that developing relationships with participants is essential for recruiting and retaining participants in Internet-based studies (Moloney et al., 2009).

Building rapport and gaining the trust of potential participants is a challenge in both online (Koo & Skinner, 2005; Moloney et al., 2009) and traditional offline studies (Sixsmith, Boneham, & Goldring, 2003), and this is a particularly difficult process when trying to recruit hidden populations (Hidaka et al., 2006). It is suggested in the literature that rapport can be gained with potential participants through online conversations in chat rooms (e.g., Fernandez et al., 2007) or via postings on online forums (e.g., Alessi & Martin, 2010); but this approach can be time-consuming given the need to engage directly with individual participants (Miller & Sonderlund, 2010). Therefore, it may be more expedient to utilise derived rapport, which can be achieved by disseminating recruitment messages via people, organisations, or communities that have an existing relationship with the potential participants. Internet-based recruitment methods which putatively utilise derived rapport and implicit endorsement include email snowballing (e.g., Koo & Skinner, 2005; Benfield & Szlemko, 2006), dissemination of recruitment messages via newsletters or emailing lists to online communities (e.g., Denson & Earleywine, 2006), and placing notices or announcements about a study on relevant websites (e.g., Katz et al., 2008). However, it seems likely that any derived rapport effect will weaken as the connection between the initial source of the information and the recipient becomes more tenuous, as evident in Benfield and Szlemko’s (2006) snowballing experience where the process appeared to stop at the third or fourth iteration.

A contrasting approach to Internet-based recruitment involves the use of Internet advertising. While this recruitment method can be expensive (Alessi & Martin, 2010), it promises immense reach with relatively little effort from researchers, which may be why it is the most commonly utilised Internet-based recruitment method for Internet-based studies involving substance users (Miller & Sonderlund, 2010). Moloney et al. (2009), as noted above, did not find Internet advertising to be a particularly successful recruitment strategy, suggesting that this was related to the lack of relationship building possible between researchers and potential participants. However, Graham, Milner, Saul, and Pfaff (2008) found that the effectiveness of online recruitment advertisements depended on their tone and placement. In their study, four different online advertisements were employed to attract people to a smoking cessation website: serious, humorous, and website-specific banner advertisements, and text-based keyword search-related advertisements. The keyword search-related advertisements attracted more people to the website than banner advertisements, but the humorous advertisement was the most successful in recruiting younger males.

The success of Graham et al.’s (2008) keyword search-related recruitment method is likely to be due to appropriate targeting (Miller & Sonderlund, 2010), with only people actively searching a relevant term being exposed to the recruitment message. Appropriate targeting can also be achieved by placing notices on websites accessed by the target population, posting messages on relevant forums, and the distribution of emails by Internet-based organisations to their members (Miller & Sonderlund, 2010). The success of Graham et al.’s humorous advertisement with younger males suggests that another important aspect of Internet-based recruitment may be the tailoring of the recruitment message to the target audience by considering their characteristics and attitudes. However, this can be a challenge for hidden populations, such as non-treatment-seeking cannabis users, as little is known about their intrinsic characteristics (Temple, Brown, & Hine, 2011). Nevertheless, Graham et al.’s (2008) successful use of humour suggests that recruitment messages with a humorous tone may be more effective for some target populations than traditional, or straight, recruitment messages. Email snowballing recruitment strategies, for example, may benefit from the use of a humorous recruitment message, with recipients perhaps being more likely to forward the email on to others, as often seems to occur with the communication of jokes and amusing anecdotes via email, than if the recruitment message is more conservative in tone.

To summarise, there appear to be a number of features that may increase the effectiveness of Internet-based participant recruitment campaigns. These include having rapport (either direct or derived) with potential participants, utilising implicit endorsements, and balancing high exposure levels (to increase the size of the audience receiving the recruitment message) with the need to ensure that recruitment messages are appropriately targeted and tailored to the target population (Benfeld & Szlemko, 2006; Graham et al., 2008; Koo & Skinner, 2005; Maloney, et al., 2009; Miller & Sonderlund, 2010).

These features were explored in the study discussed in the remainder of this article by comparing the effectiveness of three different Internet-based recruitment strategies in attracting potential participants from the hidden population of cannabis users to an Internet-based study. The recruitment strategies utilised were: (a) an email recruitment strategy consisting of the distribution of recruitment messages via website membership emailing lists and email snowballing, (b) a website recruitment strategy consisting of posting recruitment messages on online forums and placing notices on websites, and (c) an Internet advertising strategy.

Specifically, it was anticipated that the different Internet-based recruitment methods would perform dissimilarly in terms of attracting potential participants to the Internet-based study versus gaining participants and completed questionnaires, such as seen in Moloney et al.’s (2009) study. For example, exposure levels were anticipated to be primarily associated with the number of potential participants attracted to the study website, thus Internet advertising was thought likely to be most successful at this recruitment function and the email strategy the least successful. In contrast, utilising direct or derived rapport and implicit endorsement was anticipated to be primarily associated with conversion of potential participants into questionnaire completers, hence Internet advertising was thought likely to perform more poorly than the email and website strategies. It was anticipated that appropriate targeting and tailoring of messages would enhance both recruitment functions.

4. Recruiting Participants for the Cannabis Experience and Everyday Functioning Study

4.1. The Study Website and Questionnaire

The site name (thecannabisexperience.info) was chosen and registered through Domains by Proxy (domainsbyproxy.com). Hosting for the site was arranged through ADThosting (adthosting.com). The site was managed through the use of a CPanel web-based control panel for the domain. This provided the ability to manage all aspects of the domain, including email, files, backup, FTP (file transfer protocol), and website statistics. The study website was an open site, so anyone could access the study, and consisted of 92 pages. The front page explained the study and included a consent statement, asking participants to click on a button to indicate their consent before they could progress to the questionnaire. The questionnaire included item branching and automatic skips, thus the length differed for participants based on their answers to certain questions. Time and date stamping and IP (Internet protocol) address logging were employed to permit data screening, but cookies were not used in order to enable completion on multiuser computers.

The Cannabis Experience and Everyday Functioning Study questionnaire included detailed questioning of cannabis use, and also collected data on other licit and illicit substance use, psychological wellbeing and distress, cognitive functioning, as well as various aspects of past and current lifestyles (e.g., past: family structure, history of trauma/abuse, family functioning, education; current: living arrangements, employment), including demographic information (e.g., age, gender, nationality). There were four overarching content areas of the questionnaire, in order of presentation: (a) psychopathology, (b) cognitive functioning, (c) substance use and dependence, and (d) demographic and lifestyle factors. While much of the content of the questionnaire is outside the scope of the present article, results relating to demographics will be discussed.

The initial version of the questionnaire was pretested in paper-and-pencil format, while the second version was pilot tested as an Internet-based survey. The pilot and final questionnaires were converted into an Internet-based format through the use of StatPac for Windows, version 9.0 (StatPac Inc., 2004). The study was conducted with the approval of the Human Research Ethics Committee, University of New England (UNE), Australia.

4.2. Recruitment Materials and Procedure

The recruitment campaign for the study lasted 11 weeks and consisted of three recruitment strategies: (a) email recruitment, (b) website recruitment, and (c) Internet advertising.

4.2.1. Email Recruitment

The email strategy consisted of two recruitment methods: email dissemination via website membership emailing lists and email snowballing. The first method involved submitting information about the study to two cannabis-related Internet sites (Cannabinoid Science; Multidisciplinary Association for Psychedelic Studies) and requesting the information be sent to the sites’ emailing lists.

Email snowballing involved sending four email messages to 45 people who were known to the principal researcher. The first and third messages were designed to be humorous to increase the likelihood of recipients forwarding them on to others. The first email had the subject title “George W. Bush inhaled . . .” and contained a photo of George W. Bush reading to a child from an upside-down book, and the third email contained a humorous cannabis-related poem written by the principal researcher. The other emails were a brief reminder (2nd email) and a Thank You (4th email) message. In terms of timing, the reminder email was sent two weeks after the initial recruitment email; after a further two weeks had elapsed, the poem was sent, and the Thank You email was sent a month after the poem.

The Thank You email was also sent to 74 study participants who had emailed the researcher with comments or queries about the study. All of the email messages requested that the recipient forward them to people they knew.

4.2.2. Website Recruitment

The website strategy employed two different recruitment methods: notices placed on websites and postings on online forums. Cannabis use-related websites were identified through Google searches, using combinations of the following keywords: cannabis, marijuana, online, forum, and message board. The most highly listed sites were selected. The non-cannabis-related websites were either selected because they were sites listing Internet-based research studies or because they had forums or message boards that appeared to be targeted at people between 18 and 30 years of age (to cover the 20-29 years peak age range for cannabis use). These websites were identified through Google searches for online forum or online message boards, with sites high on the list then visited to determine the likely target audience for the website.

The administrators of 16 websites were contacted, eight of which were cannabis-related; only three cannabis-related sites and four Internet-based research sites agreed to a notice about the study being placed on their websites. The principal researcher also posted recruitment messages on the forums of 28 cannabis and 23 non-cannabis related sites. There were three different types of recruitment message posted on the forums: (a) a straight call for participation, (b) the humorous poem used for email snowballing, and (c) a Thank You message posted approximately two weeks after the initial posting. The study notices placed on websites consisted of the information provided in the straight call for participation.

The forums were monitored closely for approximately 1-3 hours after a recruitment message had been posted, with the message thread then checked approximately twice daily until it was no longer attracting new responses or views. This level of observation enabled prompt replies to responses from forum members, which involved responding to queries, comments, and the occasional complaint or challenge, and contributed to rapport building with potential participants. Surveillance of the message threads was also necessary to monitor postings from people who had already participated in the study or visited the website to ensure they did not invalidate the research by providing inappropriate or misleading information. While, many study participants posted positive messages encouraging others to take part in the study, some posted unhelpful comments. The researchers needed to respond quickly in the latter instances to ensure others viewing the postings were not misinformed regarding the study. All study information provided in the postings made by the researchers was equivalent to that available on the front page of the study website.

Prompt replies to responses from community members were also beneficial in relation to the dynamics of online forums--replying to a message moves its thread back to the top of the message listings. This was necessary, particularly on heavily trafficked forums, because once a message thread moved to the second or subsequent page, it was less likely to be read by forum members. The most effective method for keeping the thread active was encouraging replies from forum members; even if their responses were off-topic (as happened fairly frequently) the message thread remained on the first page of the forum. These forum dynamics were exploited by increasing the use of the recruitment posting containing the poem after it became apparent that it garnered more replies than the straight recruitment message.

4.2.3. Internet Advertising

An 18-day Internet advertisingcampaign was run through Google and covered Australia, USA, UK, New Zealand, Canada, and Ireland. These countries were selected because they represented the majority of individuals who had accessed the study website prior to the initiation of the Internet advertising campaign.

Google AdWords campaigns have two components: keyword search and content-related placements. Keyword search placements are triggered when an individual initiates a Google search using a keyword that had been nominated for the campaign, and result in a short message about the study being displayed in the sidebar of the Google search engine website. Content-related placements are when the advertisement appears on sites with content related to the nominated keywords. This includes sites with agreements to display Google advertising and also sites related to other Google products (e.g., Gmail). Regardless of the nature of the advertisement’s placement, when a person clicks on it their Internet browser is directed to relevant website, in this case the study website. The pricing for AdWords is complex, with different keywords attracting different costs depending on demand, positioning of advertisements, and the click-through-rates (CTR).

Evidently, the choice of keywords is therefore important in relation to both exposure to potential participants and the costs of the campaign. The initial keywords for this study were cannabis, cannabis seeds, growshops, and marijuana. After four days, the following keywords were added: bongs, cannabis butter, hash, hash brownies, hash cookies, hash oil, headshops, joint rolling, joints, marihuana, and spliffs. At this point in time, growshops was dropped due to poor performance and ganja and hashish were added at a later point during the advertising campaign.

4.3. Recruitment Results

During the data collection period, there were a total of 7,701 visits to the website, 6,229 (80.9%) of these were unique visitors. The website was accessed via Internet service providers (ISP) from 48 different countries, with the majority accessing from the USA (61.2%), followed by Australia (13.6%), the UK (7.3%), and Canada (6.7%).

4.3.1. Attracting Potential Participants

A little over half (55.8%, N = 4,301) of the website visitors were recruited through the Internet advertising campaign. The advertisement was displayed a total of 507,314 times, with an average click-through rate (CTR) of 0.8%. The average CTR rate for advertising triggered by keyword searches was higher, at 1.2%, with cannabis and spliffs being the most successful keywords (CTRs of 2%). In contrast, when the advertisement was displayed on webpages with related content, the average CTR was 0.4%.

The website strategy accounted for the next largest proportion of website visits (24.7%, N = 1,902), with website notices attracting 14.9% of visitors and postings on online forums attracting 9.8%. More potential participants were attracted via websites that listed Internet-based research studies (6.4%) than via cannabis-related sites (3.6%), and some visitors (4.9%) accessed the study via websites that had not been approached by the researchers, suggesting an online word-of-mouth effect. With regard to the forum posting, more potential participants (7.3%) accessed the website in response to postings on cannabis-related sites than in response to postings on non-cannabis-related sites (2.5%). The ratio of recruitment message views to site accesses was approximately 7:1 for the cannabis-related sites, and 8:1 for other sites.

The email strategy attracted 1,498 (19.5%) visitors to the study website. Unfortunately, due to a design oversight, it was not possible to distinguish between those participants attracted to the Internet site by the snowballing emails and those who responded to calls for participation distributed by the Cannabinoid Science and Multidisciplinary Association for Psychedelic Studies emailing lists. These two recruitment paths could have been distinguished in two ways. For example, different URLs could have been assigned to each strategy, with automatic redirection to the study website, allowing for recording of which URL redirection path was utilised for each participant. Alternatively, there could have been two identical versions of the study questionnaire, both with different URLs, so that participants replying to snowballed messages would have accessed one survey and the participants replying via the emailing lists would have accessed the other. The latter option is perhaps the easier of the two for researchers using online survey hosting services (such as SurveyMonkey, Qualtrics, or similar), but the former may be easier for those placing their questionnaire on pre-existing websites.

4.3.2. Gaining Completed Questionnaires

Approximately half (51.5%, N = 3,996) of the people accessing the study website stayed for less than 30 seconds. Thus, it is not surprising that most (53.5%) visitors to the study website did not commence the study and, thus, were not study participants. Further, almost three-quarters (73.1%, N = 5,629) of the site visitors stayed for less than 15 minutes--the minimum amount of time to complete the questionnaire was approximately 20 minutes. Specifically, one-third (32.9%) of the 3,556 participants (those visitors who did commence the study) failed to progress beyond the first page of the questionnaire and half (48.6%) left the site by page 7, where the questions relating to cannabis use commenced. Overall, a little over a third (38.3%) of the participants (21.8% of those classified as unique visitors to the website) completed the study questionnaire.

The final questionnaire item asked study participants to report how they had heard about the study. Half (50.5%) of the completers were recruited via the website strategy, 35.1% were recruited via Internet advertising, and 10.4% were recruited by the email strategy (the recruitment pathway of the remaining 4% is unknown as they did not respond to the question). As can be seen in Table 1, with a conversion rate of 36.1%, the website strategy was more successful in converting visits to the study website into completed questionnaires than either Internet advertising or the email strategy.

Table 1. Comparison of Recruitment Strategies in Gaining Completed Questionnaires

4.3.3. Questionnaire Completers

The 1,360 questionnaire completers had a mean age of 27.8 years (SD = 11.85) and approximately two-thirds (63.7%, N = 867) were male. A total of 43.2% (N = 587) were employed, 41.5% (N = 565) were students, with the remainder unemployed. Most (81.5%, N = 1109) had used cannabis in the previous 12 months.

A one-way ANOVA indicated a medium age effect on recruitment strategy (F[2, 1301] = 37.24, p < .001, h2 = .054), where participants recruited via email (M = 35.49 years, SD = 13.47) were significantly older on average than those recruited through websites (M = 27.34, SD = 11.49) or Internet advertising (M = 26.15, SD = 10.82). Chi-square analysis indicated that there was a gender effect in relation to the recruitment strategies (χ2[2,1304] = 16.09, p < .001), with males disproportionately more likely to be recruited via the website strategy (57% of males vs. 46% of females) and females disproportionately more likely to be recruited via Internet advertising (41% of females vs. 34% of males)and email strategies (14% of females vs. 9% of males; see Table 2).

Table 2. Gender by Recruitment Strategy

Note. Recruitment strategy data missing for 56 participants (37 Male, 17 Female);
gender data missing for 2 participants (1 Website, 1 Internet advertising).

The proportion of employed participants (16%) was also disproportionately large in the email recruitment group in comparison to the proportion of students (5%): χ2(4,1303) = 31.43, p < .001 (see Table 3). A disproportionately large percentage of recent cannabis users (55% vs. 43% of non-users) were within the group of participants recruited via the website strategy, while a disproportionately small percentage of users (9% vs. 20% of non-users) were in the email recruitment group: χ2(2,1306) = 29.88, p < .001 (see Table 4).

Table 3. Employment Status by Recruitment Strategy

Table 4. Recent Cannabis Use by Recruitment Strategy

Note. Recruitment strategy data missing for 54 participants (43 Use, 11 No Use).

5. Discussion and Conclusions

The three Internet-based recruitment methods performed differently in terms of attracting potential participants to the Internet-based study and gaining actual participants (hence completed questionnaires). Specifically, exposure levels were found to be the best determinate of the number of potential participants attracted to the study website, with the high-exposure Internet advertising campaign being the most successful, the low-exposure email strategy being the least successful, and the moderate-exposure website strategy performing in between. However, it is likely that the targeting and tailoring of the recruitment messages also played a role here.

Overall, the Internet advertising campaign was more appropriately targeted than the email or website strategies. Within the email strategy, recruitment emails disseminated via website mailing lists were targeted, but control of the targeting for the snowballed emails was relinquished by the researchers as soon as the recruitment messages had been sent to the initial group of potential participants. Similarly, while the website notices campaign was targeted (with notices placed only on cannabis-related websites or sites where people go if wanting to participate in Internet-based research studies), messages were posted on both cannabis-related and non-cannabis-related forums. However, the impact of targeting is most evident when examining the forum posting and Internet advertising results. The more appropriately targeted postings were those on cannabis-related forum and keyword search triggered advertising. These postings attracted higher numbers of visitors to the study website than, respectively, postings on non-cannabis-related forums and advertisements placed on web pages with related content.

Tailoring the recruitment message to the target audience is likely to have also affected the number of potential participants attracted to the study website. The Internet advertisement was brief, by necessity for Google AdWords, so did not incorporate tailoring as such. The recruitment emails disseminated via website mailing lists included a straight call for participation, so were not specifically tailored to the audience, but email snowballing employed tailored messages, using humour to encourage recipients to forward the message to others. Correspondingly, the website notices campaign was not specifically tailored, but the forum posting campaign was, with the humorous poem found to consistently draw more views and comments than the straight calls for participation. However, while the use of humorous recruitment messages may have increased the number of potential participants attracted to the study website, these messages may also have acted to increase the number of website visitors who did not participate, thus reducing the conversion rates of website visitors to study completers. That is, the humorous recruitment messages contained little in the way of actual information about the study, thus, potential participants did not know enough to decide if they were interested in participating or not without having to visit the study website to learn more about it--just over half of website visitors chose not to even commence the study. Additionally, this factor may explain the poor conversion rates evident for the Internet advertising campaign, which also provided little information about the study to potential participants. Although, the lack of direct or derived rapport and implicit endorsement afforded by Internet advertising appears to have also played a role.

The website strategy, which utilised both direct and derived rapport, was the most successful of the strategies at converting visitors to the website into participants who completed questionnaires. Within the website strategy, the website notices campaign relied on derived rapport and implicit endorsement, in contrast, the forum posting campaign enabled the researchers to build rapport directly with potential participants through conversations on online forums. This rapport building process took time and effort and was complicated by the illicit nature of the behaviour under investigation. Most forums were welcoming and interested, however some (typically cannabis-related) were quite hostile initially, with concerns raised about submitting information about illegal activities to an unknown person or organisation. In particular, beliefs that the research might be biased against cannabis use, and towards negative outcomes, were fairly common. These comments were addressed by the principal researcher explaining the aims of the study and the unbiased nature of the research, which led to many people choosing to participate. Thus, the rapport building process was predominantly about gaining the trust of online community members so that they felt comfortable about the research project, but also typically incorporated the provision of information about the study.

Interestingly, and not in line with our initial expectation, the email strategy, which also employed derived rapport and implicit endorsement, was the least successful at gaining completed questionnaires. Thus, it appears that derived rapport and endorsement are not enough to guarantee successful recruitment outcomes, with exposure, targeting, tailoring, and information provision all likely to be important not just for attracting potential participants but also playing a role in gaining participants and completed questionnaires.

It is also likely that the conversion of website visitors to study completers is impacted by website design (Dillman, 2000; Koo & Skinner, 2005). Feedback received through the forums indicated that the present study had three main design issues. First, individuals recruited on the basis of their cannabis use had to progress through six pages of the questionnaire before they encountered the first cannabis-related question, leading to doubts about the pertinence of the study. Second, psychopathology and cognitive function questions were at the beginning of the questionnaire, contributing to the unease of some participants with regard to the motivations behind the research, particularly the belief that the study would be biased towards negative findings. Third, the length of the questionnaire involved a larger time commitment than some participants were willing to make. This last issue is illustrated well by the website visit data, which suggests that completions would have been substantially higher if the questionnaire had been restricted in length to less than seven pages and completion time to less than 15 minutes.

While this feedback from online forum members provided some insight, it was not possible to ascertain why some website visitors chose to participate in the study and others did not, nor why some participants completed the questionnaire and others dropped out. Similarly, it was not possible to determine the total number of people who viewed the recruitment messages or to determine the differences between those who accessed the study website in response to a call for participation and those who did not after the same exposure. Further, it was not always possible to disentangle the available data to determine the relative success of the recruitment strategies, making it difficult to fully evaluate their effectiveness. Specifically, it would have been particularly useful if it were possible to evaluate the relative contributions of rapport building and information provision to questionnaire completions. Some of these issues could be addressed by modifying the study’s design. For example, the effectiveness of the different recruitment strategies could have been evaluated individually if each had been allocated a different domain name (URL) for website access. Similarly, the items covering demographics and enquiring as to how participants had heard about the study could have been placed at the beginning of the questionnaire to ensure this data was collected for study dropouts.

The results from the present study also provide some insight into the effectiveness of different Internet-based recruitment strategies for reaching different demographic groups. For example, females, older, non-cannabis-users, and employed participants were disproportionately more likely to have been recruited through the email recruitment strategy than males, younger participants, and students. In contrast, website recruitment strategies were skewed towards the recruitment of young participants and cannabis-users. There may be many possible reasons for these differences. For example, it may be related to differences in preferences for direct and/or derived rapport, the desired amount of information provision, or the amount of time spent online. Alternatively, it may be related to the age and gender balance of those receiving the email recruitment messages. While the snowball email was sent to a gender balanced group (23 females & 22 males) ranging in age from 18 to 59 years, the gender balance and age range of the other emailing lists are unknown. Most likely, a combination of all of these and other unnamed factors have had an impact. Nevertheless, the gender and age differences, in particular, suggest that different features of the recruitment strategies may be more important for attracting and retaining different segments of the population. It would be beneficial if these differences were investigated further.

Nevertheless, these findings support Miller and Sonderlund’s (2010) suggestion that a multipronged Internet-based recruitment strategy should be utilised to recruit substance users to Internet-based studies. As evident from Benfield and Szlemko’s (2006) experiences, it is possible that these findings are not specific to cannabis users, thus, we suggest that multipronged approaches should be employed within Internet-based recruitment campaigns for other populations. Such an approach is also likely to decrease the possibility of encountering problems in relation to selection bias and external validity: issues which should be considered for all Internet-based studies (regardless of whether recruitment is online or offline) because only those with Internet access can participate (Eysenbach & Wyatt, 2002; Reips, 2002; Rhodes, Bowie, & Hergenrather, 2003). As such, decisions about whether Internet-based recruitment and data collection is appropriate for any particular study will depend on the intended target population and the nature of the research questions under examination (Benfield & Szlemko, 2006; Eysenbach & Wyatt, 2002; Rhodes, Bowie, & Hergenrather, 2003). For example, for the study described in this article, the decision to employ Internet-based recruitment methods and an Internet-based study was based on a number of factors. First, with the researchers based in an Australian rural setting, a primary consideration was access to potential participants. Second, the benefits afforded by the Internet in terms of anonymity were important as the study involved collecting information on illegal activities (illicit drug use) and other sensitive information (e.g., history of sexual abuse). Third, after examining Internet access statistics, the Internet was deemed an appropriate medium for recruiting potential participants in the target population: English-speaking, younger age groups--the peak age for cannabis use is 20-29 years (Australian Institute of Health and Welfare, 2008).

To summarise, the differences between the Internet-based recruitment methods and their relative effectiveness in achieving the study objectives suggest that Internet-based recruitment strategies require adequate exposure of appropriately detailed, targeted, and tailored recruitment messages. Further, it is suggested that Internet-based recruitment campaigns will be more successful if they utilise direct and/or derived rapport and implicit endorsement, with these features likely to be important when recruiting participants from hidden populations.

6. Acknowledgments

The researchers would like to thank all of the people who participated in The Cannabis Experience and Everyday Functioning Study.

References

Alessi, E. J., & Martin, J. I. (2010). Conducting an Internet-based survey: Benefits, pitfalls, and lessons learned. Social Work Research, 34, 122-128.

Alexander, G. L., Divine, G. W., Couper, M. P., McClure, J. B., Stopponi, M. A., Fortman, K. K., Tolsma, D. D., Strecher, V. J., & Johnson, C. C. (2008). Effect of incentives and mailing features on online health program enrollment. American Journal of Preventative Medicine, 34, 382-388.

Australian Institute of Health and Welfare. (2008). 2007 National drug strategy household survey: First results. Canberra, Australia: Author.

Baggott, M. J., Erowid, E., Erowid, F., Galloway, G. P., & Mendelson, J. (2010). Use patterns and self-reported effects of Salvia divinorum: An Internet-based survey. Drug Alcohol Dependence, 111, 250-256.

Barnwell, S. S., Earleywine, M., & Gordis, E. B. (2005). Alcohol consumption moderates the link between cannabis use and cannabis dependence in an Internet survey. Psychology of Addictive Behaviors, 19, 212-216.

Benfield, J. A., & Szlemko, W. J. (2006). Internet-based data collection: Promises and realities. Journal of Research Practice, 2(2), Article D1. Retrieved February 11, 2012, from, http://jrp.icaap.org/index.php/jrp/article/view/30/51

Bowen, A., Williams, M., & Horvath, K. (2004). Using the Internet to recruit rural MSM for HIV risk assessment: Sampling issues. AIDS and Behavior, 8, 311-319.

Burke, S. C., Wallen, M., Vail-Smith, K., & Knox, D. (2011). Using technology to control intimate partners: An exploratory study of college undergraduates. Computers in Human Behavior, 27, 1162-1167.

Denson, T. F., & Earleywine, M. (2006). Decreased depression in marijuana users. Addictive Behaviors, 31, 738-742.

Dillman, D. A. (2000). Mail and Internet surveys: The tailored design method (2nd ed.). New York: John Wiley.

Dobrow, M. J., Orchard, M. C., Golden, B., Holowaty, E., Paszat, L., Brown, A. D., & Sullivan, T. (2008). Response audit of an Internet survey of health care providers and administrators: Implications for determination of response rates. Journal of Medical Internet Research, 10, E30.

Donker, T., van Straten, A., Marks, I., & Cuijpers, P. (2009). A brief Web-based screening questionnaire for common mental disorders: Development and validation. Journal of Medical Internet Research, 24, E19.

Duncan, D. F., White, J. B., & Nicholson, T. (2003). Using internet-based surveys to reach hidden populations: Case of nonabusive illicit drug users. American Journal of Health Behavior, 27, 208-218.

Elford, J., Bolding, G., Davis, M., Sherr, L., & Hart, G. (2004). The Internet and HIV study: Design and methods. BMC Public Health, 4, 39.

Eysenbach, G., & Wyatt, J. (2002). Using the Internet for surveys and health research. Journal of Medical Internet Research, 4, E13.

Fernandez, M. I., Warren, J. C., Varga, L. M., Prado, G., Hernandez, N., & Bowen, G. S. (2007). Cruising in cyber space: Comparing Internet chat room versus community venues for recruiting Hispanic men who have sex with men to participate in prevention studies. Journal of Ethnicity in Substance Abuse, 6, 143-162.

Graham, A. L., Milner, P., Saul, J. E., & Pfaff, L. (2008). Online advertising as a public health recruitment tool: Comparison of different media campaigns to increase demand for smoking cessation interventions. Journal of Medical Internet Research, 10, E50.

Hallett, J., Maycock, B., Kypri, K., Howat, P., & McManus, A. (2009). Development of a web-based alcohol intervention for university students: Processes and challenges. Drug and Alcohol Review, 28, 31-39.

Hedman, E., Ljotsson, B., Ruck, C., Furmark, T., Carlbring, P., Lindefors, N., & Andersson, G. (2010). Internet administration of self-report measures commonly used in research on social anxiety disorder: A psychometric evaluation. Computers in Human Behavior, 26, 736-740.

Hidaka, Y., Ichikawa, S., Koyano, J., Urao, M., Yasuo, T., Kimura, H., One-Kihara, M., & Kihara, M. (2006). Substance use and sexual behaviours of Japanese men who have sex with men: A nationwide internet survey conducted in Japan. BMC Public Health, 6, 239.

Im, E. O., Chee, W., Tsai, H. M., Bender, M., & Lim, H. J. (2007). Internet communities for recruitment of cancer patients into an Internet survey: A discussion paper. International Journal of Nursing Studies, 44, 1216-1269.

Im, E. O., Shin, H. J., & Chee, W. (2008). Characteristics of midlife women recruited through Internet communities/groups. Computers, Informatics, Nursing, 26, 39-48.

Joinson, A. N., Paine, C., Buchanan, T., & Reips, U. D. (2008). Measuring self-disclosure online: Blurring and non-response to sensitive items in web-based surveys. Computers in Human Behavior, 24, 2158-2171.

Katz, N., Fernandez, K., Chang, A., Benoit, C., & Butler, S. F. (2008). Internet-based survey of nonmedical prescription opioid use in the United States. Clinical Journal of Pain, 24, 528-535.

Koo, M., & Skinner, H. (2005). Challenges of Internet recruitment: A case study with disappointing results. Journal of Medical Internet Research, 7, E6.

Leece, P., Bhandari, M., Sprague, S., Swiontkowski, M. F., Schemitsch, E. H., Tornetta, P., Devereaux, P. J., & Guyatt, G. H. (2004). Internet versus mailed questionnaires: A controlled comparison. Journal of Medical Internet Research, 6, E39.

Lieberman, D. Z., & Huang, S. W. (2008). A technological approach to reaching a hidden population of problem drinkers. Psychiatric Services, 59, 297-303.

Lin, K. Y., & Lu, H. P. (2011). Why people use social networking sites: An empirical study integrating network externalities and motivation theory. Computers in Human Behavior, 27, 1152-1161.

Link, M. W., & Mokdad, A. H. (2005). Alternative modes for health surveillance surveys: An experiment with web, mail, and telephone. Epidemiology, 16, 701-704.

Looby, A., & Earleywine, M. (2007). Negative consequences associated with dependence in daily cannabis users. Substance Abuse Treatment, Prevention, and Policy, 2, 3.

Miller, P. G., Johnston, J., McElwee, P. R., & Noble, R. (2007). A pilot study using the internet to study patterns of party drug use: Processes, findings and limitations. Drug and Alcohol Review, 26, 169-174.

Miller, P. G., & Sonderlund, A. L. (2010). Using the internet to research hidden populations of illicit drug users: A review. Addiction, 105, 1557-1567.

Moloney, M. F., Aycock, D. M., Cotsonis, G. A., Myerburg, S., Farino, C., & Lentz, M. (2009). An Internet-based migraine headache diary: Issues in Internet-based research. Headache, 49, 673-686.

Nicholson, T., White, J., & Duncan, D. F. (1999). A survey of adult recreational drug users via the World Wide Web: The DRUGNET Study. Journal of Psychoactive Drugs, 31, 415-422.

Novak, S. P., Kroutil, L. A., Williams, R. L., & Van, D. L. (2007). The nonmedical use of prescription ADHD medications: Results from a national Internet panel. Substance Abuse Treatment, Prevention and Policy, 2, 32.

Reips, U. D. (2002). Standards for Internet-based experimenting. Journal of Experimental Psychology, 49, 243-256.

Rhodes, S. D., Bowie, D. A., & Hergenrather, K. C. (2003). Collecting behavioural data using the World Wide Web: Considerations for researchers. Journal of Epidemiology and Community Health, 57, 68-73.

Ritter, P., Lorig, K., Laurent, D., & Matthews, K. (2004). Internet versus mailed questionnaires: A randomized comparison. Journal of Medical Internet Research, 6, E29.

Riva, G., Teruzzi, T., & Anolli, L. (2003). The use of the internet in psychological research: Comparison of online and offline questionnaires. Cyberpsychology and Behavior, 6, 73-80.

Rodgers, J., Buchanan, T., Scholey, A. B., Heffernan, T. M., Ling, J., & Parrott, A. C. (2003). Patterns of drug use and the influence of gender on self-reports of memory ability in ecstasy users: A web-based study. Journal of Psychopharmacology, 17, 389-396.

Sixsmith, J., Boneham, M., & Goldring, J. E. (2003). Accessing the community: Gaining insider perspectives from the outside. Qualitative Health Research, 13, 578-589.

StatPac Inc. (2004). StatPac for Windows User's Manual. StatPac Inc, Minnesota.

Temple, E. C., Brown, R. F., & Hine, D. W. (2011). The ‘grass ceiling’: Limitations in the literature hinder our understanding of cannabis use and its consequences. Addiction, 106, 238-244.

Tokunaga, R. S. (2011). Social networking site or social surveillance site? Understanding the use of interpersonal electronic surveillance in romantic relationships. Computers in Human Behavior, 27, 705-713.

Walden, N., & Earleywine, M. (2008). How high: Quality as a predictor of cannabis-related problems. Harm Reduction Journal, 5, 20.

Warmerdam, L., van Straten, A., & Cuijpers, P. (2007). Internet-based treatment for adults with depressive symptoms: The protocol of a randomized controlled trial. BMC Psychiatry, 7, 72.

West, R., Gilsenan, A., Coste, F., Zhou, X., Brouard, R., Nonnemaker, J., Curry, S. J., Sullivan, S. D. (2006). The ATTEMPT cohort: A multi-national longitudinal study of predictors, patterns and consequences of smoking cessation; introduction and evaluation of internet recruitment and data collection. Addiction, 101(9), 1352-1361.

Whitehead, L. C. (2007). Methodological and ethical issues in Internet-mediated research in the field of health: An integrated review of the literature. Social Science and Medicine, 65, 782-791.

Received 19 December 2011 | Accepted 11 February 2012 | Published 18 February 2012

Copyright © 2011 Journal of Research Practice and the author