Volume 10
Number 4

Special Issue: Psychometrics

In this Issue


Special Issue: Psychometrics

An Ethological Approach to Measuring Maternal Caretaking During Critical Child Illness

The Psychometric Properties of the 21-Item Depression Anxiety and StressScale (DASS-21) among a Sample of Young Adults

Measuring Self-Efficacy: Development of the Physical Activity Assessment Inventory

Further Validation of the Motivators and Barriers of a Healthy Lifestyle Scale

Further Validation of the Body-Mind-Spirit Wellness Behavior and Characteristic Inventory for College Students

A Critical Appraisal of Two Measures for Pressure Ulcer Assessment

Development and Psychometric Testing of the Pap Smear Intention Questionnaire

HIV-Related Scales Psychometrically Validated for Rural African-American Women

Psychometric Analysis of the Adolescent Decision-Making Questionnaire

Other Articles

Public Health Nursing in Mississippi: Changes in Context and Practice

Socio-Demographic Variables and Self-Efficacy in Caucasian and African American Adults with Type 2 Diabetes

Cluster Analysis of Alabama Adolescent Health Risk and Health Compromising Behaviors

NCLEX-RN Success: Are there Predictors?

Getting Connected: The Use of the Internet for Nursing Research

Verbal Abuse: The Words that Divide – Impact on Nurses and Their Perceived Solutions


Getting Connected: The Use of the Internet for Nursing Research

December 2010

(Download PDF)

Joy Longo, DNS, RNC-NIC
Assistant Professor
Christine E. Lynn College of Nursing
Florida Atlantic University
Boca Raton, Florida


The advent of the World Wide Web and the use of personal technology have opened avenues of communication that can impact the way that studies are conducted. The growth of the Internet provides a unique opportunity to expand a researcher’s capability to recruitment participants and to data collection. Before utilizing online resources for conducting research, several factors that could potentially influence the ability to carry out the study need to be considered. These include the technical ability of the researcher and available resources, and ethical principles in relation to open access. Sampling designs, recruitment strategies and response rates for online research also need to be contemplated. When conducting a study to evaluate the psychometric properties of a newly developed research tool, the effect of online testing versus the traditional paper-and-pen method on validity and reliability need to be understood. With thoughtful planning, the Internet can be a valuable asset for carrying out research.

Keywords: Internet Research, Psychometrics, Technology, Ethics 

Getting Connected: The Use of the Internet for Nursing Research

The advent of the World Wide Web and personal technology that allows virtually instantaneous access to one another has changed the way that the transfer of information occurs between individuals. Between 2000-2009 the use of the Internet has grown 399.3% worldwide,1 substantiating it as a valuable means of communication and dissemination of information. Due to these technological advances, researchers have the capability to reach out to a wide audience of potential research participants. In the past this was more difficult due to the need to make personal contact with the subjects. The Internet can be used for a wide variety of research initiatives including recruitment of participants, administration of interventions, and data collection. The use of the Internet for nursing research is well-documented in the literature.2-6

With this seemingly unlimited access come challenges related to the technology and to the distance between investigator and subject that need to be considered when planning a study. These challenges include the researcher’s technological skills and access to resources, the protection of participants’ rights, the representativeness of the sample, and the validity of the study. Each of these areas will be addressed in further detail. Finally, an example of the use of the Internet for the purpose of testing the reliability and the validity of a newly developed instrument will be discussed.


Before incorporating the use of the Internet into a research plan, the technical skills of those involved in the study and the availability of resources need to be considered. For the researcher, these skills include the ability to make the research available online through the development of a website or linkage to an existing website, and the ability to use sources of personal communication such as email, listservs, and blogs. The needs will be dictated by the purpose of the Internet in the research. If the intention is to recruit participants, conduct interventions such education and support, or collect data, such as from a survey, a website dedicated to the study may be developed. This requires expertise in website development. If the principle investigator does not have this skill set, a co-investigator who is an expert in systems management can be added to the research team.7 Another option may be the use of technical support personnel available within the university or facility where the research is being conducted. If this assistance in not available, an outside contractor can be hired to perform the work. Additional tasks that can be performed by this person include website updates and management.

If a researcher is interested in collecting data through the use of a survey and establishing a dedicated a website is not an option, software programs such as Survey Monkey can be used. These sites have easy to follow instructions that assist researchers in placing their survey on the site. An advantage of this option is that the data is collected and tabulated for easy download to Excel spreadsheets. The site also provides security to protect the data, and if desired, extra security may be purchased.

A technological consideration that is important when using the Internet for research is aesthetic presentation.4 It is suggested that pictures and special features be included on research websites.4 This will draw people into the site. The appearance of the actual research instrument needs to be considered as well because the layout of an online survey can affect completion of the measure.8 One thing that needs to be considered when deciding to enhance a site with graphics is that this may slow the downloading time, especially if a dial-up connection is being used, and this can discourage continued participation in the study.3,9 The benefit of graphics needs to be weighed against the potential participant’s electronic capacities.3 When collecting data on the Internet through the use of a survey, presentation of the survey is important but the main objective is to ensure that the process is smooth so that the survey is completed with minimal frustration on the part of the participant.9

Another source of communication that can be utilized on the Internet is email. This service can be used for several research purposes including recruitment and data collection. Surveys can be sent via email and returned via email, fax or regular mail.10 Postal mail is associated with a time delay that does not occur with email, but other issues associated with this method arise. Some people do not regularly check their email, which can cause an overabundance of messages resulting in the message being overlooked or inadvertently deleted. In addition, email addresses may become outdated or can be blocked by security software or spam filters.10 Spam definitions vary among organizations so it is difficult to determine what will be interpreted as spam.5 These factors may impede the ability of the researcher to initiate or maintain contact with a participant.

As with all technology, an associated cost exists, and this must be considered as part of the feasibility assessment when planning a study. Though costs are associated with all research, the distribution of expenses over time may differ between Internet-based and traditional methods. If not already in place, hardware such as computers and printers, and software applications need to be purchased. If an outside contractor is hired to assist with web site development, this can be associated with an initial cost as well as hourly wages for maintenance and management of the site. For example, Bosak, Yates and Posehl (2009) explored these costs in assessing the potential of conducting a study using an Internet-based physical activity intervention.11 They reported a one-time charge for Web site development of $3,000 with additional costs for updates. The initial cost of Web hosting was $50 with an annual renewal fee of $30. Another expense was the set-up and retrieval of data estimated at $50-$100/hr. If a commercial software program, such as Survey Monkey, is used, a monthly cost can be charged for the service, but this depends on the number of responses that are needed or anticipated. Once the study is up and running, there can be cost-saving benefits. Sites such as Survey Monkey tabulate data into formats that can be downloaded as an Excel spreadsheet and easily transferred to a statistical program. This can decrease the need for a data input assistance thus off-setting the cost of the start-up and web hosting charges.7,12 Another potential economic advantage of using the Internet is a reduction in cost of postal mailings, especially when repeated mailings are required.7 The economic advantages of using the Internet need to be weighed against the cost of establishing the set-up.

Recruitment on the Internet

The coverage of the Internet allows researchers to connect with a potentially large population from which to recruit participants. There are two approaches to recruitment: 1) the open approach in which anyone who finds the research site or a link to the research can participate and 2) targeting, in which specific members of a group are contacted to participate.8,13 When using the open approach, the criteria for participation must be clear though there is a risk that people will participate who do not fit into the desired population. The second approach, targeting, can include posting messages about the study and directions to a host site on discussion boards, newgroups, or websites dedicated to the population of interest and posting advertisements in journals, newspapers or other forms of media to direct potential participants to the survey.13,14 Another method of targeting is the use of email addresses from individuals on listservs or from local, regional, or national organizations. Posting advertisements on websites or obtaining emails from listservs require permission from the website or listserv owner and is facilitated by first having an established relationship with the group.5,8,13

Though it would seem access to potential study subjects is unlimited, recruitment on the Internet does have barriers. Response rates can be low13 so multiple methods of recruitment to supplement enrollment are recommended and incentives such as gift certificates or electronic money can be offered.4,5 The vouchers can be transmitted via email. If there is concern about sending incentives by this method due to confidentiality, gift certificates can be purchased from online retailers and the unique certificate number can be displayed to the subject when their participation in the study is completed.12 The participants can then use this number to purchase items without compromising their privacy by using an email address.  

Sampling on the Internet

Though there is vast access to participants on the Internet, one of the major concerns when recruiting is the representativeness of the sample and the introduction of bias due to the characteristics of those that use the Internet. According to a survey by the U.S. Census Bureau (2009), 31.3% of the households in the survey (n=119,296) did not have Internet access at home. The householder age group with the fewest connections at home was 55 years and older (41.8%) and the age groups most connected to the Internet was 45-54 years.15 Access also varied by race with 45.5% of the Black households and 47.2% of the Hispanic households without Internet access.15 Other limitations of the Internet that could influence the sample include the availability of Internet connections in rural areas and associated costs which may lead to disproportionate use by higher socio-economic groups. There is no sampling frame that can provide a random sample of Internet users, so this lack of representativeness can affect generalizability and needs to be considered as a limitation of the study.12,13

Other issues related to the sample and bias can occur over the Internet. One issue is self-selection of participants which may be more apparent in Internet research where only those interested in a specific topic may go to a particular website.13,16 In addition, there is the possibility that participants will give socially desired responses though this has been disputed because the anonymity offered by online surveys actually decreases social desirability response set and increases self-disclosure.8,17 Another issue relates to the honesty of the participants as distance and anonymity may reduce accountability.8 Though inclusion and exclusion criteria may be explicitly stated, people can create “alter egos”.13p400 that are not true representations of themselves. Dishonesty can also take place if participants submit multiple responses, particularly if there is an incentive for participation.18 This can be addressed by the identification of the Internet protocol address (IP) of the computer where the response came from. Each computer has a unique IP, so the researcher can check to see that only one person has responded from a given computer address. This could create a problem if it is expected that participants will share a computer, such as in a work setting. A solution to this dilemma is to match consecutive responses from the same IP address on key demographic characteristics, and if there is a match, only the first response is used.19 Another approach is to assign codes for each entry. These strategies can improve sampling despite the inherent flaws of Internet-based sampling strategies.

Protection of Participant’s Rights

Once sampling is completed, ethical principles of respect for human dignity, beneficence and justice must be adhered to with consideration given to how these principles are met when research is conducted at a distance. Human dignity entails the right to self-determination and to full disclosure.20As with all types of research, informed consent is required though a signed consent cannot be obtained in a face-to-face meeting. An alternative is an online script containing an introduction to the researcher, a description of the research, the procedures, and risks and benefits may be used for this purpose.8 If data is being collected using an Internet survey, an option is the use of implied consent in which the participant can read the consent script prior to completing the survey. If they agree to participate, they complete the survey and return it. Another option is to have a statement with a check box at the bottom of the script affirming agreement. Once the participant places a check in the box, access to the survey is granted.5,8 For further verification of understanding, a check box for each element of the consent may be required limiting the participants’ ability to proceed unless all boxes are checked.12 An advantage to the distance between the research and the participant may be that the right to self-determination may be easier in that a potential participant may simply ignore a request without having to reply directly to the researcher or log out of the computer.

The second ethical principle to be considered is beneficence. Beneficence means to do good or do no harm. Though the threat of physical risk is low in Internet research, psychological distress may occur when recalling past events or stressful situations during data collection. If participants experience emotional distress during the study, they can terminate participation.17 Information regarding mental health services should be provided, but most services are localized and not accessible to an online participant; therefore, the researcher needs to be available to debrief and to assist in obtaining help if needed. Researcher contact information must be readily available, so it is advisable that the participant print out the consent script for this purpose.  Another method of debriefing is to provide post debriefing materials on a Website that address particular conditions related to the research.12 Another matter associated with the principle of beneficence may arise when recruiting from a site that is dedicated to helping an individual, such as an online support group. If a researcher intrudes upon the group, the members may no longer feel secure in participating, and thus lose the support of the group.12 As in all research, the risks need to be weighed against the benefits for the participants.  

The third ethical principle that needs consideration is justice. This is the one that is most often violated in online research.21 Though use of the Internet is the way of life for many people, disparities still exists among users, as discussed under generalizability. Lack of resources to access the Internet interferes with equal opportunities for participating in potentially beneficial research. If the decision was made to recruit only on the Internet, this would be a concern. To compensate for this, alternative methods of recruitment can be employed.

Another ethical concern is the possible breach in confidentiality that can occur when using the Internet.7 Though a host site may appear secure, no system is 100% safe21 and participants need to be informed of this possible scenario. Data can be vulnerable as it is being transmitted and while it sits in temporary directories on intervening computers before it is delivered to the final addressee.12 Potential approaches to address this risk include not collecting identifying information or keeping demographic information separate from survey information.12 If someone does hack into the system, this information will not be accessed. The possibility of a breach always exists, however, so measures need to be taken to prevent compromising the data.

Reliability and Validity of Internet Research

When using a new method for conducting a study, the researcher must assure that the validity of the study is upheld. For research done online, this question is raised in regard to data collection and whether an Internet survey will perform like a traditional pen-and-paper survey.  Evidence suggests that the distribution of responses obtained from an online study is not directly comparable to established norms, but that measures can perform equivalently using both traditional and online methods.8,22 For example, in an evaluation of the online format of the Psychological Adaption to Genetic Information Scale, principle components analysis with varimax rotation explained 57.7% of total variance and had a Cronbach’s alpha of .90.6 In a study of bereavement, a traditional version of a questionnaire was compared to an online version. The researchers found that use of the Internet afforded a greater diversity of participants while not altering the psychometric characteristics of the instrument. A comparison was made between online and traditional versions of 16 self-report instruments used to include patient interventions.23 In another study the authors compared 16 self-report instruments used to evaluate patient interventions. No significant differences where found, construct validity was similar, and the Internet test-retest reliability was high.24 It is possible that the presentation of an instrument can be changed due to software or hardware and speed of transmission, but these studies suggest that results from instruments used online can be comparable to data obtained by traditional methods.

Response Rates

Validity can also be influenced by response rates, and Internet surveys are prone to lower response rates thus posing a threat to validity.13 In a study of orthopedic surgeons’ treatment of femoral neck fractures, both Internet and mailed surveys were used, yielding a significant difference (p<0.01) between the response rates of the Internet surveys (45%) and mailed surveys (58%).25 Reasons for low response rates for Internet surveys include lack of motivation to participate and frustration with spam emails.4 Response rates may also be affected by the timing of survey, particularly when sent out by email, so holidays and vacation seasons should be taken into consideration.4 The ease of use is an important factor in determining if a subject will fully participate in a survey, so assuring that the process is succinct and does not require excessive time for downloading will alleviate this concern.16

Reporting response rates is also a challenge in regard to Internet data collection. When an open access approach is used, it is not possible to describe a response rate or to define the study population.8 When using an email targeting approach, a receipt can be requested for the email and the response rate could be reported based on the proportion of returned surveys to the return of receipts for inquires.13 When a survey is hosted on a designated website, daily visitors can be monitored.4 Despite these options, it may be not be possible to report accurate response rates.

Validity of an Internet-based study may also be questioned due to the issue of missing data, which is a problem with this type of research. In a study comparing Internet-based and traditional pen-and-paper methods of survey distribution, few items were found to be missing from one or more scales of the traditional survey but many web surveys were submitted blank or only 1 or 2 of the scales were completed.26 This could be attributed to the participant being called away and not finishing the survey.26 This result was in contrast with a study that found that Internet questionnaires did not require as much follow-up to assure a higher completion rates when compared to mailed questionnaires.24 One recommendation is to designate all fields as required. If a participant chooses not to complete an item, they can terminate participation.17 Another option is to arrange for skipped items to be highlighted and generate an error message.8 In order to allow participants to skip sensitive questions, a “reminding message” may be used instead, thus allowing the participant to continue in the study.4 This will ensure that data can be collected while protecting the rights of the participants.

Issues related to addressing these issues were explored. Following is a scenario in which the validity and the reliability of a newly developed instrument were tested using the Internet as a means of recruitment and data collection.

Implementation of an Internet-Based Study

A study was undertaken to develop an instrument to measure horizontal violence, or nurse-to-nurse aggression, in nursing. Research is needed to understand how nurses’ work behaviors impact patient outcomes and patient safety, yet this is often a difficult group to recruit for research. In order to engage nurses, the research must have meaning and salience for them.26 Nurses have busy schedules so it is burdensome to participate in any study, and if data collection occurs during work hours, there is concern about handing off patient care.5,26 To facilitate recruitment of this often difficult to reach population, a decision was made to place the instrument on the Survey Monkey host site. Technological assistance was obtained from a programmer employed by the college. Two forms of recruitment were used. Open access was implemented by having an announcement about the research with a link to the survey on a college website. Targeted access was implemented by having a local not-for-profit healthcare agency send out email notification of the survey directing potential participants to the survey link on the Website. Representations of this agency also discussed the research at local healthcare meetings in which chief nursing officers were in attendance so that information about the research could be transferred to staff nurses in the respective facilities. Reminder messages were also sent via email approximately one month after the initial mailing. Initially the responses were slow. About midway through data collection it was discovered that the survey could only be accessed from a computer once, thus eliminating the chance of multiple submissions from the same person. In this situation it was desired that multiple submissions from one computer be allowed so that nurses on a unit could use a central computer to complete the instrument. This was corrected on the host site. During data collection it was also discovered that some potential participants would access the survey but not actually complete it. One possible reason for this was that there was no option to skip questions. This was also corrected on Survey Monkey. In the end, data collection took 9 months and 259 completed surveys were obtained. Reliability of the instrument was reported to be a Cronbach’s alpha of .95 and validity is currently being evaluated by confirmatory factor analysis.

The major issues associated with Internet-based research that were encountered during the study were related to low response rates and technological choices concerning response choices and options. At one point during the study when response rates were low, consideration was given to testing the instrument using a pen-and-paper method to see if the response rate would improve. This would have also allowed comparisons between methods, but in the end, the needed responses were obtained via the Internet. A future study could entail using the traditional method to administer the instrument to assure equivalence of the formats.


The Internet can provide a viable portal for conducting research. In planning a study, the researcher needs to be prepared to address specific issues related to the use of this technology including recruitment, sampling, ethics and study validity. The points raised here are intended to help with the decision to use the Internet to facilitate research. As advances are made in technology and access to the Internet broadens, additional issues may arise related to this method of conducting research, so continued attention to these issues must be paid when planning a study online.


  1. Miniwatts Marketing Group. (2010). Intent usage statistics: The Internet big picture. Retrieved from
  2. AbuAlRub, R. F. (2006). Replication and examination of research data on job stress and coworker social support with Internet and traditional samples. Journal of Nursing Scholarship, 38(2), 200-2004.
  3. Huntington, A., Gilmour, J., Schluter, P., Tuckett, A., Bogossian, F. & Turner, C. (2009). The Internet as a research site: Establishment of a web-based longitudinal study of the nursing and midwifery workforce in three countries. Journal of Advanced Nursing, 65(6), 1309-1317.
  4. Im, E., & Chee, W. (2004). Recruitment of research participants through the Internet. Computers, Informatics, Nursing, 22(5), 289-287.
  5. Im, E-O., Chee, W., Lim, H.J., Bender, M., Tsai, H., Yang, S., & Lee, H. (2006). Recruitment of oncology nurses for Internet research: Issues and future directions. Oncology Nursing Forum, 33(2), 249-254.
  6. Read, C. Y., Perry, D. J., & Duffy, M. E. (2005). Design and psychometric evaluation of the psychological adaptation to genetic information scale. Journal of Nursing Scholarship, 37(3), 203-208.
  7. Im, E., & Chee, W. (2003). Issues in Internet research. Nursing Outlook, 51(1), 6-12.
  8. Whitehead, L. C. (2007). Methodological and ethical issues in Internet-mediated research in the field of health: An integrated review of the literature. Social Science & Medicine, 65, 782-791.
  9. Crawford, S., McCabe, S. E. & Pope, D. (2010). Applying Web-based survey design standards. Journal of Prevention & Intervention in the Community, 29(10), 43-66.
  10. Baernholdt, M. & Clarke, S. P. (2006). Internet research in an international context. Applied Nursing Research, 19, 48-50.
  11. Bosak, K.A., Yates, B. & Posehl, B. (2009). Feasibility of an Internet physical activity intervention. Western Journal of Nursing Research, 31, 648-661.
  12. Kraut, R., Olson, J., Banaji, M., Bruckman, A., Cohen, J. & Couper, M. (2004). Psychological research online: Report of Board of Scientific Affairs’ Advisory Group on the conduct of research on the Internet. American Psychologist, 59(2), 105-117.
  13. Holmes, S. (2009). Methodological and ethical considerations in designing an Internet study of quality of life: A discussion paper. International Journal of Nursing Studies, 46, 394-405.
  14. Belcher, J. V. R. (2004). Specific challenges in using the Internet in research. In J. J. Fitzpatrick & K. S. Montgomery (Eds.) Internet for nursing research: A guide to strategies, skills, and resources (pp.81-92). New York: Springer.
  15. U. S. Census Bureau. (2010, February). Current population survey. Retrieved from
  16. LeCoursiere, S. (2003). Research methodology for the Internet: External validity (Generalizability). Advances in Nursing Science, 26(4), 257-273.
  17. Cantrell, M. A. & Lupinacci, P. (2007). Methodological issues in online data collection. Journal of Advanced Nursing, 60(5), 544-549.
  18. Bowen, A. M., Daniel, C. M., Williams, M. L. & Baird, G. (2008). Identifying multiple submissions in Internet research: Preserving data integrity. AIDS Behavior, 12, 964-973.
  19. Gosling, S. D., Vazire, S., Srivastava, S. & John, O. P. (2004). Should we trust web-based studies? A comparative analysis of six preconceptions about Internet Questionnaires. American Psychologist, 59(2), 93-104.
  20. Polit, D., & Beck, C.T. (2008). Nursing research: Generating and assessing evidence for nursing practice (8th Ed.). Philadelphia: Lippincott Williams & Wilkins.
  21. Im, E-O., & Chee, W. (2004). Institutional review board issues in Internet-based research. In J. J. Fitzpatrick & K. S. Montgomery (Eds.), Internet for nursing research: A guide to strategies, skills, and resources (pp.104-119). New York: Springer.
  22. Roberts, L. D. (2007). Equivalence of electronic and off-line measures. In R.A. Reynolds, R. Woods & J. D. Baker, Handbook of research one electronic surveys and measurements (pp. 97-103). Hershey, PA: Idea Group Reference.
  23. Tolstikova, K., & Chartier, B. (2010). Internet method in bereavement research: Comparison of online and offline surveys. OMEGA, 60(4), 327-349.
  24. Ritter, P., Lorig, K., Laurent, D., & Matthews, K. (2004). Internet versus mailed questionnaires: A randomized comparison. Journal of Medical Internet Research, 6(3), e29.
  25. Leece, P., Bhandari, M., Sprague, S., Swiontkowski, M.F., Schemitsch, E.H., Tornetta, P., Devereaux, P.J. ,& Guyatt, G.H. (2004). Internet versus mailed questionnaires: A controlled comparison. Journal of Medical Internet Research, 6(4), e39.
  26. Kramer, M., & Schmalenberg, C. (2009). Incentives and procedures effective in increasing survey participation of professional nurses in hospitals. Nursing Administration Quarterly, 33(2), 174-187.