patient perceptions about a bilingual patient portal in the Los

Preparing to load PDF file. please wait...

0 of 0
100%
patient perceptions about a bilingual patient portal in the Los

Transcript Of patient perceptions about a bilingual patient portal in the Los

Journal of the American Medical Informatics Association, 26(12), 2019, 1525–1535 doi: 10.1093/jamia/ocz115
Advance Access Publication Date: 2 August 2019 Research and Applications

Downloaded from https://academic.oup.com/jamia/article-abstract/26/12/1525/5542899 by UCLA Biomedical Library Serials user on 21 May 2020

Research and Applications
Su salud a la mano (your health at hand): patient perceptions about a bilingual patient portal in the Los Angeles safety net
Alejandra Casillas,1 Giselle Perez-Aguilar,1 Anshu Abhat,2 Griselda Gutierrez,2 Tanya T. Olmos-Ochoa,3 Carmen Mendez,2 Anish Mahajan,2 Arleen Brown,1 and Gerardo Moreno4
1Division of General Internal Medicine and Health Services Research, Department of Medicine, David Geffen School of Medicine at UCLA, Los Angeles, California, USA, 2Department of Health Services, Harbor-UCLA Medical Center, Torrance, California, USA, 3VA Greater Los Angeles Healthcare System, Los Angeles, California, USA, and 4Department of Family Medicine, David Geffen School of Medicine at UCLA, Los Angeles, California, USA
Corresponding Author: Alejandra Casillas, MD, MSHS, Division of General Internal Medicine and Health Services Research, Department of Medicine, David Geffen School of Medicine at UCLA, 1100 Glendon Avenue, Suite 850, Los Angeles, CA 90024, USA; [email protected]
Received 22 January 2019; Revised 18 April 2019; Editorial Decision 3 June 2019; Accepted 13 June 2019
ABSTRACT
Objectives: Driven by beneficial patient-centered outcomes associated with patient portal use and the Affordable Care Act, portal implementation has expanded into safety nets—health systems that offer access to care to a large share of uninsured, Medicaid, and other vulnerable populations. However, little attention has been paid to the factors that affect portal accessibility by the vulnerable patients served by these health systems— including those who are limited English proficient (LEP). Materials and Methods: The Los Angeles County Department of Health Services (LAC DHS), the second-largest safety net system in the nation, launched its first patient portal, and one of the few bilingual English-Spanish interfaces in existence, in March of 2015. To explore portal awareness and perceptions, we conducted focus groups with LAC DHS patients, in English and Spanish (LEP). The Technology Acceptance Model was used to guide thematic analysis of focus group data. Results: Of the 46 participants, 37 were patients and 9 were caretakers; 23 were English-speaking and 23 Spanish-speaking LEP. All patients had diabetes or hypertension. Over half had an annual household income <$10 000, yet 78% of English-speaking and 65% of Spanish-speaking LEP participants reported at-home Internet access. Participants’ discussion centered around 3 major thematic narratives: (1) participants’ awareness or attitudes about the LAC DHS portal; (2) role of culture, language, or community with regard to portal accessibility and utility; and (3) perceived needs for successful portal implementation. Conclusions: Safety net participants identified concrete benefits to the portal and emphasized the need for portal engagement that offered accessible education, support, and resources in clinical and community settings. The portal offers an additional opportunity to engage the patient and family with trusted and validated health information, and should be further developed in this capacity. This study provides a better understanding of preferred improvements of patient portal engagement that guide broader health technology efforts to address electronic health disparities.
Key words: patient portal, digital divide, electronic health, health disparities, LEP, vulnerable patients

VC The Author(s) 2019. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: [email protected]

1525

1526

Journal of the American Medical Informatics Association, 2019, Vol. 26, No. 12

Downloaded from https://academic.oup.com/jamia/article-abstract/26/12/1525/5542899 by UCLA Biomedical Library Serials user on 21 May 2020

INTRODUCTION
Online patient portals offer a new standard for patient-centered health care1,2 and allow patients to see visits, check results, and refill medications.3 Growing evidence that portal use is associated with patient-centered care has spurred systems to focus on portals as an access point for the future of primary care.4,5 The financial incentives of the Meaningful Use Program, as part of U.S. healthcare reform in 2014 triggered a rapid uptake of portals across health systems nationwide6–9—including in the safety net. Almost 2 decades after the inception of patient portals, systems that make up the safety net and provide care to large proportions of minority, lowincome, limited English proficient (LEP), and other vulnerable patients, are implementing patient portals for the first time.4,10 And although the studies linking portal use to improvements in healthcare outcomes are encouraging, we know that vulnerable populations are substantially less likely to access a patient portal.11–22 Recent evidence demonstrates that patients with baseline barriers to health communication and care don’t access patient portals and other digital health tools in the same ways as do formally educated, English-speaking patients, with whom the technologies are often tested.21 One group that is notably left behind is the 22.3 million people in the United States who are LEP.21
Patients with LEP and chronic conditions are at increased risk for poor clinical outcomes and medication nonadherence.23–25 Because much of the required self-care must occur outside of the healthcare system, improving the health of vulnerable groups will require enhanced patient-centered chronic disease management strategies. Such strategies may be facilitated via a patient portal—and yet these patients do not utilize portal management tools that may improve clinical care.19 This is in part due to the few number of multilingual patient portals that exist.17,21 Even in large integrated health systems, such as Kaiser Permanente, where portals have been accessible for almost 20 years,18 minority race or ethnicity is the strongest negative predictor of patient portal registration, and uptake falls further for Spanish-speaking Latinos and older minority adults.18,22 These disparities in portal access or use across systems, mirror what is termed the “digital divide”— decreased access to, and lower use of, digital technology among racial or ethnic minorities in the United States. This divide is also observed among those with lower income, older age, less educational attainment, rural residence, and low health literacy.26–36 Given the previous, there is valid concern that health technologies, like the patient portal, exacerbate existing racial or ethnic and socioeconomic healthcare disparities—only to “yield further advantages for the already advantaged,” as stated in the “Inverse Care Law” argument.37
The Los Angeles County Department of Health Services (LAC DHS), the second-largest health safety net system in the nation, launched its English-Spanish patient portal in March 2015—one of the few bilingual portal interfaces in the nation. Through an integrated system of 19 public health centers and 4 county hospitals— and through partnerships with community-based clinics—LAC DHS is the safety net of a large area of southern California: caring for a majority proportion of Medicaid, low-income, low-literacy, immigrant, and LEP patients (and other vulnerable populations). The current LAC DHS MyWellness patient portal allows patients to view clinical data (labs, radiology reports, provider notes [OpenNotes]), refill medications, request or change appointments, message with their healthcare team(s), read about their prior and current diagnoses (links to educational content), and view or download their medical records. However, registration of the LAC DHS portal remains

at <15% across sites. Reasons for this limited uptake are unclear. Recent studies have explored awareness and acceptability of portals among safety net patients, citing communication factors such as low literacy and education.4,10,13,17,20,36,38–56 However, these studies have mostly left out patients who are LEP. Other work has demonstrated lower rates of patient portal use among Spanish speakers,18,21 but a 2017 study in a Spanish-speaking safety net population reported a high level of interest in a patient portal.57
If health technology is destined to serve even the most vulnerable patient, medical informatics research must answer the following questions by giving such patients a voice: What do safety net patients think about patient portals? How can portals be made more accessible from a patient engagement standpoint? In this study, we start to answer these important exploratory questions by characterizing portal perceptions for patients in safety net systems with large numbers of LEP patients, most of whom have never used a patient portal. The implementation of a bilingual portal in Los Angeles’s diverse safety net is a rare opportunity for such unique work.4
The goals of this exploratory study were to (1) describe attitudes toward the portal when presented as a possible tool for healthcare management, (2) describe perceived needs for effective portal implementation among both English-speaking and Spanish-speaking LEP patients in the LAC DHS health system, and (3) delineate the role of language and culture on portal perceived utility, including those patients that are LEP. This study seeks to inform the development of future interventions to help close the gap on these electronic health disparities.13,15
MATERIALS AND METHODS
Design and study setting
We conducted 8 focus groups with a purposive sample of community-dwelling adult patients (n ¼ 46) who spoke English or Spanish. Participants were recruited from 3 primary care clinic waiting rooms in the LAC DHS system. LAC DHS forms the core of the healthcare safety net for indigent populations in Los Angeles County—the largest, most ethnically diverse county in the United States. LAC DHS serves more than 10 million residents and provides over 2.5 million ambulatory visits every year.58–60
Participants and recruitment
Adults over 18 years of age were eligible to participate if they (1) were able to sit for about an hour, (2) had diabetes or hypertension (self-report), (3) had at least 2 primary care clinic visits in the last 6 months (self-report), (4) were proficient in English or were Spanish speaking and LEP (self-report), and (5) were able to answer the eligibility criteria screening questions. If the patient met the eligibility criteria for the study and needed the help of their caretaker to attend the discussion, the caretaker was also invited to participate in the focus group discussion. Primary care patients with demonstrated cardiovascular disease risk (had diabetes or hypertension) were primarily recruited, because these patients are already engaged in the primary care system and have concrete chronic disease management needs (which the portal may address). Prior portal studies have also focused on patient populations with cardiovascular disease risk for this reason.42,43 Of note, we did not require patients to have registered for the LAC DHS patient portal or have been active portal users for this exploratory study, largely because the investigators and community partners sought perceptions from “average” LAC DHS patients, most of which have not accessed a patient portal. If

Journal of the American Medical Informatics Association, 2019, Vol. 26, No. 12

1527

Downloaded from https://academic.oup.com/jamia/article-abstract/26/12/1525/5542899 by UCLA Biomedical Library Serials user on 21 May 2020

patients had accessed or used the portal, they were still eligible for this study.
From June to July 2017, we recruited participants from the waiting areas of a hospital-based internal medicine clinic, hospital-based geriatrics clinic, and a family medicine community clinic at 2 locations in the LAC DHS system. We posted study flyers about the focus groups on the walls of the waiting rooms to help inform potential participants about why we were visiting the clinic. A.C. and G.P.A. visited the clinic sites on multiple days until at least 7-10 patients were signed up for each focus group (purposive sampling to recruit 4 groups in English and 4 groups in Spanish).
Patients were approached about the study in the waiting room (before or after their clinic visit) and screened in a private area if they chose to participate. As part of the screening script, we informed patients that “we are conducting group discussions with patients in the LA County health system to learn how we can improve access to health information and communication, specifically through an internet patient information portal. This study will help inform how to improve these services for patients in our health system.” The information about the study was then followed by the screening prompt, “Would you like to continue with the screening? The screening will take about 5 minutes. I will ask you a few questions about yourself (like age), your health and clinic visits. You do not have to answer any questions you do not wish to answer or are uncomfortable answering, and you may stop at any time. Your participation in the screening is voluntary.” If the patient wished to continue, we asked them brief questions regarding the eligibility criteria, as stated previously.
If eligible, patients who expressed interest provided their contact information. We later called patients and scheduled them for a focus group session. Each participant who attended was provided with a $30 gift card and dinner during their scheduled focus group.
Data collection and focus groups
Before the focus group discussion, we administered a short questionnaire to gather information on demographics and technology use (Internet; computer, tablet, or smartphone use; social media use; and online shopping or banking). We also administered the short 8item Electronic Health Literacy Scale (e-HEALS) to gauge patient and caregiver electronic health literacy.61
The discussion questions (in English and Spanish) were based on a guide used at Kaiser Permanente and tailored by the study team and community partners (this focus guide is available upon request).39 After introductions, patients were asked about awareness regarding the LAC DHS MyWellness patient portal (whether they had heard about it, how they heard about it). They were introduced to the portal functionalities with detailed screenshots. Patients were then shown a 10-minute video to demonstrate some of the portal’s current functions (video created for the purposes of these focus groups). The video included the following portal functionalities: (1) how to register and log in, (2) language toggle, (3) sending a message to the doctor, (4) viewing lab results, (5) viewing medication list and refill prescriptions, (6) schedule or review appointments, and (7) medical records (viewing or downloading visit information, such as discharge summaries from a hospital visit).
Following the video, patients were asked what they thought was useful and how they envisioned the portal could affect their daily life. The group discussions wrapped up with a conversation about what LAC DHS could do to help improve engagement with the portal.

Focus groups lasted 60-80 minutes and consisted of about 5-10 patients and caregivers. A.C. facilitated the focus groups. The discussions took place in clinic meeting rooms during prescheduled evenings. Focus group discussions were digitally audio-taped, transcribed, and reviewed for accuracy and de-identified. The study was approved by the University of California, Los Angeles, Institutional Review Board.
Analysis
A.C. and G.P.A. independently read and summarized the 8 focus group transcripts. Inductive methods were used to identify the spectrum of codes (ie, themes) encountered for each of the focus groups with an accompanying example quotation.62,63 A.C. and G.P.A. then cross-referenced each of the independent list of codes, for each focus group, and develop a complete set of codes.
The definition for each code, with final example quote, was developed by A.C., with iterative feedback from the entire group, until clear consensus was achieved (codebook also available upon request). With the codebook, A.C. and G.P.A. coded the transcripts using ATLAS.ti version 11.7.64 Four rounds of independent coding by A.C. and G.P.A. took place, and differences in coding were resolved through discussion between A.C. and G.P.A. After this stage of the analysis was complete, the codes or themes were then grouped into 3 major narratives— the academic and community partners (LAC DHS study team) inductively arrived at these 3 unifying narratives after group discussions.
Themes within each narrative were then matched to factors within the Technology Acceptance Model (TAM) (Figure 1).65,66 The TAM identifies the theoretical pathways that explain a person’s behavioral intention or actual use of a technology. Using the TAM, we observed how themes within our population mapped onto an established conceptual framework of technology acceptance (with the patient portal as the specific example).
Although the TAM was originally meant to be a quantitative model, this established framework guides the qualitative understandings of our findings, and the theorized order of the relationships between the themes (Figure 1). The key variables of the TAM include a person’s perceived usefulness (PU) of the technology, their perceived ease of use (PEU), and their attitudes toward the technology (ATT). The model extends to include a person’s belief in their own ability to use the technology, “computer” self-efficacy (CSE), their perception of how others in their social network perceive the technology and their use of that technology, subjective norms (SN), and their belief that the resources to support their technology use are present, facilitating conditions (FC). The relationship among these variables then leads to actions or thoughts that demonstrate an intention to use the technology, behavioral intention (BI), which is followed by actual system use (USE).
RESULTS
Description of participants
A total of 71 individuals were approached about the study, and 46 participated (65% participation rate): 37 patients and 9 caretakers. Table 1 summarizes the demographic and clinical characteristics of the participants. Of the 46 participants, 59% were women, and 50% were Spanish-speaking with self-reported LEP, 63% were Latino, 22% were African American, and 9% were white. Table 2 summarizes participants’ prior experiences with digital devices and Internet access or use. Although over half of participants were al-

1528

Journal of the American Medical Informatics Association, 2019, Vol. 26, No. 12

Downloaded from https://academic.oup.com/jamia/article-abstract/26/12/1525/5542899 by UCLA Biomedical Library Serials user on 21 May 2020

Figure 1. The Technology Acceptance Model. ATT: attitudes toward the technology; BI: behavioral intention; CSE: “computer” self-efficacy; FC: facilitating conditions; PU: perceived usefulness; PEU: perceived ease of use; SN: subjective norms; USE: system use.

ready looking up health information online or using Facebook, 22% of the English speakers and only 4% of Spanish speakers had ever used a portal. Table 3 shows patients’ electronic health literacy, as measured by the validated e-HEALS scale (composite score ranges from 8 to 40, with 40 being the highest electronic health literacy). Mean e-HEALS scores were not significantly different between the English (n ¼ 26) and Spanish speakers (n ¼ 25), and were similar to the scores found among previously studied populations.61
Thematic narratives
Participants’ themes regarding a meaningful patient portal experience in the safety net setting centered around 3 major narratives: (1) participants’ awareness or attitudes about the LAC DHS portal, (2) role of culture or language or community with regard to portal accessibility and utility, and (3) perceived needs for a successful portal implementation in the safety net setting. Themes are matched to factors in the TAM framework (Figure 1).
NARRATIVE I. AWARENESS OR ATTITUDES ABOUT THE LAC DHS PORTAL
Focus group participants reflected on their perceptions of how the LAC DHS portal might be beneficial for their health management and cited empowerment as a potential byproduct of accessing their health information through the portal.
Perceived portal utility and benefit
This theme mapped onto 2 TAM factors: perceived usefulness (PU) and perceived ease of use (PEU). Although most patients were not aware of the LAC DHS portal before the focus groups, participants discussed the barriers to health care in the safety net that could be spe-

cifically improved by the current portal, after they watched demonstration video of the observed functionalities. These included access to medical records (ie, would not have to pay expensive fees for printing records at the county hospital), access to labs or tests, better medication adherence secondary to easier medication refills, less time spent on phone calls and transportation, and fewer missed appointments. Participants mentioned how the patient portal could help to fix the “broken” system—the portal was compared to a “foundation” by one caretaker in a Spanish focus group.
Participants also discussed the portal as a centralized “hub” for health information where they would get personalized information about their medication’s side effects, their specific health condition, and lifestyle instruction (eg, exercise videos and diet recipes). They discussed the portal’s impact on health communication, perceiving that they would be able to access helpful health information to show to treatment teams when they sought doctors outside of the LAC DHS system. Many stated that they would be better aware of any changes made to their treatment plan because the “concrete details are found in the portal regardless of relationship problems with the doctor.”
One caretaker discussed the care of her husband, who has multiple chronic diseases:
“Because being bounced between a nephrologist that never gets back to us and between our primary doctor that works for a busy clinic, we could never get the answer. This way now we’re going to get an answer, which helps us better manage his health. A lot of things should have been controlled maybe over a year ago, but we didn’t know how to. We had to go through all the steps, the hassles. This would have probably cut the time by one-third and him suffering less and getting everything addressed sooner. So. . .it will improve not only his health, but the time management to come down here and wait. . .”

Downloaded from https://academic.oup.com/jamia/article-abstract/26/12/1525/5542899 by UCLA Biomedical Library Serials user on 21 May 2020

Journal of the American Medical Informatics Association, 2019, Vol. 26, No. 12

Table 1. Demographic and clinical characteristics of focus group participants

Total Participants (N ¼ 46)

Age, y Female Race/ethnicityb
White (non-Hispanic, non-Latino) Black or African American Latino/Hispanic Asian Other Highest educationb Less than Grade 8 Less than high school High school/GED Some college College Annual family/household income <$10 000 $10 000-$29 999 $30 000-$49 999 >$50 000 Chronic medical condition(s) Diabetes Hypertension Coronary artery disease Kidney disease Liver disease Asthma Chronic obstructive or pulmonary disease/diagnosis Take medication(s) prescribed by MD Health status Excellent Very good Good Fair Poor

56.1 6 10.4 58.7
8.7 21.7 63.0
4.4 2.2
23.9 19.6 30.4 17.4
8.7
54.6 31.8 13.6
0
60.9 63.0 28.3 28.3
2.2 6.5 4.4
93.2
8.7 6.5 17.4 45.7 21.7

Values are mean 6 SD or %. aP < .05. bP < .01.

English-speaking participants (n ¼ 23)
54.3 6 10.5 56.5
17.4 43.5 26.1
8.7 4.4
4.3 8.7 34.8 34.8 17.4
39.1 39.1 21.8
0
60.9 60.9 21.7 30.4
0 8.7 4.4
90.9
4.4 8.7 21.7 39.1 26.1

1529
Spanish-speaking Participants (n ¼ 23)
58.0 6 10.1 60.9
0 0 100 0 0
43.5 30.4 26.1
0 0
71.4 23.8
4.8 0
60.9 65.2 34.8 26.1
4.4 4.4 4.4
95.5
13.0 4.4
13.0 52.2 17.4

Patient empowerment
Linking to the TAM’s attitudes to technology (ATT) and “computer” self-efficacy (CSE), participants mentioned feelings of preparedness, independence, and security secondary to the access that the portal would provide. They reported that they would feel more prepared to participate in self-care because of the immediacy of the information and more prepared or empowered to partner in decisions with their physician and other members of the healthcare team. One patient noted they would feel “on common ground.” Participants thought that they would take the time to understand their medical information “peacefully” with the portal— on their own terms or time and not frazzled by the rush of a short medical appointment.
They predicted less worry in figuring out how to reach their medical team outside of the clinic. Because of the portal providing access to their own information, patients envisioned themselves as more proactive in self-care and positive health behaviors, thereby, perhaps, displaying a higher level of self-efficacy in relation to disease management. Participants also talked about the portal as a motivator to make big lifestyle changes—like learning to use digital technology. Particularly among the Spanish speakers, those who

were less tech-savvy perceived health care as a useful reason to engage with the Internet vs something like Facebook: “¡Si se aprenden las cosas malas, las cosas buenas mucho mejor!” (“If you use the Internet to learn bad things, better to learn to use it for something good!”). One Spanish-speaking patient discussed a sense of security that would come along with a portal, having their “salud a la mano,” or “their health at hand”:
“Thinking about what will happen or what will this happen to you. In short, one is always doubting. Instead (with this portal), well, you have it at hand, your health is at hand. Any time you investigate and you realize what happened. . . because we realize that doctors only have so much time to be with each patient, and it is not very much. Instead, on the computer you take your own time to see.”
Another patient likened a portal to a tool that enabled a routine check-in for one’s health:
“It’s just something about making it a routine part of your life to go over your health status or something. . . you’re more mindful. . .I look at it like a bank account coming once a month.”

Downloaded from https://academic.oup.com/jamia/article-abstract/26/12/1525/5542899 by UCLA Biomedical Library Serials user on 21 May 2020

1530

Journal of the American Medical Informatics Association, 2019, Vol. 26, No. 12

Table 2. Digital device and internet use among focus group participants
Has personal access to a computer/laptop/tablet at home Has access to the Internet at home Has access to the Internet outside the homea Respondent personally uses the Internet Family member or a friend helps respondent use the Internet Uses a phone that sends/receives textsa Uses a phone that connects to Internet Has ever used Internet for banking/bills Has ever used the Internet for shoppinga Has ever used the Internet for health info Has ever used the Internet for info related to personal chronic
disease or medical issues Has an electronic message account that is used on a regular basis (“e-mail”) Has ever sent an electronic message to their MD Has ever received an electronic message from their MD Has ever used Facebook on a computer/laptop/tablet Has ever used Facebook on a phone Has ever used a personal electronic patient portal to access health care services
aP < .05. bP < .01.

Total participants (N ¼ 46) (%)
55.2 77.8 64.4 71.1 37.8 84.4 69.6 32.6 30.4 54.4 56.5
55.6 4.4 9.1
47.8 54.4 13.0

English-speaking participants (n ¼ 23) (%)
65.2 90.9 86.4 86.4 45.5 95.5 78.3 43.5 47.8 60.9 69.6
52.2 8.7
13.6 52.2 52.2 21.7

Spanish-speaking participants (n ¼ 23) (%)
39.1 65.2 43.5 56.5 30.4 73.9 60.9 21.7 13.0 47.8 43.5
59.1 0 4.6
43.5 56.5
4.4

NARRATIVE II. ROLE OF CULTURE, LANGUAGE, AND COMMUNITY
Focus group participants discussed communication benefits of the portal, specifically in light of their own prior barriers with the healthcare system, that had been affected by factors such as culture, language, and community. Patients’ discussion of how they would use the portal also highlighted the central impact of these factors.
Communication bridge
In relation to the TAM’s perceived usefulness, participants perceived the portal as a validated and centralized hub for health information—particularly for patients with a non-English native language and different culture. They mentioned the lack of trustworthy health information on the Internet– a subject that was repeatedly noted among the LEP Spanish speakers. These patients also noted the advantage of being able to bypass communication and language barriers experienced in person and around the health system through a patient portal. One Spanish speaker noted the following about bypassing language barriers:
“More communication between patient and doctor, maybe that. . . one can translate the information the visit from Spanish to English, then if the doctor speaks English and the patient only Spanish, then there is another way to translate everything and one can communicate more directly, more easily with the doctor through that.”
Community portal
Patients perceived that a portal would have positive effects within their social networks, very much in relation to the TAM’s definition of subjective norms (SN), how a social network perceives the technology, and the patient’s use of that technology. The discussion of this theme included the portal having an impact on family or friend participation with the patient’s health care by engaging

family members in health decisions, giving caregivers needed access (especially in case of emergencies), and showing medical information relevant to the patient (ie, easier for the patient to explain condition to loved ones, instead of the patient having to explain all their health issues and events to family members). Patients saw the portal as a means toward sharing medical information with a trusted source and better understanding this information through the support of their community or family members. Some mentioned that the portal would also impact the health or lifestyle of the patient’s friends or family, by being a resource for health information for family members themselves. One caregiver noted that access to a portal would get her adult children more involved in her husband’s care:
“. . .I am the only one who gives medication to my husband. I tell my children- ‘Your father has been taking the medication for 3 years now and you guys don’t know which medicines to give him and at what time either’. . .when the page is there already, they would only need to open the page and they’ll know what to do. . .”
Patients indicated that family support, guidance to help patients navigate the portal, and practice with younger or tech-savvy family members would be necessary to use the portal well. One Spanishspeaking patient noted that her son was made aware of the portal through the hairdresser, and she returned to the salon to ask the hairdresser:
“My son told me that the hairdresser had told him about this. . .where he’s going to cut his hair, that the lady had told him if he could help her to program her phone for it. . .that with this she could look at all her doctors’ appointments and everything, the exams and everything they did to her. So, he went to me and said, ‘Why don’t you get this program, so you can look at your appointments?’ Then he tried to do it for me, but I do not know what happened and he could not. Then I came here (to the beauty salon) and I said to my daughter, ‘look, there’s the lady . . .’ and another lady who passed by said to me said, ‘ask her

Journal of the American Medical Informatics Association, 2019, Vol. 26, No. 12

1531

Downloaded from https://academic.oup.com/jamia/article-abstract/26/12/1525/5542899 by UCLA Biomedical Library Serials user on 21 May 2020

Table 3. e-HEALS among focus group participants
I know what health resources are available on the Internet 1- Strongly disagree 2- Disagree 3- Undecided 4- Agree 5- Strongly agree Mean item score I know where to find helpful health resources on the Internet 1- Strongly disagree 2- Disagree 3- Undecided 4- Agree 5- Strongly agree Mean item score I know how to find helpful health resources on the Internet 1- Strongly disagree 2- Disagree 3- Undecided 4- Agree 5- Strongly agree Mean item score I know how to use the Internet to answer my questions about health 1- Strongly disagree 2- Disagree 3- Undecided 4- Agree 5- Strongly agree Mean item score I know how to use the health information I find on the Internet to help me 1- Strongly disagree 2- Disagree 3- Undecided 4- Agree 5- Strongly agree Mean item score I have the skills I need to evaluate the health resources I find on the Internet 1- Strongly disagree 2- Disagree 3- Undecided 4- Agree 5- Strongly agree Mean item score I can tell high quality health resources from low quality health resources on the Internet 1- Strongly disagree 2- Disagree 3- Undecided 4- Agree 5- Strongly agree Mean item score I feel confident in using information from the Internet to make health decisions 1- Strongly disagree 2- Disagree 3- Undecided 4- Agree 5- Strongly agree Mean item score Mean e-HEALS score (range 8-40)
Values are %, unless otherwise indicated. e-HEALS: Electronic Health Literacy Scale.

Total participants (N ¼ 46)
11.4 13.6 15.9 45.5 13.6
1.22
13.6 11.4 18.2 45.4 11.4
1.23
15.9 13.6 15.9 40.9 13.6
1.31
16.3 7.0
20.9 48.8
7.0 1.21
13.6 18.2 13.6 38.6 15.9
1.31
16.7 19.1 16.7 38.1
9.5 1.29
11.4 27.3 22.7 27.3 11.4
1.22
15.9 20.5
6.8 43.2 13.6
1.35 8.51

English-speaking participants (n ¼ 23)
8.7 13.0 17.4 52.2
8.7 1.12
8.7 17.4 13.0 47.8 13.0
1.20
8.7 17.4 21.7 39.1 13.1
1.18
8.7 4.4 17.4 60.9 8.7 1.04
8.7 8.7 21.7 43.5 17.4 1.16
13.6 22.7 13.6 40.9
9.1 1.27
8.7 34.8 26.1 21.7
8.7 1.14
17.4 17.4 13.0 43.5
8.7 1.31 8.07

Spanish-speaking participants (n ¼ 23)
14.3 14.3 14.3 38.1 19.0
1.35
19.1 4.8
23.8 42.9
9.5 1.29
23.8 9.5 9.5
42.9 14.3
1.46
25.0 10.0 25.0 35.0
5.0 1.31
19.1 28.6
4.8 33.3 14.3
1.43
20.0 15.0 20.0 35.0 10.0
1.34
14.3 19.1 19.1 33.3 14.3
1.31
14.3 23.8
0 42.9 19.0
1.42 9.11

1532

Journal of the American Medical Informatics Association, 2019, Vol. 26, No. 12

Downloaded from https://academic.oup.com/jamia/article-abstract/26/12/1525/5542899 by UCLA Biomedical Library Serials user on 21 May 2020

how to get the program with all your doctors’ appointments and information’ . . .”
NARRATIVE III. RECOMMENDATIONS FOR SUCCESSFUL PORTAL ENGAGEMENT AND IMPLEMENTATION IN THE SAFETY NET
Participants’ primary recommendations focused on increasing patient awareness and providing forums for patients to practice with the portal.
Awareness, engagement, and messaging
Participants discussed ideas to improve portal registration, related to the TAM framework’s facilitating conditions (FC), such as community talks about portals, advertising the LAC DHS portal (pamphlets or flyers, videos in clinic). Several recommended that LAC DHS make the portal “opt-out” (automatically register patients in clinic) and introduce the portal when patients had a medical issue or problem so to connect the portal to a person’s experience. Participants also mentioned the need for physician or provider approval of portals, as a way to validate its use, and making sure to mention that “it is a free service”:
“It’ll make more emphasis on what I need to do. By talking to the doctor, I’ll know how imperative it is instead of some friend of mine who says, ‘Oh, why don’t you take a look at this?’ The doctor will give it more credibility, basically. . .”
Education and practice (ease of use)
Participants mentioned ways to make portals easy to use, which relates to the TAM’s behavioral intent (BI) on behalf of the participant. Suggestions included helping people become more familiar with computers through in-clinic computer labs, simplified pamphlets on how to use the portal, and support for portal practice and bilingual educational tutorials in clinic (portal practice or education while waiting for doctor). Many said that LAC DHS clinics needed a person in clinic to educate patients about the portal (what it was, how to start using it). Patients also asked for the ability to give feedback on the portal (patient voice, user experience), to have a “feedback loop”: “someone who specifically deals with portal problems. . .”
DISCUSSION
Despite the expansion of patient portals into the safety net, and the threat of the heightened disparities predicted by the Inverse Care Law,17,37 little attention has been paid to the process of engaging vulnerable patients into a patient portal, especially among patients who are LEP.10,13,67 Because portal uptake among patients in safety net settings remains low, and almost nonexistent among LEP patients, even with the federal prioritization of portal expansion in these health systems, this is an area that is ripe for evidence-based portal engagement strategies. In this exploratory study, we found that safety net patients are using the Internet and digital tools, but are not aware of the patient portal in their health system. These safety net patients (both English speaking and LEP) perceived benefits to using a portal in many creative and unique ways, that were tailored to the experience of their struggles. Patients asked for robust patient-centered portal education that would include community or family networks as a unique focus point of engagement and participation.

The discussion regarding health technology in diverse populations has been limited to the explanation offered by the digital divide—that vulnerable populations will not have access to, or be able to use, digital technology.26–37 Though gaps in portal use are partially due to structural barriers like access to and knowledge of the Internet, and familiarity with Internet-connected devices (computers, tablets, or smartphones),29–35,68–70 emerging research suggests that the factors driving differences in portal use among diverse patients go beyond technological access or literacy.40 Recent studies suggest that physician validation of the portal, repeated education about the portal’s tools, and use of a patient portal by friends or family are important,57,71–73 and these points were all brought up by patients in our study.69,72 There is interest in portals that has been documented among low-income, LEP populations, Medicaid recipients, and patients in public hospital settings, similar to these discussions.44,48,57,74,75
A 2015 review on state of the science for portal engagement67 concluded that patients’ interest and ability to use portals was strongly influenced by personal factors such as age, ethnicity, education level, health literacy, health status, and role as a caregiver. Notably, provider endorsement contributed to patient’s willingness to engage with the patient portal in this review—a factor observed in other recent studies, and featured prominently in our discussions.49,50,57 A 2017 study examined portal readiness in 5 free clinics serving low-income patients and found that “perceived ability to manage one’s own health” was a motivator for portal readiness among these patients,76 not unlike the code of “patient empowerment” in this study.
This study extends the existing literature by identifying specific perspectives and preferences regarding portals from vulnerable and LEP patients in a safety net setting, also in the context of a validated framework for patient-centered technology adoption, the TAM. These findings point to engagement strategies for safety net patients that prioritize: (1) physician validation or recommendation of the portal,49,50,57 (2) messaging about the portal that is useful to a patient’s struggle with the healthcare system (ie, can bypass language or communication barriers),76 and (3) educational scripts that incorporate caretakers or family or community members (an effective strategy that is already present in the chronic disease selfmanagement literature, particularly among Spanish-speaking Latinos with diabetes mellitus).77,78
As next steps from this formative work, the LAC DHS is working to revise portal engagement content (videos, informational cards) to include the role of the caretaker or family members as a way to encourage patients to use the portal and highlight the technological help or resources available in their own communities. We are also exploring nearby community spaces (libraries, schools) as partnered centers for portal engagement. The primary rehabilitation center at the LAC DHS is developing a physical space to welcome patients, and they plan to include a form of patient portal instruction in this area. We have also partnered with the Health Educator program of the LAC DHS ambulatory care network to develop and then evaluate a patient portal curriculum for existing chronic disease group visits. As echoed by some of the patients in these focus groups, bilingual support interventions to increase patient portal registration and use in the safety net are in high demand, but have not been developed and evaluated for these populations, especially for patients who are LEP.13,15,26,49,67 Based on this formative work, the LAC DHS is also developing tailored Spanish-speaking instructional content for patient registration and use in the efforts mentioned previously.58–60

Journal of the American Medical Informatics Association, 2019, Vol. 26, No. 12

1533

Downloaded from https://academic.oup.com/jamia/article-abstract/26/12/1525/5542899 by UCLA Biomedical Library Serials user on 21 May 2020

Among the limitations of this study are small sample size and generalizability to other safety net healthcare systems or other persons who speak languages other than English or Spanish. Issues related to patient portals and their implementation will undoubtedly vary across portal vendors, implementation setting, and community type (urban vs rural), which we did not measure here. Quotes are also subject to bias from the transcript readers. Most of the patients in this study had never used any patient portal; however, this was not a study about portal usability (a portal usability study is now ongoing at the LAC DHS as a next step from this formative work). The investigators sought portal perceptions from “average” LAC DHS patients in these focus groups, and the majority had not used this portal.
Results from this formative work on patient portals, which include perspectives from a LEP population, can inform portal implementation for safety net systems serving diverse patient populations across the nation. As implementation is tailored to health systems serving diverse patients, it is likely that these groups (especially those that are LEP) will need different modifications from the larger population to eliminate the digital divide. This study provides adjuvant evidence to motivate such health technology developers to create portals with this intention: incorporating the needs of vulnerable, medically underserved, and LEP populations from the outset, rather than developing portals for the “high-tech” populations. As an illustrative next step to our formative work, the LAC DHS has also begun a formative partnership with its electronic medical record vendor to commence with patient portal usability studies, so as to help changes that enhance the user interface for our patients, in the next versions of the LAC DHS portal.
CONCLUSION
Patient portals offer the potential to expand access to health services for all patients, including millions of newly insured low-income patients who gained access to insurance in 2014, and also has the potential to improve quality of care and patient experience, while reducing use of resources in overburdened settings.4 More broadly, this work is an opportunity to better understand the general mechanisms by which vulnerable populations interface with health technology, and thus, help close the emerging gap on electronic health disparities.
FUNDING
The study was funded by Research Centers for Minority Aging Research National Institutes of Health/National Institute on Aging grant no. P30AG021684 and UCLA Clinical and Translational Science Institute National Center for Advancing Translational Sciences grant no. ULITR001881 pilot investigator awards (to AC and AB); National Institutes of Health/National Institute of Neurological Disorders and Stroke Award grant no. U54NS081764 (Los Angeles Stroke Prevention/Intervention Research Program in Health Disparities).
CONTRIBUTIONS
All authors contributed significantly to the work.
PRIOR PRESENTATIONS
Society of General Internal Medicine National Meeting. Oral Podium Presentation. April 2018, Denver CO.
CONFLICT OF INTEREST STATEMENT
None declared.

REFERENCES
1. Otte-Trojel T, Rundall TG, de Bont A, van de Klundert J, Reed ME. The organizational dynamics enabling patient portal impacts upon organizational performance and patient health: a qualitative study of Kaiser Permanente. BMC Health Serv Res 2015; 15: 559.
2. Peacock S, Reddy A, Leveille SG. Patient portals and personal health information online: perception, access, and use by US adults. J Am Med Inform Assoc 2017; 24: e173–7.
3. Amante DJ, Hogan TP, Pagoto SL, English TM. A systematic review of electronic portal usage among patients with diabetes. Diabetes Technol Ther 2014; 16: 784–93.
4. Lyles CR, Aulakh V, Jameson W, Schillinger D, Yee H, Sarkar U. Innovation and transformation in California’s safety net health care settings: an inside perspective. Am J Med Qual 2014; 29 (6): 538–45.
5. Househ MS, Borycki EM, Rohrer WM, Kushniruk AW. Developing a framework for meaningful use of personal health records (PHRs). Health Policy Technol 2014; 3 (4): 272–80.
6. Kruse CS, Bolton K, Freriks G. The effect of patient portals on quality outcomes and its implications to meaningful use: a systematic review. J Med Internet Res 2015; 17 (2): e44.
7. Kamo N, Bender AJ, Kalmady K, Blackmore CC. Meaningful use of the electronic patient portal—Virginia Mason’s journey to create the perfect online patient experience. Healthc (Amst) 2017; 5 (4): 221–6.
8. Blumenthal D, Tavenner M. The “meaningful use” regulation for electronic health records. N Engl J Med 2010; 363 (6): 501–4.
9. Agency for Healthcare Research and Quality. 2010 National Healthcare Quality and Disparities Reports. Rockville, MD: AHRQ; 2014.
10. Lyles CR, Fruchterman J, Youdelman M, Schillinger D. Legal, practical, and ethical considerations for making online patient portals accessible for all. Am J Public Health 2017; 107 (10): 1608–11.
11. Halamka JD, Mandl KD, Tang PC. Early experiences with personal health records. J Am Med Inform Assoc 2008; 15 (1): 1–7.
12. Zhou YY, Kanter MH, Wang JJ, Garrido T. Improved quality at Kaiser Permanente through e-mail between physicians and patients. Health Aff (Millwood) 2010; 29 (7): 1370–5.
13. Lyles C, Schillinger D, Sarkar U. Connecting the dots: health information technology expansion and health disparities. PLoS Med 2015; 12 (7): e1001852.
14. Casillas A, Moreno G, Tseng CH, Grotts J, Morales L. A digital language divide? The relationship between Internet medication fills and medication adherence among Limited English Proficient patients. J Racial Ethn Health Disparities 2018; 5 (6): 1373–80.
15. Ancker JS, Barron Y, Rockoff ML. Use of an electronic patient portal among disadvantaged populations. J Gen Intern Med 2011; 26 (10): 1117–23.
16. Gordon NP, Hornbrook MC. Differences in access to and preferences for using patient portals and other eHealth technologies based on race, ethnicity, and age: a database and survey study of seniors in a large health plan. J Med Internet Res 2016; 18 (3): e50.
17. Lyles CR. Capsule Commentary on Moreno et al., Disparities in the use of internet and telephone medication refills among linguistically diverse patients. J Gen Intern Med 2016; 31 (3): 322.
18. Garrido T, Kanter M, Meng D, et al. Race/ethnicity, personal health record access, and quality of care. Am J Manag Care 2015; 21 (2): e103–13.
19. Goel MS, Brown TL, Williams A, Hasnain-Wynia R, Thompson JA, Baker DW. Disparities in enrollment and use of an electronic patient portal. J Gen Intern Med 2011; 26 (10): 1112–6.
20. Lyles CR, Sarkar U. Health literacy, vulnerable patients, and health information technology use: where do we go from here? J Gen Intern Med 2015; 30 (3): 271–2.
21. Moreno G, Lin EH, Chang E, et al. Disparities in the use of internet and telephone medication refills among linguistically diverse patients. J Gen Intern Med 2016; 31 (3): 282–8.
22. Roblin DW, Houston TK 2nd, Allison JJ, Joski PJ, Becker ER. Disparities in use of a personal health record in a managed care organization. J Am Med Inform Assoc 2009; 16 (5): 683–9.

1534

Journal of the American Medical Informatics Association, 2019, Vol. 26, No. 12

Downloaded from https://academic.oup.com/jamia/article-abstract/26/12/1525/5542899 by UCLA Biomedical Library Serials user on 21 May 2020

23. Shin HB, Bruno R. Language Use and English-Speaking Ability: 2000. Census 2000 Brief. 2003. https://www.census.gov/prod/2003pubs/c2kbr29.pdf. Accessed August 20, 2018.
24. Wilson E, Chen AH, Grumbach K, Wang F, Fernandez A. Effects of limited English proficiency and physician language on health care comprehension. J Gen Intern Med 2005; 20 (9): 800–6.
25. Karliner LS, Auerbach A, Napoles A, Schillinger D, Nickleach D, PerezStable EJ. Language barriers and understanding of hospital discharge instructions. Med Care 2012; 50 (4): 283–9.
26. Sequist TD. Health information technology and disparities in quality of care. J Gen Intern Med 2011; 26 (10): 1084–5.
27. Smith SG, O’Conor R, Aitken W, Curtis LM, Wolf MS, Goel MS. Disparities in registration and use of an online patient portal among older adults: findings from the LitCog cohort. J Am Med Inform Assoc 2015; 22 (4): 888–95.
28. Arora S, Ford K, Terp S, et al. Describing the evolution of mobile technology usage for Latino patients and comparing findings to national mHealth estimates. J Am Med Inform Assoc 2016; 23 (5): 979–83.
29. File T. Digital Divides: A connectivity continuum for the United States. Data from the 2011 Current Population Survey. 2012. http://paa2013. princeton.edu/papers/130743. Accessed February 20, 2018.
30. File T. Digital Divides: A Connectivity Continuum for the United States. Presented at the Annual Meeting of the Population Association of America 2013 April 11–13, 2013; New Orleans, LA Social and Economic Statistics Division, U.S. Census Bureau, U.S. Department of Commerce.
31. File T. Computer and Internet Use in the United States. Current Population Survey Reports, P20-568. Washington, DC: U.S. Census Bureau; 2013.
32. File T, Ryan C. Computer and Internet Use in the United States: 2013. American Community Survey Reports, ACS-28. Washington, DC: U.S. Census Bureau; 2014.
33. Kontos E, Blake KD, Chou WY, Prestin A. Predictors of eHealth usage: insights on the digital divide from the Health Information National Trends Survey 2012. J Med Internet Res 2014; 16 (7): e172.
34. Laz TH, Berenson AB. Racial and ethnic disparities in Internet use for seeking health information among young women. J Health Commun 2013; 18 (2): 250–60.
35. Livingston G. Latinos and digital technology, 2010. Pew Hispanic Center website. Published February 9, 2011. http://www.pewhispanic.org/2011/ 02/09/latinos-and-digital-technology-2010/. Accessed February 20, 2018.
36. Ratanawongsa N, Barton JL, Lyles CR, et al. Computer use, language, and literacy in safety net clinic communication. J Am Med Inform Assoc 2017; 24 (1): 106–12.
37. Schillinger D. Literacy and health communication: reversing the ‘inverse care law’. Am J Bioeth 2007; 7 (11): 15–8; discussion W1–2.
38. Fontil V, McDermott K, Tieu L, et al. Adaptation and feasibility study of a digital health program to prevent diabetes among low-income patients: results from a partnership between a digital health company and an academic research team. J Diabetes Res 2016; 2016: 8472391.
39. Lyles CR, Allen JY, Poole D, Tieu L, Kanter MH, Garrido T. “I want to keep the personal relationship with my doctor”: Understanding barriers to portal use among African Americans and Latinos. J Med Internet Res 2016; 18 (10): e263.
40. Lyles CR, Harris LT, Jordan L, et al. Patient race/ethnicity and shared medical record use among diabetes patients. Med Care 2012; 50 (5): 434–40.
41. Lyles CR, Harris LT, Le T, et al. Qualitative evaluation of a mobile phone and web-based collaborative care intervention for patients with type 2 diabetes. Diabetes Technol Ther 2011; 13: 563–9.
42. Lyles CR, Sarkar U, Ralston JD, et al. Patient-provider communication and trust in relation to use of an online patient portal among diabetes patients: the Diabetes and Aging Study. J Am Med Inform Assoc 2013; 20 (6): 1128–31.
43. Lyles CR, Sarkar U, Schillinger D, et al. Refilling medications through an online patient portal: consistent improvements in adherence across racial/ ethnic groups. J Am Med Inform Assoc 2016; 23 (e1): e28–33.

44. Mirsky JB, Tieu L, Lyles C, Sarkar U. A mixed-methods study of patientprovider e-mail content in a safety-net setting. J Health Commun 2016; 21 (1): 85–91.
45. Rowsell A, Muller I, Murray E, et al. Views of people with high and low levels of health literacy about a digital intervention to promote physical activity for diabetes: a qualitative study in five countries. J Med Internet Res 2015; 17 (10): e230.
46. Sarkar U, Gourley GI, Lyles CR, et al. Usability of commercially available mobile applications for diverse patients. J Gen Intern Med 2016; 31 (12): 1417–26.
47. Sarkar U, Lyles CR, Parker MM, et al. Use of the refill function through an online patient portal is associated with improved adherence to statins in an integrated health system. Med Care 2014; 52 (3): 194–201.
48. Schickedanz A, Huang D, Lopez A, et al. Access, interest, and attitudes toward electronic communication for health care among patients in the medical safety net. J Gen Intern Med 2013; 28 (7): 914–20.
49. Tieu L, Sarkar U, Schillinger D, et al. Barriers and facilitators to online portal use among patients and caregivers in a safety net health care system: a qualitative study. J Med Internet Res 2015; 17 (12): e275.
50. Tieu L, Schillinger D, Sarkar U, et al. Online patient websites for electronic health record access among vulnerable populations: portals to nowhere? J Am Med Inform Assoc 2016; 24 (e1): e47–54.
51. Clarity C, Gourley G, Lyles C, et al. Implementation science workshop: barriers and facilitators to increasing mammography screening rates in California’s Public Hospitals. J Gen Intern Med 2017; 32 (6): 697–705.
52. Garg SK, Lyles CR, Ackerman S, et al. Qualitative analysis of programmatic initiatives to text patients with mobile devices in resource-limited health systems. BMC Med Inform Decis Mak 2016; 16: 16. doi: 10.1186/ s12911-016-0258-7.
53. Lyles CR, Altschuler A, Chawla N, et al. User-centered design of a tablet waiting room tool for complex patients to prioritize discussion topics for primary care visits. JMIR Mhealth Uhealth 2016; 4 (3): e108.
54. Lyles CR, Sarkar U, Osborn CY. Getting a technology-based diabetes intervention ready for prime time: a review of usability testing studies. Curr Diab Rep 2014; 14: 534.
55. Mirsky JB, Tieu L, Lyles C, Sarkar U. Readability assessment of patientprovider electronic messages in a primary care setting. J Am Med Inform Assoc 2016; 23 (1): 202–6.
56. Tuot DS, Leeds K, Murphy EJ, et al. Facilitators and barriers to implementing electronic referral and/or consultation systems: a qualitative study of 16 health organizations. BMC Health Serv Res 2015; 15: 568.
57. Ochoa A, Kitayama K, Uijtdehaage S, et al. Patient and provider perspectives on the potential value and use of a bilingual online patient portal in a Spanish-speaking safety-net population. J Am Med Inform Assoc 2017; 24 (6): 1160–4.
58. Hay JW, Katon WJ, Ell K, Lee PJ, Guterman JJ. Cost-effectiveness analysis of collaborative care management of major depression among low-income, predominantly Hispanics with diabetes. Value Health 2012; 15 (2): 249–54.
59. Wu S, Ell K, Gross-Schulman SG, et al. Technology-facilitated depression care management among predominantly Latino diabetes patients within a public safety net care system: comparative effectiveness trial design. Contemp Clin Trials 2014; 37 (2): 342–54.
60. Huckfeldt PJ, Meeker D, Peters A, Guterman JJ, Diaz G Jr, Goldman DP. Diabetes management for low-income patients in Los Angeles: two strategies improved disease control in the short term. Health Aff (Millwood) 2012; 31 (1): 168–76.
61. Norman CD, Skinner HA. eHEALS: the eHealth literacy scale. J Med Internet Res 2006; 8 (4): e27.
62. Kahlke RM. Generic qualitative approaches: pitfalls and benefits of methodological mixology. Int J Qual Methods 2014; 13 (1): 37–52.
63. Sally T, Reimer KS, Me J. Interpretive description: a noncategorical qualitative alternative for developing nursing knowledge. Res Nurs Health 1997; 20: 169–77.
64. Atlas.ti Qualitative Data Analysis. Berlin, Germany: Scientific Software Development; 2013.
65. Davis F. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q 1989; 13 (3): 319–40.
PatientsPortalStudySafetyPatient Portal