Complex organizations with diverse missions (e.g., educating undergraduates and caring for patients) can overcome challenges by unifying around common goals, being transparent in their communications, and nimbly adapting.
Programs that screen essential workers for Covid-19 must balance their ability to identify concerning symptoms or exposures against the demands to staff essential services supporting the daily operations of an organization or clinical care.
Automation can substantially increase efficiencies in a system that otherwise would have required extensive human-to-human interaction.
How organizations have made decisions under the pressures of the pandemic can illustrate potential strategies for advancing innovation and transforming health care in the future.
In May 2020, leaders at the University of Pennsylvania (Penn) and the University of Pennsylvania Health System (UPHS) were deliberating how to safely return nearly 70,000 employees, students, and others to campus during the Covid-19 pandemic. While social distancing, handwashing, and masking were enforced on our campuses, these behaviors were encouraged to varying degrees by policy makers in the three states where our campus members live or work (Pennsylvania, New Jersey, and Delaware), and individual compliance with these practices outside of our campuses was unenforceable and unobservable. Symptom and exposure detection emerged as an essential layer of protection for our campuses. At the time, temperature screening at building entrances had low sensitivity in detecting Covid-19 symptoms, did not screen for concerning exposures, created delays and crowds, and turned people away from campus without offering additional guidance or resources. We needed a virtual entryway to screen individuals before coming into contact with others on our campuses and, when necessary, we need to streamline access to testing, self-isolation guidance, contact tracing, and medical care.
We needed to identify people with symptoms or exposures concerning for Covid-19. Without routine, frequent, rapid surveillance testing, our system had to rely on self-reporting while being scalable and adaptable. As a result, we focused on digital solutions complemented by existing clinical staff who could determine when individuals with concerning symptoms or exposures could return.
PennOpen Pass is a daily electronic symptom and exposure screening system that was developed and implemented in a phased approach over a period of several months to reduce community spread of Covid-19 and to manage those exposed or infected, with patient triage and care coordination based on clinical algorithms (, ).
After a one-time enrollment process lasting approximately 5 minutes, end users are required to complete symptom and exposure checks each day in seconds with the use of a mobile-friendly Web application (). Users receive a green pass if they do not report either concerning symptoms or recent contact with someone with suspected or confirmed Covid-19. Users with concerning symptoms or exposures receive a red pass and are guided through the Red Pass Management System (RPMS), an automated evaluation process backed by a team of trained clinicians who determine the user’s next steps, tailored to their situation. Intake questions adjudicate whether they should truly retain a red pass or if they should be cleared to a green pass because their situation is less concerning ( ). Even if users do not complete all intake questions, their data populate a Web-based dashboard and workflow management tool, critical for care coordination and contact tracing support ( ). Our communities are highly interconnected geographically, with many people routinely crisscrossing our clinical and university campuses. Creating a centralized management tool that facilitated coordination and provided a uniform response to screenings was critical.
The PennOpen Pass/RPMS user-facing teams included: (1) a centralized call center staffed from 4 a.m. to 10 p.m., 7 days a week to provide technical and logistical support to users; (2) a team ofregistered nurses, cross-utilized from other telemedicine programs, to provide clinical support to PennOpen Pass users and to finalize the determination of a green versus red pass in situations ofclinical uncertainty; and (3) a diverse and large team of personnel from Penn (including representatives from Student Health Service and Campus Health as well as contact tracers) and UPHS (including representatives from Occupational Medicine and Infection Control and Prevention) who were already dedicated to mitigating Covid-19 risks on our campuses.
Contact tracing was conducted at two levels. First, workplace contact tracing was conducted by Penn contact tracers and UPHS infection control teams. Second, community contact tracing was conducted by a team led by Penn’s Center for Public Health Initiative. The PennOpen Pass teams were tightly linked to both groups of contact tracers.
PennOpen Pass Design and Implementation
We wanted our solution to be usable on mobile devices and capable of producing an instantly understandable visual pass to display at building entrances. Early in the process, we ruled out developing an iPhone and Android mobile app because of potential adoption barriers. Instead, we built a mobile-friendly Web application that facilitated mobile or desktop symptom surveys so that daily passes could be printed or shown on screens like an airplane boarding pass (Figure
). To accommodate campus users without phones or easy access to the Web application, verbal attestation signs were developed at entryways so that users could acknowledge the absence of concerning symptoms or exposures to entry personnel, modeled after “Nothing toDeclare” pathways for customs inspection. To date, the verbal attestation process has been used by 0.5% to 5% of users per day.
PennOpen Pass was built by our internal software development team, despite the fact that the team was facing many other pandemic-related challenges. Given that many organizations faced similar challenges, it might seem that identifying a close-enough external solution would have been more efficient. If a screening functionality was the only requirement, outsourcing would have made sense. However, to create a unified and coordinated approach that accounted forthediverse populations across hospital, ambulatory, and university settings, frequent design iterations were needed from a team who understood and was willing to work through the operational complexities of all of our campuses. Using internal resources for this task allowed us to adapt to rapidly evolving public health guidance and specific campus needs. This flexibility was essential for widespread stakeholder buy-in. Although a daily survey could have been created in a third-party system, there were numerous features that required customization and integration, such as the ability to prevent a person with a red pass from completing a new screening until clinically cleared. In addition, using internal software teams gave us complete flexibility to integrate complex logic and PennOpen Pass within other enterprise systems.
Our product team followed an Agile software-development process to rapidly iterate the Web application and quickly test concepts through small-scale rapid-cycle pilots.In June 2020, we ran a 2-week pilot to evaluate adoption, user experiences, clinical workflows, and operational requirements.
Design issues with the questionnaire initially led to a high rate of user errors in generating red passes, accounting for 80% of red passes and resulting in an unnecessary burden on the teams who were managing red passes as well as on the users themselves, who were delayed in being able to enter campus buildings. Accordingly, we applied user-centered design principles by modifying the questionnaire design (e.g., simplifying the language used in the screening questions and response choices to improve interpretability), improving the user-centered design (e.g., enhancing the usability of on-screen buttons to minimize click errors), and giving users agrace period to retake their questionnaire if they generated a red pass by mistake. These changes decreased the red pass rate from 1.5% during the pilot to 0.8%, but they did not decrease the total number of symptoms or exposures reported in the days soon after. Questionnaire redesign also resulted in faster survey completion, reducing the average completion time from 36 to 13 seconds. We also removed friction from the log-in process on thebasis of user feedback, such as the removal of daily dual-factor authentication for the majority of user groups.
With these design improvements, the team completed the first step in a phased rollout to more than 30,000 Penn faculty, staff, and students by July 2020. With a small implementation team facing a 12-school university campus with different needs, we leveraged local operational groups from each school (e.g., law, engineering, design, medicine, etc.). We developed a detailed implementation toolkit with templated workflows, standardized communications, and held virtual office hours to answer questions and foster peer-to-peer learning and dissemination (). In July 2020, each of Penn’s 12 schools went live in a three-step, month-long process of: (1) preparing for go-live, (2) enrolling users, and (3) mandating usage for building access. Starting in September 2020, the same model was used to deploy the program across thehealthsystem.
Fixing the Plane While Flying It
In the time between the Penn and UPHS rollouts, we developed and implemented the RPMS to enhance scalability as we prepared to enroll an additional 40,000 health system employees. While there had been a lull in cases in the Philadelphia region over the summer in 2020, we recognized the urgent need to implement PennOpen Pass at our health system by the early fall of 2020, as national case counts were rapidly rising. The time urgency required us to “fix the plane while flying it,” rolling out PennOpen Pass to our UPHS campuses while prototyping and iteratively redesigning RPMS despite all the clinical protocols and software specifications that still needed refinement and testing at the time of launch. A daily workgroup of staff from the PennOpen Pass call center, nursing leadership, occupational medicine, and infection control reviewed clinical scenarios and technical problems to adapt RPMS into a functional, end-to-end red pass screening system that could be used throughout UPHS and by the small number of Penn students, faculty, and staff who were working on campus during the fall of 2020.
The major hurdles during design and development were more organizational and human than technical. PennOpen Pass and RPMS touched every part of the university and health system, requiring extensive discussions, stakeholder engagement, and buy-in for decision-making (). Visual mapping, prototyping, and pilots helped inform decision-making, achieve alignment, and identify requirements and workflows that would meet the needs of our diversecampus.
Clinical Considerations and Staffing Needs
PennOpen Pass and RPMS needed to balance adequate detection of concerning situations (essential for keeping the campus and health system safe) against overdetection and running the risk of unnecessary strains on staffing for essential campus and health system roles. We convened infectious disease physicians, epidemiologists, and operational leaders to join our core workgroup and provide expert guidance on how to achieve this balance. We also trained registered nurses with a designated nurse lead and medical director. The clinical leads were responsible for disseminating and implementing institutional guidelines related to testing and quarantining, which evolved as public health guidance shifted.
To determine which symptoms PennOpen Pass should screen for daily, we referred to Covid-19 symptoms outlined and updated by the Centers for Disease Control and Prevention.The tension between adequate detection and overdetection was felt acutely when considering which symptoms (either individually or as part of a cluster of symptoms) should trigger a red pass. Our greatest challenge was the limited evidence available to guide our decision-making process. Many symptoms associated with Covid-19, such as sore throat, are nonspecific. Symptoms of other communicable diseases, such as fever and newly developed cough, were concerning enough for Covid-19 that they warranted immediate testing and self-isolation. Symptoms that have become almost pathognomonic for Covid-19, such as loss of taste and smell, were easier to include as individual red pass triggers. The finalized algorithm and list of symptoms triggering a red pass and further evaluation are included in .
Given that approximately 30% of people capable of transmitting the disease were asymptomatic,, we emphasized identifying, quarantining, and testing individuals with concerning exposures, in addition to those with symptoms. Classifying exposures was more challenging than classifying symptoms, reflecting not just the mechanism of exposure, butalso uncertainty about the contact’s own risks. A substantial proportion of our campus members resided in poor and largely minority neighborhoods that were disproportionately burdened by the pandemic. While we needed a screening process that would be sensitive enough to detect concerning exposures, we also had to back it up with human-centric contact to account for the needs of individual users and the communities in which they resided. We recognized that we might need to rely on PennOpen Pass for the foreseeable future and hoped to gain the trust and long-term acceptance of our users.
The Challenges and Value of Integrating Automation
Initial projections indicated that $3.8 million would be required to manage users with red passes for 12 months using a manual, human-staffed program. That figure, already daunting, was made even less attainable because few staff could be reallocated to this task, a hiring freeze had limited new personnel, and the pandemic had already caused a financial strain. Automating much of the process, from identification to management to clearance for returning to work, was immediately recognized as a key requirement. RPMS provided the in-house capability to test and validate processes that were hypothesized to be automatable while ensuring the flexibility to adapt to user feedback, stakeholder requests, and the changing evolving nature of the pandemic. However, the constantly changing landscape and multiple user demands were challenging.
First, we faced internal barriers to the concept of automation itself. Many members of our team defined their roles as providing essential human touch and hard-to-replicate expertise in the context ofemotionally sensitive conditions. Acceptance of the role of automation became easier as the impossibility of supporting manual approaches became more apparent with experience.
Second, the timeline to a minimum viable product was short. Our “fix the plane while flying it” approach acknowledged that critical learning would occur along the way, but hiccups experienced by users during the early stages of development created lingering perceptions among users that the program was ineffective or unreliable. Overcoming these challenges required a level of transparency, vulnerability, and rapid responsiveness to concerns at a scale that many of us had not previously experienced.
Third, while we initially believed that assessing exposures could be largely automated with the use of a set of follow-up questions from RPMS, we quickly learned that users’ exposure scenarios were too nuanced and idiosyncratic to fall into simple patterns. Individual stories evolved over time as users progressively recalled details or gained collateral information from their contacts in the subsequent hours or days. We transformed RPMS questions to gather basic information, such as when an exposure occurred and whether or not individuals wore a mask.
Fourth, while the initial intake process was variable from person to person, we learned that the process of clearing individuals for returning to work could be largely automated. For those with symptoms, we followed public health guidance after receiving their test results. For exposures, we designed RPMS to conduct time-based follow-up to determine if users remained asymptomatic or, in alignment with public health guidance later in the pandemic, if they had followed test-based strategies for returning to campus at 10 days instead of waiting the full 14 days. We also leveraged RPMS to flag individuals with more complicated situations such as household exposures, reserving those individuals for a more hands-on, clinician-managed process.
Adaption and Expansion
Evolving public health guidance necessitated a flexible, ongoing implementation process and in-house software development to keep us nimble. For example, PennOpen Pass was adapted to reinforce travel warnings during the winter holidays, and its quarantining or self-isolation recommendations have been adjusted for users who have been vaccinated. The program has experienced a number of expansions. First, it has been expanded to support symptom and exposure screening of patients and their visitors prior to entering our health care facilities. Second, the success of PennOpen Pass in the fall of 2020 generated confidence among leaders at Penn that the Penn community members who returned to university campus could be safely monitored and cared for when Covid-19 concerns arose.
In January 2021, campus was “reopened” to allow more than 3,000 students to live on campus as well as to reintroduce an increased number of in-person activities (e.g., opening athletic facilities for student athletes and opening dining halls) for thousands of students living both on and off campus. PennOpen Pass checkpoints were integrated into all buildings, including dormitories and common spaces. That same month, PennOpen Pass was integrated with a Covid-19 surveillance testing program that tested all undergraduates twice per week, all graduate students once per week, and certain Penn community members (i.e., those who accessed campus grounds or facilities 4 or more hours per week) once per week. If individuals received a positive test or did not comply with scheduling a screening test, their pass would automatically be converted to a red pass. The surveillance testing program and the PennOpen Pass system were seen as complementary. One program could detect asymptomatic infections, and the other provided an effector arm for implementing public health mitigation techniques through the red pass system and PennOpen Pass’ team of contact tracers and clinicians.
A core workgroup was assembled to develop PennOpen Pass and the RPMS from the Penn Medicine Center for Health Care Innovation, Penn Medicine Information Services, the Center for Health Incentives and Behavioral Economics, and executive leaders from Penn and UPHS. After university-wide implementation, the core workgroup managed a transition of the program to the current operational structure under the Penn Medicine Center for Connected Care ().
From June 2020 through February 2021, more than 75,000 campus members enrolled in PennOpen Pass. During that time, the program supported 4,183,462 daily surveillance checks and the PennOpen Pass and RPMS teams issued and managed 25,117 red passes. Exposures accounted for 54% of red passes. The most common symptoms triggering red passes triggered were cough (18%), fatigue (16%), and fever (15%). Among the 37,034 UPHS employees who enrolled in PennOpen Pass —users who were most likely to receive testing through the health system when they triggered a red pass — 1,276 positive tests occurred. To provide context, in the five southeast Pennsylvania counties surrounding Philadelphia during the December 2020 surge, the median number of new cases per 100,000 in the last 7 days was 53.7 (range, 44.7 to 64.8). By comparison, the median number of new cases per 100,000 in the last 7 days among UPHS employees who enrolled in PennOpen Pass in December was 63.3 (range, 44.6 to 72.3). Among those who tested positive, 44% attested to symptoms, 11% attested only to exposure, and 45% attested to both symptoms and exposure.
Through automation and iterative improvements, the final PennOpen Pass system resulted in reductions in the median time of red pass clearances from a peak of 45 hours in mid-November to 27 hours by the end of December 2020. The cost of administering PennOpen Pass with our current staffing model combined with technology costs is projected to be $0.9 million over a 12-month period. This figure is substantially less than the initial $3.8 million projected for a PennOpen Pass system without automation.
Where to Start
During the process of developing and implementing PennOpen Pass, we learned several key lessons for the future of innovations in health care:
A shared common mission can be a motivating force to drive new initiatives forward. While the pandemic was an experience that many of us would not want to repeat, it brought people from disparate and siloed parts of our campuses together with a sense of extreme urgency to achieve a common goal. As the pandemic wanes, we need to identify strategies for how to identify future shared missions and work toward achieving those goals together.
Delivery challenges in health care can be overcome. Often in health care, solutions to challenges are plagued by pessimism about what is not legally possible, a reimbursement structure that is not ready, or situations in which the technology does not exist. The development of PennOpen Pass and RPMS demonstrated the feasibility of crafting a new approach that required working through a variety of legal and data-privacy challenges without a reimbursement model and of building a customized technological solution when none existed. The pandemic has taught us that we can overcome seemingly insurmountable challenges with ingenuity, imagination, and collaboration.
Automation can be integrated into care, and health care will not lose its humanity. Initially, automation of the red pass system was met with hesitation, even among our own team. However, as we opened the PennOpen Pass system to 70,000 users, it became clear that the estimated 0.5% daily red pass rate would require a substantial amount of work to process users with symptoms and/or exposures and to efficiently return individuals to work. This work could largely be done in an algorithmic fashion. Automating routine tasks helped our clinicians focus on providing humane care to those who needed it the most: the worried and the clinically complex. The PennOpen Pass system provides a technological, operational, and clinical backbone that can be retooled after the pandemic for more efficient, large-scale health and safety-screening initiatives such as postoperative follow-up or population health-management approaches for other conditions for which the early detection of clinical decline is critical.
Our experience with Penn OpenPass and the RPMS demonstrated that the technical challenges that defined our initial solution sat alongside even larger, often time-consuming, organizational challenges of stakeholder management and operational implementation. Yet we were motivated by the importance of stemming the pandemic, keeping our campus safe, and helping our campus members feel safe. While PennOpen Pass was created under the pressures of the pandemic, the lessons we learned will help us advance health care innovations in the future.
Katy Mahraj, Krisda H. Chaiyachati, David A. Asch, Glenn Fala, David Do, Doreen Lam, Amy Miller, Nancy Mannion, Vanessa Stoloff, Ashlee Halbritter, Ann Marie Huffenberger, Julie Shuttleworth, Judith A. O’Donnell, Judith Green-McKenzie, Kash Patel, Roy Rosin, Greg Kruse, P.J. Brennan, and Kevin G. Volpp have nothing to disclose.
1K Beck, M Beedle, A van Bennekum. Manifesto for Agile Software Development. Accessed February 7, 2021. https://agilemanifesto.org/.
2Centers for Disease Control and Prevention. Symptoms of Coronavirus. Accessed December 22, 2020. https://www.cdc.gov/coronavirus/2019-ncov/symptoms-testing/symptoms.html.
C Menni, AM Valdes, MB Freidin. Real-time tracking of self-reported symptoms to predict potential COVID-19. Nat Med2020; 26:1037-4010.1038/s41591-020-0916-2.32393804.
DP Oran, EJ Topol. The proportion of SARS-CoV-2 infections that are asymptomatic: a systematic review. Ann Intern Med2021; 173:362-710.7326/M20-3012.33481642.
MA Johansson, TM Quandelacy, S Kada. SARS-CoV-2 transmission from people without COVID-19 symptoms. JAMA Netw Open2021; 4:e203505710.1001/jamanetworkopen.2020.35057.33410879.
We thank Dawn Augustino, Greg Barendt, and the Penn Medicine Software Development Team, Michael Begley, Sean Burke, Carolyn Cannuscio, Andy Cruz, Deirdre Darragh, Jennifer Feldman, Eugene Gitelman, Erika Gross, Mike Restuccia, C. William Hanson III, Virginia Haughey, Davis Hermann, Sharon Jankauskas, Damien Leri, Gina Marinilli, Susan McGarvey, Tom Murphy, Anthony Nguyen, Maureen O’Leary, Jennifer Pinto-Martin, Scott Schafer, Lauren Steinfeld, the Center for Public Health Initiatives, and the PennOpen Pass Call Center, nursing teams, clinical staff, contact tracers, and public health teams.