Sweepstakes Design and Implementation
The Philly Vax Sweepstakes consisted of a series of three, high-payoff lotteries for nearly $400,000 in total prizes. Drawings were held every two weeks between June 7 and July 19, 2021. In each of the three drawings, 12 prizes were awarded to Philadelphia adults who had received a first dose of their COVID-19 vaccine: six $1,000 prizes, four $5,000 prizes, and two $50,000 “grand prizes”.
All adult residents of Philadelphia who received at least one dose of the COVID-19 vaccine were eligible to win a prize in the Philly Vax Sweepstakes. The sweepstakes drawing pool was seeded with the names and contact information for 1,064,805 Philadelphia adults from a purchased commercial database (27). In addition, to ensure they were included, residents could actively register for the sweepstakes online at phillyvaxsweeps.com or by phone. Both registration channels were managed by a professional sweepstakes vendor, Universal Promotions, Inc. By the close of the Sweepstakes, 75,356 people (6% of adult Philadelphians) had actively registered (see Figure 1 for registration volume over time and Supplemental Materials for more on how the Sweepstakes was advertised). Winners’ names were drawn from a deduplicated database.
Residents whose names were drawn but who had not received their first dose of the COVID-19 vaccine before the day of the drawing were ineligible to claim a prize. We reached out to each resident whose name was drawn using all available contact information.[2] If proof of first-dose vaccination could not be verified by the jurisdiction in which a resident received their shot, a new name was drawn for that prize. By design, this feature of the lottery created the potential for regret (11-18).
The experimental component of the Philly Vax Sweepstakes was designed as follows: the twenty Philadelphia zip codes with the lowest per capita vaccination rates as of May 27, 2021 (11 days before the sweepstakes) (28) were included in the experiment, and three Philadelphia zip codes were randomly selected from this set for “treatment,” defined as vastly increased odds of winning the sweepstakes. The other seventeen zip codes from this pool became “controls”. Each “treatment zip code” was announced two weeks before the drawing in which its residents would have heightened odds of winning prizes, and residents of this zip code enjoyed 59 to 98 times higher chances of winning a prize for that drawing (depending on the zip code’s population) compared to other Philadelphians. Specifically, each resident in the first treated zip code (19126) had a 1 in 2,081 chance of winning, or 98 times the chances of residents in other parts of Philadelphia (1 in 203,542). Each resident in the second treated zip code (19133) had a 1 in 3,304 chance of winning, or 61 times the chances of residents in other parts of Philadelphia (1 in 202,307). Finally, each resident in the third treated zip code (19142) had a 1 in 3,427 chance of winning, or 59 times the chances of residents in other parts of Philadelphia (1 in 202,184). Our experiment’s design 1) allowed us to causally evaluate the impact of increasing some Philadelphians’ chances of winning and 2) targeted more lottery resources toward underserved communities.
The Philly Vax Sweepstakes officially launched on Monday, June 7, 2021, with a press conference featuring the Mayor of Philadelphia and the announcement of the first treatment zip code. Following the Monday, June 7th announcement of the first treatment zip code, a new treatment zip code was randomly selected fortnightly and announced on the Philly Vax Sweepstakes website, with the second treatment zip code announced on Monday, June 21st and the third announced on Tuesday, July 6th (Monday, July 5th was a national holiday).
On the day the sweepstakes was announced (June 7th), it was a featured story on at least 5 local news channels (29-33), and it was a front-page story in the most prominent local newspaper, the Philadelphia Inquirer (34), on June 8th. Over 35 different news outlets published original stories about the Sweepstakes at some point during the six-week program (see Supplemental Materials Table S1 for the full list of news articles), which ended with the third and final drawing of winners on Monday, July 19th. Sixty-two percent of those stories were published during the first treatment period, 29% were published during the second treatment period, and 9% were published during the third treatment period.
The sweepstakes was marketed by the City of Philadelphia through twice weekly press releases, Nextdoor posts, text messages from the emergency information notification system, automated phone calls, a press conference announcing the first round winners, and a press release with quotes from the second round winners. There were also radio advertisements on Philadelphia’s two largest Hip Hop and R&B stations and emails and text messages to patients about the lottery from Penn Medicine, a large, regional health system (see Supplemental Materials for more information about how and when the Sweepstakes was advertised).
Google search trends data for “Philly Vax Sweepstakes” indicates that considerably greater attention was focused on the Philly Vax Sweepstakes immediately following its launch on June 7, 2021 compared to later in the summer (see Supplemental Materials Figure S3).
Participants
The 20 zip codes included in our experiment included 587,508 adult Philadelphians and these zip codes had an average vaccination rate of 33% prior to the start of the Philly Vax Sweepstakes (i.e., as of June 6, 2021). Figure 2 shows a map of Philadelphia and its surrounding counties, highlighting the locations of our three treatment zip codes in Philadelphia -- 19126, 19133, and 19142 -- and our seventeen control zip codes. These treatment and control zip codes comprised the twenty Philadelphia zip codes with the lowest per capita vaccination rates as of May 27, 2021. Table 1 provides summary statistics on the demographic composition of residents of these communities as well as the percent of each population with at least one COVID-19 vaccination dose prior to the start of the Philly Vax Sweepstakes.
Data and Measures
Our primary outcome was the number of first-dose vaccinations of adults ages 18 and older in the zip code of interest each week per 100,000 people.[3] We calculated weekly vaccinations per 100,000 people by dividing the total first-dose vaccinations for adults in the geography of interest for a given week by the total adult population in that region according to the American Community Survey (35) and then multiplying by 100,000. First-dose vaccinations included first doses of the Moderna and Pfizer vaccines as well as the single-dose Johnson & Johnson vaccine. The number of weekly first-dose vaccinations in each Philadelphia zip code for residents ages 18 and older was provided by the Philadelphia Department of Public Health (36) on August 18, 2021.
Analysis
We evaluated the impact of our geographically targeted lottery treatment using difference-in-differences analyses comparing weekly vaccinations per 100,000 adult residents over time. Specifically, to evaluate the impact of “treating” three zip codes with vastly increased odds of a prize, we compared the difference in vaccination rates between each of our treatment zip codes and our control zip codes before versus during the two-week period of the sweepstakes when a particular treatment zip code had higher odds of winning. In our Supplemental Materials, we also present an attempt to evaluate how the Philly Vax Sweepstakes affected vaccinations in Philadelphia County, overall, versus other Pennsylvania counties.
Prior to implementation, the design for the Philly Vax Sweepstakes was reviewed and approved by the institutional review boards (IRBs) of the University of Pennsylvania and the City of Philadelphia. Our study’s analysis plan was also pre-registered at osf.io/gxsa4.
[2] Residents had at least 48 hours to claim their prize after being successfully contacted. If they did not, or if they could not be successfully contacted after all available means were exhausted, a new name was drawn.
[3] We focused on individuals who were 18 years old or older when calculating the vaccination rate because individuals had to be at least 18 years old to participate in the Philly Vax Sweepstakes.