The Mayor's Office for Economic Opportunity311Search all NYC.gov websites

The NYC Behavioral Design Team

 

Our Partnership

NYC Opportunity has partnered with ideas42, a leading non-profit behavioral design lab, to run the New York City Behavioral Design Team (BDT). There is extensive social science research showing that government programs that are designed and implemented with careful attention to how people use them produce better results. The behavioral design approach emphasizes learning directly from residents about how they use and experience government services. Behavioral design adapts services and systems to meet needs of those being served by accounting for their specific context as well as the quirks of human cognition, and assesses those adaptations using rigorous evaluation methods.

Since its launch in 2016, BDT has worked with more than a dozen agencies across City government to evaluate existing programs and develop ways of using behavioral design to enhance them. Behavioral design experts and some of the most innovative thinkers in City government have collaborated on formulating practical solutions to problems in education, employment, health, government efficiency, energy and the environment, economic mobility, equity, and other areas. There is already considerable evidence that these interventions are producing strong results. In one project, BDT worked with the Fire Department of New York (FDNY) to assess whether a filing fee waiver could increase the diversity of applicants who took its qualifying examination. For the project, a randomly selected group of applicants had their fees waived while another group did not. The results showed that the fee waiver increased overall filing rates substantially, with large gains among black and female candidates. Other BDT projects have significantly improved the rate at which SNAP recipients recertify for benefits, CUNY students file for federal student aid, and City employees get flu shots.

Based on BDT’s record of success, the Laura and John Arnold Foundation, which provided initial support for the project, has provided a new round of funding to extend its work for an additional two years.


Resources

BDT Playbook: A Model for Integrating Behavioral Design in City Government | Ideas42

Behavioral Science Around the World: A Profile of 10 Countries | World Bank Group

Projects

*RCT - Randomized Controlled Trial

311

Reducing Misfiled Service Requests on 311’s Mobile Platform

The 311 Mobile App team was receiving a large number of misfiled service requests in an early version of the application, and wanted improved, more frequent feedback on the app’s functionality from users. The BDT conducted qualitative research with a small sample of users and provided behaviorally-informed recommendations to redesign the smartphone app’s menu and reframe its feedback function

  • Pre/Post evaluation showed that changing menu choices and reframing language around feedback decreased misfiled service requests from 59% to 9%.

NYC Administration for Children’s Services (ACS)

Improving Positive Reinforcement Systems in Youth Detention Centers

The complexity and contradictions within ASPIRE, the behavior management system used at ACS’s youth detention centers in NYC, had caused an uneven application of rules by staff and a lack of buy-in amongst youth residing in centers. To encourage youth’s investment in the system, reduce unsafe behaviors, and increase prosocial behavior, the BDT collaborated with ACS to redesign the points and levels structure and revise incentives. Design recommendations were based upon the input of youth and staff in centers and addressed behavioral barriers of limited attention, scarcity, and present bias.

  • The redesigned behavior management system (“STRIVE”) is currently being implemented in every youth detention centers in NYC and staff are being trained in its principles.

NYC Department of Education (DOE)

Increasing Sign-Up for Gifted & Talented Testing in Low-Income School Districts

Students in low-income districts are less likely than their peers in higher income districts to take the Gifted & Talented (G&T) admissions test. The BDT redesigned emails and postcards to encourage preschool parents to sign their students up for the test. The redesigned email used a congratulatory tone to prime the positive identity of a proud parent and used icons and succinct language to highlight key information. The postcard reframed the G&T program as one in which all types of students participate and included a sample test question to reduce ambiguity around testing level.

  • An RCT showed that the redesigned email increased testing requests by 6.5%. The results of the study on specific districts is expected in March 2019.

Reducing “Summer Melt” Among College-Intending Seniors

Many high school seniors are accepted to college but neglect to enroll in the months following graduation. To reduce this “Summer Melt,” the BDT designed and implemented a series of two-way text messages during the summer of 2016. The text messages pulled various psychological levers, including plan making to keep prospective students on track for enrollment.

  • Engagement among students was high: 81% of participating students responded at least once.

NYC Department of Finance (DOF)

Reducing the Number of Parking Tickets Entering Judgment

About a fifth of parking tickets issued to drivers in NYC enter into judgment, among which over three-quarters could have been addressed sooner. Unaddressed tickets are placed into judgment, incurring additional fees for both the driver and the court system. To reduce the number of unpaid tickets entering judgment, the BDT redesigned the envelope containing penalty notices sent to individuals with unpaid tickets. The envelope tested the use of a non-blaming tone to reduce the tendency to ostrich from unpleasant news, as well as attempted to reduce present bias by highlighting the potential losses as a consequence of delaying payment of the ticket.

  • An RCT showed that expensive tickets belonging to drivers that received the behaviorally-designed envelope 90-days after receiving their ticket were 9.7 percentage points less likely to enter into judgment.
  • DOF plans to scale the behaviorally-designed envelope to all drivers whose unpaid tickets prompt the 90-day notice.

Reducing Tax Lien Sales

Tens of thousands of property owners in New York City are at risk of having a lien put on their properties, due to non-payment of property taxes. To reduce the number of property owners who are at-risk for losing their property, the BDT redesigned the 60-day notice that the DOF sends to these clients. The redesigned notice clarified what the tax lien sale is, where to find more information, and what options property owners have for payment. In addition the notice used loss aversion and a deadline to inspire action, as well as included a checklist of simplified steps for requesting a payment plan.

  • 60-day notices were scaled citywide.

Encouraging Licensed Businesses to File Taxes Online

In 2016, only 6% of licensed businesses on record were registered for the City’s online tax portal. To prompt registration among business owners and tax preparers, the BDT redesigned two emails encouraging registration. The redesigned emails presented social norms messaging about businesses registered on the portal, provided a concrete estimate for the time required for the task, and used icons to highlight the key benefits of online tax filing.

  • An RCT found no statistically significant effect, likely due to low power. However, 30-additional businesses registered after receiving the behavioral email compared to the business-as-usual email.

Increasing Engagement with the Notice of Property Value

Homeowners in NYC are often surprised by or concerned with their Notice of Property Value (NOPV), which notifies them of their property’s market and assessed values for the coming tax year. To reduce confusion and improve engagement, the BDT designed a behaviorally-informed mailing insert to accompany the NOPV for every homeowner in NYC in February 2017 (tax year 2017-2018). The notice clarified what the NOPV is, how to decipher it, and what action (if any) a homeowner should take.

  • Mailed inserts were scaled citywide.

Reducing Vehicle Booting Among NYC Drivers

When NYC drivers approach $350 in unpaid parking fines and interest, their vehicles become at-risk for booting. To release a boot, drivers often pay over $700 – an unanticipated financial cost that can be particularly disruptive for those living in low-to-moderate income households. To motivate drivers' to take action and lower the risk of booting, the BDT redesigned a warning email delivered to drivers approaching the $350 threshold. The redesign makes salient the consequences of inaction to reduce procrastination, cuts hassles to make payment easier, and reframes the email in a personalized, non-blaming tone to reduce aversion associated with the ticket.

  • An RCT was launched in May 2019 with the results expected in December 2019.

NYC Department of Health and Mental Hygiene (DOHMH)

Increasing Physician Uptake of Asthma Medication Administration Forms

Asthma medication administration forms (MAFs) allow school staff to administer asthma medication to students in respiratory distress, reducing emergency room visits and lost classroom time. MAFs must be completed by a physician and signed by a parent/guardian, but in 2014-15 only 43% of approximately 70,000 diagnosed students had MAFs on file. To encourage submission, the BDT designed and tested a new MAF that was mailed to physician offices and supplemented with a call. The new design reduced the number of steps on the form, presented all information for physician on the front and for the parent on the back, and added arrows to make salient the most important information (i.e. parent’s signature). Because physicians access documents online, only a few physicians’ offices used the new form. After conducting post-mortem interviews and surveys with physicians, the BDT learned that physicians widely approve of the new form.

  • The behaviorally designed MAF was scaled to all physicians citywide, with improved distribution in 2017-18.

Increasing Submission of Asthma Medication Administration Forms (MAF Round 2)

Continuing its work on increasing the submission of MAFs, the BDT conducted qualitative interviews with community school case managers, school physicians, and school nurses. To simplify what is asked of parents, the BDT created a simple and personalized slip (sent home with the form) with clear instructions on how and when to complete and submit the form. This slip implements a self-affirmation note that reinforces parents’ identities as good parents. Additionally, the BDT created a decision tree that simplified responsibilities for school nurses to ensure that students can receive emergency medication while in school.

  • The self-affirmation notes and decision trees were scaled to all NYC school nurses at the start of the 2017-18 school year.

NYC Fire Department (FDNY)

Reducing Micro-hassles Improves Firefighter Diversity: Removing an Exam Filing Fee

Historically, women and underrepresented group members have become firefighters at lower rates than white men. In the 2017 firefighter recruiting season, the FDNY set explicit targets for increased firefighter diversity and engaged the BDT to design and test ways to increase the number of female and underrepresented candidates entering the process. The BDT targeted the first step in the recruitment process: filing to take the civil service exam. Waiving a small fee associated with filing overcomes a bureaucratic hassle, which may disproportionately discourage underrepresented group members and women from filing for the exam.

  • An RCT showed that fee waivers increased registration amongst female candidates by 83% and registration amongst black candidates by 84%.
  • Registration amongst male and white candidates also increased, but only by 22% and 38%, respectively.

Converting Exam Filers to Test Takers: Behavioral Reminders for Potential Firefighter Candidates

Many individuals who file to take the DCAS Firefighter Exam often fail to appear for their scheduled exam. FDNY engaged with the NYC BDT to test the effectiveness of a reminder campaign, using a randomly-assigned email, text, email and text, and automated voice response (AVR) phone campaign. Based on previous research on plan-making, the BDT redesigned these communications to include a planning prompt to encourage candidates to make a plan for how they would arrive on time to their scheduled exam.

  • An RCT showed that candidates who received both a text message and an email were 3.6% more likely to appear for the exam than those who received the other forms of outreach.

Diagnosing Former Firefighter Candidate Attrition Through a Survey

Women and underrepresented group members in the past have attrited from the New York Fire Department’s hiring process at higher rates than white males. BDT designed a survey to understand the barriers that female and underrepresented candidates face using psychometrically-validated questions that test hypotheses about how relevant psychologies, such as belonging uncertainty and stereotype threat, may influence persistence through FDNY’s hiring process.

  • The survey closed in November 2018, and final results were presented in April 2019..
  • Survey responses demonstrated that Black and female candidates report experiencing threat related to their race and gender, respectively, regardless of whether or not they were hired.
  • FDNY, in response, is in the process of leveraging opportunities to reduce perceived threat throughout the hiring process. For example, FDNY’s future communication materials and outreach/programming for candidates are designed to reduce perceived threat.

Supporting female firefighter candidates to appear for and pass their Candidate Physical Ability Test (CPAT)

In the past, nearly twice the number of eligible female candidates fail to appear for the CPAT than male candidates. Female candidates may be primed to think of stereotypes of what others expect of women in this physically-demanding setting. To address barriers that prevent women from showing up to and performing on their physical exam, the BDT developed an exercise for candidates to reflect on their motivations for becoming a firefighter and a planning card to help candidates form personal training plans and track their weekly progress.

  • The reflection exercise and planning card were rolled out to candidates in June 2019, and the next physical exam is scheduled to be held in the Fall 2019.

Increasing the number of firefighter candidates who claim residency points

Many firefighter candidates with long-term residency in NYC who take the written firefighter exam fail to claim the additional points offered to long-term NYC residents – this difference of a few extra points often bumps candidates into eligibility. To encourage candidates to claim residency points by email, the BDT designed an email and text message campaign, using social norms and loss aversion to grab candidates’ attention, as well as reduced the number of steps required to claim points by drafting a pre-populated email for candidates to easily complete and send to the Department of Citywide Administrative Services (DCAS). Half of the candidates received the social norms messaging while the other half received the loss aversion messaging.

  • A randomized controlled trial showed no statistically significant difference in claimed points for those who received social norms messaging compared to loss aversion messaging.
  • A small sample of underrepresented candidates who received a more intensive campaign were 7.4 percentage points more likely to claim residency points, which is statistically significant. This suggests that a more intensive behavioral campaign targeting encourages candidates who are less likely to claim their points.

NYC Department of Housing Preservation & Development (HPD)

Improving “Good Fit” Matches in NYC Housing Lotteries

The NYC affordable housing lottery online application system (“Housing Connect”), which currently hosts close to 2 million users, faced a “goodness of fit” problem, or a mismatch between which lotteries applicants apply to and whether their household is ultimately eligible for and interested in those housing developments. To support clients to accurately report annualized income and select good fit lotteries, the BDT provided a set of behavioral design recommendations that considered key barriers clients face when visiting Housing Connect. Recommendations sought to improve the choice architecture of the portal, making strong matches more salient and reducing the hassles, ambiguity aversion, and present bias that clients encounter with the current design.

  • Recommendations are being used to inform the redesign the Housing Connect portal, which will include a behaviorally-informed income questionnaire and housing suggestions tailored for applicants.

NYC Human Resources Administration (HRA)

Increasing Form Submission Rates Among SNAP Recertification Clients

SNAP churn (losing benefits, then returning within 90 days) hurts recipients and burdens HRA. To improve SNAP recertification rates, the BDT designed a reminder notice spurring clients to take the first required recertification step: submitting a recertification form. The notice leveraged loss aversion by reframing recertification as a potential loss of benefits and used graphics and numbering to make action steps clear and seem more manageable.

  • An RCT showed the reminder reduced failure to submit recertification forms by 5.5%.
  • The reminder notice was scaled to all clients who do not submit their forms by the reminder mailing date.

Increasing Housing Assistance Uptake Among Households at Risk of Homelessness

New York City offers several housing assistance programs to households at risk of becoming homeless. To prompt the uptake of valuable housing assistance, the BDT redesigned a notice sent to at-risk households urging them to call the City’s Neighborhood Homelessness Prevention Outreach (NHPO) program for potential assistance. The redesigned notice used a testimonial to reframe services as an opportunity benefitting others, and included a plan-making prompt to help households carve out time in their day to call.

  • An RCT showed no statistically significant differences in call-in rates or rent assistance between the original and the redesigned notices.

Increasing Interview Completion Rates Among SNAP Recertification Clients

New York City offers several housing assistance programs to households at risk of becoming homeless. To prompt the uptake of valuable housing assistance, the BDT redesigned a notice sent to at-risk households urging them to call the City’s Neighborhood Homelessness Prevention Outreach (NHPO) program for potential assistance. The redesigned notice used a testimonial to reframe services as an opportunity benefitting others, and included a plan-making prompt to help households carve out time in their day to call.

  • An RCT showed no statistically significant differences in call-in rates or rent assistance between the original and the redesigned notices.

Raising Awareness to Increase Utilization of Free Legal Representation Among Clients Facing Eviction Cases in Housing Court

New York City is the first city in the country to enact a ‘universal-access-to-counsel’ (UATC) law that provides residents facing eviction in Housing Court with free legal assistance. To raise awareness and increase utilization of these valuable services, the BDT worked with the Human Resources Administration’s Office of Civil Justice to redesign outreach materials, such as flyers, that are sent to New Yorkers facing eviction court proceedings advising them of the resources available and encouraging them to access these services. The behaviorally-designed flyer used social norms messaging to highlight other people’s use of the program, outlined the wide range of situations in which people could benefit from having a lawyer or accessing legal guidance, and provided an actionable next step for receiving legal services. These designs more clearly delineated for readers when and how lawyers are useful, more effectively empowering tenants to act if and when they feel at-risk.

  • The impact of this intervention will be measured using an RCT, and results are expected by early 2020.

SNAP Recertification, Periodic Reports Reducing SNAP Churn through Periodic Report Reminders

SNAP churn results from failure to recertify annually, but is also driven by failure to complete the six-month periodic report. The BDT designed two behaviorally-informed emails to prompt online periodic report submission. Both emails leveraged loss aversion, outlined action steps and time expectations, and made deadlines salience. One email also used enhanced active choice.

  • The impact of this intervention will be measured using an RCT, and results are expected the Fall of 2019.

NYC Department of Consumer Affairs: Office of Financial Empowerment (OFE)

Increasing Sign-Ups for Tax Preparation Messaging

The City has important tax information to share with residents, but it often lacks the contact information to do so. To increase sign-ups for tax preparation messaging, the BDT redesigned the Tax Prep Portal website to reduce ambiguity around services and eligibility. The redesigned website used icons to display information in a digestible way, social norms messaging to enhance relevance, and default options to reduce choice conflict.

  • While an RCT was attempted to measure the impact of the redesigned website, the study was under-powered.
  • The redesigned website was scaled citywide for future tax years.

Office of Labor Relations (OLR) + WorkWell NYC

Increasing Flu Vaccines among City Employees

Within the New York City workforce, over 400,000 employees and their family members were unvaccinated from the flu in the 2015-16 flu season, risking their own health and potentially spreading the flu to others. To increase uptake of flu vaccines and maintain a healthy workforce during the 2016-17 flu season, the BDT partnered with WorkWell NYC to design a set of three behaviorally-informed emails to send to all NYC employees encouraging them to protect themselves and others from the flu by getting a vaccine. One version of the redesigned email used the strategy of “enhanced active choice,” whereby employees were prompted to make a choice that had a clear right answer.

  • Behavioral emails doubled click through and registration rates. The enhanced active choice version of the email increased vaccine uptake at worksite locations by 5%.
  • The most successful arm was scaled in 2017-18 and a 10% increase in vaccine uptake was observed year over year.

Registering City Employees for Telemedicine

Telemedicine is a low-cost option for insured employees to engage quality healthcare services, and in 2017 only 180 of 588,000 employees and their dependents in the City’s GHI/EBCBS CBP plan had enrolled. To encourage City employees to enroll, the BDT supported the redesign of the second of three emails that the OLR sent to City employees in January and February of 2017.

  • Email designs were scaled citywide after the piloted email blast enrolled about three times as many people as the first email.

Health Insurance Card Insert

In 2017 only 0.03% of employees and their dependents in the City’s GHI/EBCBS CBP plan had enrolled in telemedicine, and City health insurance began offering ZocDoc as an option for finding a primary care physician. The BDT designed a behaviorally-informed to-do list encouraging City employees to register for ZocDoc and telemedicine. The to-do list was included in the mailing that sent City employees their new health insurance cards in summer 2016. The checklist was sent to employees with last names beginning with the letters A-K and a control version (with plain formatting) was sent to those with last names beginning with the letters L-Z.

  • A pseudo-random assignment was designed, but no comparative data were released by the insurer

NYC Mayor’s Office of Recovery & Resiliency (ORR)

Increasing Survey Completion for Flood Insurance Affordability Study

Only 11% of NYC households in flood-prone areas had responded to an online survey, which provided valuable information to inform policies and programs that address expanding flood maps due to climate change. In order to boost engagement with targeted homeowners, the BDT worked with ORR to design and test–via a randomized controlled trial—a behaviorally-informed "last chance letter" that simplified language, depicted the concrete benefits of participation, and imposed a soft deadline.

  • An RCT showed a 4.6 percentage point increase in survey completion, which means households were 15.5x more likely to complete the survey if they received our “last chance letter.”
  • The “last chance letter” was scaled to every household eligible for the affordability study.

Supporting Efforts to Boost Survey Uptake among High Flood-Risk Households

In a final effort to gather information on the flood resiliency and insurance affordability among flood-risk households, the BDT supported the design of door hangers to accompany door-to-door canvassing efforts to increase uptake of a survey. The design incorporated elements from an earlier randomized controlled trial with ORR: using icons to emphasize the benefits of participation, highlighting a soft deadline, and simplifying the steps to take action. In addition, individuals were given their unique PIN for survey completion on the doorhanger.

  • Personalized action steps were delivered to every household eligible for the affordability study.

NYC Department of Small Business Services (SBS)

Increasing New/Renewed Certifications of Minority & Women-Owned Businesses

While there are benefits to being certified as a Minority & Women-Owned Enterprise (M/WBE), many business owners never pursue or do not renew their certification. To increase the number of M/WBEs certified with the City, the BDT led a behavioral design workshop with SBS’s M/WBE team and co-designed a behaviorally-informed cover page for the application, as well as a behaviorally-informed, mobile-optimized email reminding already-certified businesses that they are required to recertify to maintain their status. The email implemented social norms messaging, simplified action steps, and made salient the benefits of certification.

  • The email and cover page were implemented citywide for all eligible businesses.

NYC Taxi and Limousine Commission (TLC)

Increasing Enrollment in a Wheelchair Accessible Vehicle Incentive Program

An incentive program that pays taxicab drivers to drive Wheelchair Accessible Vehicles (WAVs), aimed at increasing the number of accessible cabs on the road, experienced low enrollment and engagement. This prompted the TLC to look into their strategies for engaging drivers, and together the TLC and the BDT redesigned materials mailed to a subset of drivers, including enrollment instructions, a sample application, and a letter, all intended to reduce the hassles and ambiguity of applying to the program.

  • An RCT showed drivers who received the behaviorally designed mailing were 15% more likely to enroll in the program, but the underpowered study did not find this difference statistically significant.
  • The behaviorally designed mailing was scaled to all eligible taxicabs.