NYC’s Recovery Data Partnership: Cross-Sector, Values-Driven Data Sharing for Pandemic Response

By Matthew Leger • NOVEMBER 17, 2020

Back in late March and early April, an outbreak of coronavirus was surging in New York City. At that time, it was the largest hot spot for new cases in the country and would soon become the world’s epicenter. In the face of the outbreak, the city began to shut down; millions were quarantined in their homes, jobs were being lost in droves, and people began rapidly leaving the city. New York City as we all know it was no more.

Very early on in the economic shutdown, Joseph Berkman-Breen, policy advisor for the Mayor’s Office of Policy and Planning (MOPP), was working to identify public and private datasets that could help response efforts. At the same time, many NYC private and nonprofit entities were proactively reaching out to the city eager and ready to help with response and recovery efforts, including sharing data. Berkman-Breen quickly realized that these external organizations would have extremely valuable information that the city government did not and began working with these organizations to develop data sharing partnerships. Kelly Jin, Chief Analytics Officer and Director of the Mayor’s Office of Data Analytics (MODA), caught wind of Berkman-Breen’s efforts at the same time that many agency leaders were reaching out to her looking for data, and she quickly contacted him to see how her team could assist.

Together, they began forming one-off data sharing partnerships with companies and nonprofits. These initial partnerships were focused on providing data to a single city agency with a specific and defined need, but the team soon realized that the shared datasets could also bring value to other agencies. Over the course of several brainstorming sessions, the team decided to launch a formalized data sharing partnership and an open call for data. From there, the NYC Recovery Data Partnership was born.

Building the foundations of an innovative cross-sector data sharing partnership

The Recovery Data Partnership (RDP) is co-chaired by the directors of three offices: MODA, MOPP, and the Mayor’s Office of Operations. The city’s Chief Privacy Officer, Laura Negrón, and counsel in the Mayor’s Office of Information Privacy provide legal services and guidance. Driven by the leadership of these offices, the partnership brought together legal and policy experts, data analysts, project managers, and others to define the mission of RDP and iron out the technical and operational details of the partnership. Mayor De Blasio formally announced the launch of RDP on July 23. The founding cohort of data partners included StreetEasy, which provides neighborhood level real estate market data; LinkedIn, which provides real-time hiring data; Cuebiq, which provides aggregate mobility and density data; as well as Foursquare and SafeGraph, which monitor foot traffic at the neighborhood level. The RDP team is now working to onboard a second cohort of data partners to be announced shortly.

In establishing this initial cohort of partnerships, the team confronted two key challenges to effectively scaling the program. First, they found that negotiating data sharing agreements with external partners one at a time was an incredibly labor intensive and time consuming task. To address this, the team developed a standardized data sharing agreement template that they could present to potential partners at the outset of negotiations. This not only helped to streamline the data sharing agreement process but also enabled them to embed uniform, responsible data sharing practices into each and every relationship. Data partners are required to collect their data with lawful consent, publicize their participation in the partnership, and provide comprehensive data documentation to the RDP team. The team has publicly shared the data sharing agreement template so that other cities across the country can replicate this in their own operations.

Second, they realized that the datasets coming in from partner organizations, prior to founding RDP, were generally siloed, and intended to be shared with only one or two agencies. This made it difficult to share data with other departments that might benefit from the data but weren’t involved in the initial discussion or agreement. To address this, the team stood up a data repository, including a governance structure to provision access to the repository. This meant that any approved agency can pull data for specific, approved use cases. Doing so helped streamline data sharing across agencies and freed up the team to focus on identifying the city’s data needs and the partners that could help fulfill them.

A delicate balancing act: Ensuring actionability, while protecting privacy and integrity

Given the proprietary and sensitive nature of the data being shared, the partnership must achieve a delicate balance that ensures actionability for city decision makers while also protecting data privacy and integrity. To start, RDP expends a lot of energy up front ensuring that any data brought in will help directly address COVID-19 response and recovery needs, and that the data partner meets the city’s standards. This emphasis on purposeful, values-driven data sharing starts during the legal agreement negotiations where the city evaluates potential partners on several criteria including the type of data, how it is collected, and how relevant it may be to COVID response and recovery efforts. The team, including the Chief Privacy Officer and Mayor’s Office of Information Privacy counsel, conducts an extensive review of all datasets and sharing agreements before bringing new partners in.

Next, in order to access the data in the repository, each requesting agency must fill out an application. Each agency has to outline the relationship between their work and COVID response, and how the data they intend to access will help in those efforts. They have to discuss any other datasets they’ll bring in, as well as who will be working with the data. Once filled out, the application is sent to the RDP program team, which reviews it to ensure it meets the partnership’s standards, especially for data privacy and protection. An external group of advisors, made up of data experts, contribute their perspectives to help improve use cases, and act as ambassadors for the program, using their networks to pull in other data partners that can better serve the city’s needs. These steps are designed to ensure that data is used in ways that will ultimately help New Yorkers.

According to Jin, it is a tough balance to try and make the data accessible to agencies that need it and streamline the process of sharing that data, while also ensuring integrity of the process and the data itself. However, she notes that the hard work that goes into finding that balance is absolutely crucial. To ensure they are not slowing down the data sharing process, the partnership places an emphasis on clearly communicating their values, the process for getting approved, as well as what makes for a good application. To the extent possible, the partnership has also established data sharing and use standards that align with existing data governance structures the city already had in place prior to the pandemic to prevent confusion for city agencies. RDP’s leadership is also constantly gathering feedback from agencies about the process and making improvements.

Purposeful, values-driven data sharing drives RDP’s success

In advancing RDP, Jin pointed to three key factors that have enabled success. First and foremost, the primary driver lies in prioritizing purposeful, values-driven data sharing. “This isn’t just data sharing for the sake of data sharing,” she said in an interview. “We are really narrowing down the scope to ensure we are collecting data that is really helpful and that we are pulling in partners that truly care to help.” To demonstrate and clearly communicate these values, the partnership has outlined the guiding principles that direct their work: equity & fairness, transparency & accountability, and privacy.

The second key to success according to Jin has been targeting data partners that are based in NYC, understand NYC, and have NYC-level data. “There are lots of national companies out there that can provide detailed information only at the national or state level,” she said. “While that information can be helpful, this partnership is about pulling in data that is drilled further down and is granular enough for us to take action at the neighborhood level.”

Lastly, the RDP team works to ensure data is contextualized and detailed prior to being shared with agencies. One of the biggest challenges with analyzing third party data is that it can be difficult to understand what it says and what its limitations are unless the source explicitly outlines those details. The partnership works to provide this level of specificity with the assistance of data partners, the program team supporting the effort, and external advisors – all so end-users can account for those limitations in their analysis. As part of the data sharing agreement, potential partners have to provide detailed explanations about the data that describe what it says, how it could best be used, and any potential limitations. RDP also maintains communication with partners so that the team or other agencies can follow up with any questions they might have.

Looking ahead: A new precedent for cross-sector data sharing

While the partnership is meant to address short term COVID-19 response and recovery needs, Jin is also looking at how this partnership can benefit the city in the long term. If proven successful and efficient in the midst of a pandemic and economic downturn, she believes that this partnership has the potential to set a new precedent for data sharing between the private, nonprofit, and public sectors to achieve mutual goals in “normal” times. While the partnership is still relatively new, there are several innovative use cases that have stemmed from its founding.

First, NYC Emergency Management (NYCEM) has built two major data dashboards: one for tracking response related activities daily and one for tracking recovery metrics on a weekly basis. The daily dashboard was first initiated during the early response phase of the pandemic. However, as cases started to decline and the city began a phased approach to reopening, NYCEM started developing weekly recovery dashboards to track how the city was recovering in the following sectors: healthcare, economic, housing, government, social services/community, and transportation. These high-level dashboards have helped inform emergency management operations and recovery efforts as well as provide other agencies with situational awareness about what is happening in different sectors or places throughout the city.

Additionally, the Department of City Planning’s (DCP) has been a thought partner and close collaborator with the Partnership team since project inception. Early on, DCP took on a significant role in building RDP’s data infrastructure – its data engineering experts did extensive research on the best technologies to use for processing and publishing the data, created automated pipelines that transform partner datasets, and made them accessible to the end users at city agencies. For ease of use, DCP standardized the datasets, built a user-friendly interface for users to access the data once they receive permission, and wrote detailed metadata to accompany each dataset. DCP’s transportation and mobility group have also been using the data to provide regular updates to a broader set of city agencies that monitor movements across, as well as in and out of, the city.

An added benefit of RDP, although it was unintended at first, is that it has served as a “central hub for conversations about data needs for city agencies,” said Berkman-Breen. In the early days of the partnership, the group’s leaders have already seen an increase in communication between agencies about what data they have and have seen working groups informally spring up as different agency analysts come together and try to answer similar questions.

At the end of the day, “it is about making sure we’re making analytically driven decisions in response to COVID-19,” Berkman-Breen continued, “and that entails having a clear sense of what's going on in the lives of New Yorkers, and in our major economic industries. To do that, we need to bring in as much data as we can to help us get a clearer picture of what the city looks like right now or at any given moment in time.” While city government has a significant amount of data to work with, these partnerships have played a critical role in filling in information gaps and providing more timely information that enables a more proactive response. It has also set the foundation for better relationships and collaboration between sectors that have important implications for the future of NYC as it attempts to rebound from the devastation the pandemic has caused.

About the Author

Matthew Leger

Matt Leger is a Research Assistant for the Innovations in Government Program at the Ash Center. He has a diversity of experiences in research across the public and private sectors, as well as in academia with a primary focus on understanding how technology can be used to help address some of society’s greatest challenges. Matt has worked with the Smart Cities Strategies team at the International Data Corporation (IDC); the NYCx team in the NYC Mayor’s Office of the Chief Technology Officer; and at the research institute CTG-UAlbany. He holds a Bachelor of Arts in Public Policy and a Master’s Degree in Public Administration both from the Nelson A. Rockefeller College of Public Affairs and Policy at the University at Albany in Albany, NY.