Read more articles by Steve Goldsmith

By Stephen Goldsmith • January 26, 2018

This brief is a part of Bloomberg Philanthropies' What Works Cities initiative. Originally published in March of 2017, the brief was updated in January of 2018 to include four new case studies. To download the brief as a PDF, please click here.


What Works Cities, an initiative of Bloomberg Philanthropies, pairs mid-sized cities with expert partners— the Behavioral Insights Team, the Center for Government Excellence (GovEx) at Johns Hopkins University, the Government Performance Lab at the Harvard Kennedy School, Results for America, and the Sunlight Foundation— for technical assistance in better using data and evidence. After three years of work with 100 cities across the United States, the appropriately named program identified the key characteristics of a city devoted to using data to comprehensively identify, well, what works. Driven by the desire to share that knowledge more broadly, What Works Cities designed its Certification program to recognize high-performing cities, to create an objective standard of success, and to help cities at any point in the data journey understand how they can improve their practices. As Jenn Park, Associate Director for What Works Cities, said, “We want to be able to show the world what the best cities are doing. The Certification program is made to be able to do just that—publicly validate, recognize, and celebrate cities that are doing this work at the highest level.”

Certification measures a city’s work across criteria in the domains of open data, data governance, performance analytics, low-cost evaluations, results-driven contracting, and repurposing for results. The What Works Cities Standard—commit, measure, take stock, and act—has guided the What Works Cities initiative from the beginning, and the Certification criteria are divided into those four areas. The Standard represents phases of a city’s work to use data and evidence effectively, beginning with a mayor’s public commitment and concluding with using a deep understanding of city data to inform major policy and program decisions. Simone Brody, Executive Director of What Works Cities, described the Standard as “the North Star of what this work should look like.” She noted that, based on demand from cities for a tactical guide to improving practices, Certification takes the theoretical Standard and translates it into concrete indicators.

Any city with a population over 30,000 is eligible to apply for Certification, and after a robust evaluation of their efforts, high-achieving cities will be recognized with silver, gold, or platinum certification. Applicants will be able to benchmark themselves against their peers and get a clear sense of where their individual practices are and in what areas they can move forward. Although What Works technical assistance is limited to midsized cities, participating in Certification allows far more cities to access resources and participate in a growing community of cities.

What Works Cities Certification fits into the existing landscape of initiatives in this space by recognizing governments that have developed a broad, citywide capacity for using data and evidence, rather than awarding specific successful initiatives. By measuring aspects such as establishing a person or team responsible for data standards and protocols, developing a process for releasing open data, and measuring the outcomes of key procurements, Certification focuses on the fundamentals of data-driven government in a way that other recognition programs do not. Elevating the day-to-day city work and processes that results in dramatic successes is an important contribution to the field.

Brody emphasized that even a city just starting out can benefit from the process, noting that the keys to success are accessible to any city: “What we’ve found is most important to being effective at this is a real commitment from senior leaders in cities, a real belief this is going to improve outcomes, and then giving folks in city government the space to be innovative and try new tools and practices. Any city can do that if they want to, and we’ve seen dramatic progress in cities just starting out, even in a few months.”


What Works Cities Certification evaluates applicant cities on 50 criteria within open data, data governance, performance analytics, low-cost evaluations, results-driven contracting, and repurposing for results. The criteria are focused on the people, programs and policies necessary to improve the effectiveness of government. Below, we highlight examples of the cities in the What Works community that are already achieving selected criteria at the highest levels, both to provide inspiration and to illustrate what a top example of each criterion looks like.

Seattle's Open Data Policy

Commit: Does your local government have a codified open data policy?

The City of Seattle, Washington was one of the first U.S. cities to pursue open data, creating the first iteration of its open data portal in 2010 under Mayor Michael McGinn, a prominent proponent of government transparency. Since then, Seattle has established itself as a leader in open data, consistently increasing the volume and accessibility of available information. In 2015, Seattle was named a What Works City, and according to Seattle’s Chief Technology Officer (CTO) Michael Mattmiller, it came at the perfect time for building the city’s open data capabilities. “We were already thinking about how to re-engage the city and expand the open data portal’s use,” Mattmiller said. “One of the areas that we identified to focus on was this notion of creating an open data policy.” The city began work with What Works Cities partner the Sunlight Foundation to develop this policy. “It was very helpful for us to have the model policy language, to have specific policy objectives that we could work towards,” said Mattmiller. “But, we also realized that we had some unique aspects of Seattle that we had to mediate.”

For Seattle, it was important to develop a policy that fit the needs of the community, particularly in the realm of privacy. Mattmiller explained, “Before the What Works Cities engagement started, we had several missteps in our community about how we collected and used residents’ data…When we thought about opening more datasets, we felt the tension that our community was going to have between seeing this as a win for transparency and economic development and concern about what that data might do in terms of causing privacy harms.” The partnership with What Works Cities provided an impetus to mold an open data policy that worked for Seattle residents. The city partnered with the University of Washington and received a grant to develop a municipal privacy program, creating an action committee to establish a set of privacy principles to include in the open data policy. The city then reached out to the Seattle community for feedback on the policy, making a number of changes to the types of data to be opened. Seattle also established a network of open data champions in the city government to examine data before publication to monitor for invasive personal info and potential mosaic effects—combinations of datasets that together provide private information. To implement its privacy and open data policies, Seattle has partnered with the Future of Privacy Forum to identify and help mitigate risk present in its open data program, and contributed to research led by the Berkman-Klein Center at Harvard to share its policies with other cities.

However, publishing and protecting data was only the beginning for Seattle; the city then had to encourage city employees and citizens to use that data. Seattle began convening the open data champions and other city employees each month in an “Open Data Breakfast of Champions,” bringing in guest speakers to talk about applications of data. “Through these meetings we keep people enthusiastic and give them best practices,” said Mattmiller. Seattle also held a Data Camp, in which the city took employees off the job for three days for training on how to use the open data portal in addition to other data skills. Moreover, to promote resident engagement with data, the city created the position of Civic Technology Advocate, a data leader that goes into communities and hosts meetings, hackathons, and design labs to spread the goals of the open data program and empower residents to use this data. As a result, the city has seen the development of a number of useful tools and applications. For example, thanks to a Park Hackathon, developers created a tool using Parks Department trail data that helps users navigate Seattle’s parks.

By creating an open data policy that engaged users but fit the needs of Seattle employees and residents, the city was able to invigorate and institutionalize its open data program.  

San Francisco’s Intuitive Data Management- New case study

Commit: Does your local government classify data according to sensitivity and need for protection?

Measure: Has your local government established or adopted data standards (e.g., address and date formats, preferred geospatial projections)?

Take Stock: Does your local government have a designated person or team responsible for managing data?

In the City of San Francisco, managing the city’s data is a family affair.

“Everyone is responsible for data governance,” says Joy Bonaguro, San Francisco’s Chief Data Officer (CDO) and leader of the city’s DataSF team.

Data governance—the practice of ensuring the availability, usability, integrity and security of data—is a relatively new concept and one that municipal employees may see as needlessly bureaucratic. That’s why Bonaguro has sought to slowly introduce the process into San Francisco’s day-today operations and as part of other value-added services.

According to Bonaguro, DataSF does background research in data management, identifies best practices in other jurisdictions and sectors and develops departmental standards accordingly. Most critical, however, is the next step: ensuring that departments actually follow these standards.

“A lot of people in our world are a little allergic to new policies and standards without an implementation approach that provides reassurance that we’re actually doing it,” said Bonaguro.

That is why DataSF only introduces standards that meet a three part policy test: 1) they address a broadly shared pain or value, 2) policy and standards are an appropriate fix, and 3) implementation is feasible.

The heart of DataSF’s approach to data governance rests in its annual inventory and its open data service.

San Francisco starts from the foundation of an annual data inventory. The city documents all systems and data in departmental possession. The inventory provides the basis of data governance by defining data roles and responsibilities and generating important metadata, including technology used, data coverage and more about each dataset. Additionally, the inventory classifies information as either public, sensitive or protected. This classification feeds the city’s cyber policy, by helping to identify those systems in greatest need of protection. This classification also serves to flag data that must be processed through the City’s Open Data Release Toolkit, which codifies a risk management approach for data publishing.

To facilitate the annual inventory, each department designates a data coordinator. According to Jason Lally, DataSF’s Data Services Manager, the data coordinators group and the inventory process has formalized many of the roles and responsibilities of data governance. In order to assist these employees in their novel and evolving role, DataSF maintains a set of resources for the data coordinators, including a frequently updated guide that advises coordinators on best practices.

The city’s open data program steps in to support data governance through facilitating and enforcing standards.

“A great example of [the relationship between our open data and data standards work] is our metadata and licensing standards,” said Bonaguro.

In order to upload data to the city’s open data portal or the new Open Data Explorer tool, departments must ensure their metadata conforms to the city’s standards. If they try to publish non-conforming data, the DataSF team is alerted and can direct departmental data coordinators to reformat.

“When I arrived in San Francisco, it was the Wild West in terms of publishing,” said Bonaguro. “Now everyone wanting to publish data has to come through our publishing process, which provides a wonderful control point for enforcing standards.”

Once data is published, DataSF supports improving data quality by automating the continuous profiling of data by generating statistics and summaries of every published dataset and field. That can help data publishers quickly identify quality issues, e.g. by highlighting that a field has date ranges or values that don’t make sense. This combined with their Data Quality Guidebook, helps demystify and simplify data quality processes.

Collectively, DataSF’s approach emphasizes a fabric of data governance that gently enforces and continuously improves the management of city data.

Setting Strategic Goals in South Bend

Commit: Has your local government defined and made publicly available time-bound, measurable local government-wide strategic goals (e.g., reduce homicide by 20% in three years)?

The City of South Bend’s notable ability to set strategic goals has improved city government in a major way: helping Mayor Pete Buttigieg deliver on critical priorities and driving structural changes in the way the city addresses problems and services. By setting clear goals that drive work throughout the city, and reporting on those goals to residents, Buttigieg has created a high-performing government that is accountable for results. One outstanding example is the publicly-stated strategic goal of addressing 1,000 vacant or abandoned properties in 1,000 days, which started in early 2013. Mayor Buttigieg wanted to tackle the issue of blight, which residents told him was a priority during his campaign, in a visible way that allowed citizens to track the city’s progress.

The city’s commitment to addressing the vacant properties was measurable and available on the city’s website. Even when the process had issues, the public value was clear. Local media picked up on a bug in the progress tracking system that erroneously showed 100 pending properties as already addressed. The city’s Chief Innovation Officer Santiago Garces said this media revelation led to structural changes in the way that the city was tracking its progress with code enforcement. These changes, which included simplifying inspector checklists, requiring inspectors to take pictures of the properties, and assigning a central data analyst to do quality assurance, allowed the city to “improve the speed at which we were addressing the properties and we actually exceeded the goal that we had set,” Garces said.

Other strategic goals laid out by the South Bend city government include ensuring transparency and equity in policing, enhancing physical and technological infrastructure, and addressing mobility. In addition to addressing public concerns and creating action-driven strategic goals, the city consulted with the Drucker Institute to help with the framing of goals as well as the Center for Priority Based Budgeting, What Works Cities, and GovEx. Garces said working with outside groups was critical to build the “operational capacity and framework” and making tangible goals that address public concerns has been critical in building trust with residents.

A key thread running through the strategic goals is the emphasis on reporting progress and critical information to the public. The city is working to create transparency-oriented microsites on the open data portal that will report data and contextual information about specific goals to the public. The first such site, which is set to be released in Spring 2017, will focus on the strategic goal of “making sure the city has a 21st-century police department.” Garces added that these microsites will help the city better tell the story of what the city is trying to achieve and how it is progressing toward that goal.

Kansas City's Data Inventory

Measure: Does your local government maintain a comprehensive data inventory?

Kansas City, Missouri’s comprehensive data inventory shows the importance of clear internal structures and processes to a successful, sustainable open data program. When Chief Data Officer Eric Roche realized how much time he was spending updating out-of-date, non-automated open data in the city’s portal, he embarked on a project to understand and inventory the data in all departments to develop a more systematic approach to open data publishing in the city.

Roche took a methodical approach to building the inventory: he drew on the relationships he had built through the performance management program, asked for organizational charts, and talked to individual departmental representatives. Through What Works Cities, the Sunlight Foundation and GovEx provided guidance on the inventory process. Roche acknowledged that not all department officials were data systems experts, but nonetheless, he and his team were able to find the answers they needed: “We asked what kind of work the departments do, how they track that work, where they store it and then backed our way into the more technical questions.”

This process has proved fruitful; Roche has been able to identify people who “speak data” in several city departments and this has led to positive relationships that yield results beyond the inventory. This peer-to-peer work in the city government has been the key ingredient in the building of a comprehensive data inventory for Kansas City. Roche said that the biggest lesson he learned was to “start small”— the city originally planned to complete an inventory of seven departments in 60 days, but adapted the plan to incorporate departments in an ongoing way that also builds capacity for data in other city departments.

Roche said the development of a citywide data inventory has given the city a thorough, well-documented resource that facilitates a more effective open data strategy. It allows the city to prioritize data releases based on key priorities and what can be automated, instead of just the “low-hanging fruit.” Technical difficulties are a common barrier to publishing certain city datasets, Roche said, as data systems are not always compatible with publishing online, but the inventory has served as a critical resource for the city in navigating such challenges. “The inventory gives me the ability to move on to the next thing,” Roche said. “It gives me the sense that there’s more out there, there’s a lot more valuable data to grab at any given moment.”

Tracking Success on Results Vegas- new case study

Measure: Does your local government publish progress on local government goals on at least a quarterly basis (e.g., via a dashboard, update to local government’s strategic plan, etc.)?

Responsibility might not be the first word that comes to mind when you think of Las Vegas, but behind the uproarious mischief that attracts millions to Sin City sits a government that has placed a premium on accountability. By setting and tracking progress towards strategic goals, the city has ensured that it remains responsible for delivering effective service to its residents.

While Las Vegas has had an open data portal since 2013, the city really began its push towards accountability after partnering with What Works Cities in 2015. This relationship came at the perfect time for Las Vegas: in 2014, the City Council had developed a new set of four priorities— Economic Diversification, Education, Homelessness, and Transportation Mobility—and had asked city departments to develop goals aligning with these priorities. However, Las Vegas found that “departments were coming up with goals but had not yet figured out how to measure the outcomes,” said Victoria Carreón, Administrative Officer for the city.

At the same time, the city was conducting a self-assessment of its performance management processes, which revealed many opportunities for improvement. While departments had developed 600 measures for assessing their work, many focused on outputs rather than outcomes, and only 20 percent of city staff said that metrics reflected key departmental priorities.

This provided an invaluable opportunity for What Works Cities to work with Las Vegas and redesign the city’s performance management approach. In December 2015, What Works conducted a pilot program with city departments to work on developing strategic, outcome-driven goals. “Each department in the pilot redefined its goals to focus more on outcomes, developing one key performance indicator and supporting measures,” said Carreón.

Using these insights, Las Vegas then set out to revolutionize the ways the city sets, tracks, and delivers on its goals. “After the What Works Cities engagement, we set a citywide thematic goal to be completed in six to nine months,” Carreón explained. The goal was the implementation of Results Vegas, a new system for tracking city goals via a public-facing dashboard that would involve contributions from all city departments.

The first step towards reaching this goal was working with city departments to develop the measures that would eventually go onto the Results Vegas website. Led by Carreón, the city’s Office of Administrative Services expanded upon the What Works Cities pilot in order to work with departments to complete a broad overhaul of metrics with a focus on measurable outcomes. The city then showcased the fruits of this work in a citywide visioning document called City by Design, which communicated council priorities and related goals in a “user-friendly consumer guide,” according to Carreón.

In December 2016, the city then gathered the relevant departmental metrics on the Results Vegas website. Developed by the city internally, Results Vegas includes interactive dashboards for city focus areas, displaying goals and data on progress—some updated annually, some quarterly, and some once a month. The city manager’s office reviews progress on these goals on a regular basis in order to inform interventions.

According to Carreón, during this process, What Works Cities and partner the Center for Government Excellence (GovEx) at John’s Hopkins connected Las Vegas with other cities that had already created similar websites. “Knowing what cities a little further along had done was instrumental,” she said. Las Vegas incorporated elements from the efforts of many other cities into Results Vegas: “There’s a little bit of New Orleans, Chattanooga, Seattle, Portland, and Kansas City in there,” Carreón explained.

The next step was ensuring that this performance work became embedded in the day-to-day operations of the city. The Office of Administrative Services asked departments to redesign their strategic business plans, aligning with the measures they had developed previously. The city then sought to integrate these metrics into its budget, starting by changing the timeline for business plans to align with the budget cycle. “Normally, budget decisions had all been made before departments came out with strategic business plans, so departments didn’t have much of a chance to make their case,” Carreón explained. Now departments’ business plans are due in February, a week after they submit their budget requests for the next fiscal year. As a result, “Departments can use their business plan as a narrative justification for budget requests,” said Carreón.

And, in order to ensure a continued commitment to performance management, the city has amended its performance meetings. “We used to have each department meet individually once or twice a year with the city manager executive team,” said Carreón. In these meetings, attendees usually talked about critical issues in their departments, not on ways of meeting broader city goals. In an effort to emphasize cross-departmental priorities, the city has begun organizing meetings around four themes: Growing Economy, Neighborhood Livability, Community Risk Reduction, and High Performing Government. Now, between three and six departments meet with the city manager at a time to discuss a specific theme, and the city holds eight meetings per year.

Now that the city has developed the framework to pursue performance-driven work, Carreón envisions her team in the Administrative Office expanding its role. “We want to move towards not only trying to help departments set, but also reach goals” via “a menu of services” that includes strategies like predictive analytics and behavioral interventions. Carreón’s team has already begun training city staff on how to use these services, moving the city towards a comprehensive performance-driven enterprise.

Results-Driven Policy in Boston

Measure: Does your local government measure outcomes, impacts, and/or cost-effectiveness of key procurements, contracts, and/or grants? (i.e., monitor performance data in real-time and troubleshoot with contractors to achieve the goals of the contract and/or grant)

Boston, Massachusetts has a robust open data, performance, and analytics program, so when the opportunity to engage with What Works Cities experts arose, government leaders looked to apply the power of data to their contracts through results-driven contracting strategies. They focused on the Department of Public Works Construction Management Division, which manages numerous contracts each year, with the help of Elijah de la Campa, a Fellow from the Government Performance Lab (GPL) at the Harvard Kennedy School.

Each year, Boston spends nearly $8 million on an asphalt resurfacing program for its 800 miles of streets. To ensure an equitable distribution of repairs, the city divides this work into three geographical regions and accepts bids for each. While the prior contracts included technical standards related to the quality of asphalt resurfacing, there were few mechanisms in place to enforce or incentivize vendors to adhere to the standards. Modifying the contracts for this program offered the city a chance to increase the overall quality of repaving efforts, to improve communication and transparency with vendors, and to enhance the articulation and measurement of outcomes crucial to the asphalt resurfacing process. 

The Department of Public Works and GPL began assessing the existing procurement process by gathering information from stakeholders. De la Campa emphasized the qualitative and human-centric nature of this work, as he spent considerable time meeting with city engineers and vendors to understand concerns with the program, how they could be best addressed, and the viability of different types of performance payment.

In its new asphalt resurfacing contracts, the city has defined outcomes of interest related to pavement quality, the speed and progression of paving operations throughout the city, parking management, and environmental management, among others. Because the data to rigorously measure these aspects of performance did not yet exist, the city set up new processes for its engineers to track data. The contracts are now written with a clear set of outcome metrics, which are incentivized with a new performance-based payment structure. In addition to offering performance payments for meeting pre-specified progression of work benchmarks, the city will grade each contractor’s performance three-quarters of the way through the paving season, and then award additional in-season work for the final quarter according to vendor performance. The vendors benefit from the clear information about the city’s expectations and the incentives for high-quality performance.

Boston has now hired its pavers for 2017 using the new contracts, and will implement the performance-based payment structure for the first time this paving season. The new approach of results-driven contracting has many more applications throughout the city’s operations to help the city deliver better services to its residents.

Pairing Performance and Analytics in New Orleans- new case study

Take Stock: Does your local government have a designated person or team responsible for performance management?

For the City of New Orleans, performance management is a means of combating one of the most fundamental problems of federated governments: division. “The root of much public sector mediocrity is siloed departments,” said Oliver Wise, Director of New Orleans’ Office of Performance and Accountability (OPA). “Individual departments are not engineered to focus on cross-cutting projects or the experience of residents.”

The need to unify departmental priorities and hold agencies accountable for meeting their goals was the inspiration to create OPA, a centralized performance management team in the city. “We wanted to prioritize outcomes instead of widgets and develop strategic approaches that cut across departments,” Wise explained. While New Orleans could have integrated performance management into individual departments, creating a central office with a 360-degree view of city operations has allowed the government to transcend the individual departmental perspective and facilitate collaborations that support citywide priorities.

The office has a unique model, pairing performance management with analytics in an effort to deliver maximum value to city departments. “We started as a performance management shop, and our theory of change was to set goals and use data to track performance, ratcheting up tension and accountability to compel those goals to be met,” said Wise. With these goals in mind, integrating analytics was an obvious next step. “With analytics, the value proposition is providing departments with the intelligence to do work smarter, which supports better performance.”

This combination of performance and analytics has proven successful in addressing a number of core city issues. Perhaps most prominent was the city’s BlightStat approach to blighted properties, for which Mayor Landrieu’s administration set a goal of addressing 10,000 blighted addresses in four years—and then delivered ahead of schedule.

More recently, OPA has applied its performance and analytics approach to police recruitment and retention. In the last several years, New Orleans has faced a severely understaffed police department. “At the beginning of the administration, we didn’t have the money to hire any new officers for around five years,” Wise explained. “At the same time, the Police Department was going through a lot of major reforms—which were good but made life different for officers. A lot of officers were unhappy and left.”

To address this shortage of officers, OPA reached out to the New Orleans Police Department, the Civil Service Department, and the Justice Foundation to create a Police Recruit Stat group that meets each month to review metrics on officer applications, hires, and retention. To drive process improvement in these areas, OPA has turned to analytics. “On the recruitment side and now the retention side of policing, we’re asking why people aren’t applying, who is leaving, and why. These are questions analytics can help answer,” said Wise. The city is currently using “machine learning algorithms like random forest models, gradient boosted decision trees, and flexible discriminant analysis to uncover the most important variables that lead officers to leave,” Wise continued. Examining these questions has informed changes that respond to officer priorities and concerns—like a 15 percent pay raise and investment in state-of-the-art police equipment.

While working on individual projects, OPA has also sought to embed performance management into the culture of New Orleans’ government. To that end, in 2013 the city passed the Performance Management Policy, which formalized the performance management process, requiring departments to monitor progress on performance measures for funded budget offers. According to Wise, this policy will help “ensure that performance management transcends the people who occupy municipal offices.”

Scottsdale’s Behavioral Interventions- new case study

Act: In the last 12 months, has your local government initiated low-cost or randomized evaluation of priority local government programs or services in 5 of the local government’s largest departments and/or programs?

With help from What Works Cities, the City of Scottsdale has pursued behaviorally- informed interventions in a great variety of areas. The city has leveraged insights from behavioral psychology to design low-cost evaluations—A/B tests that compare the effectiveness of a number of interventions on a representative sample, also called randomized control trials (RCTs). Thus far, the city has developed low-cost evaluations encouraging employees to complete health risk assessments and set aside money for retirement, nudging residents to donate to local charities and pay water bills online, and recruiting a diverse group of new members to the police force and fire department.

According to Cindi Eberhardt, Scottsdale’s City Volunteer Program Manager and behavioral insights team leader, the choice of initiatives was strategic on the city’s part. “After we decided to partner with What Works Cities, we went to our executive team and asked who would be interested in sponsoring a trial,” she explained. Doing so ensured that the departments conducting trials had a genuine interest in and would follow through on the evaluations. The city’s police and fire department, for example, immediately spoke up about their interest in help recruiting a diverse cohort of new employees.

The other consideration was to identify policy areas that both addressed critical city goals and possessed the characteristics required to conduct low-cost evaluations. “Our approach was to look at Council priorities and identify all the places we could use data from RCTs,” said Brent Stockwell, Assistant City Manager. Conducting RCTs essentially requires two things: sufficient data and a human behavior to influence. For example, encouraging employees to fill out health risk assessments is a good subject for an RCT because there exists good data on whether or not employees complete the assessments and the desired behavior is obvious. On the other hand, a proposition like getting city employees to eat more healthily is not a good subject, because there is no way to track employee eating habits and the desired behavioral change is not clearly defined—should they eat more vegetables and drink less soda, or eat more protein and fewer carbs, or should this depend on the employee?

In an effort to embed this behavioral work into the city’s day-to-day operations, Scottsdale has created an internal team to work on behaviorally-informed interventions. While still in its early stages, the team has drafted a charter and brought in department leaders to discuss potential new areas for low-cost evaluations. Eberhardt described these meetings as focusing on three goals: “One, we wanted to educate department leaders on what the city has done. Two, we wanted to talk about potential areas in which we could provide value to their departments. And three, we wanted to discuss some of the challenges involved in finding data and determining outcomes to track.” The team is currently putting its structure and training in place, and Eberhardt says that they hope to finish four new projects by the end of the fiscal year next June.

However, as Eberhardt and Stockwell would admit, making behavioral interventions a regular feature in city policy has had its challenges. “With some of the projects we’ve done, the departments have gone back to business as usual,” said Stockwell. Part of the challenge is overcoming inertia in city departments, many which have completed the same tasks in the same way for many years.

According to Stockwell, facilitating strong organizational health is one means of overcoming this inertia. “You need an organization where people don’t think they have the answers to everything. You don’t test interventions if you know you’re always right. It’s critical to have that humility and curiosity,” said Stockwell. “Organizational health was part of what made the interventions in the police department easier and successful,” he continued.

Improving communications with departments can also help encourage continued commitment to behavioral interventions. On one hand, Eberhardt stressed the need to manage expectations upfront, communicating to departments that behavioral policies are not a silver bullet to all their problems. At the same time, “It’s important to have additional conversations with sponsors and individuals to ensure you emphasize the value of the work,” she said. Eberhardt pointed to the Behavioral Insights Team’s recent article on Medium “Eight Things Cities Can Do Today to Generate Evidence and Outcomes” as an example of the type of work that cities should produce and disseminate. The designers of a behavioral intervention should stress that such policies can produce significant results, but ensure that departments do not become disenchanted if interventions do not lead to radical change.

The other key lesson learned in Scottsdale was the need to create mechanisms for following up on initial evaluations. Eberhardt emphasized the value of management agreements that establish when a department will implement a behavioral policy or duplicate a project in another area. “You need capacity in place to follow up and make sure changes are actually implemented,” she explained. Doing so can ensure a city produces not only valuable insights from trials, but also meaningful behavioral change on a large scale.

Louisville's Performance Management Program

Take Stock: Does your local government convene a performance management program (i.e. Stat meetings)?

The City of Louisville, Kentucky’s performance management system, LouieStat, sets the bar for city government performance improvement. Mayor Greg Fischer united lessons from his business background with existing government stat models and unveiled LouieStat in 2012 to focus on two areas: planning and operations. “We needed to figure out how to plan, and we created consistent guidelines and language and a single coordinated strategic planning process that would help us measure the strategic areas of focus,” said Daro Mott, Chief of Performance Improvement in Louisville. “We also needed something that was more operational, which would have us measure the critical business processes—the processes that deliver the core of citizen services. We really needed to create a program that could answer the question of how Louisville could continuously improve on service delivery.”

Mott said that breaking the work into distinct strategic and operational categories was critical for the success of the system. “Operations should flow from the strategy of the city…If you start with data that you already have, you may not develop the right performance measures. You need to ask, ‘What are we planning to do and what data will help us understand how well we’re doing the work?’” This way, a city’s performance management efforts will center around its strategic priorities, rather than boosting performance on arbitrary metrics.

As a part of the planning process, Mayor Fischer developed a six-year plan with 21 city goals and asked each agency to develop its own goals and plans to achieve them. The Mayor’s senior leadership meets with senior staff from 18 of 20 departments four times a year and with other staff members between these forums. In these meetings, attendees discuss progress, look at metrics for the department and identify areas of weakness, evaluate the impact of city programs, and make data-driven decisions about where and how to best allocate resources. The Mayor attends many of these forums himself, and also meets with Mott on a regular basis to analyze Louisville’s performance on a citywide level. Mayor Fischer said what he calls a “weakness orientation” is key to making these meetings productive instead of punitive: “Bad stat programs are human- and people-focused and create more of a blaming culture. Ours is a celebration culture, focused on identifying broken processes or bad data and then fixing that and celebrating the people who do the work.”

In order to promote buy-in from so many departments, LouieStat from the beginning sought to demonstrate its utility to agencies. According to Mott, “What really got us more buy-in was facilitating process discovery workshops with departments by which we documented the critical business processes of each department and talked about measures linked to these processes.” In doing so, the Mayor’s Office introduced departments to performance management—and showed how performance management could help identify and track metrics to improve service delivery.

The performance management culture has become increasingly embedded in Louisville’s agencies. Mayor Fischer points to this as a critical aspect of developing a culture of performance; he said, “We provided training for people to understand how to solve problems, which has given them a sense of not just empowerment, but fulfillment and hopefully joy in their work, where now they feel they are in control of making things better.” The Office of Performance Improvement has trained at least one staff member in each agency to lead the LouieStat process and analyze that department’s data. Most data analysis now happens at the departmental level, and agencies have come to embrace a performance-based approach, learning to adapt LouieStat to their various needs.

Washington D.C.'s Low-Cost Policy Evaluations

Act: In the last 12 months, has your local government used the results from low-cost or randomized evaluations to improve local government programs or processes?

In the past year, Washington, D.C. has dramatically increased its efforts to use low-cost evaluations in policymaking. Boosted by commitments to data-driven decision making from the Mayor and City Administrator, the city launched The Lab @ DC last year. The Lab brings diverse scientific skillsets in house to enable the city to use low-cost interventions and other research methods throughout its operations.

By basing the team in the Office of the City Administrator, The Lab builds on existing relationships, processes, and data infrastructure. Chief Performance Officer Jenny Reed noted that the connection to performance management surfaces ideas and also ensures that the work is tied to the city’s priorities. Lab Director David Yokum said that in order to identify opportunities for low-cost evaluations, “Having scientists inside government is a strength. You really need to know a lot about the agencies, what they are capable of doing, what their budgetary constraints are, what their IT looks like – you need all those pieces to make the scientific judgment of what the opportunities are.” 

The Lab is already embarking on a variety of efforts, including testing redesigned paperwork for applications to Temporary Assistance for Needy Families (TANF). For a project with the Metropolitan Police Department (MPD), The Lab designed a randomized controlled trial for the rollout of body-worn cameras. Because the MPD was already planning to distribute the cameras and already collected relevant administrative data, adding in the randomized distribution had an extremely low marginal cost. The randomized trial will allow the city to compare the outcomes for officers with cameras to those without cameras to answer important questions about the technology’s effectiveness. Support from the Behavioral Insights Team through What Works Cities this year will facilitate additional projects.

Yokum said that even doing a small concrete project, such as testing two subject lines for an email, can inspire departments to come back with more ambitious ideas for ways to apply the same methods to other areas. The goal of the team is to support talented employees in all departments and raise the city’s collective capacity to use evidence to drive policy. Although it is still relatively new, The Lab @ DC demonstrates promise to scale to an evidence-driven District government. 

Using Data to Repurpose Funds in Jackson

Act: In the last 12 months, has your local government shifted funds away from a practice, program or policy that, through rigorous data analysis and evaluation, has consistently failed to achieve desired outcomes toward a more effective and efficient practice, program or policy?

Last year, Jackson, Mississippi faced a challenge familiar to many cities: a need to make significant budget cuts while trying to preserve jobs and maintain service delivery. Because of Mayor Tony Yarber’s commitment to data and the city’s prior achievements in developing an open data portal and launching a performance management program, the solution was obvious: turn to data to restructure and repurpose funds.

Beginning in May 2016, four months before the budget season began, Jackson began a comprehensive effort to analyze its spending, programs, and results to identify opportunities for efficiencies. The team responsible consisted of directors, deputy directors, executive staff, fiscal officers, and on-the-ground support workers. This team worked with the Johns Hopkins Center for Government Excellence (GovEx) through What Works Cities for technical assistance in data analysis.

With the help of GovEx, the city analyzed its budget in comparison to comparable cities and identified areas of disproportionate spending. GovEx also trained every departmental data coordinator and fiscal officer in how to analyze and visualize their own data with Tableau to enable ongoing data use. The city then used this work to analyze each department’s programs more granularly, which leveraged the city’s JackStats performance management framework to identify which were producing results aligned with the city’s priorities.

The city made changes in many departments based on the analysis. The Human & Cultural Services Department merged low-performing senior and child care centers with higher-performing ones. In other departments, staff positions related to underperforming programs were repurposed to higher-impact areas in order to avoid layoffs. The city also looked at departmental structures to identify existing functions that would be more efficient under the purview of another department, such as moving tree and limb removal from Parks and Recreation to Public Works, which owned the necessary equipment.

Mayor Yarber and his budget team saw record turnout at community outreach events related to the budget. Basing decisions on the data helped the Mayor and his senior staff have difficult conversations with departments and with the community. The dashboards and visualizations that the budget team used are available to the public and all departmental employees, ensuring that the data behind the decisions is transparent.

Justin Bruce, Director of Innovation and Performance, emphasized the progression of work with data that developed the capacity that made this effort possible. He said, “Open data allowed us to take data, clean it, work with it, and actually track progress and performance. JackStats helped us look at the data at a more granular level to show us why and how we are meeting goals.” The latest effort, he said, “allowed us to take performance management to a different level, not just looking at what we are doing but how efficiently and effectively we are accomplishing our goals.”

Jackson was able to cut its budget by 7.6% with this process. The city went from a $14 million deficit in 2014 to a $6 million surplus this fiscal year, in addition to creating a new reserve fund. The city plans to continue its priority-based budgeting and repurposing work in the same way going forward. Bruce said, “Every time we touch an issue that has data to guide the situation, we’re always going to do an analysis of that data and always going to determine what’s most effective and efficient and ultimately, what’s best for our citizens.”


The stories above illustrate some of the many ways that leading cities are leveraging the power of data and evidence. By objectively recognizing successes like these and providing a roadmap for critical data practices, What Works Cities Certification will enable even more cities to improve the way they work and deliver better services to their residents. For the first time, an objective organization has produced comprehensive and reliable criteria for high performance that will produce a roadmap to operational excellence for mayors aspiring to improve the quality of life in their cities.

Download PDF

This brief was written in conjunction with Harvard Kennedy School’s Katherine Hillenbrand, Project Manager; Eric Bosco, Research Assistant/Writer; and Chris Bousquet, Research Assistant/Writer. The San Francisco case study was written in conjunction with Joy Bonaguro, Chief Data Officer, City and County of San Francisco.