Evaluation 1: Your local government has a policy or ordinance that encourages the use of rigorous evaluation methods for practices, programs, and/or policies.
Evaluation 2: Your local government has defined standards, methodologies, or tools to help staff rigorously evaluate practices, programs, and/or policies.
How do you know if your work is making a difference? Perhaps you gauge impact by tallying how many residents are served, seeing efficiency improve year over year, or saving money while still maintaining excellent service. In short, you use metrics. Effectively using metrics on a citywide scale, however, requires data, a dedicated evaluation process, and an innovative leader. Enter Carmen Moreno-Rivera, who Louisville, Kentucky, Mayor Greg Fischer appointed to be the city’s third Chief of Performance Improvement in early 2019.
Moreno-Rivera built on Louisville’s strong tradition of evaluation and performance planning by implementing a systematic process and structure to gain insights into the planning, implementation, and efficiency of Louisville’s programs. Louisville’s approach is a valuable model for other cities that are hoping to gain more information about their programs and educate employees about the importance of data and evaluation. Thanks in part to Moreno-Rivera’s work, What Works Cities (WWC) has recognized Louisville’s dedication to their evaluation criteria.
Moreno-Rivera leads a team of three in the Office of Performance Improvement (OPI), including lead evaluator Rebecca Hollenbach. Given her previous experience at the city’s Department of Public Health and Wellness, Hollenbach brought her expertise in rigorous program evaluation to OPI. The team is compiling a program inventory, identifying which programs already had evaluations and which did not, and what kinds of program planning were already in place. They developed two tracks: one for basic program planning, and one for rigorous evaluations. OPI also adopted the Center for Disease Control’s program evaluation framework to apply to their evaluation processes.
Implementing an evaluation system has had cascading effects, influencing how the Office of Management and Budget (OMB) partners with OPI to make programmatic decisions, how programs collect data, what kinds of trainings are available to city employees, and allowing Moreno-Rivera and her team to tackle a broader set of priorities within the context of implementing evaluations. During the transition to this more systematic approach, Moreno-Rivera found that departments have varying levels of data readiness — some already have the data needed to evaluate, but for others the available data is not reliable or simply does not exist. To that end, the departments without data are now focusing on capacity building—which hasn’t always been prioritized—and OPI is conducting trainings for employees about how to collect data, why it’s important, and what a rigorous evaluation looks like. For many units, it starts with trying to link together all their data collection skills, learning program planning as needed, and finally arriving at the evaluation plan.
In addition, implementing a systematic evaluation process can lead to increased effectiveness and cost-savings for city programs. For example, as part of Louisville’s prior work with WWC in 2016, the Behavioral Insights Team and OPI worked with the Codes and Regulations Agency to redesign Louisville's code violation letters in order to improve the responses to violation and citation letters. The letters were simplified and redesigned, and when OPI evaluated these changes, they found a decrease in the number of re-inspections and a small increase in fees collected, ultimately saving the city on costs while also providing a modest increase in revenue. And in 2018, OPI, with Develop Louisville, helped evaluate the redesign of the city’s vacant and abandoned land applications. Previously, many applications were incomplete and could not be processed, or required costly follow up with applicants. After the changes, the percentage of incomplete applications dropped significantly, from 45 percent to 8 percent, again saving city resources.
Even with these benefits, there are still obstacles. Often, the OPI team faces hesitance to change from other departments, with some colleagues telling Moreno-Rivera “government just doesn’t work that way.” But, as Moreno-Rivera describes it, part of the resistance is that people aren’t familiar with formal evaluation programs, and it is part of their work to engage others and support their learning. These cultural obstacles can run up against the sense of urgency from Mayor Fischer — a strong advocate of performance improvement initiatives — but Moreno-Rivera knows that good evaluations take time. She has a goal to improve where continuous evaluations are embedded into programs, and as the city is faced with budget cuts, this means that agencies can be more pragmatic about advocating to keep the programs that are most successful. It is a different way of thinking for city employees, given OMB’s traditional role in making the final decision for programs. But this evaluation work can empower staff, and it can ensure that programmatic decisions will be more data-driven and efficient, rather than allowing funding availability to determine which programs exist and which don’t.
Through this work, OPI and OMB have established a close relationship, and Moreno-Rivera wants to make sure that relationship continues to develop. She is working towards having OPI and OMB jointly manage the evaluation process, ensuring that the Budget team is well-positioned to understand and implement program evaluation. Her team is working with OMB to train their staff on data collection and the evaluation process to increase government-wide capacity and build sustainability. The OMB team is also excited by this partnership, since it provides them with a new way of looking at programs and using data to make decisions about effectiveness and cost.
Moreno-Rivera and the rest of the OPI team have big plans for the evaluation work in Louisville. Since the creation of their department in 2014, they have always been a scrappy, dedicated group that not only introduces new data-driven methods but takes the time to mentor others across government, so they too can engage in the evaluation process. This holistic method is making a significant difference in Louisville and is a major reason that What Works Cities awarded them Certification at the gold level in 2019.
Moreno-Rivera shared advice for other cities that are interested in improving their programs through data and evaluation.
1. Meet staff where they are.
To encourage others to adopt the evaluation process, it is important not to assume that everyone is coming to the conversation with the same working knowledge. Sometimes when OPI leads trainings, its apparent that people are confusing an evaluation with an audit or a grant requirement. To ensure that everyone is on the same page, her team breaks down each piece of running an evaluation and scaffolds each component to show how all of the parts fit together into a worthwhile process.
2. Learn how to tell your data story from different angles depending on who is in the room.
Instead of leaning on complicated jargon or theoretical frameworks, the OPI team learned how to tell the same core story in distinct ways in order to help everyone understand the process and the benefits of evaluation. Given that the Louisville Metro Government is one organization with 27 unique units and departments, Moreno-Rivera tells her team that it’s ok—and even advantageous—to adjust their strategy depending on their audience when talking about the complexities and payoffs of evaluation. A department that frequently does evaluations, like a public health department, might not need the same approach that an emergency services department does.