Big Data, Analytics and a New Era of Efficiency in Government

By Stephen Goldsmith • May 23, 2013

This post originally appeared on

Buy something from Amazon and you will soon find the online retailer telling you what other products you might like. Linger too long on your preferred cell provider's website for new phones and you will quickly encounter a live person to chat with online about the options. Each day, companies of all sizes discover new ways to gain customer insights that allow them to target products and services with unprecedented specificity.

This same technology can fundamentally change the way government operates, breaking down hierarchies and silos, enabling preventive action, incorporating citizens into every aspect of governance and increasing overall efficiency. Data analytics offer us unprecedented opportunities to improve the effectiveness of government.

The key to these opportunities is "big data," the ever-growing volume of information created and captured by the modern digitized world, from cloud-based systems to sensors to smart devices. New data-mining techniques allow governments to break through legacy-system barriers that seemed insurmountable only a couple of years ago. We soon will see new solutions in every area of government, from how public agencies hire, train and promote, to how performance is measured, how problems are identified and preempted, and how personalized services are delivered.

Although the term seems enterprise-oriented, a significant component of the big data in the civic realm comes from the community. Citizens generate data when they converse with their governments via social media, when they participate in online "ideation" forums, and when they call 311 or use city apps to report problems or rate services. This citizen feedback can be curated into solutions to guide governments' rulemaking, problem solving and resource allocation.

The greatest public value and insights come when governments, through their open data and transparency initiatives, produce usable information that allows meaningful public participation in the delivery of public services. The proliferation of open data sets on sites such as encourages private sector, nonprofit and interagency use of once-locked information for the public good. It also prompts important discussions on data standardization and interoperability as well as around privacy concerns.

Yet these massive amounts of data will drive efficiency only when organized and analyzed in a manner that supports decision-making. Governments are just beginning to meaningfully incorporate data analytics into their operations, but the results so far have been highly promising: Predictive algorithms allow police departments to anticipate future crime hotspots and preemptively deploy officers or buildings departments to determine which structures are most likely to have code violations in order to efficiently allocate limited inspector time. Analyzing accumulated data from subway smartcards can predict the effects of transit disruptions and give broad insight into transit-system operations. Integrating data from different human-services agencies can greatly increase the effectiveness of social workers and others as they assist at-risk youth. Agencies and their workers can use digital tools both to collaborate and to gain new insight from their combined data resources.

Our new project at the Harvard Kennedy School, Data-Smart City Solutions, aims to help catalyze local government use of data, analytics and civic engagement technology. We are highlighting model programs, identifying best practices, curating resources, supporting cities embarking on new data projects, and connecting leading industry, academic and government officials in the field.

We're looking at a technology-driven sea change in governance over the next few years as these technologies rapidly become more affordable and accessible. Awareness of these new tools and opportunities can help the public sector break out of its hierarchical and rule-driven structures and reinvent itself through voices from the field, case studies documenting proven results, and fresh ideas.


About the Author

Stephen Goldsmith 

Stephen Goldsmith is the Derek Bok Professor of the Practice of Urban Policy and the Director of the Innovations in American Government Program at Harvard's Kennedy School of Government. His latest book is Growing Fairly: How to Build Opportunity and Equity in Workforce Development.

Email the author.