The City of Chicago Office of Inspector General (OIG) has completed an audit of the Department of Innovation and Technology’s (DoIT) management of the City’s investment in information technology. The objective of this audit was to determine if DoIT manages information technology investments in accordance with best practices outlined in the United States Government Accountability Office’s Information Technology Investment Maturity (ITIM) framework. Specifically, we examined how DoIT ensures that the City selects the right technology projects, manages them effectively, and evaluates performance after completion.
Effective management of an IT portfolio requires consistent and repeatable organizational processes. While certain projects may succeed without consistent enterprise-wide management, such successes are more often attributable to exceptional individual efforts, rather than effective, efficient, and repeatable institutional processes.
To assess the consistency and repeatability of DoIT’s processes, OIG compared documentation of DoIT’s processes and the outcomes of projects, such as budget or schedule information, to GAO’s ITIM framework. The framework describes five stages of process maturity. At the lowest level—Stage 1—organizations make IT investment decisions in an unstructured, ad hoc manner. This suboptimal approach may result from a lack of well-designed formal procedures, inconsistent implementation of such existing procedures, or a combination of the two. At the highest level—Stage 5—organizations have optimized their processes, and IT investments drive strategic organizational change. DoIT is in Stage 1 and is working toward Stage 2.
DoIT did not consistently adhere to best practices for project selection, which increased the risk of projects delivering fewer benefits, costing more, and/or taking longer than expected to complete. In addition, DoIT’s data collection practices hamper effective monitoring and evaluation of project and portfolio performance, thereby limiting the Department’s ability to identify opportunities for improvement.
DoIT designed a scoring tool to assess projects on a common set of predefined criteria, with the goal of ranking projects and selecting those that would most benefit City operations. OIG review of eight projects started in 2016 and 2017 determined that DoIT did not use the ranking process at all. Notably, DoIT did not have a complete inventory of the projects initiated during the years under review. Moreover, DoIT completed the required assessment prior to selecting only three of the eight OIG-reviewed projects. As a result, the City may have selected projects that did not best meet the departments’ specific and the City’s overall needs. The Department did not consistently collect critical information needed to rank projects and make selection decisions. In addition, the Chicago Police Department, Chicago Fire Department, and Office of Emergency Management and Communications each declined to use the project selection process DoIT developed. Therefore, DoIT could not rank these departments’ projects against those proposed by other departments for purposes of setting priorities for spending City resources.
DoIT did not ensure that launched projects met performance goals and did not consistently monitor progress. Five of the six projects reviewed took longer than scheduled to complete, with two taking more than twice as long as originally planned. Moreover, DoIT did not have a process or criteria for determining whether ongoing projects were meeting user department needs and should be continued or terminated.
DoIT did not evaluate projects across its portfolio and, therefore, did not adjust its investment processes based on lessons learned. The Department did not consistently evaluate project performance after project completion. Some project managers told us that, while they typically discuss lessons learned from projects, those discussions are not memorialized or used to improve project and portfolio management.
OIG recommends that DoIT rank all proposed projects using predefined criteria. The Department should also develop procedures for collecting more robust cost, benefit, and risk data to facilitate comparative evaluation of the merits across departments, i.e., City-wide. DoIT should work with the Office of Budget and Management (OBM) and the Mayor’s Office to ensure that the various boards, groups, and other entities authorized to oversee IT strategy and spending are fully engaged in maximizing the return on the City’s investments throughout the project lifecycle.
DoIT should also set performance goals related to cost/benefit and risk for each project, monitor performance against those goals, and report on performance to the appropriate governance body. Finally, project oversight should include evaluation of outcomes and long-term performance. Taking a broad view of the City’s portfolio of projects will improve the Department’s decision making at the proposal stage.
In response to our audit, DoIT agreed with OIGs recommendations and stated that it has undertaken changes that will address the findings. These changes include updating relevant policies, requiring project managers to adhere to all written policies for selection, monitoring and evaluation of projects, achieving full engagement by the IT Governance Board, and requiring all City departments to engage in the standardized IT oversight processes.