The city of Dallas had no backstop to prevent an employee from accidentally deleting massive amounts of data. That’s the main takeaway from the city’s investigation into how more than 20 terabytes of police files were deleted during a migration from a cloud storage system to an internal server. The city of Dallas never implemented a formal storage strategy, despite posting it to its website for the public to see. The report presents managers as absent or apathetic and flags the city for training, culture, and management shortfalls.
In August, we discovered incidents that happened in March and April: an IT staffer somehow had the power to delete millions of pieces of police data during a migration from the cloud to a local server, which the city was doing to save a little money. The City Council and the district attorney weren’t told for four months; the media learned of it after DA John Creuzot alerted the county’s defense lawyers.
The city released its report of what happened on Thursday. (And published it in … Calibri font? Really?) It is a chronicle of failings: the employee ignored warnings from the city’s system as he was deleting files. The city has “no set rules” for how to archive backed up data because it didn’t implement anything from a strategy document that laid out how to archive backed up data. And then three IT managers signed off on the individual’s work, but they apparently didn’t understand what they were signing off on.
“[T]he ITS Infrastructure Services managers either did not understand the actions to be performed, the potential risk of failure, or negligently reviewed the Change Request prior to providing authorization and approval to proceed with the Change Request.”
It continues:
“We understand the magnitude and seriousness of the situation and will continue to work diligently with DPD to recover as much data as possible. To date the recovery team has recovered 140,353 potential files that were deleted.”
Six figures! Sounds like a lot until you see the true scale of the deletion: the city believes 8.7 million files were deleted, which is more than 20 terabytes worth. Of those, 4.6 million may be recoverable if they are stored elsewhere, like on a police laptop. Most of what has been vanquished involved the Family Violence Unit, “information gathered by DPD detectives for prosecutable, adjudicated, on-going cases.” The DA’s office still is unsure of the potential impact to these cases, but the report says 17,494 were affected. The DA’s office classified 1,000 of those as priority cases.
Some of the findings in this 131-page report are jaw-dropping: the city has a strategy document for how it should manage data on its external website, but that document is “out of date” and, oh yeah, “it has not been implemented as a formal activity and process within the City’s data environment.”
The city isn’t regularly auditing its enormous data haul to ensure that it’s backed up properly.
“The failure to have a complete, centralized, and enforced Data Management capability managed and enforced by a Data Governance committee is a factor in the conditions which led to an environment in which the loss of data is possible. This failure, particularly regarding unstructured data was a significant factor, among many others that culminated in the loss and resulting inability to recover the Dallas Police Department data.”
The kicker here? The city knew what it needed to do. The data management vendor the city uses, Commvault, provided “detailed documentation” instructing how to migrate data. The employee didn’t follow it and his bosses “had insufficient oversight of the migration.” This “inadequate monitoring … directly contributed to the loss.” Plus, “[i]nstructions were either never presented, read, or reviewed by the technicians and management prior to actions leading to data deletion,” and executives within the IT department didn’t require routine training to stay up to date on “the latest knowledge of these functions.”
The entire report is filled with these anecdotes—poor oversight, poor culture, poor management, and very few safeguards. There was a fire in a bedroom and the city closed the door, walked into the living room, and turned on the Mavericks game. Then they acted surprised when the hallway was aflame.
The report points the problem right at the top of City Hall’s governance structure:
“Gaps in documented management directives and clear expectations around management control systems instantiating and commitment to the City of Dallas’s core values of excellence, ethics, empathy, and equity exist.”
This isn’t the end of it. The FBI is investigating because Dallas police lacked the knowledge to determine whether a crime was committed. (Which, fair. Weird situation.) The city is going to hire a third-party to investigate and recommend changes, which, judging from the city’s report, should probably be starting from scratch.
