Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.


05:00 PM
Connect Directly

Lessons Learned In San Francisco Hack Job

An IT administrator with the city of San Francisco is alleged to have created a private administrative account on systems within the city's FiberWAN project.

At one time or another, it happens in almost every IT shop: A handful of people, or even one person, has the sole responsibility for and knowledge of critical systems. So much so that if any one of them were hit by a bus, your IT department would be severely hampered. Virtually every IT manager recognizes this problem, but far fewer do anything about it. San Francisco is now learning its lesson the hard way.

Terry Childs, an IT administrator with the city of San Francisco, whom the city had tried to fire in recent months, is alleged to have created a private administrative account on systems within the city's FiberWAN project. According to prosecutors, he's keeping the password a secret. In so doing, he's locked everyone else out of the system's networking gear and landed himself in jail, with bail set at $5 million. The city is trying to regain access to its network and has brought in experts from Cisco to help break into the compromised systems. The details are murky but also not vital to learning from this painful incident. Simply put, a trusted employee is accused of using that trust to hose his employer.

Organizations must trust employees to act honestly and ethically while at the same time acknowledge that employees pose one of the greatest threats. Employees must have access to systems, and that means they can do damage. InformationWeek's 2008 Strategic Security study found that 53% of respondents considered authorized users/employees one of the greatest threats, and 48% of respondents considered all employees one of the greatest threats. While most employees are ethical, all it takes is one disgruntled person to cause you headache and misery.

No technology or process can fully stop an insider, especially a knowledgeable insider like Childs, from causing harm--but what can be managed, and what's shocking about this incident, is its apparent magnitude. Technology can help. Strong authentication systems could have made creating unique credentials more difficult, and change management products might have alerted someone to Childs' activities. But technology alone isn't the solution; good management plays a part, too.

Jonathan Feldman, director of IT for Asheville, N.C., says problems like these can crop up when the IT department relies on heroes versus using a team approach. He describes the IT hero as a lone wolf, a go-to person who is solely responsible for critical systems. IT heroes often don't like to give up control over their systems, Feldman points out. These employees pose a real risk to the organization because if they leave the job, are unable to work, or simply decide to strike back, no one else has the knowledge to work around them.

What's At Stake
Childs changed router and switch administrator passwords

The network serves 60% of the city's needs, including some police systems

Childs is suspended with pay while he sits in jail
The solution is to cross-train IT staff so that no one person alone understands and controls key systems; spread responsibility for systems over several people; and adopt a practical change management process. By doing those three things, the harm that any one person can inflict on your IT department is lessened because there are redundant skills and knowledge in play.

Building an IT team rather than relying on lone wizards requires management discipline and may seem a more costly approach; IT heroes are an easy shortcut for even the best-intentioned managers. However, the payoff in building a team is a more resilient IT department. "You know you are successful when you can let IT personnel go on vacation without having to ask for their cell phone and hotel numbers in case something comes up," says Feldman.

Rogue employees can still wreak havoc if they want. The best you can do is to take steps to minimize the damage.

Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
Former CISA Director Chris Krebs Discusses Risk Management & Threat Intel
Kelly Sheridan, Staff Editor, Dark Reading,  2/23/2021
Security + Fraud Protection: Your One-Two Punch Against Cyberattacks
Joshua Goldfarb, Director of Product Management at F5,  2/23/2021
Cybercrime Groups More Prolific, Focus on Healthcare in 2020
Robert Lemos, Contributing Writer,  2/22/2021
Register for Dark Reading Newsletters
White Papers
Cartoon Contest
Write a Caption, Win an Amazon Gift Card! Click Here
Latest Comment: This comment is waiting for review by our moderators.
Current Issue
2021 Top Enterprise IT Trends
We've identified the key trends that are poised to impact the IT landscape in 2021. Find out why they're important and how they will affect you today!
Flash Poll
Building the SOC of the Future
Building the SOC of the Future
Digital transformation, cloud-focused attacks, and a worldwide pandemic. The past year has changed the way business works and the way security teams operate. There is no going back.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
PUBLISHED: 2021-02-27
An issue was discovered in through SaltStack Salt before 3002.5. salt.modules.cmdmod can log credentials to the info or error log level.
PUBLISHED: 2021-02-27
In SaltStack Salt before 3002.5, eauth tokens can be used once after expiration. (They might be used to run command against the salt master or minions.)
PUBLISHED: 2021-02-27
An issue was discovered in SaltStack Salt before 3002.5. Sending crafted web requests to the Salt API can result in salt.utils.thin.gen_thin() command injection because of different handling of single versus double quotes. This is related to salt/utils/thin.py.
PUBLISHED: 2021-02-27
i-doit before 1.16.0 is affected by Stored Cross-Site Scripting (XSS) issues that could allow remote authenticated attackers to inject arbitrary web script or HTML via C__MONITORING__CONFIG__TITLE, SM2__C__MONITORING__CONFIG__TITLE, C__MONITORING__CONFIG__PATH, SM2__C__MONITORING__CONFIG__PATH, C__M...
PUBLISHED: 2021-02-27
An issue was discovered in SaltStack Salt before 3002.5. The salt-api's ssh client is vulnerable to a shell injection by including ProxyCommand in an argument, or via ssh_options provided in an API request.