Book Review: The VisibleOps Handbook

The VisibleOps Handbook: Implementing ITIL in 4 Practical and Auditable Steps

by Gene Kim, Kevin Behr, and George Spafford
Publisher: Information Technology Process Institute
ISBN: 0975568612
Number of Pages: 100
Date Published: June 15, 2005


VisibleOps is one of my favorite computer geek books of all time.  This book is a no-nonsense, straight forward guide to running a highly successful IT department.  But, VisibleOps is not just some flavor of the week self-help management book.  The lessons and goals presented in VisibleOps are the culmination of years of observation and research by the authors, who happened to notice that successful organizations had IT departments that operated in very similar ways.  This book is a distillation of those observations into a methodology that is easy for anyone in IT to grok.  Loosely based on the ITIL framework, VisibleOps cuts straight to the chase with four basic steps. 
 
The Four Steps of Visbile Ops

Phase 1. Stabilize the PatientPhase 2. Catch & Release and Find Fragile ArtifactsPhase 3. Establish Repeatable Build LibraryPhase 4. Enable Continuous Improvement

Stabilize the Patient

In the first phase of VisibleOps, the goal is triage.  Can you reduce the number and impact of outages?  Some of the key ways to accomplish this goal is to implement and strengthen Change Management processes, only allow scheduled changes, and have a defined maintenance window.

Another huge benefit to the Change Management process that often gets overlooked is its ability to act as a communication tool and a way to publish a schedule of changes.  With these processes in place, you will have better visibility for outage responders:

1. What changed? 
2. How to back out that change

Fragile Artifacts

The second phase is all about using a risk based approach to identifying and cataloging critical systems.  Some of the key indicators include:

• Systems with the highest Mean Time To Recovery (MTTR)
• Systems with low change success rates
• Systems with the highest downtime costs

But being able to understand and identify the cost of downtime requires understanding the business processes that each system supports.  That is why this phase is based on the Configuration Management process and includes implementing a Configuration Management Database (CMDB).  Once these processes are in place, you should see a reduction in variance, increased conformity in your systems, and it will be easier to detect anomalies within the environment.

Repeatable Build Library

In order to overcome the limitations imposed by the Fragile Artifacts, you must create a way to commoditize these systems.  Phase three is all about implementing proper Build and Release Management processes to further reduce variance and increase your understanding of what your systems are actually doing.  The thing that makes systems fragile in the first place is your lack of understanding about how that system operates.  

Once you are able to obtain that level of understanding, it is much easier to swap out interchangeable components than it is to ad-hoc a resolution out of random troubleshooting steps that you can't really explain WHY those steps "fixed" the issue.

Continuous Improvement

You would think that phase four would be self explanatory.  It is anything but that.  In terms of implementation, I have found that this can be the absolute most difficult because it requires a major shift in the culture of most organizations.  The VisibleOps Handbook provides some key indicators and metrics that can help track your progress on this journey.  It does not, however, provide much advice on how to steer your Titanic to avoid icebergs along the way.

Reflection

The thing I love the most about the Visible Ops approach to ITIL and managing IT in general, is how corporeal it is.  The word "visible" in the title obviously wasn't an accident; it is visible because the steps for implementation, the explanation of the methodology, really everything about it is so clearly evident that [almost] anybody should be able to thumb through this booklet and pick up some ideas that they can put to use right away and see results almost as fast.

Security Interconnected

In a recent blog post, Is Your Security Harming Someone Else’s Business?, Tripwire CTO, Dwayne Melancon, talks about mapping out the relationships your business has with other outside entities to "connect security to the businesses we depend on and those who depend on us."

This is a novel concept considering that many organizations have such a hard time mapping out their own internal processes, let alone ones that stretch outside their environment.  One of the main points that Dr. Eric Cole discussed this year during his SANS 2013 keynote in Orlando was that when he has been called into organizations to do an investigation or analysis, the first thing he asks for is a network diagram and a list of locations of critical data.  He then conducts a discovery of critical data on the client's network and maps out the true location of critical data to find that it rarely matches the client's list.

One of the questions that Melancon asks in his blog is, do you know what impact changes in your organization might have on contractual commitments with outside parties?  Many times the people engaged in writing, reviewing and signing contracts within an organization do not have the level of technical understanding to know what good security practices are, let alone whether they are properly included in the contract.  In the financial sector, there are regulatory requirements for this sort of thing (see Interagency Guidelines Establishing Information Security Standards)

Under the Security Guidelines, each financial institution must:

• Develop and maintain an effective information security program tailored to the complexity of its operations, and 
• Require, by contract, service providers that have access to its customer information to take appropriate steps to protect the security and confidentiality of this information.

You Get What You Pay For (and Prepare For) 

If the people writing and signing these contracts do not understand InfoSec, then this whole process seems a bit like going off a cliff and not knowing what's up ahead.   

I was reviewing some BCP/DR documents for a small financial institution not long ago and found a contract with a technology service provider (TSP) that was storing off-site backups for them.  The TSP provided the proof of breach insurance that was requested and it showed that the coverage limit was $1 MM per incident.  That sure doesn't sound like much given the amount of money lost by small and medium sized banks recently due to phishing and account take over attacks which led to ACH/Wire Fraud.  But then after taking a closer look at the contract, I found this statement in the section titled "Limitation of Liability":

"If VENDOR becomes liable to the CUSTOMER under this Agreement for any other reason, whether arising by negligence, willful misconduct or otherwise, (a) the damages recoverable against VENDOR for all events, acts, delays, or omissions will not exceed in aggregate the compensation payable to VENDOR […] for the lesser of the months that have elapsed since the Operational Date […]"


Say what? [Unnecessary Risk]

Somebody obviously didn't take the time to read this garbage before whipping out their John Hancock.  Let's just say the amount paid to this vendor each year is much less than the value of the data the customer was backing up with this vendor.  

Managing the security of interconnected systems is not just an IT issue.  It is a business issue which means taking the time to read and understand the contracts that your business is agreeing to abide by.  Does your company have a process for reviewing contracts?  Is your IT/InfoSec team involved in that process?  Are contractual requirements communicated to your IT/InfoSec teams?  If you are missing these steps, then it will be impossible to do any sort of impact analysis across interconnected outside entities... just say'n.

Windows Firewall with Advanced Security

I don't know very many people that take the Windows Firewall seriously.  That's too bad because the Windows Firewall with Advanced Security in Windows Vista, 7, Server 2008, and newer operating systems has some features that can help you baseline what "normal" traffic is on your systems.  

Below is a sample script I use when building out a 2008R2 server. (Yeah, I know Netsh is supposed to be going away, so maybe I'll re-write this in PowerShell for a future post.)

Configure Logging 

First of all I want to turn on logging for everything inbound and outbound.  By default, the settings for logging are disabled (only verified this for Windows 7 and Server 2008R2).  I want to set the log size to the max 32 MB and log all dropped and allowed connections.  

Sample Script:

echo "Turn on Logging for all connections..." netsh advfirewall set Domainprofile logging filename %systemroot%\system32\LogFiles\Firewall\pfirewall.log netsh advfirewall set Domainprofile logging maxfilesize 32767 netsh advfirewall set Domainprofile logging droppedconnections enable netsh advfirewall set Domainprofile logging allowedconnections enable netsh advfirewall set Privateprofile logging filename %systemroot%\system32\LogFiles\Firewall\pfirewall.log netsh advfirewall set Privateprofile logging maxfilesize 32767 netsh advfirewall set Privateprofile logging droppedconnections enable netsh advfirewall set Privateprofile logging allowedconnections enable netsh advfirewall set Publicprofile logging filename %systemroot%\system32\LogFiles\Firewall\pfirewall.log netsh advfirewall set Publicprofile logging maxfilesize 32767 netsh advfirewall set Publicprofile logging droppedconnections enable netsh advfirewall set Publicprofile logging allowedconnections enable

This is important because the default firewall settings only drop inbound connections that do not explicitly match a defined rule.  Outbound connections are allowed anywhere, anytime by default.  (I'll come back to this topic in another post later.)

Disable Unused Rules

Also, best practice is to disable anything you aren't using, and since I'm not running IPv6 on my network, I want to disable all of the IPv6 related rules.  It is always good to double check that IPv6 is actually unchecked on your NIC as well.

Sample Script:

echo "Turn off IPv6 Inbound Rules..." netsh advfirewall firewall set rule name="Core Networking - Destination Unreachable (ICMPv6-In)" dir=in new enable=no netsh advfirewall firewall set rule name="Core Networking - Dynamic Host Configuration Protocol for IPv6(DHCPV6-In)" dir=in new enable=no netsh advfirewall firewall set rule name="Core Networking - IPv6 (IPv6-In)" dir=in new enable=no netsh advfirewall firewall set rule name="Core Networking - Multicast Listener Done (ICMPv6-In)" dir=in new enable=no netsh advfirewall firewall set rule name="Core Networking - Multicast Listener Query (ICMPv6-In)" dir=in new enable=no netsh advfirewall firewall set rule name="Core Networking - Multicast Listener Report (ICMPv6-In)" dir=in new enable=no netsh advfirewall firewall set rule name="Core Networking - Multicast Listener Report v2 (ICMPv6-In)" dir=in new enable=no netsh advfirewall firewall set rule name="Core Networking - Neighbor Discovery Advertisement (ICMPv6-In)" dir=in new enable=no netsh advfirewall firewall set rule name="Core Networking - Neighbor Discovery Solicitation (ICMPv6-In)" dir=in new enable=no netsh advfirewall firewall set rule name="Core Networking - Packet Too Big (ICMPv6-In)" dir=in new enable=no netsh advfirewall firewall set rule name="Core Networking - Parameter Problem (ICMPv6-In)" dir=in new enable=no netsh advfirewall firewall set rule name="Core Networking - Router Advertisement (ICMPv6-In)" dir=in new enable=no netsh advfirewall firewall set rule name="Core Networking - Router Solicitation (ICMPv6-In)" dir=in new enable=no netsh advfirewall firewall set rule name="Core Networking - Teredo (UDP-In)" dir=in new enable=no netsh advfirewall firewall set rule name="Core Networking - Time Exceeded (ICMPv6-In)" dir=in new enable=no echo "Turn off IPv6 Outbound Rules..." netsh advfirewall firewall set rule name="Core Networking - Dynamic Host Configuration Protocol for IPv6(DHCPV6-Out)" dir=out new enable=no netsh advfirewall firewall set rule name="Core Networking - IPv6 (IPv6-Out)" dir=out new enable=no netsh advfirewall firewall set rule name="Core Networking - Multicast Listener Done (ICMPv6-Out)" dir=out new enable=no netsh advfirewall firewall set rule name="Core Networking - Multicast Listener Query (ICMPv6-Out)" dir=out new enable=no netsh advfirewall firewall set rule name="Core Networking - Multicast Listener Report (ICMPv6-Out)" dir=out new enable=no netsh advfirewall firewall set rule name="Core Networking - Multicast Listener Report v2 (ICMPv6-Out)" dir=out new enable=no netsh advfirewall firewall set rule name="Core Networking - Neighbor Discovery Advertisement (ICMPv6-Out)" dir=out new enable=no netsh advfirewall firewall set rule name="Core Networking - Neighbor Discovery Solicitation (ICMPv6-Out)" dir=out new enable=no netsh advfirewall firewall set rule name="Core Networking - Packet Too Big (ICMPv6-Out)" dir=out new enable=no netsh advfirewall firewall set rule name="Core Networking - Parameter Problem (ICMPv6-Out)" dir=out new enable=no netsh advfirewall firewall set rule name="Core Networking - Router Advertisement (ICMPv6-Out)" dir=out new enable=no netsh advfirewall firewall set rule name="Core Networking - Router Solicitation (ICMPv6-Out)" dir=out new enable=no netsh advfirewall firewall set rule name="Core Networking - Teredo (UDP-Out)" dir=out new enable=no netsh advfirewall firewall set rule name="Core Networking - Time Exceeded (ICMPv6-Out)" dir=out new enable=no


ICMP Echo Request

Next I want to enable the default echo request rule so I can ping this new server. 

Sample Script:

echo "Turn on Inbound Echo Request so I can ping this box..." netsh advfirewall firewall set rule name="File and Printer Sharing (Echo Request - ICMPv4-In)" dir=in new enable=yes

There may be some other rules that you should disable depending on your environment, (such as DCHP or IGMP), but this should provide a decent example to get started with Netsh and the Windows Firewall with Advanced Security.

Surprising Results Improving Weak Passwords

Last year I saw a presentation by John Strand of Black Hills Information Security titled "Everything They Told Me About Security Was Wrong" during which he talked about how ridiculously easy it is to crack most people's passwords.  John gave several examples about how the typical alpha-numeric-special-character complexity requirements mandated by most security frameworks or regulations are actually making it easier to break into and harder on users to remember their passwords. 

Then along came correct horse battery staple.  Just for reference, here are a few examples from the Brute Force Calculator (these examples are based on a system running an Intel i7-2600K CPU @ 3.40GHz).

Password Length	Password Type	Character Set	Total Key Space (password combinations)	Time Required to Brute Force
7	NT MD4	Mixed Alpha Numeric All	6.5545E+13	15 days
8	NTLMv2	Mixed Alpha Space	6.3456E+13	15 days
8	NT MD4	Mixed Alpha Numeric All	6.16123E+15	4 years
15	NT MD4	Mixed Alpha Space	7.45436E+25	50 billion years
15	NTLMv2	Mixed Alpha Space	7.45436E+25	2 trillion years
18	NTLMv2	Mixed Alpha Space	1.10978E+31	357 quadrillion years
19	NTLMv2	Mixed Alpha Space	5.88185E+32	19 quintillion years
20	NTLMv2	Mixed Alpha Space	3.11738E+34	1 sextillion years

Implementing the Changes

So, I decided to see if I could get my organization to go along with the idea of much longer but less complex passwords.  There were two parts to implementation, the first was the social/political side of things, getting the right approvals internally to change the policy, explaining to users why the change was good for them and the company, and convincing the auditors that this change actually improves security.

The second part was configuring Windows Group Policy to enforce the new policy.  This turned out to be not as straight forward as I expected.  Using the Windows GPO Editor to set passwords minimum length stops at 14.

Some Google-fu netted me this result that provides a couple of examples for fixing Microsoft's shortsightedness for improving password security.

Joeware's ADMod tool makes it easy to modify the Default Domain Policy, here's the command to set the minimum length to 15 characters

    admod -default minpwdlength::15

Then to see if it worked...

Dealing With Auditors

You know you are going to have a fun time when an IT auditor asks a question like, "Can you explain how to read this network diagram?" Another point John made during his prezi, was that no organization is ever 100% compliant with every applicable regulation or security requirement.  No one.  So what happens when your organization doesn't meet the specific letter of the law for a particular requirement?  You implement a compensating control.  After walking through a few examples of password cracking tools, such as John the Ripper and Cain/Abel, it wasn't as tough a sell as I was expecting.

Results

The auditors signed off for the changes to the password policy without too much trouble. The auditors weren't the only concern I had about changing the policy.  My guess was that these changes were going to create a problem for users as well as IT having to do account unlocks all the time.  After having this policy in place for several months now, one of the biggest surprises I found from going through this exercise is that account lockouts are currently at an all time low across the entire company.  That is what you call a Win-Win!