One thing I saw mentioned in The Phoenix Project was the phrase "technical debt", during chapter 15, page 164. In one of the phone conversations between Bill and Erik they are discussing how to correctly implement step three of the Theory of Constraints (ToC), how to subordinate all processes to the capacity of the constraint.
"So, here's your homework," [Erik] says. "Figure out how to set the tempo of work according to Brent. Once you make the appropriate mapping of IT Operations work to work on the plant floor, it will be obvious. Call me when you've figured it out."
"Wait, wait," I say hurriedly before he hangs up. "I'll do the homework, but aren't we missing the entire point here? What caused all the unplanned work is Phoenix. Why are we focusing on Brent right now? Don't we need to address all the issues with Phoenix inside of Development, where all the unplanned work actually came from?"
"Now you sound just like Jimmy [John the CISO], complaining about things you can't actually control," he sighs. "Of course Phoenix is causing all the problems. You get what you design for. Chester, your peer in Development, is spending all his cycles on features, instead of stability, security, scalability, manageability, operability, continuity, and all those other beautiful 'ities.
"On the other end of the assembly line, Jimmy keeps trying to retrofit production controls after the toothpaste is out of the tube," he says, scoffing. "Hopeless! Futile! It'll never work! You need to design these things, what some call 'nonfunctional requirements,' into the product. But your problem is that the person who knows the most about where your technical debt is and how to actually build code that is designed for Operations is too busy. You know who that person is, don't you?"
I groan. "Brent."
Time Slicing
I ran into this exact scenario a few years ago. I had a superstar senior engineer on my team that had technical knowledge, intuition and dedication beyond anyone I've ever worked with before or since. The problem was that he was burnt out on fire-fighting, getting emails and phone calls at 3 am to fix some emergency issue that someone higher up the food chain crammed into production at the last minute as an emergency change. The CIO I worked for at the time wanted to micro manage my team by telling me that everyone on my team had to be in the on-call rotation. I replied that it is ridiculous to waste the best engineer on the team's time answering emails and doing an emergency code push at 4 am when he should be spending his time designing and building a better deployment process, and to get a few entry level Operations people to answer the phone and reply to email.
I explained it to my boss this way (in part thanks to the lessons I learned about ToC from "The Goal" and "Critical Chain"), the more different tasks you add to one person or one team, the more time you waste trying to stop your mind from thinking about the last task you were just in the middle of and the longer it takes you to focus on the new emergency sitting in front of you. I called this concept "Time Slicing". When your team is spending more cycles time slicing than they are getting any real work done or completing a task, the less efficient the workflow will be.
The Snowball
I tried to draw a comparison to Dave Ramsey's get out of debt philosophy. The more different debts you have, the less progress you can make towards any of them if you are trying to focus on them all with equal attention (and every project we were trying to juggle was priority #1). Dave Ramsey's approach is to pay off the smallest debt first, regardless of the interest rate. This may seem counter intuitive to some people who look at the total amount of interest being paid out and say this doesn't make sense. You are wasting money on interest payments. But, by focusing on the smallest debt, the timeline for completing that task decreases, thereby freeing up additional resources to focus on the next smallest debt. This approach builds momentum (like a snowball rolling downhill), and most of all improves morale by generating a sense of accomplishment. Eventually, things began to improve. We created a brand new Tech Ops team to handle the firefighting and reserved the Engineer team for larger project work. As momentum increases, so does throughput and project completion rates improve. The key is not to have so many competing tasks/projects "time slicing" your team to death.
In The Phoenix Project, Bill tackles this problem by first getting Brent out of the first responder role and sets up a team of engineers to handle escalations and only that team gets to interrupt what Brent is working on if they need help with an issue. Then Brent can only explain what to do and how to do it. Brent isn't allowed to fix the issue himself. This starts to enhance the culture of cooperation and cross training so that Brent doesn't continue to be a silo of knowledge.
One thing I've noticed is that technical people with that level of skill are usually very open about sharing their knowledge, but they are never given time to do so. The business side of the house is usually more interested in faster resolution times than they are about redundancy of operational knowledge. This is disappointing, because with enough people intimately familiar with an issue, the better the opportunity for someone to find a creative solution for the long term.
Another great point in the book about increasing throughput is explained in chapter 19 during the "leadership off-site". Starting on page 198 through 202, Bill, Erik and Steve (CEO) discuss implementing a project freeze in order to reduce the number of tasks that the critical resources need to focus on (effectively erasing all of the small debts and leaving only one huge debt for the time being). Steve is extremely reluctant to allow this, seeing it as a huge waste of company resources akin to "subsidized potato farmers paid not to grow crops" (or in my example, wasting money on interest payments). Erik, in typical Jonah cadence draws on Steve's experience as a plant manager to relate due-date performance, WIP, inventory levels all back to taking on new orders and releasing work onto the plant floor. The result is obvious.
As common sense as this approach should be, many business executives, senior management and project managers just simply do not grasp this concept. If all of your resources are allocated to projects, systems keep breaking down and require "all hands on deck" to get things running again, project due dates are not being met, and quality of project deliverables are going down... in what world view does it make sense to add another project to the queue? Shouldn't it be time to pay down some technical debt first?
Admittedly, I'm a little behind the times here. In my defense, I got "promoted" into some other management roles for a few years where I wasn't much of a hands-on sysadmin other than just playing around with some gadgets in my home lab. I've recently been trying to update my scripting skill-set from VBScript to PowerShell and so far I'm having a blast. I'm not a developer, nor do I play one on TV, but PowerShell seems to be able to do about anything.
I typically don't like editors for writing scripts. Notepad works just fine for me (although I do like Notepad++). Then a few weeks ago I found PowerGUI. Wow, this opens up a whole new world of possibilities for me. As I mentioned in an earlier post, I'm not much of a Linux guy... and as I've heard others state it, "I'm a recovering Windows admin", and PowerGUI offers a user interface that is extremely easy for me to work with.
PowerGUI allows users a way to build and manage a repository of scripts in an interface that is both easy on the eyes and extremely functional. Scripts can be grouped together into a powerpack that can be exported and shared with others. And there is a nice community of PowerShell gurus that have published some cool powerpacks ready to lock and load.
I've been dissecting the default Network powerpack by Kirk Munro and putting together a list of features I would like to add to it or just customize my own powerpack. Hopefully more to come in the next few weeks.
So a few years ago, I'm sitting at my desk and the mail cart rolls by and a box gets dropped off at my desk. Strange. I hadn't ordered anything and wasn't expecting anything in the mail. It's a about an 8"x8"x8" cardboard box and I didn't recognized the address on the label, from somewhere in California. Ok, let's tear this puppy open and see what it is. And I pull out a black tee-shirt and on the front in white silk screen is the XKCD Sandwich comic.
Ok, that's cool. But where the heck did this come from? A few hours later that same day, I'm on the phone with one of my favorite customer support reps at Rapid7, and he says, "So how do you like shirt? You're a linux guy, right?" Now that's funny. I'm really not 'a linux guy'. I can eventually find my way around when I have to, but certainly not an expert. I still wear the shirt around the house, and I think my lovely wife hates it. She is not a techie, and she doesn't get the joke. Anyway, where's my sandwich?
The sad part of this picture is that most InfoSec people carry this same stigma portrayed in the comic about being some overbearing, demanding, I've-got-an-InfoSec-pwn-your-system trump card up my sleeve sort of jerk. I get it, I can be the same way. It is easy to rant and rave about how badly everyone is doing their job and how well I'm doing my job of ranting and raving.
When it comes to InfoSec, there is no silver bullet. Wouldn't it be nice (in terms of security for the general population) to simply say:
user@terminal$ sudo implement-security --now
[sudo] password for user: ********
And then be done with it? Sure all the folks in InfoSec would have to find another line of work, but think about all that extra time you would have to fill since you aren't responding to incidents (and nothing to rant and rave about in the echo chamber).
So the real key to success, and part of the reasons for picking the SudoSec handle is this...
1. Principle of Least Privilege
The reason for having the sudo command in the first place is because you aren't root. Sure you can (ab)use the sudo command and do anything you want and act as if you were root, but you aren't. Before ranting and raving, perhaps it is worth checking whether you personally have the correct access/permissions/rights to rant and rave? The sudo command helps restrict access down to something like least privilege. The 'Principle of Least Privilege' should apply to our roles in life and not just our interactions with the keyboard and screens on our desks. Then when an important issue/need arises, think of the sudo command as having the ability to take initiative and to address the problem, but also asking permission to respond with the commands needed to actually implement a solution.
user@terminal$ sudo cat i.told.you.so | cut -d arrogance > helpful.solution
2. Defense in Depth
I know so many people in Information Technology jobs that are completely stuck (or in their eyes, content) in their one little corner of the universe. Be it a typical systems administrator or applications developer or network engineer or help desk specialist; these people are usually very good at what they do. Some of them I consider to be experts because of the depth of knowledge they have on the inner workings of their particular [insert system/application]. But many of them are clueless about how their system/application/role interacts with other systems/users/employees/customers. Isolated silos of knowledge, no matter how deep that pool of knowledge is, does not make for depth of knowledge for the organization. In the same way that proper defense in depth for security requires layered controls, having an understanding of the dependencies among systems, the data flow of those systems, and the process flow of who interacts with that data are the layers of understanding that truly differentiates the average admin type from the superstars that I want on my team when a real incident arises.
user@terminal$ sudo chmod u+superstar *
by Gene Kim, Kevin Behr, and George Spafford
Publisher: IT Revolution Press; 1st edition
ISBN: 0988262592
Number of Pages: 345
Date Published: 1/10/2013
Let me first say that it is incredible how many different lessons are packed into Gene Kim's latest book, “The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win.” It was a really quick read because the story line was so easy to follow. (Hey, where are the hidden cameras in my office anyway? I actually know all the characters in this story.) But I've spent a few extra days going back over the parts of the book where the real instruction is interleaved into the plot.
The Phoenix Project combines the teaching of the several "management / improvement" genre books such as:
- Eliyahu M. Goldratt's "The Goal" which teaches the Theory of Constraints (ToC),
- Patrick Lencioni's "Five Dysfunctions of a Team",
- David J. Anderson's use of kanban boards to control the release of work and Work in Progress (WIP) for Development and IT Operations,
- Mike Rother's “Toyota Kata: Managing People for Improvement, Adaptiveness and Superior Results”,
- As well as TPM / LEAN strategies and many others...
- And last by not least, Kevin Behr, George Spafford and Gene Kim's previous research in Visible Ops Handbooks.
The narrative tells the story of Bill who reluctantly gets promoted to VP of IT at Parts Unlimited, an automotive parts manufacturing and retail company, after his previous boss and boss's boss get canned due to poor performance. Bill inherits all of the typical problems that come with an overworked, dysfunctional IT team that is so busy fighting fires, that they don't have time to look at the long view of their situation. And as if that weren't enough, the whole company is on a death march to deploy their latest home grown system, "Phoenix" that was intended to save the company's poor performance and lack of profitability for the past several quarters.
Sprinkled throughout the novel, the themes from the Visible Ops Handbook are intertwined as Bill takes measures to implement a new Change Management process, identify critical systems and get his team out of the fire fighting business and into something that resembles productivity.
The Four Steps From Visible Ops:
- Stabilize the Patient
- Catch & Release and Find Fragile Artifacts
- Establish Repeatable Build Library
- Enable Continuous Improvement
Bill has some good ideas and starts to make some progress for his team, but it is obvious that there is more work piled on Bill's team than they have cycles to work on. Bill quickly realizes that one of his key resources, Brent, is the bottleneck that is holding up a lot of work from getting done as well as a silo of critical knowledge for managing and maintaining his systems. Using the techniques from Goldratt's ToC, Bill instinctively starts to put some processes in place to manage work getting assigned to Brent.
Things really start to get interesting for Bill when he meets a potential new board member named Erik Reid, who turns out to have worked with Part
Unlimited years ago to help solve a crisis at their manufacturing plant by implementing what sound remarkably like the same solutions recommended by Jonah in "The Goal". Erik befriends Bill and takes on the role of mentor with the same Socratic approach to helping Bill find a way to fix all the troubles he is having. Erik first introduces Bill to "The Three Ways" in Chapter 7, page 91.
- "The First Way helps us understand how to create fast flow of work as it moves from Development into IT Operations..."
- "The Second Way shows us how to shorten and amplify feedback loops..."
- "The Third Way shows us how to create a culture that simultaneously fosters experimentation, learning from failure, and understanding that repetition and practice are the prerequisites to mastery."
Erik's first assignment for Bill is to identify the four types of work that he manages. By the time Bill gets around to taking Erik seriously (and for a raving mad man), it is already chapter 15, page 160, and the wheels are really starting to fall off at Parts Unlimited. There have been outages to the POS systems and a "small credit card breach". Bill gets the first assignment right recognizing that the four categories of work are:
- Business Projects
- Internal IT Projects
- Implementing Changes
- Unplanned Work
Bill and Erik discuss other progress that Bill's team has made with trying to simplify the input of change requests into the change management process, the use of kanban boards in the CAB meetings, and the first step in ToC identifying Brent as the constraint. Bill's next assignments are to figure out how to take needless work out of the system as much as it is to control WIP already in the system, and further define how to control the flow of work to Brent.
The Five Steps From The Theory of Constraints:
- Identify the constraint (the resource or policy that prevents the organization from obtaining more of the goal)
- Determine how to exploit the constraint (get the most capacity out of the constrained process)
- Subordinate all other processes to above decision (align the whole system or organization to support the decision made above)
- Elevate the constraint (make other major changes needed to break the constraint)
- If, as a result of these steps, the constraint has moved, return to Step 1. Don't let inertia become the constraint.
Despite all of Bill's progress, he is quickly falling out of favor with his new boss, CEO Steve. After another major outage and some harsh words, the two finally make up and get the team back together to figure out how to actually fix the problems that the company has created for itself. Erik helps Bill convince Steve to institute a project freeze to free up resources to work on the critical Phoenix project, which they are banking is the only hope to save the company from going under.
By Chapter 20, page 208, Erik and Bill are discussing WIP again and defining what a work center is... "every work center is made up of four things: the machine, the man, the method and the measures." Having personnel assigned to too many work centers is so ineffecient and is the crux of Brent being the constraint to so many projects. Erik introduces an interesting concept on page 213 showing that the wait time for any piece of work can be calculated by dividing the percentage that a resource is busy by the percentage that resource is idle. "When a resource is ninety-nine percent utilized, you have to wait ninety-nine times as long as if that resource is fifty percent utilized."
Another interesting character in the story is John, the CISO. John is a fanatic about security, but in typical Infosec fashion runs around barking out orders that he claims are mandated by some policy or regulation or auditor. In Chapter 22, Erik finally puts John in his place by showing that none of the controls that he has attempted to put in place have any impact on the business' ability to manage risk. After that John disappears for several weeks. And finally re-emerges as a changed man ready to finally look IT through the lens of the business' priorities.
Once Bill and John realize what the business' metrics for success are, they come to a hard realization that the Phoenix project will never fulfill the needs of the business in the project's current form. Bill puts together a proposal to build a SWAT team that will work in parallel with the Phoenix project but with permission to break all of the rules in order to help the business make its numbers. The new processes that emerge from this SWAT team are the fundamentals of #DevOps.
Like I mentioned earlier, the book is phenomenal, extremely well written and easy to follow. [Spoiler Alert] Of course everything works out by the end of this fictional tale, but most of the lessons really do work in real life... IF
Now here's where the story becomes somewhat of a fantasy apart from real life. In several of the situations I have been in or seen unfold in the past, senior management hasn't been willing to really look in the mirror and recognize that they don't really understand technology. And if they are having trouble understanding it then they are really going to have trouble managing it. Some of this stuff should be basic project management 101. It always amazes me how many project managers and senior management figures at some companies measure success by completing a project on time, and slightly over budget. But they don't analyze the success of the project in terms of actual return on effort to the business (maybe because they don't want to admit their pet project didn't achieve the objectives they claimed it would).
Not everyone in the IT world gets a personal Jonah or Erik to guide them through the murky waters of IT management and give credibility to the ideas and initiatives that you want to champion. If you find yourself with a team to manage, hopefully this book along with some of the other resources mentioned can strengthen your decision making abilities to help your team and your company to succeed in this crazy new world of #DevOps.
One other thing to note is that the artwork on the cover of the book is pretty cool!
"Well, this certainly wasn't on my list of things to do today." I thought as I opened the door and stepped out of my car. Then I saw the blood dripping down from my forehead. My next thought was to find a napkin or paper towel or something to try to stop or at least cover up the bleeding...
Psalm ch118.v1,4-8
O give thanks unto the
Lord; for he is good: because his mercy endureth for ever... Let them now
that fear the Lord say, that his mercy endureth for ever. I called
upon the Lord in distress: the Lord answered me, and set me in a large
place. The Lord is on my side; I will not fear what can man do unto
me. The Lord taketh my part with them that help me... It is better to
trust in the Lord than to put confidence in man.
I was headed south on my way to one of my employer's ATM locations just a few miles from my office to fix a broken fascia and reinstall the monitor. (Working on these ATMs is probably my least favorite part of my job.) On the way there, I came up to an intersection where the east/west lanes have a stop sign, but the north/south lanes do not stop. I've been though this intersection literally hundreds of times before and I've always heard that it is a dangerous spot. As I topped the little hill just to the north of the intersection, I looked down at the speedometer which said I was moving at about 60 MPH. I saw a car at the stop sign on the west side of the intersection pull out and make left hand turn (northbound). Then I saw a white pickup drive up to the stop sign, stop, and then pull out right in front of me. I had no time to step on the breaks. I didn't see any on-coming traffic in the northbound lane, so I swerved left as much as I could to lessen the impact. I can remember most of the sequence of event very clearly. I can still see the front driver's side fender of the pick up rolling straight towards me. I can remember pulling the steering wheel toward the left and bracing for the impact. I can visualize my head hitting the rearview mirror and falling back into my seat, then the airbags deploying, then I couldn't see the road anymore. I knew the car was still moving, so I tried turning my body towards my driver side door and curling up in the seat to avoid any more potential impact(s). Then the car hit something and my right shoulder slammed into the steering wheel. That's when I got out of the car.
The car that was behind me pulled over and dialed 911. Several other vehicles stopped to check things out. The driver of the pickup came over to check on me. He was fine; I thought I was going to be just fine once I got cleaned up. I called back to my office to let them know what happened. And I remembered that a close friend of mine and fellow church member had just walked in the door to do some business as I was leaving. I asked if he would come pick me up since my car was obviously in no shape to be on the road. I called my wife to let her know what happened and left her a brief but calm voicemail (she never answers her phone).
The first (local) officer on the scene asked me to sit down. I was perfectly content to stand. Next an off duty (another local) officer that had stopped came over to question me. I answered his questions and politely declined the medical attention for religious reasons. Finally the highway patrol officer arrived and was the one that ended up working the accident. He had a few questions for me as well and I reiterated the fact that I was going to decline the medical attention. I'm sure that I looked a sight with streaks of dried blood all over my face/head. He said I was going to need stitches. But I must have been convincing enough. He radioed back to the ambulance not to bother showing up. That was a surprise and a relief.
When the highway patrol officer came over to give me a print out of his initial accident report, he asked, "So you're not even going to take an asprin or anything?"
"No, sir", I replied.
"Man, I feel sorry for you in the morning." He responded.
By the time my friend arrived (in the time it took him to drive less than ten miles), there were already three other church members that just happened to be passing by at that time who had already stopped to check on me. My friend took me back to his house and helped get me cleaned up (my wife isn't the best at dealing with a bloody mess, and I wasn't sure if I could really get myself cleaned up).
The car was deemed a total loss.
Warning... Bloody Pictures Below...
I got a new haircut out of the deal :)
A few of the absolutely amazing things about my accident and injuries:
- The accident happened on a Wednesday afternoon. I worked from home on Thursday just in case I started bleeding all over the place. And when I went back to the office on Friday, I didn't even need a band-aid; it was actually healed that much!
- It took longer for my black eye to subside than it did for the gashes on my head to heal over.
- No concussion; I never even had a headache! 60 mph collision and NO HEADACHE!
- The picture below is from Thursday 12/27, about 30 hours after the accident (I didn't need stitches after all).
Psalm ch118.v17
I shall not die, but live, and declare the works of the Lord.
I didn't die. Maybe I should have, but since I didn't, I will declare the works of the Lord. God's mercy preserved my life and granted a speedy recovery.
Looking back on it after the fact, if I
would have slammed on the breaks, I quite possibly would have collided
directly into the driver side door of the pickup and the outcome could
have been much different for both myself and the other driver.
Send me a message on twitter if you want: @SudoSec
