Category Archives: Silos

Veteran of the Process Wars

Posted on by
Reply

Tokyo. I’m still in Tokyo. I wake up, rub the sleep from my eyes, and roll out of bed. As I rise I take a quick look at my watch for any new emails. Nothing. There’s not much email anymore. Not since the process wars of 2016.

Many people won’t talk about the process wars. They changed the way most of the InfoTech industry works and how we are allowed to think of our jobs. In the past, I might have been responsible for running hundreds of servers. Now, the machines are in charge. I’m only allowed to feed the machines code.

We are grouped together in small teams that write code as a unit. We are “kept in line” by dogs that are trained to attack us. They say it’s for our own good. That it’s for the betterment of industry. If we try to touch the servers, if we try to do anything other than write code and feed it to the machines, the dogs will bite us.

We are kept in line by process and controls, but not like we used to be kept in check. Before, we used to have weekly meetings reviewing the work we wanted to perform. The meetings were always a joke amongst my peers. Typically they were run by a VP that had no clue what we wanted to change on the machines. If we wanted something bad enough, we could social engineer our way to what we wanted.

No more today. When we write the code to feed the machines, we have to write tests. They tell us these tests are for our own good. The tests confirm that the machines are set up exactly how they want them to be. We actually write the tests first, and they always fail the first time through. Then we write code to bring the tests into compliance.

If the tests weren’t bad enough, we also have automated tests to make sure our code conforms to “Style Guidelines”. They tell us these guidelines are to ensure conformity and consistency. I say they are there to hold us down and control us. They also require us to “lint” our code before we feed it to the machines. This once again is to ensure “conformity”.

Even with all of this verifying for conformity, we still aren’t allowed to directly feed the code directly to the machines. We must check the code into a repository where more machines take over to verify that we have successful tests and we meet the conformity guidelines. The machines also verify that our code works with code written by others, in my group and other groups.

Then, a Conformity Checker reviews my changes to make sure they are compliant with the policy. We are no longer independent; we are no longer allowed to game the review board. We feed the machines compliant code or else we end up on the streets. Three strikes and we’re out.

Which is why I’m in Tokyo. I’ve had two strikes at my company in the United States. They shipped me over to the Tokyo division for reeducation. Japan culture is heavily based on order and process. During the process wars they helped lead the revolution. Many of the compliance tools were written by or enabled by Japanese technology leaders. Yukihiro Matsumoto (codenamed Matz) was responsible for designing the programming language I now feed to the machines. Gosuke Miyashita wrote the tools that I am required to use to test my code before feeding it to the machines. And then there was Kohsuke Kawaguchi, the creator of the master machine that ensures all the code is compliant, automatically with no humans to game the process.

It’s all very neat and orderly now. I take requests for code from the owners of the machines, I write compliant code, and the machines automatically verify my work. Eventually the machines apply my code, and the owners get exactly what they want. I’ll wake up the next however many hundreds of mornings and do just this. No more, no less. It’s all very neat and orderly now.

You Build Kingdoms Because Your Mother Didn’t Love You

Posted on by
1

Mother-Child_face_to_faceDestruction of silos is all the rage in DevOps and has been since the beginning of the movement. Patrick Debois wrote a very intelligent piece on why silos exist and how they came about as a management strategy. While the post explains why hierarchy style of management came about in the US (General Motors and Sloan), it doesn’t cover some of the personal motivations as to why silos or management kingdoms come about.

Parkinson’s Law

Over the last several years Bike Shedding – or more appropriately Parkinson’s Law of Triviality – has become very popular in technology.  But in all the trivial debate, it seems more technologists have missed C. Northcote Parkinson’s other law, aptly named Parkinson’s Law.

Parkinson’s Law simply states “…that work expands so as to fill the time available for its completion.” Any life long procrastinator will immediately know this is true, as does anyone that has attempted to do any level of project management. Further, Parkinson explains that this expansion of work, also creates an expansion of people doing the work. While Parkinson was focused on governmental organizations, Parkinson’s Law can also apply to other organizations.

Parkinson attributes this expansion of work to two factors:

Factor I.—An official wants to multiply subordinates, not rivals; and

Factor II.—Officials make work for each other.

The Law of Multiplication of Subordinates

Now as work expands, for whatever reason, Parkinson explains that the worker (worker A) must find ways to handle the workload. Parkinson explains that they must find a solution and have three options:

  1. Resign
  2. Split the work with colleague B
  3. Hire subordinates.

And as Parkinson points out, any rational actor is going to choose #3, and in doing so will at a minimum hire 2 subordinates, workers C and D. Hiring one subordinate would be effectively equal to #2, splitting the work with C instead of B, and thus increasing the pool of competition (A, B, and C would effectively be at the same level at this point). Thus the rational choice is to hire 2 or more subordinates, and in doing so A can leverage C and D against one another, holding a possible promotion out as a carrot in order to keep C and D in check.

Of course, work expands, eventually C and D become too busy, and thus they must make the same choice that A had to when they were hired. Rational actors as they are, they choose option #3, each hire 2 subordinates (at least), and please welcome E, F, G, and H to the company. Worker A now has a beautiful fucking kingdom; self-loathing because of an unloving mother notwithstanding.

The Law of Multiplication of Work

As Parkinson points out, seven people are now doing what one once did. Instead of simply expanding to fill time, work now begins to multiply. Why? Well the workers begin to create busy work for each other. The example Parkinson gives follows as such:

“An incoming document may well come before each of them in turn. Official E decides that it falls within the province of F, who places a draft reply before C, who amends it drastically before consulting D, who asks G to deal with it. But G goes on leave at this point, handing the file over to H, who drafts a minute, which is signed by D and returned to C, who revises his draft accordingly and lays the new version before A.”

Parkinson continues his example documenting the busy work that this kingdom produces, much of it useless, and leaves the example with A leaving the office for the day:

“Among the last to leave, A reflects, with bowed shoulders and a wry smile, that late hours, like grey hairs, are among the penalties of success.”

Success indeed my King, Success indeed.

Sound Familiar?

If at this point, you haven’t seen the slightest reflection of an org you know, work at, or have worked for then please let us all know the magical organizational utopia that employs (or has employed) you. Snark and rage aside, this really highlights the problems of many organizations; big kingdoms built to produce very little of value other than process and busy work. And that is why the DevOps Silo Rage gets so much airtime. Process and busy work are there to further the growth of the kingdom, not feed the soul of the individuals at the bottom.

This also highlights why DevOps focuses so heavily on borrowing from things like Lean Manufacturing. Lean emphasizes getting rid of unnecessary process and waste, in order to focus on value creation activities. It also empowers the individuals – E, F, G, and H in the example above – to shape how the value creation process should actually work and what processes are wasteful.

Now reflect on A. What if (s)he came in one day and E, F, G, and H wanted to revamp all the “meaningful” process that keeps them in check. What do most kings (or queens) do when a revolt happens? Kingdom in jeopardy, they squash the rebellion and execute the leaders of the rebellion. Sometimes the monarchy throws carrots to the rebels to appease them just enough to keep the rebellion down.

And that is my rub with the Marketing Driven DevOps drivel being produced today. It’s a fucking carrot to appease the rebels in order to keep the status quo, kingdoms intact, and incumbents in bed with the monarchy. It’s an illusion to pretend you’re doing something new, and at the end of the day thinking, “All this hard work is just my price for my success.”

Parkinson’s Law – The Economist – November 19th, 1955 – http://www.economist.com/node/14116121

What if Everything We’ve Been Doing is Wrong?

Posted on by
2

60-wrong-way

After I wrote my last post, I was talking with Donnie Berkholz as we traveled to FOSDEM. Donnie commented on how powerful of a post it was, yet it left the reader hanging. He, and other readers, wanted more. So I’ve taken the liberty of breaking down more of the reasons Enterprise IT needs a “special kind of DevOps” as posted by Andi Mann. I don’t want anyone to think I am picking on Andi personally. Rather, his post reminds me of all the excuses Enterprises give as to why “We can’t change”. As Mick told Rocky, “There ain’t no can’ts!”

  • They cannot achieve the same levels of agility and personal responsibility as a smaller or less complex organization.

Why Not? Principles that teach agility and speed have long been used at large companies such as Microsoft. (Yes, feel free to say Microsoft is a bad example, they are still one of the world’s largest software companies.) Additionally, if one doesn’t want to take personal responsibility for what they produce for a company, maybe they are in the wrong job for the wrong company?

  • They cannot stream new code into production and just shut down for a couple of hours to fallback if it fails.

This is fool-hardy to begin with. The goal of methods such as Continuous Integration is to be constantly building releases and testing them to catch problems before they are released to production. Also, the idea is to test small changes, so you know exactly what breaks, rather than large chunks of code. Large enterprises “cannot stream new code” because they haven’t built the necessary flows in front of production releases to effectively and efficiently test and verify code changes. This requires IT organizations to fully automate their processes all the way down to server builds, a process they often are incapable of doing because of an attachment to the “old way of doing things”.

  • They rarely ever have ‘two pizza teams’ for development or operations (indeed, they are lucky if they have ‘two Pizza Hut teams’).

The size of the team is nearly always irrelevant. Within each Pizza Hut there are tables, and each table consumes the pizza buffet. The goal of DevOps is to increase the flow of the work through those tables so the teams can eat their pizza and leave quicker. As I’ve said before, focusing on the Silos is the wrong way to solve the problem. Rather focus on the grain elevators that move the grain to produce something meaningful.

  • They cannot sign up for cloud services with a credit card without exceeding their monthly limit and/or being fired.

Get an MSA/PO with the cloud vendor or build a Private Cloud. Cloud or no cloud, building strong automation on top of existing VM or server infrastructure can help alleviate many problems in service delivery.

  • They cannot allow developers to access raw production data, let alone copy it to their laptop for development or testing.

Scrub the data. DevOps or not, this is a problem that we’ve solved years ago. When I worked at a major e-commerce site, real data was often required for testing, but that data was always cleaned of any sensitive PII. This is not an issue that is unique to DevOps.

  • They cannot choose to stream new code into production in violation of a change freeze, or even without the prior approval of a CAB.

Once again, one assumes that DevOps is all about willy nilly pushing of code to production. One aspect of DevOps is about increasing the flow of the work through the system by optimizing the centers where value is added. As I’ve discussed before, principles and practices of DevOps actually help things like Change Control.

  • They cannot just tell developers to carry pagers ‘until their software is bedded in’ (not least because their developers have always carried pagers, and on a full-time basis).

If Devs already carry pagers, then they’ve already been told to carry pagers, hence, “they” can indeed tell their Devs to carry pagers. Additionally, bedding in of the software should happen in the lower environments as discussed previously. If you’ve done things right before production, pagers become a tool that is used when things go really badly. It’s a form of monitoring and incident response that becomes meaningful again because you aren’t being paged for endless break fix work.

  • They cannot put developers and operators together because one team works 24×7 shifts in 7data centers while the other works 16-hour days in 12 different locations.

Well, good, they at least have 16 hours a day together. Highly distributed remote teams are becoming more and more common. Technology is evolving to help bring this concept of remote work and people are finding creative ways to work around it. I’m also against the idea that DevOps is all about merging dev and ops onto one team, because that is not the point. The idea, as already stated, is to increase the flow of work between Dev and Ops and build a culture of continuous improvement between the two groups (three groups if you include the business). Dev, Ops, Business, who gives a shit. The point is working towards a common goal, no matter where you sit.

What large IT shops cannot do is be satisfied anymore with the status-quo. They cannot accept the ways of the past any longer, and they have to start thinking about blowing up their way of doing things. They cannot let the castles and fiefdoms of the past get in the way any longer.

I think the single most powerful question any IT shop can ask themselves is, “What if everything we’ve been doing over the last X years is completely wrong?” Start there, and reevaluate everything you’ve been doing to achieve (or not achieve) the results your customers require.

More about Goats and Silos

Posted on by
1

Owens Valley Silo Stairs B&WThe morning after my talk on Goats and Silos at the Cloudstack Collaboration Conference I was sitting at a table with Mark Burgess and John Willis. I was busy working through my email, so I only half heard their conversation, but one thing Mark said really stuck with me. Basically Mark pointed out the importance of “breaking down the silos in our mind.”

This of course stuck out to me, and as I began to think about it, that is exactly what this whole idea of Goats and Silos is about. Much of the talk in DevOps is about breaking down organizational silos, which is hard to impossible for us at the individual contributor level. But there is nothing stopping us from breaking down our preconceived notions and biases. within our minds. Go out and explore across the organizational silos in order to break down your own silos. And if you are a manager, give your goats the rope to go and explore across these silos.

Later when Mark left the table, I started talking with John about the idea of “breaking down the silos of our mind.” John reminded me of a talk by David Foster Wallace. Wallace speaks of our cognitive bias, how the exact same experience can be interpreted completely different by two people, and how humans have a default setting of being self centered.

Later in the week, a friend was telling David Nalley and I about a monitor he keeps on his desk. This monitor was used by a trader at a brokerage firm. The trader was using a VDI instance and the instance froze up because of a problem. Needless to say, the trader was in the middle of trying to execute a large deal, and wasn’t able. Frustrated, he punched the monitor and cracked the LCD. That monitor now sits there as a constant reminder that “the work we do matters.”

That is what it is really about; breaking down those silos in our head so we remember the person on the end of what we produce.