Category Archives: Goats

DevOps at Standard Bank – The Goat Farm – S1E6

Posted on by
Reply

It’s been awhile since the last episode, but we are back with a bang! In this episode we talk to Standard Bank, the largest bank in Africa, about the challenges they faced in taking a DevOps approach in their organization.

Compliance at Velocity was one of the tracks at this year’s ChefConf. Our guest Josef Langerman discusses corporate compliance and the scale of how broad and wide regulations can affect an enterprise’s approach to DevOps, leveraging Agile, and delivering the right solutions for customers/guests.
Listen to Josef’s recount of Standard Bank’s journey – including discovery of change
champions, driving a new, DevOps culture, and establishing a set of themes to
continuously improve and advocate for new ways to satisfy the company’s needs.

We recorded this episode at ChefConf 2015 and we were happy to have Jason Walker of Target as our guest host. If you want to find out more about the topics we discussed, check out the links below.

Download MP3 – iTunes – Stitcher – RSS

Guest Info:

Josef Langerman – LinkedIn – Twitter

Information Technology executive with experience across the Airline, Retail and Investment Banking Industries. My focus is on maximizing development throughput and large scale software development using DevOps and Agile approaches. I am also passionate about higher education and IT research. My teaching and research focus is on Project Management and Software Development.

 

Show Notes:

Jason Walker at ChefConf:

Rachel Chalmers at ChefConf:

 

Running Internal Events – The Goat Farm – S1E5

Posted on by
Reply

A guy in Belgium inspires a bank in the Netherlands to hold an internal DevOps Days. A weekly newsletter in the UK picks up a presentation from that internal event, and a team in Minneapolis, MN is inspired to hold their own event.

Internal events are becoming more and more popular in Enterprise IT. Cloud Symposiums, Automation Symposiums, DevOps Leadership Summits, and DevOps Days are all internal events I have participated in this year alone. Ross and I talk to Heather Mickman (Target), Brent Nelson (Target), and Mark Heistek (ING Bank) about the events they have run in their organizations, how they got started, what challenges they faced, and any tips for people wanting to run their own events.

If you’d like to see some of the tweets and activities from Target’s last two DevOps events, you can search twitter for the hashtag #dotgt.

We also talk briefly about “The Prince of DevOps“, and reviews we’ve gotten about the podcast (sorry for the heavy breathing last time).

Download MP3 – iTunes – Stitcher – RSS

Guest Info:

Heather Mickman – LinkedIn – Twitter

Heather Mickman is the leader for the API and Integration team at Target and a DevOps enthusiast.  Throughout her career, Heather has continuously embraced hard technology challenges from consulting large Fortune 50 companies on Supply Chain approach, implementing warehouse automation technologies, running large Ops & Support organizations, and establishing enterprise security approaches.  She has a passion for technology, building high performing teams, driving a culture of innovation, and having fun along the way.  Heather lives in Minneapolis with her 2 sons and 2 dachshunds.

Brent Nelson – LinkedIn – Twitter2014-ProfilePic

Husband, father and life-long resident of Minnesota. I’ve been with Target for 26+ years and for the last year have been an internal DevOps collaboration and social media evangelist involved in hosting internal DevOpsDays events, creating/delivering internal educational materials, co-curating the #make_awesome_happen Flipboard ezine and much more.

fotoMarkHeistekMark Heistek – LinkedIn – Twitter

Father of two children, sport fanatic, having fun in life and working at ING Bank Netherlands since 2008. Currently in a continuous delivery team to facilitate in an enterprise continuous delivery pipeline. Furthermore a Continuous Delivery and DevOps evangelist in and outside ING.

Veteran of the Process Wars

Posted on by
Reply

Tokyo. I’m still in Tokyo. I wake up, rub the sleep from my eyes, and roll out of bed. As I rise I take a quick look at my watch for any new emails. Nothing. There’s not much email anymore. Not since the process wars of 2016.

Many people won’t talk about the process wars. They changed the way most of the InfoTech industry works and how we are allowed to think of our jobs. In the past, I might have been responsible for running hundreds of servers. Now, the machines are in charge. I’m only allowed to feed the machines code.

We are grouped together in small teams that write code as a unit. We are “kept in line” by dogs that are trained to attack us. They say it’s for our own good. That it’s for the betterment of industry. If we try to touch the servers, if we try to do anything other than write code and feed it to the machines, the dogs will bite us.

We are kept in line by process and controls, but not like we used to be kept in check. Before, we used to have weekly meetings reviewing the work we wanted to perform. The meetings were always a joke amongst my peers. Typically they were run by a VP that had no clue what we wanted to change on the machines. If we wanted something bad enough, we could social engineer our way to what we wanted.

No more today. When we write the code to feed the machines, we have to write tests. They tell us these tests are for our own good. The tests confirm that the machines are set up exactly how they want them to be. We actually write the tests first, and they always fail the first time through. Then we write code to bring the tests into compliance.

If the tests weren’t bad enough, we also have automated tests to make sure our code conforms to “Style Guidelines”. They tell us these guidelines are to ensure conformity and consistency. I say they are there to hold us down and control us. They also require us to “lint” our code before we feed it to the machines. This once again is to ensure “conformity”.

Even with all of this verifying for conformity, we still aren’t allowed to directly feed the code directly to the machines. We must check the code into a repository where more machines take over to verify that we have successful tests and we meet the conformity guidelines. The machines also verify that our code works with code written by others, in my group and other groups.

Then, a Conformity Checker reviews my changes to make sure they are compliant with the policy. We are no longer independent; we are no longer allowed to game the review board. We feed the machines compliant code or else we end up on the streets. Three strikes and we’re out.

Which is why I’m in Tokyo. I’ve had two strikes at my company in the United States. They shipped me over to the Tokyo division for reeducation. Japan culture is heavily based on order and process. During the process wars they helped lead the revolution. Many of the compliance tools were written by or enabled by Japanese technology leaders. Yukihiro Matsumoto (codenamed Matz) was responsible for designing the programming language I now feed to the machines. Gosuke Miyashita wrote the tools that I am required to use to test my code before feeding it to the machines. And then there was Kohsuke Kawaguchi, the creator of the master machine that ensures all the code is compliant, automatically with no humans to game the process.

It’s all very neat and orderly now. I take requests for code from the owners of the machines, I write compliant code, and the machines automatically verify my work. Eventually the machines apply my code, and the owners get exactly what they want. I’ll wake up the next however many hundreds of mornings and do just this. No more, no less. It’s all very neat and orderly now.

Get Your Head Out of Your aaS

Posted on by
Reply

3815168722_faee10cf62_bI’ve been floating between the worlds of Cloud and DevOps for a while now and it is interesting to see the Cloud world finally start to realize the real value is in DevOps. It’s great that more people are starting to pay attention to this cultural and professional movement. What is not great is how the Cloud experts tend to get wrapped up in some debates that are trivial and meaningless, in the larger scheme of things. Take for instance two persistent debates I am seeing over IaaS vs. PaaS, and then which PaaS is better. I hate to be the one to break it to these camps, but it doesn’t matter; at the end of the day you are selling plumbing fixtures that crap flows through.

To understand what I mean, lets take a step back. In 2008, I started pursuing my MBA at The Ohio State University. One of the core requirements of the degree was Operations Management. In Operations Management, you learn manufacturing optimization through ideas such as Lean and Six Sigma. The book “Learning to See” was part of the course material and it focused on optimization of manufacturing processes through visualization, also known as Value Stream Mapping. As the course progressed, I had a personal epiphany. As we kept walking through manufacturing processes, and Value Streams, what I quickly realized was that the work we did in IT was all about manufacturing a good or service someone would be consuming. Automation in the IT world is about (or should be about) optimizing these Value Streams and (hopefully) eliminating waste from the system. My Operations Management course really taught me to see (pun intended) and to think differently about how we worked in IT.

I took this new found knowledge back to my work where it was summarily ignored by my boss and coworkers, and lacking support I shelved my ideas. Little did I know many of the Lean principals I had learned would be at the forefront of how IT is changing today, and was already being changed at that time in 2008, I just didn’t know it.

When somebody asks me what DevOps is, I often respond with the simple idea that “DevOps is about increasing the flow of work through IT.” I borrow this idea heavily from “The Phoenix Project“, but I find it is the most simplest way to capture the essence of this cultural and Imageprofessional movement. And that is where Value Stream Mapping and the ideas of Lean come into the conversation. Books like the “The Phoenix Project“, and notable DevOps contributors such as John Willis expound the values of these techniques to optimize the IT Manufacturing chain, be it Development work or Operations work.

Value Stream Maps are relatively simple. They identify the flow of a raw material through Screen Shot 2014-04-03 at 11.07.33 PMvarious processes that add value. They also identify areas of waste in the system, and they help in building the Future State Map, or the Value Stream that you want to achieve in the future after optimizing the system. The most basic and valuable thing about Value Stream Maps is how they allow you to easily visualize your work, and once it is visualized it is easy to understand and optimize.

If you look at the first current state map, you can easily see how relabeling the boxes to reflect common IT tasks, say in a server build process, makes this a powerful tool for IT. Replace the box names with another process – maybe code build, testing, and release – and you see once again how Value Stream Mapping is a key tool in fixing our broken IT.

Now that we’ve established a method for the optimization of our IT processes, let’s go back to thinking about Cloud and the debates around Iaas, PaaS, and the PaaS vendors. Take the second Value Stream Map. Say this diagram more accurately reflected server builds and the time it took to install an OS was one hour. We optimize this process through our IaaS based Cloud, public or private, and get the time down to 5 minutes. That is awesome, we’ve saved 55 minutes and really optimized that process. Go team!

If “premature optimization is the root of all evil”, then local optimization is the Devil’s half brother. In the above example we saved 55 minutes, but the total time of work flowing through the system is still 67 days, 23 hours. And that is where we come back to Cloud. IaaS is a local optimization. It is great, it is awesome, but it is a very small piece of the puzzle. PaaS is another local optimization, but instead of optimizing one process it optimizes three or four. Which is great, but many IT organizations are going to “adopt Cloud for business agility and speed, then be sadly surprised when their local optimization does little to fix their screwed up system. Cloud can be a great enabler, but it is only a small piece of the larger system. It is high time more of us focus on the larger system.

What if Everything We’ve Been Doing is Wrong?

Posted on by
2

60-wrong-way

After I wrote my last post, I was talking with Donnie Berkholz as we traveled to FOSDEM. Donnie commented on how powerful of a post it was, yet it left the reader hanging. He, and other readers, wanted more. So I’ve taken the liberty of breaking down more of the reasons Enterprise IT needs a “special kind of DevOps” as posted by Andi Mann. I don’t want anyone to think I am picking on Andi personally. Rather, his post reminds me of all the excuses Enterprises give as to why “We can’t change”. As Mick told Rocky, “There ain’t no can’ts!”

  • They cannot achieve the same levels of agility and personal responsibility as a smaller or less complex organization.

Why Not? Principles that teach agility and speed have long been used at large companies such as Microsoft. (Yes, feel free to say Microsoft is a bad example, they are still one of the world’s largest software companies.) Additionally, if one doesn’t want to take personal responsibility for what they produce for a company, maybe they are in the wrong job for the wrong company?

  • They cannot stream new code into production and just shut down for a couple of hours to fallback if it fails.

This is fool-hardy to begin with. The goal of methods such as Continuous Integration is to be constantly building releases and testing them to catch problems before they are released to production. Also, the idea is to test small changes, so you know exactly what breaks, rather than large chunks of code. Large enterprises “cannot stream new code” because they haven’t built the necessary flows in front of production releases to effectively and efficiently test and verify code changes. This requires IT organizations to fully automate their processes all the way down to server builds, a process they often are incapable of doing because of an attachment to the “old way of doing things”.

  • They rarely ever have ‘two pizza teams’ for development or operations (indeed, they are lucky if they have ‘two Pizza Hut teams’).

The size of the team is nearly always irrelevant. Within each Pizza Hut there are tables, and each table consumes the pizza buffet. The goal of DevOps is to increase the flow of the work through those tables so the teams can eat their pizza and leave quicker. As I’ve said before, focusing on the Silos is the wrong way to solve the problem. Rather focus on the grain elevators that move the grain to produce something meaningful.

  • They cannot sign up for cloud services with a credit card without exceeding their monthly limit and/or being fired.

Get an MSA/PO with the cloud vendor or build a Private Cloud. Cloud or no cloud, building strong automation on top of existing VM or server infrastructure can help alleviate many problems in service delivery.

  • They cannot allow developers to access raw production data, let alone copy it to their laptop for development or testing.

Scrub the data. DevOps or not, this is a problem that we’ve solved years ago. When I worked at a major e-commerce site, real data was often required for testing, but that data was always cleaned of any sensitive PII. This is not an issue that is unique to DevOps.

  • They cannot choose to stream new code into production in violation of a change freeze, or even without the prior approval of a CAB.

Once again, one assumes that DevOps is all about willy nilly pushing of code to production. One aspect of DevOps is about increasing the flow of the work through the system by optimizing the centers where value is added. As I’ve discussed before, principles and practices of DevOps actually help things like Change Control.

  • They cannot just tell developers to carry pagers ‘until their software is bedded in’ (not least because their developers have always carried pagers, and on a full-time basis).

If Devs already carry pagers, then they’ve already been told to carry pagers, hence, “they” can indeed tell their Devs to carry pagers. Additionally, bedding in of the software should happen in the lower environments as discussed previously. If you’ve done things right before production, pagers become a tool that is used when things go really badly. It’s a form of monitoring and incident response that becomes meaningful again because you aren’t being paged for endless break fix work.

  • They cannot put developers and operators together because one team works 24×7 shifts in 7data centers while the other works 16-hour days in 12 different locations.

Well, good, they at least have 16 hours a day together. Highly distributed remote teams are becoming more and more common. Technology is evolving to help bring this concept of remote work and people are finding creative ways to work around it. I’m also against the idea that DevOps is all about merging dev and ops onto one team, because that is not the point. The idea, as already stated, is to increase the flow of work between Dev and Ops and build a culture of continuous improvement between the two groups (three groups if you include the business). Dev, Ops, Business, who gives a shit. The point is working towards a common goal, no matter where you sit.

What large IT shops cannot do is be satisfied anymore with the status-quo. They cannot accept the ways of the past any longer, and they have to start thinking about blowing up their way of doing things. They cannot let the castles and fiefdoms of the past get in the way any longer.

I think the single most powerful question any IT shop can ask themselves is, “What if everything we’ve been doing over the last X years is completely wrong?” Start there, and reevaluate everything you’ve been doing to achieve (or not achieve) the results your customers require.

You’re Not a Beautiful and Unique Snowflake

Posted on by
4

“You are not special. You’re not a beautiful and unique snowflake. You’re the same decaying Enterprise IT Org as everyone else. We’re all part of the same compost heap. We’re all singing, all dancing crap of IT.” — Apologies to Chuck Palahniuk

Enterprise IT, The SnowflakeI’ve seen a few exchanges from “Enterprise IT” vendors on twitter about the need for “a different kind of DevOps” for Enterprise IT. This culminated with a blog post from Andi Mann from CA on “Big Enterprises Need Big DevOps“. I’ll avoid the proverbial piss taking that could take place on the title alone and instead focus on the content.

First, let me say that Andi is spot on in the problems he mentions with Enterprise IT. Andi highlights that code cannot be “streamed into production” because of change controls. Audit and Compliance is critical for many large IT organizations. Enterprise IT can’t go buy cloud services with a credit card, and so on. In the end, Andi proposes that a new form of DevOps, Big DevOps, is needed to handle the unique nature of Enterprise IT.

But like a first year med student that is trying to impress the professor with an intelligent response, Andi is focusing on the symptoms of the problem, rather than the causes of the problem. Giving a patient a prescription for pain killers because he has a headache will do nothing if the cause of the headache is the patient constantly banging his head against his desk. The only people who benefit from that scenario is the doctor who gets to pay for his boat with the extra office visits, and the prescription drug salesperson that is making their quota (and taking the doctor to steak dinners).

The problem with many Enterprise IT shops is that they think they are a special and unique snowflake. They won’t stop talking long enough to understand how they might actually be their own worst enemy in creating all this process that is not “small DevOps Compliant”. Instead of understanding how the tenets of DevOps can achieve the same goal as many of their legacy processes, they are immediately dismissive.

Take for instance the issues around audit, compliance and change control. Many legacy change controls were put in place because changes to the environment were impossible to track across one or hundred systems. But the ideas of automation and Infrastructure as Code have evolved to help alleviate this problem. Wrapping things like Source Control Management, and Test Driven Development around your automation allows you to 1) have tested infrastructure code, 2) audit what is changing in your environment , 3) have an audit trail of who changed things, and 4) know exactly when it changed. Compare that to legacy change control processes if you will.

If you want to be successful with any large scale organizational change, you need to assume that everything you are currently doing is wrong and be open to change. Attempting to conform the organizational change to the organization just leaves you with the same organization you had in the first place.

Which brings me around to this post from ZeroTurnaround on “Why your organization hates DevOps and won’t implement it this year (again)“. They make excellent points that echoes and reinforces the points made in this post. Enterprise IT won’t do anything about DevOps or Cloud or anything else this year. They are too happy with the status quo. They want the change to conform to them and their processes. But change doesn’t work like that. Change is often hard, but if you dislike change, you’ll dislike irrelevance even more*.

*Props to @jonisick for that great quote.

More about Goats and Silos

Posted on by
1

Owens Valley Silo Stairs B&WThe morning after my talk on Goats and Silos at the Cloudstack Collaboration Conference I was sitting at a table with Mark Burgess and John Willis. I was busy working through my email, so I only half heard their conversation, but one thing Mark said really stuck with me. Basically Mark pointed out the importance of “breaking down the silos in our mind.”

This of course stuck out to me, and as I began to think about it, that is exactly what this whole idea of Goats and Silos is about. Much of the talk in DevOps is about breaking down organizational silos, which is hard to impossible for us at the individual contributor level. But there is nothing stopping us from breaking down our preconceived notions and biases. within our minds. Go out and explore across the organizational silos in order to break down your own silos. And if you are a manager, give your goats the rope to go and explore across these silos.

Later when Mark left the table, I started talking with John about the idea of “breaking down the silos of our mind.” John reminded me of a talk by David Foster Wallace. Wallace speaks of our cognitive bias, how the exact same experience can be interpreted completely different by two people, and how humans have a default setting of being self centered.

Later in the week, a friend was telling David Nalley and I about a monitor he keeps on his desk. This monitor was used by a trader at a brokerage firm. The trader was using a VDI instance and the instance froze up because of a problem. Needless to say, the trader was in the middle of trying to execute a large deal, and wasn’t able. Frustrated, he punched the monitor and cracked the LCD. That monitor now sits there as a constant reminder that “the work we do matters.”

That is what it is really about; breaking down those silos in our head so we remember the person on the end of what we produce.

The Evolution of an Idea

Posted on by
1

goats in towerFive months ago a presentation was given in Amsterdam. In that presentation a comment was made that I woke up with a goat in my hotel room. That coupled with the fact that almost every presentation at the conference had a picture of a grain silo in it made some to think they were at a farmer’s conference, not a technology conference.

A quick google search by Arjen Wolfs  found a calculus problem called “The Goat and Silo”, and a challenge was thrown down. I was challenged to build a presentation around “The Goat and Silo”. But as I researched this idea, and talked to others, I found that the idea was not as half cocked as I thought.

Initially, when I talked to people about the idea they thought I was crazy, but as I explained it more they realized the analogy of a goat tied to a silo was an excellent comparison to some IT organizations and the people within them. I wrote an article for Information Week describing the idea of hiring goats, and the response helped to validate the idea. Further validation on the idea was provided at Cloud Connect Chicago where I presented the ideas of goats, silos, and the misconception that silos must be “torn down”.

So now we circle back to Amsterdam, and the Cloudstack Collaboration Conference. I’m excited to be heading back to some of the people that initially hatched the concept, and reflect together with them on the journey over the last five months. Am I crazy (maybe), or do the ideas really reflect what we see on a day to day basis in IT. But what is more exciting is getting to interact with the great attendees and speakers, as well as attend the great talks scheduled.

Image credit: http://www.flickr.com/photos/valkyrieh116/1209010274/in/photostream/