The Goat Farm – Episode 13 – Measuring Success at Capital One

We all think DevOps is a better way to work, but how can you begin to measure aspects of your DevOps transformation. In this episode we talk to Adam Auerbach and Topo Pal of Capital One, and learn more about the work they are doing. We discuss how their DevOps journey started, how it’s now a CIO mandated journey, and how they build some open source tooling to help them measure the speed at which they are moving.

Download MP3 – iTunesStitcherRSS

Show Notes:

Scaled Agile Framework (SAFe)

Capital One DevOps Dashboard – Hygiea

15 principles of CD (mostly binary, used to create heatmap of maturity for Capital One platforms)

  • Github (or similar) with branching strategy
  • Code Coverage (90% preferred, at business discretion)
  • Static Analysis (e.g. PMD, CPD, FindBugs)
  • Static Security
  • Open Source/third party vulnerability scan/support
  • Automated instance provisioning in each region
  • Immutable servers
  • Artifact management
  • Automated build, deploy, and testing on commit (can be by feature)
  • Automated integration testing on successful test
  • Automated performance testing
  • Automated/repeatable rollbacks (including data migrations)
  • Push button/automated deployments to production
  • Automated generation of COs
  • Blue/green (zero downtime/canary) releases
  • Feature activation (wire on/wire off)

 


adam_auerbachAdam Auerbach – TwitterLinkedIn

Adam Auerbach is the Sr Technology Director for Advanced Testing and Release services for Capital One Financial Corporation.  Adam is responsible for Capital One’s enterprise performance and automated testing departments as well as enterprise release management. Since joining Capital One, he has provided leadership for the agile transformation of their quality assurance group and led the enterprise adoption of DevOps and ATDD. Before joining Capital One, Adam was with Chase and other financial and insurance companies, in various leadership positions focusing on quality and agile practices.

 

topo_palTapabrata Pal (Topo) – TwitterLinkedIn

Tapabrata Pal has 20 years of IT experience in various technology roles (developer, operations engineer, and architect) in the retail, healthcare, and finance industries. Over the last five years, Tapabrata has served as director of Capital One’s Enterprise Architecture group, and led the company’s DevOpsSec initiatives. He is currently director and individual contributor focusing on next-generation infrastructure. Tapabrata is also the community manager and a core committer of an Open Source project “Hygieia” that won “Open Source Rookie of the Year” for 2015.
Previously, Tapabrata spent some time in academics doing doctoral and post-doctoral research in the field of solid state physics.

The Goat Farm – Episode 12 – DevOps When Startups Become Enterprises

In this episode we talk to Andy Domeier of SPS Commerce. As startups grow into larger companies, they face the same scaling challenges that larger enterprises tend to encounter. Andy gives us his 11 years of experience of watching SPS Commerce grow from a startup to an enterprise, and how they’ve handled these challenges. We also focus on some of the technology SPS using to help scale the people, and scale their technology capabilities.

Download MP3 – iTunesStitcherRSS

Show notes:


Andy Domeier – LinkedInTwitter
andydomeier

Andy has been in Technology Operations leadership with SPS Commerce for the past 11 years.  SPS grows very aggressively creating an environment of persistent growth challenges.  Andy’s focuses within the organization include:  monitoring and operating complex changing systems, priority organization and alignment, and the organization of Knowledge.

DevOps at IBM – The Goat Farm – Episode 9

How does IBM manage to run web sites for some the World’s largest sporting and television events? With the practices of DevOps of course! In this episode, Ross and Michael talk to Brian O’Connell of IBM.

Brian tells us of his journey to DevOps practices through stumbling onto the ideas of Chef and Infrastructure as Code. We talk about the cultural shift required when it comes to who owns delivery of changes and ownership of those changes. Brian also tells us how they leverage the “build, measure, learn” product development loop.

The sites Brian and team help run are some of the more high profile, and highly visited sites in the world. Brian talks about the challenges when trying to introduce DevOps to such high profile sites, and mistakes that were made along the way. We also talk about some of the tooling Brian and team use, and how they effectively deploy enterprise software packages.

Download MP3 – iTunesStitcherRSS

Show notes:


Brian O’Connell – TwitterLinkedInBrian O'Connell

Brian O’Connell is a Senior Technical Staff Member at IBM that leads a team focused on DevOps, predictive analytics, big data, and cloud technologies.

Brian joined IBM in 2001, starting as a software engineer. He built many software systems to support the continuous availability and events infrastructure.  His expertise includes architecting and developing scalable server applications, concurrency, advanced visualizations, and big data.

From 2007 until 2011 Brian was the lead infrastructure technology advocate and designer for the World Wide Sponsorship Marketing (WWSM) client. His role included strategic technical direction, evaluating technology pilots and the end to end delivery of highly visible web events. In that role, he successfully delivered all IBM sponsorship web sites including The Masters, Wimbledon, Roland Garros (French Open), US Open Tennis, US Open Golf, Australian Open, and The Tony Awards. Brian designed systems to manage the infrastructure and applications used by the client including a focus on defining plans, strategies and architectures for the installation, operation, migration and management of complex information systems.
Brian has had more than 250 patents issued, is an IBM designated Master Inventor and a Franz Edelman laureate.

Adrian Cockcroft of Battery Ventures – The Goat Farm – Episode 8

In this episode we talk to the famous (or infamous) Adrian Cockcroft of Battery Ventures. Adrian is known for his work at Netflix and his work to migrate them to a Cloud first strategy, then before that for his book on Sun performance tuning.

Adrian has been doing a lot of work talking to CIOs of large enterprises and helping them understand where ideas such as DevOps, microservices, Cloud are taking the industry. He allows tells us how he is helping CIOs realize how their IT organizations must transform to adopt these new ideas. This episode is all about how the horses are growing horns to become the unicorns.

(Editor’s note: We are really sorry about the audio on this episode. Adrian was in Portland, Michael was in Amsterdam, and Ross was in Minneapolis. While we could have cut a bunch of the bad audio, the content was so good we didn’t want to drop anything. Apologies.)

Download MP3 – iTunesStitcherRSS


Show Notes


Adrian Cockcroft – LinkedInTwitter

Adrian Cockcroft has had a long career working at the leading edge of technology. He’s always been fascinated by what comes next, and he writes and speaks extensively on a range of subjects. At Battery, he advises the firm and its portfolio companies about technology issues and also assists with deal sourcing and due diligence.

Before joining Battery, Adrian helped lead Netflix’s migration to a large scale, highly available public-cloud architecture and the open sourcing of the cloud-native NetflixOSS platform. Prior to that at Netflix he managed a team working on personalization algorithms and service-oriented refactoring.

Adrian was a founding member of eBay Research Labs, developing advanced mobile applications and even building his own homebrew phone, years before iPhone and Android launched. As a distinguished engineer at Sun Microsystems he wrote the best-selling “Sun Performance and Tuning” book and was chief architect for High Performance Technical Computing.

Jonny Wooldridge on Enterprises vs Startups – The Goat Farm – Episode 3

In this episode Ross and I talk to Jonny Wooldridge, formerly of Marks & Spencer and currently at The Cambridge Satchel Company. We ask Jonny his thoughts on what DevOps is like in an Enterprise vs a Startup, how to jumpstart adoption, how to handle “legacy systems”, and get his thoughts on concepts such as “Pace Layering” and “Bimodal IT”.

Ross and I also talk about why the language we use is important when talking about DevOps and DevOps concepts.

Download MP3 – iTunesStitcherRSS

Guest Info:

Jonny Wooldridge – LinkedInTwitter

Jonny Wooldridge is CTO of The Cambridge Satchel Company and has a history of leading agile cross-functional teams in dynamic and fast paced start-ups in London including lastminute.com, Opodo.com and Photobox.com. Prior to joining The Cambridge Satchel Company he was Head of Web Engineering at the British multinational retailer M&S. He was instrumental in introducing DevOps to the enterprise whilst working on a 3 year / £150 Million project to re-platform the website, order management systems & customer service tools.

He is passionate about Lean and DevOps topics, particularly in challenging environments (like the average enterprise!) and earlier this year started a blog at enterprisedevops.com.

Veteran of the Process Wars

Tokyo. I’m still in Tokyo. I wake up, rub the sleep from my eyes, and roll out of bed. As I rise I take a quick look at my watch for any new emails. Nothing. There’s not much email anymore. Not since the process wars of 2016.

Many people won’t talk about the process wars. They changed the way most of the InfoTech industry works and how we are allowed to think of our jobs. In the past, I might have been responsible for running hundreds of servers. Now, the machines are in charge. I’m only allowed to feed the machines code.

We are grouped together in small teams that write code as a unit. We are “kept in line” by dogs that are trained to attack us. They say it’s for our own good. That it’s for the betterment of industry. If we try to touch the servers, if we try to do anything other than write code and feed it to the machines, the dogs will bite us.

We are kept in line by process and controls, but not like we used to be kept in check. Before, we used to have weekly meetings reviewing the work we wanted to perform. The meetings were always a joke amongst my peers. Typically they were run by a VP that had no clue what we wanted to change on the machines. If we wanted something bad enough, we could social engineer our way to what we wanted.

No more today. When we write the code to feed the machines, we have to write tests. They tell us these tests are for our own good. The tests confirm that the machines are set up exactly how they want them to be. We actually write the tests first, and they always fail the first time through. Then we write code to bring the tests into compliance.

If the tests weren’t bad enough, we also have automated tests to make sure our code conforms to “Style Guidelines”. They tell us these guidelines are to ensure conformity and consistency. I say they are there to hold us down and control us. They also require us to “lint” our code before we feed it to the machines. This once again is to ensure “conformity”.

Even with all of this verifying for conformity, we still aren’t allowed to directly feed the code directly to the machines. We must check the code into a repository where more machines take over to verify that we have successful tests and we meet the conformity guidelines. The machines also verify that our code works with code written by others, in my group and other groups.

Then, a Conformity Checker reviews my changes to make sure they are compliant with the policy. We are no longer independent; we are no longer allowed to game the review board. We feed the machines compliant code or else we end up on the streets. Three strikes and we’re out.

Which is why I’m in Tokyo. I’ve had two strikes at my company in the United States. They shipped me over to the Tokyo division for reeducation. Japan culture is heavily based on order and process. During the process wars they helped lead the revolution. Many of the compliance tools were written by or enabled by Japanese technology leaders. Yukihiro Matsumoto (codenamed Matz) was responsible for designing the programming language I now feed to the machines. Gosuke Miyashita wrote the tools that I am required to use to test my code before feeding it to the machines. And then there was Kohsuke Kawaguchi, the creator of the master machine that ensures all the code is compliant, automatically with no humans to game the process.

It’s all very neat and orderly now. I take requests for code from the owners of the machines, I write compliant code, and the machines automatically verify my work. Eventually the machines apply my code, and the owners get exactly what they want. I’ll wake up the next however many hundreds of mornings and do just this. No more, no less. It’s all very neat and orderly now.

Get Your Head Out of Your aaS

3815168722_faee10cf62_bI’ve been floating between the worlds of Cloud and DevOps for a while now and it is interesting to see the Cloud world finally start to realize the real value is in DevOps. It’s great that more people are starting to pay attention to this cultural and professional movement. What is not great is how the Cloud experts tend to get wrapped up in some debates that are trivial and meaningless, in the larger scheme of things. Take for instance two persistent debates I am seeing over IaaS vs. PaaS, and then which PaaS is better. I hate to be the one to break it to these camps, but it doesn’t matter; at the end of the day you are selling plumbing fixtures that crap flows through.

To understand what I mean, lets take a step back. In 2008, I started pursuing my MBA at The Ohio State University. One of the core requirements of the degree was Operations Management. In Operations Management, you learn manufacturing optimization through ideas such as Lean and Six Sigma. The book “Learning to See” was part of the course material and it focused on optimization of manufacturing processes through visualization, also known as Value Stream Mapping. As the course progressed, I had a personal epiphany. As we kept walking through manufacturing processes, and Value Streams, what I quickly realized was that the work we did in IT was all about manufacturing a good or service someone would be consuming. Automation in the IT world is about (or should be about) optimizing these Value Streams and (hopefully) eliminating waste from the system. My Operations Management course really taught me to see (pun intended) and to think differently about how we worked in IT.

I took this new found knowledge back to my work where it was summarily ignored by my boss and coworkers, and lacking support I shelved my ideas. Little did I know many of the Lean principals I had learned would be at the forefront of how IT is changing today, and was already being changed at that time in 2008, I just didn’t know it.

When somebody asks me what DevOps is, I often respond with the simple idea that “DevOps is about increasing the flow of work through IT.” I borrow this idea heavily from “The Phoenix Project“, but I find it is the most simplest way to capture the essence of this cultural and Imageprofessional movement. And that is where Value Stream Mapping and the ideas of Lean come into the conversation. Books like the “The Phoenix Project“, and notable DevOps contributors such as John Willis expound the values of these techniques to optimize the IT Manufacturing chain, be it Development work or Operations work.

Value Stream Maps are relatively simple. They identify the flow of a raw material through Screen Shot 2014-04-03 at 11.07.33 PMvarious processes that add value. They also identify areas of waste in the system, and they help in building the Future State Map, or the Value Stream that you want to achieve in the future after optimizing the system. The most basic and valuable thing about Value Stream Maps is how they allow you to easily visualize your work, and once it is visualized it is easy to understand and optimize.

If you look at the first current state map, you can easily see how relabeling the boxes to reflect common IT tasks, say in a server build process, makes this a powerful tool for IT. Replace the box names with another process – maybe code build, testing, and release – and you see once again how Value Stream Mapping is a key tool in fixing our broken IT.

Now that we’ve established a method for the optimization of our IT processes, let’s go back to thinking about Cloud and the debates around Iaas, PaaS, and the PaaS vendors. Take the second Value Stream Map. Say this diagram more accurately reflected server builds and the time it took to install an OS was one hour. We optimize this process through our IaaS based Cloud, public or private, and get the time down to 5 minutes. That is awesome, we’ve saved 55 minutes and really optimized that process. Go team!

If “premature optimization is the root of all evil”, then local optimization is the Devil’s half brother. In the above example we saved 55 minutes, but the total time of work flowing through the system is still 67 days, 23 hours. And that is where we come back to Cloud. IaaS is a local optimization. It is great, it is awesome, but it is a very small piece of the puzzle. PaaS is another local optimization, but instead of optimizing one process it optimizes three or four. Which is great, but many IT organizations are going to “adopt Cloud for business agility and speed, then be sadly surprised when their local optimization does little to fix their screwed up system. Cloud can be a great enabler, but it is only a small piece of the larger system. It is high time more of us focus on the larger system.