InfoQ catches up with Jeff Smith on Centro transformation to a DevOps culture, which will feature in his forthcoming keynote at DevOpsDays NZ. Smith also discusses his recent DevOpsDays Indianapolis talk on the misalignment which can arise due to the different lenses through which collaborators see the world.
Jeff Smith, manager of production operations for Centro, a Chicago based organisation which provides a platform for digital marketing, will deliver a keynote at DevOpsDays NZ this November in Wellington, New Zealand, with a talk titled Moving from Ops to DevOps: Centro’s Journey to the Promiseland. Smith also spoke at DevOpsDays Indianapolis in July about the misalignment which can arise in an organisation, simply due to the differing motivational lenses through which professional silos examine the same subject matter.
InfoQ caught up with Smith to discuss Centro’s journey and compare it with Grubhub’s DevOps transformation, which he spoke about in 2016 at DevOpsDays Minneapolis.
InfoQ: You talked at DevOpsDays Indianapolis this July about the misalignment which can arise from the differently biased «lenses» through which various parts of an organisation look at the same problem. Can you tell us a little more about this?
Jeff Smith: Sure. It’s this problem where we don’t really see the problem from the same perspectives. It reminds me a lot of the Blind Men And The Elephant parable from India. A group of blind men who have never encountered an elephant before are feeling the elephant and describing it to the other blind men. But because each of them is touching a different part of the elephant, their perspectives don’t always overlap. It’s the same problem in organizations. We’re grossly misaligned in both our goals and our incentives, which results in wasted work and wasted effort. Establishing the context in which each of us is working in helps us to achieve each of our goals more efficiently. If my goal is to build the most robust Infrastructure possible and your goal is to contain costs, we’re going to have an impasse if we don’t communicate. Through dialogue, we can either both meet our goals or at least present the conflict to leadership in order to strategically decide which one is more important.
InfoQ: How did Centro deal with this misalignment in understanding?
Smith: Like most organizations, it’s a constant struggle. The key thing is to always, always ask for context around the problem being solved. Another thing is to be astute and make sure you’re getting the problem and not just feedback on someone’s proposed solution. This is referred to as the XY Problem. If someone is asking you for feedback on their proposed solution, versus their actual problem you will end up with a sub-par solution.
An example of this is something most people are probably familiar with, someone asking for production access to a system. The truth is they don’t need production access, they just need to be able to execute task X. The solution they’ve come up with, however, involves them SSH’ing to a box and running a command. But if we rephrase and say «I need to be able to run this script in production without involving other people», it opens the range of possibilities up. Maybe we automate it and turn it into a Jenkins job. Maybe we schedule it on a regular basis to run. Simply granting production access may solve your problem, but create a host of audit problems for another team.
InfoQ: Your talk for DevOps NZ will be about Centro’s journey from Ops to DevOps. What do these two terms mean to you?
Smith: OPS is basically the function of what my team does. I like to think of it as the technical version of business operations. Operations try to manage the process of bringing our code to the customer. Production systems are part of that, but there are many pre-production systems that are a piece of that as well and it all falls under the purview of OPS in my view. This includes build servers, testing environments, code repository management etc. DevOps is about the approach we take to doing that. DevOps is not a team, it’s not a third silo, it’s a method of collaborating between Dev and Ops. That collaboration usually involves some cultural changes. OPS has to give a bit more control to the developers of the environment, as well as a bit more visibility. Development needs to take a large role of responsibility for managing and operating their code once it hits production. The world of «only OPS touches production» and «development ends once its off my laptop» are gone. We need a tighter method of working together and that’s what DevOps is in my view.
InfoQ: How was this journey affected by Centro’s own particular starting state and context?
Smith: Centro was kind of struggling with what DevOps actually meant for the org. When I was originally interviewing for the position, the title was called «DevOps Manager», which for me exposed some of the core issues with how they’ve been trying to manage their transition. When I started we had a lot of Iron Fisted OPS sorts of policies. But there were a few key players in the org, that saw the benefits of a different style of working and craved more ownership. One developer, in particular, had an amazing amount of metrics around the systems he was responsible for. He had a real interest in knowing what was happening in the systems he was responsible for. Every company has a few of those people and the key is to find them and unleash them! Seek their input on the problems and the solutions the team is facing. Find all the allies you can and leverage them.
InfoQ: What types of challenges did Centro and its teams have to overcome during this transformation?
Smith: The biggest challenge remains to be instilling a sense of ownership for the code that gets produced and deployed to production. Because we operate a monolith, there are a fair number of developers that commit to the same codebase. This can produce the bystander effect, where all of the involvement serves as a sort of diffusion of responsibility. How does someone know it was their change that broke things? Maybe it was some other change! The best way we’ve found to instil ownership is to make sure problems are assigned to specific people. If it gets assigned to general areas, what ends up happening is everyone assumes someone else is looking at it. But when an issue is owned by a specific person, it’s on them to either solve the problem or generate enough evidence to point it at the real issue owner and transfer it to them. Explicit ownership of problems is key.
InfoQ: How has the culture at Centro changed since its DevOps transformation?
Smith: People want to know why a thing is being done. That’s probably the biggest change I’ve witnessed. The act of asking «why?» shows that there is a level of engagement that goes beyond just getting a request off their plate so they can move on to the next thing. When people don’t understand something, they ask good probing questions in order to understand something. One simple question about a failed job run might end with an in-depth discussion of how Write Ahead Log replication in the database works. People just naturally want to learn more!
InfoQ: What measures do you take to avoid reverting back into practice based silos?
Smith: Discipline on how we approach problems. A knee-jerk reaction when something goes wrong is to add another layer of approval or another layer of supervision, but it doesn’t solve any of the actual issues that lead to the situation you’re in. We also really enjoy doing post-mortems, but we focus on the human side of the problem rather than the order of events. What were people thinking when they made a particular decision? Why did that seem like a rational choice? What can we learn about the incident to make sure we haven’t slipped into an old way of doing things? We have to talk about things that go wrong in a much deeper sense than the way we typically talk about failure in retributive cultures.
InfoQ: How have non-technical partners responded to Centro’s journey to a DevOps culture?
Smith: Non-technical resources honestly don’t interface with us at the level where they see the cultural differences, but they see the change in capabilities. When a person can have their own mini-environment to test out a specific feature, it’s powerful. And when they can get it in minutes as opposed to days, it speaks to the types of changes and the speed at which we intend to move.
InfoQ: You previously spoke at DevOps Minneapolis about Grubhub’s DevOps transformation. How did these two experiences compare?
Smith: The experiences are largely different because Grubhub was a company born in technology.
