Podcast Ep. 17: Embedding DevOps in Large Organizations, with Barry Matheney

Share on facebook
Share on twitter
Share on linkedin
Share on email

Episode Description:

Repeat guest Barry Matheney joins Dan Neumann for this week’s episode of the Agile Coaches’ Corner podcast!

Barry is an agile thinker, an expert in the DevOps space, and a colleague of Dan’s at AgileThought—serving as a Senior DevOps Consultant. Prior to his role at AgileThought, he was the Director of Enterprise Applications at Kforce, Inc.

Last time Barry was on the podcast, they discussed embedding DevOps into Scrum teams with Eric Landes. This week, they’re taking it up a notch and discussing even bigger projects: They discuss what everyone on an agile team should do; how teams can practice agility even in a non-agile company; how teams can plan with a legacy system in mind when going about new development efforts; and many more concepts around embedding DevOps in large organizations.


Listen on Google Play Music


Key Takeaways:

  • What everyone on an agile team should do:
    • Challenge each other to make the product the best it can be (and not just worry about the individual components)
    • Look at team formation and set standards to adhere to
    • Practice personal accountability
    • Challenge all of the technical resources to make sure they’re protecting the team
    • Consult team and make sure rules and policies still apply
    • Take a grassroots approach
  • How teams can be agile in a non-agile company:
    • Punch holes in the silos rather than tear down the silos (AKA create a conduit between the teams and learn how to integrate their thoughts and processes)
    • Have a team/working agreement in place (especially to challenge each other with new ideas)
    • Diversify the teams so you do not repeat limiting patterns
    • Build some of the software and test it instead of arguing over a whiteboard
  • How to create safety around new development efforts (especially when integrating with a legacy system):
    • Plan with the legacy system in mind (while looking for ways to eliminate parts of it gradually)
    • Migrate a legacy system incrementally by using the strangler pattern
    • Remember that it’s not just a technology challenge; it’s a business challenge (engage with the business side about the value of taking a different approach vs. a big-bang replacement effort of the old system)
  • Other concepts for embedding DevOps in large organizations:
    • Minimize interdependencies on any upstream or downstream system
    • Ensure teams are challenging the technical resources (through the working agreement)
    • Include the development team in the entire lifecycle
    • Have empathy for all involved
    • Continuously improve and learn over time (instead of delving into a much larger, riskier change)


Mentioned in this Episode:


Barry’s Book Picks:


Like what you heard? Check out our podcast page for more episodes full of interesting discussions, agile insights, and helpful resources.



Intro: [00:03] Welcome to Agile Coaches’ Corner by AgileThought. The podcast where practitioners and leaders seeking advice to refine the way they work and pave the path to better outcomes. Now here’s your host, coach and agile expert, Dan Neumann.

Dan Neumann: [00:16] Welcome to this episode of Agile Coaches’ Corner. I am your host Dan Neumann, and joining me today is my colleague, Barry Matheney who is also an agile coach here at AgileThought. And just a quick reminder before we dive in, of course these opinions are throws of myself and Barry and not necessarily those of AgileThought or other folks or other companies. Last time Barry, you and I were doing a podcast. We talked about embedding DevOps into Scrum teams with Eric Landes. So it was very much a, a smaller focus and this time let’s take that up a level to talk about kind of bigger gnarlier projects.

Barry Matheney: [00:58] Yeah, so, and I think that it becomes more challenging as you get larger, right? And you’re going into, you know, I think conceptually a lot of people like to talk about doing things in Greenfield, right? There’s nothing, there’s no landmines there. There’s nothing that’s been happened. You know, there’s no history that we have to worry about. A lot of times when you go to large corporations or large organizations, you’re going to find out that, oh, we want to do this one product and you may have one team or two small teams or something like that that are going off and do something cool and fun, but then they find out they have to fit into the rest of the big organization. Right? So, oh, now you’ve got to integrate with 17 other organizations or 17 other products or 17 other applications and how do you do that? Right? And how do you make sure, and I’m not talking just from an agile perspective, but from a technical perspective somewhat, right? Um, I think there’s a lot of opportunities and especially when you’re forming new teams or there’s, there’s some, um, people that don’t necessarily challenge. And one of the things that I think that everybody on the team has to do is challenge each other to how are we going to make this thing the best product for the team and not just worry about the individual components and the individuals within it.

Dan Neumann: [02:17]  That opportunity to start a greenfield project is, is fantastic. And to what you were saying, that doesn’t come around every day. Often it’s enhancements to older systems or integrations with legacy systems where we don’t just get to start with all the newest, latest and greatest, everything new team members, new tool suites, new technologies, new agile teams, right. And so, uh, you know, really being able to challenge it, challenge each other is an important facet to that. Yeah. As an agile coach, that’s where team formation becomes a really important activity so that we are setting standards for that team that people would adhere to.

Barry Matheney: [03:01] Yeah. In my experience, it’s been funny. Uh, the more senior people wind up being the ones chosen to go work on the cool new projects, right. Naturally, that’s a natural thing that happens for the cool new teams are getting it spun up with not just legacy integrations but legacy people. Right? And there are pecking orders in every organization when you go in and people who think they’re, um, you know, there rewarded for what they do and they’re, observed by everybody else as being really good at what they do. Right. But that doesn’t necessarily mean that we, we don’t want to challenge everyone on the team. Right. So one of the things that recently happened is, and I know you’re talking about books at the end mostly, but one of the things that happened with somebody challenged me to go read a book called The Question Behind The Question. And the point of the whole book is personal accountability. And that’s something that still has to happen in every agile team, just like it does in every other walk of life. Right. The point really being, I think it’s incumbent on everyone on the team to be sure that they challenged, uh, all of the technical resources to make sure that they are protecting the team. And what I say by that, what I mean is there are lots of legacy applications you’re going to have to be integrating with. How do we protect the team so that we can go at our own set of velocity and not be tied to those legacy applications, those legacy systems? How do we, you know, how do we architect and build things and make sure that we don’t break existing stuff with every change that we make to make sure that, uh, we can go with the velocity that the team is capable of and not bound by some preexisting organizational legacy stuff.

Dan Neumann: [04:50]  You’d mentioned the rewarding of the legacy people. And of course they’ve been in their organization while they know how to make things work in the old approach. And sometimes when we then take a new team and form it with those, the same patterns and challenges and limitations tend to emerge. So I like that concept of how do you, um, really diversify the thought on them. So looking at team formation, it may not be taking the most senior folks in the organization or are the ones who have kind of proven themselves over time on a current architecture because there’s a really good chance that architecture is going to emerge again in the new system.

Barry Matheney: [05:29] And I agree with that. I think that one of the things that I’ve, I’ve experienced says that those that are senior are the ones that they see it as a reward to go off and do something new and cool on a new platform or a new tool or a new delivery. I do think that the diversity of the team is an absolute requirement just to get new thoughts and new ways of looking at things. And you know, again, we get conditioned to a norm, right? And it becomes, Oh, I’ve tried this, I’ve run into this wall six times. I’m not going to run into that wall anymore. Well, new people don’t have that same level of conditioning. So having them challenge each of us and say, hey, why don’t we go talk to x to make sure that that rule still applies, right? Rules and policies change over time. And those are the challenging things that we deal with on a team on a day to day basis. And they, they impact our design decisions, they impact our architectural decisions, they impact everything that we do. And we go back to that. That’s a water Scrum fall thing that we talked about last time, right? We still have those boundaries around us in most situations. Most companies haven’t gone to being agile from top to bottom. So when we experience those new teams being stood up, then we wind up having to operate within the confines of our agile team, which doesn’t necessarily span the organization very.

Dan Neumann: [06:51]  Could you expand, um, the, the concept of the teams that are trying to be agile and trying new things, being within organizations maybe where there is not a lot of agility at the broader level.

Barry Matheney: [07:04] Yeah. And so from a, from a perspective of, you know, large companies that have existing organizational infrastructures that aren’t necessarily aligned with agile, your development teams joined up being the agile. And the change comes from the bottom. And from a DevOps perspective, we would love to say that our teams can have organizational members from everybody from security, from operations, from the business, from, from, from right. But the reality is you’re still gonna wind up being somewhat centric, if you will, without the dedicated participation of everybody that we would hope to have on your normal agile team. So what we want to do is, and I started talking about punching holes in the silos rather than tearing down the silos, because in large organizations, you’re typically just not going to go in and say, hey operations, you are going to disseminate you and put you in all these teams and it’s just going to function that way. Right? So what we want to do is create a conduit, if you will, between those teams and all the other organizations and how you learn to integrate their thoughts and their processes and account for what the requirements are in terms of delivering working solutions to them. So, and I’ll just elaborate a little bit. So in terms of Lean, every, everything that we do, we have exactly two customers. And those two customers are the ultimate consumer of the product that we’re producing and whoever’s next in the assembly line, right? And so we have to satisfy both of those to be successful. And if we are missing, you know, handing over to operations when we’re done with a product or operations is having to work extra hours to support a product or whatever the case may be, then we failed in the delivery of one of those customers to end users. May love it. But at the same time we’ve created a whole bunch of, um, of rework and a whole bunch of recycling of, of things that we should have been, could have addressed from the beginning.

Dan Neumann: [09:09]  The metaphor of silos. Being a guy who’s spent quite a bit of time in the Midwest, you know, you see plenty of silos and between them, a lot of times our grain elevators. And so yes, you, you have a effectively a hole in the silo, things move from one to another and ultimately out to the final destination, whether it’s a truck or a train or whatever. And so you have that kind of the visual of yes, we’ve got the silos but we need to also make sure things move between those silos. Um, and yes, the organization maybe evolves over time, but some organizations, uh, even if they have a small it staff, we, you still need to have many different hands involved, especially as the regulatory environment comes into play where there are multiple sign offs and multiple specialties involved in rolling out a change. So I liked that punching holes in the silos versus tearing down the silos. So creating a conduit to other groups.

Barry Matheney: [10:02] Yeah, it’s just a matter of practicality, right? Because again, you know, think of, so you’re going to ask about books. One of the books that I’m reading is American Icon and it’s about Ford Motor Company. Ford Motor Company has been around for so long and they are not going to change inherently who they are. You know, that’s just now the book is talking about the transformation that they had to survive and how many iterations or they’ve gone through for that. But the point being big companies, it is very, very hard. Again, when you have compliance and regulatory and all those things, the existing practices are separation of duties, separation of concerns, making sure that you have all the validity of sign offs and all those types of things. Okay, so the question becomes how do we do that and how do we enable our teams to go faster at the same time.

Dan Neumann: [10:52]  So we’ve talked a little bit about the large scale collaborations and in team formation. So making sure that we have agreement, a team agreement in place, a working agreement in place, and for folks who are interested in that topic, there’s a previous podcast with Christy Erbeck and I’ve discussing that we’ve talked about making sure you have diversity, particularly diversity of thought on those teams so that we aren’t just reproducing some of the same patterns that have evolved and ultimately created limitations. And then we’ve talked about protecting the team from a technology standpoint as well as then making sure that part of that team agreement is to have a way to appropriately challenge each other with new ideas. And one way to do that in agile teams, you know, we value working software. So instead of arguing at a white board over a particular design, build something, build a little bit of it, build a slice, test that slice, and then of course tearing down, tearing down the silos between the teams.

Barry Matheney: [11:51] Well, and you’re not going to be able to test rapidly enough if those silos are penetrated in some way. Right. So it’s interesting that you’re talking about work agreements, because I think that’s one of the key things we’ll, you can, there are a lot of great examples of how to do this, but you can, you know, as a team, let’s assume for a second that you have some flexibility and you have somebody that’s willing to try some new processes with you on the operations team. It’s time to explore. And this time to say, here, you know, I’m not trying to change the company but I want our product to be as good as we can be, right, and I’m probably not going to change our company from this small team, but my experience has consistently been in as soon as people see the results that are capable when you operate this way, they want some of that too, right? So then it begins to just spread across the organization and instead of a top down approach to organizational change, you wind up with a grassroots approach to hey, this is a better way to work and this is a better way to do business and this is a better way to deliver working software. And we also know that the value of the software that we’re delivering is what it should be because we’ve run those tests and small micro systems, if you will. We delivered them, got our feedback and then figured out, yes, this is something we want to explore. We want to continue to go do.

Dan Neumann: [13:12]  Yeah. Let’s dig in a little bit on that. You’d mentioned micro systems. Of course, microservices is a pattern that a lot of people try to embrace to allow some of that flexibility. What are some other ways of creating that safety as we go about new development efforts, especially as we maybe have to integrate with a legacy system?

Barry Matheney: [13:32] Yeah, so I mean, obviously there’s a lot of design principles and practices and those things that we can get into, but the, uh, I think that, so I’m going to go back to my question behind the question and you know, being, uh, and I’ll call myself a recovering developer, right? Because that’s my background. Um, it is, uh, it’s very easy to fall into a space of here’s what I’m supposed to deliver. Here’s what you’re asking me to go to, here’s the path of least resistance to get you what you want. And the fastest way, that’s what agile does, right? We don’t want to go fast. Yeah. But hang on, let’s isolate and protect the scene at the same time. Right? So what does that mean from a developer perspective? That means the evolution of the developer goes from, and this is kind of a new thing that I’ve been talking about, is you’re going from getting something working to making it to where it can’t fail. And that’s a huge shift, right? So if I’m protecting the team, I’m isolating the team from those existing systems. I am making sure there’s a layer of abstraction so that if something does change downstream from us, we are not going to be impacted or minimally impacted at most. Right. Um, simplifying our testing approaches because I’ve written stubs and all of those types of things to make sure that I don’t, I’m not, uh, I’m not tied to the velocity of those external legacy systems. I want to be completely independent. So how do I facilitate that? So these are just DevOps best practices, right? Loose coupling. You talked about microservices, microservices, are it’s interesting, you can start what’s called the strangulation pattern on large legacy systems, right? And you can create a set of quote unquote microservices from your product perspective. It’s just a service that you call with a layer of abstraction that you guys have built to, to take out the technical details and the gory details and all of the legacy history that that system has associated with it. So those are, those are some of the things, right? And uh, it doesn’t have to be a technical solution. It can be very simple solution. So let’s assume you have a legacy system. How do you start to evolve that legacy system to, to apply those types of patterns? The pattern could be, you know, in a database, and we’ve actually done this before. If we had store procedures in the database, we versioned each one of those so that the old ones still existed in, unless it was some compelling business or a compliance or regulatory reason to go change it, we left it there for the consuming applications to continue to use. Right? So just depends on what your interfaces are and how you want to extract those. And that’s not a big technical solution. We did it via naming convention, right? So it’s not glorious, it’s not glamorous, but it gets the job done. And sometimes that’s what’s necessary, especially when you’re going through brownfields situation is like we’re talking about in large organizations.

Dan Neumann: [16:36]  I can honestly say, I can’t think of a replatforming effort that has ever gone well. You know, it’s like it’s old technology. We, we’ve outgrown it. We’re going to build the new system with a new architecture. But by the way, we have to honor all the business rules and the logic from the old one except for where we want it to be different. And those, that every single one’s turned into a dumpster fire. I mean, they’re just, they’re brutal. Um, I don’t know, and that’s why, that’s why I love the strangler pattern.

Barry Matheney: [17:12] No, yeah, the, and I, I can’t agree with you more, but again, I’m going to go back to this. So how we involve our business people to understand is the key component to that. Because to me, that’s not a technical issue. We can write whatever functionality and whatever solution and deliver that at whatever cadence we want to go. Uh, I can tell you that, you know, there have been systems that were implemented where just take it off the shelf and implement it. Oh, but it’s replacing this other system. So now then we have to go, you know, spend forever customizing something. Our business is so special. If there’s an off the shelf product for it, odds are is that your business is not that special. Right? And you may think that it is or there may be some nuances. I’m not trivializing it, but that doesn’t mean that you want to replicate the existing system and the new system that you’re going to, or you should be using that as an opportunity for improvement rather then, uh, assuming that it’s going to be, uh, uh, equal for equal replacement type of thing.

Dan Neumann: [18:16]  And that is why I love the strangling pattern. And you know, I’ve even seen it go so far as we were, um, re implementing a bug intentionally because that bug resulted in a number that people really cared about and making it, making it right, but different was worse than leaving it wrong, but static and um, fascinating, fascinating behavior with replatforming. So I love the, I love the strangling pattern cause it gives you a chance to gradually replace the system as opposed to taking a much more risky approach of, of a big bang. And that gets into protecting the team from all the chaos that can happen from doing a big flip of the switch to release a new product.

Barry Matheney: [19:01] Yup. Yup. And I think that, uh, one of the things that I’ve seen is if you do a little bit of planning when you’re looking at those legacy systems, then you can get really start to design microservices, if you will, to do those strangulations and then you can really start exchanging those. And Oh, by the way, so now, then that gives me a great entrance to do the elimination of that backend system. Right. And now that I’ve got the patterns in place so that I can start that microservices movement of eliminating or replacing with the newer technology or something less costly or whatever. Right. So, uh, I’ve often seen it where the implementation of a replacement system leads to the replacement of multiple downstream systems because the behaviors have changed and the expectations of the assistants have changed as well.

Dan Neumann: [19:52]  So to recap a couple of those things, you know, plan with the legacy system in mind and look for ways to kill off parts of it gradually. And it’s not just a technology piece, what you are alluding to engage with the business folks about the value of taking a different approach versus a big bang replacement effort of the old system because it’s not just a technology challenge, it’s a business challenge and we need to, you know, business people and the development team working together daily. It’s, it’s kind of a thing in the agile principles and so really embracing that and bringing that to reality for your teams.

Barry Matheney: [20:31] Yeah, and I think that again, the core behind why people are reluctant to change from existing systems and existing behaviors. Uh, don’t know. I don’t know the psychology behind the whole thing. I can’t just tell you that I’ve seen it multiple times where, Oh, I had this number of this screen and this system and I expect to see that same number on the same screen and the whatever the subsequent system is supposed to be. Right. Um, so I think that when you wind up in a place where you’re, I guess it goes back to if you think about it, when you prove the value of the features that you’re implementing from a monetary business perspective, it becomes a much easier problem to solve. Right. And until your team is mature enough to start making those strides, it’s going to become a difficult conversation with your business person because it’s just what they’re accustomed to.

Dan Neumann: [21:33]  Yes. And proving the value, showing the working software, building some trust with your stakeholders goes a really long way. Are there any other kind of large scale collaboration concepts that come to mind for you?

Barry Matheney: [21:51] So, oh, there’s a lot, but um. Um, so to me, the, the biggest key is minimize the interdependencies on any upstream or downstream system. Right. And when I say interdependencies, I mean some system that is 99 degrees away from you may be consuming one attribute that you’re producing and you’re the system of record for. Be cognizant of that, but at the same time understand that, uh, their practices should be the ones that make sure that they’re getting it in a way where you can change under the covers what you’re doing. Right? So in other words, they shouldn’t be reading your database to get it, provide it to them through a service, through an API, through something that abstracts them out. And now then you have all the freedom in the world to go do what you need to go. Um, I think that that is, that’s the biggest thing. And something else that I think that is just core to behavior. So we talked about one of the things that was the, the feature toggles in terms of microservices and in terms of feature proof, right? Um, it’s, uh, this goes back to getting business value and understanding the business value. Feature toggles is something that where when you, you deployed the software, the system that feature may or may not be on, most probably not. And then through some dashboard or if you know, feature user interface, the business person is actually the one that can operate it and they can get real time statistics what the behaviors are and whether there’s real value and adding that feature or not. Right. Um, I think those are, those are key in determining value for the features that you’re choosing. So I’m going to go back to something that we’ve talked about or may not have talked about, is there’s a lot of articles about lazy developers, right? And it’s not bad to be at lazy developer. Well, if it lazy development, if a, if a lazy developer, it looks at a problem and says, oh, that’s just one line of code in an existing system, but it’s going to change the interface. Right. That’s probably not the right approach because now that I have impacted all the testing and all the integration and all the other components of those systems, it’s, I, I feel it’s going back to the question behind the question that it’s incumbent on the team members through that working agreement to make sure that they’re challenging the technical resources to say, are we making the right decisions for the long term of the team? Right. Um, and that’s a maturation process. That’s not something that’s going to happen overnight. And each one of those teams, for sure.

Dan Neumann: [24:30]  What you alluded to there reminded me of the conversation with Eric Landes about full cycle developers where we really want the development team to have involvement in the entire life cycle. And so if you suboptimize where the Dev just makes changes to the code and they don’t own responsibility for it going forward, the right choice air quotes, right choice is to make the simple fix to that one line within the existing function changed the interface and your job is quote done and it becomes somebody else’s problem. A QA person, typically the people doing user acceptance testing, the people consuming that service to go then change their calls to it. But if you take a mindset of now I own the entire life cycle of this tool, lazy is actually taking the other approach it. So it’s, it’s more enlightened, right? So you can make the new function or new version or new version of the service and then gradually pull over the consumers, uh, in a, in a safer way. And so I think that’s where the responsibility for owning more of the system really can help drive some behavior changes. As long as, as long as I own the problem or the result that I’m making, I’m hopefully going to be more inclined to then make less suboptimized choices with it.

Barry Matheney: [25:57] Yeah. And I think that that goes back to the, if you’re talking about a large organization, what’s the probability of that developer being the one that’s going to get called if things go bump in the night? Right. So from a personal accountability perspective, as that developer, I need to put myself in that other person’s shoes. You know, we’ve always done what we’ve called warranty periods where guess, what the first line of defense was back into the product team, right? They’re the, all the help test calls were just immediately routed back into the team room. And it was meant you will deal with this before we handed over to a production. And that was your working agreement. Right? So that’s kind of the model that the Google site reliability engineers and all those types of things come into play of, you haven’t met the level of maturity and sophistication from an operational perspective. And to allow us to consume this or put it out into the wild so that it can be managed and supported at a level where it’s not impactful to the organization. Um, you know, and one of the approaches to doing that, and you know, the, the, there was a lot of conversation about, oh, you need to be t-shaped, you know, have one level of specialty. And then it was pie shaped where you have to have two specialties and abroad spectrum and then it went to comb shaped. Right. Well, I agree. And that’s kind of the full cycle developer approach that Netflix follows. I don’t know that developers and typical product teams and regular organizations, right? I’m not talking about Unicorns again. I don’t know that you’re going to get the opportunity to go get that breadth of skill set. In other words, you’re not going to rotate everybody through an on call rotation. You’re not going to make them be a DBA. You’re not going to make them or allow them, not make them, but allow them the opportunity to go play all those different roles, right? So that being the case, it’s going to be difficult to, um, it’s going to be difficult for them to develop as cone shaped skills. But what you can do is get them to learn what are the concerns for those people that play those roles so that I can make sure if that through my cone and through my technical implementations I can eliminate as much of that risk and again isolate us from uh, uh, protection perspective to make sure that we are delivering products that are going to have a great reputation in the industry. A great reputation in our company specifically because that’s where a lot of the negative impact will come is from the internal people.

Dan Neumann: [28:27]  Yes. Learning to have empathy for the other folks involved is critical. And then the other piece, you alluded to the Unicorns, they got there overtime, they have a different context and it’s kind of a fool’s errand to just tried to duplicate what somebody else has done, whether it’s a whole ocracy of Zappos or whether it’s the way Amazon does things or a Netflix or, or whatever, you know, they got there, for their own local contexts and organizations need to kind of figure out what’s going to work best internally. And that’s where I love the Kanban approach of continuously improve over time as opposed to trying to a much larger disruptive, riskier change.

Barry Matheney: [29:11] Well that’s not just continuously improving, but it’s continuous learning too. And that’s another one of the core DevOps fundamental principles, right? Your organization has to position everyone in a role that’s going to have code or a product operating in a production environment that servicing customers who are, that customer is not only do I want to continue to customer experience, I want to continue and improve the delivery experience. I want to continue and improve the operations experience. I want to continue and improve the cost base. I want to, you know, again, like you said, it’s not just continuous learning but it is continuous improvement too.

Dan Neumann: [29:49]  Very good. Well thank you for your time Barry. We would typically ask about books, but during this, uh, you’d alluded to the book American Icon, about Ford’s journey in business. And then the other one was Question Behind the Question and really looking at ways to create a personal accountability and team level accountability. So really want to thank you for that. And thank you for your time today.

Barry Matheney: [30:14] Thank you, I appreciate it.

Outro: [30:18] This has been the Agile Coaches’ Corner Podcast brought to you by AgileThought. Get the show notes and other helpful tips from this episode and other episodes at agilethought.com/podcast.

Stay Up-To-Date