Developer Control Planes: A CTO's Point of View

DANIEL: Hello, and welcome to the Ambassador Labs podcast, where we explore all things about cloud-native platforms, developer control planes, and developer experience. I'm your host, Daniel Bryant, Head of DevRel here at Ambassador Labs. And today I have the pleasure of sitting down with Nicki Watt, CEO and CTO at OpenCredo, a tech consultancy based out of London in the UK.

Join us for our fantastic discussion covering topics such as building platforms on top of cloud technologies, managing data in the associated risk when developing, and exploring why the cloud-native developer experience is important for all levels in the org chart to understand.

And remember, if you want to dive deeper into the motivations for, and benefits of a cloud-native developer control plane, or a new Kubernetes, and want to learn more in our free Kubernetes developer learning center, please visit getambassador.io to learn more. Welcome Nicki. Many thanks for joining us today. Could you introduce yourself to the listeners please?

NICKI: Sure thing. Thanks, Daniel. Yeah, so my name is Nicki Watt, and I'm the CTO/CEO at a company called OpenCredo. We're a hands-on software development consultancy, and we help clients to adapt and adopt emerging technologies to solve their business problems.

We operate in multiple areas, cloud-native development, data platforms, platform engineering are a core part of what we do. And I suppose I personally have been a techie most of my life and been involved in quite a few of those projects along the way. Although, not doing nearly as much hands-on stuff now, as I was before.

DANIEL: Brilliant, Nicki. I was saying off Michael, you and I have worked together at OpenCredo in the past.

NICKI: Yeah.

DANIEL: And we had similar roles and unfortunately we'd never actually worked on the same project together, because we were often doing different things.

NICKI: Yeah.

DANIEL: But we were both acting as developers, like what we now call SRE as well, and we were building platforms. I'd love to get your thoughts, what do you see as the evolution of platforms over the last, say five years? Have we gone from things like Cloud Foundry and Heroku, now onto Kubernetes? Have we gone better or worse, do you think in the last five years?

NICKI: It's an interesting question. I think we're still evolving, if I'm honest. I think we've certainly went from the Heroku Cloud Foundry type things and I think there was a need to get more control for some of the projects.

And so as a result, and I think with the likes of Kubernetes coming along, that provided much more granular control over some of the things and hence, the ability to, I think curate a different type of platform experience for different types of teams, which was great.

However, I think, well, Kubernetes is one of the main platforms at the moment that comes with so many knobs and you can do absolutely anything with it. But I think people have also got themselves in quite a knot recently because there's so many different ways to do it.

And so people are, in some cases, veering back towards wanting a bit more of an opinionated way of actually building some of these platforms. So I think we're in a bit of an interesting space at the moment where some people who are maybe more mature or companies more mature in their development or delivery, are quite happy with their Kubernetes setup and/or platforms, and they're happy to do that.

But others are actually looking for a bit more of a curated experience. And so there are things sitting on top of Kubernetes, I think that are coming down the line to try and help clients with that, or companies.

DANIEL: Yeah. Super interesting. That's exactly what I'm seeing as well. Where do you think folks are focusing most at the moment? Because again, talking off-mic, we were talking about code, ship and run. As developers, we need to be able to build stuff and debug and test, got to be able to get it out reliably and fast into a, say production with the ship and then running it.

You've got to be able to observe it and maintain it and so forth. Where do you think the most focus is being put at the moment in the ecosystem on that code, that ship or that run?

NICKI: That's interesting. I mean, specifically on the platforms?

DANIEL: Yeah.

NICKI: Yeah. So specifically I think on the platforms, there's a lot around the shipping and the running side of things. I think absolutely you have to build and code in order to develop that. But I think because some of the platforms are quite difficult sometimes to actually run and observe, there's a huge demand for that type of skillset and role that has come up in the industry.

So I actually think that there's quite a lot of people focusing on that and trying to make it more simple. So you just have to look at the job markets for Kubernetes, infrastructure engineers, like DevOps, that's just exploded, I think at the moment.

So I think being able to ship and get that whole continuous deployment going properly in these platforms, is one of the key areas where things are moving.

I think in terms of the run, it's reducing because there are a lot more Kubernetes as a service type-

DANIEL: Oh, exactly. Yeah.

NICKI: ... offerings and more platforms as a service type offerings, which are coming up. I mean, even for some of our clients, we used to build Kubernetes clusters for people-

DANIEL: Yeah, I remember that.

NICKI:... more in Amazon, but most clients nowadays are looking to actually use some of the built-in options that come with the cloud providers or services that are out there. But being able to actually take what's written and deploy it and do blue greens and do all of those type of things, which actually add value, I think is where the majority of innovation and the focus is happening at the moment.

DANIEL: Super interesting. How important do you think the Dev part of this is, because again, I remember back in my early Java days, it was, spin up an IDE, IntelliJ, Oracle JDev, I think it was back in the day, coding away in my little world. I could start them on there, I could hot reload and I could be super productive.

And now I've got to run minikube, I've got to build Docker containers and all this stuff. How important do you think it is that we should focus on that developer, almost like a day-zero type experience?

NICKI: I mean, I've always been a fan of people having multiple skills and being able to do everything. So it's not having completely separate roles for developers, just do the developing and ops do ops type thing, hence, DevOps obviously. But I think it's important that developers are able to understand and have mechanical sympathy actually because-

DANIEL: Oh, I love it. Martin Thompson, wasn't it?

NICKI: Martin Thompson, that's correct. Yeah. So having that understanding and mechanical sympathy for actually what's going on under the covers, means that you build and develop applications better because you understand how distributed systems work and the challenges that come along with that.

Crucially important for developers to be able to at least have sympathy for that, but ideally to be able to actually run it themselves. Or at least if you have a team that's able to take full responsibility within the team, I think that is crucially important.

DANIEL: I see. Do you see that in general with the industry, are developers being empowered and are they capable of being empowered for that full life-cycle?

NICKI: I think it depends. We have, I'm speaking from our experience as a consultancy, having gone into different clients and I think the culture has a lot to do with that and the way that organizations are set up. So in order to empower developers to be able to do that, your organization needs to understand the benefit of organizing teams in a way to actually allow that to happen.

And so I would say for organizations that have understood that benefit, yes, sometimes the developers are empowered to do that. In other cases, however, there are still silos and there's still also a desire I think sometimes with some of the larger clients to build a platform that will contain developers, if I can put it that way, make sure that they don't deviate too much and go off-piste.

And then you do find that actually developers don't have that freedom, depending on how constrained the platform itself is. So I think it's a bit of a mixed bag on that front, but certainly getting to the point where developers are empowered to do that and you just have better teams when people can do the full gamut of things.

DANIEL: It's so tricky now. I remember back from my time at OpenCredo and other places that I've worked, it's that delicate balance of, you almost want guardrails for developers. I never found the right phrase for it, but you want to encourage and promote good behaviors, like good citizens almost in the world.

But at the same time, if you put too much control over folks, they naturally rebel, because even developers, all smart people, very logical, clearly see the boundaries and want to push them. And I wonder, how perhaps with a consultant mindset, how do you go about educating the middle management or above to say, "Hey, developers, they're smart people. Let's build guardrails rather than cages or what better analogy."

NICKI: Yes. So I mean, interesting that I actually wrote an article and I did a talk on this, which is called Platform Engineering as a Community Service. And part of that is to recognize that when you're building a platform, it's not just the developers that you are building for.

It's also people like the data scientists, but also the management side of things, because they are expecting certain value to come from a platform. You don't just build a platform for nothing. So they're looking for economies of scale and the likes of that.

So I think part of this is to convince them that, or not convince them, but basically try and sell the upside, if I can put it that way, of providing developers with enough freedom to be able to still innovate.

If you are so constrained, you're unable to innovate too fast. And what we've seen in a few cases is that platforms that are too constrained and they don't have guardrails, but they've got very firm rules, actually slows people down. And so providing examples of maybe where this has worked before, where you still are able to innovate, which is where I think a lot of businesses now really want to focus.

DANIEL: Agreed, yep.

NICKI: So there's not so much on cost savings. There is an element of cost savings, but a lot of companies are trying to innovate at the moment. So really pushing on the innovation side of things, means that you have to have a different approach to the way that your platform works. If it's purely about constraining and cost savings or whatever, then you'll build it one way.

But if your goal is innovation, you have to build and structure differently. And one of the things that I mentioned in that talk as well, is that you have to have C-level buy-in for this because when you try and build these things from the ground up purely as an engineering initiative, it tends to fail.

And that's not because technically it's problematic, but it's because it also requires organizational change. And this is also something that I think in speaking with organizations, it's important to realize that there's an organization level evolution, which has to happen as part of that.

I think when C-levels and executives, they understand that, that actually they need to adapt their organization, the allowing developers more freedom and stuff is something which is a little bit easier to sell. So you've also got to sell on the organizational side of things to allow innovation to happen at the technical level as well.

DANIEL: Very interesting, Nicki. And something I've heard you mention, which I haven't actually heard other folks mention. I probably haven't asked the questions in fairness, is the data, right? And I'm thinking in terms of innovation, data is almost like oil, sort of the cliche. How do you think modern platforms and engineers work on all those platforms? How are they set up to deal with the data? Do you think it's good enough at the moment or not?

NICKI: I think that's still a whole area where people are trying to work this out. If I'm honest, I don't know that anyone has a super great answer to things. I think being able to build a specific data platform for something, is challenging but doable.

When you're trying to build a platform where you're not quite sure what you need, but this needs to deal with data and you've got different people who might want different things, I think it becomes more challenging.

And this is where I think new approaches like the data mesh-

DANIEL: Yeah, I'm thinking that too.

NICKI: ... and that side of things is actually coming out. So some of the principles that were applied in the service mesh world, we're now looking at actually, maybe we need to apply some of these on the data side of things.

And actually, again, it's not just the technology, it's teams, the way you're structured, who owns things that you need to start looking at. So I think this is an area which is still evolving, especially with the rise of ML and AI and that whole side of things.

There's a whole new level of challenges that come with trying to integrate that into an automated platform because it's not just code anymore. Suddenly you've got data and you've got other challenges.

So there's another level of requirements that not many people have got very good answers, I don't think yet in this space, but it's certainly moving in that direction.

DANIEL: Yeah. I think similar experience here, Nicki. I love Zhamak Dehghani's data mesh stuff.

NICKI: Yes.

DANIEL: I've been lucky enough to meet Zhamak through InfoQ and chat, and super interesting, but totally took away the culture, the organization, the processes, as much as almost the tech is a bit of a sideshow, right?

NICKI: Yes.

DANIEL: Which I thought was super interesting. And just on that note, do you think the developer experiences would be fundamentally different from someone building ML, AI type stuff than typical applications like our classic Spring Boot app where you manage some data or whatever, at a simple level? Do you think they're fundamentally two different experiences, two different platforms maybe even?

NICKI: Interesting. I think it also depends on where the emphasis is on the building of the data side of things. So for example, data engineering and data science are somewhat different. And the machine learning engineer, I think in a way, is trying to straddle some of those worlds.

I think you get different types of developers as well, so some that are more focused on the data engineering side of things, or they're more geared towards that. Then you have those that are more on the data science, which is more around experimentation and trying to explore data. And those are two fundamentally different experiences in my book.

The data engineer is far more focused on engineering, automation, and then your data scientist is around the experimentation. So from a development perspective, I think that there's a difference between those two.

But if you're going to compare it to, for example, like building a Spring Boot application, I think there is a difference, I think between those two because you've got different challenges.

So having to deal with data as a potentially versionable thing is very different-

DANIEL: Ooh, interesting.

NICKI: ... to having to deal with code because code is static, whereas data changes and you can control that a lot easier than you can, data. I think there's different challenges to consider. But I mean, fundamentally there's the engineering aspect to it. I'm not sure if I've actually answered that very well.

DANIEL: No, that's a great thinking voice. As you were talking, I think for me, it's a learning journey as well. And I bumped into a few folks like customers on Ambassador Labs or folks in the ecosystem echoing what you're saying.

And the next step they were saying is, continuous delivery is even harder because if you really want to test with a preset set of data like where you've got mock, stubs, that kind of stuff, it's like the actual model, the success of the model that you're building is the criteria.

And how do you manage that data? And data's got privacy concerns. Even the ship bit is equally as hard as the code, right?

NICKI: Yes, because I mean, the testing I think is one of the most challenging sides of things when it's not deterministic. So when you've got models that are coming up with their own logic for making decisions and things, it's very difficult to write unit tests for that and you can write, "This is what I expect the outcome to be."

So from a developer perspective, you might say, "Ah, I've got unit tests, I've got integration tests," and it's very set up, but when you've got this data piece in the middle, it's a little bit more difficult, I think to do some of that.

And like I say, with having to deal with things like GDPR and those type of things, developers in the past haven't necessarily had to worry too much about that. But if you are working with client data and PII type data, then suddenly you have to become aware of this and the impact of that.

That's why I think there's maybe also some tooling of things that maybe will get developed to try and help with some of this. But as yet, it's evolving.

DANIEL: Yeah. Super interesting, Nicki. Just following on from that thought, do you think the cloud has actually made it easier to support those requirements around the data, because I know for example, you can spin up a RDS in Amazon and say, "EU only," or whatever, and bolt Kubernetes cluster on for processing and you go, "As long as it stays in the EU, it's good."

But then obviously, I remember you and I doing a lot of Terraform back in the day, a lot of HashiCorp Terraform to spin up all these things. I'm sure you still do that. I still know and love my HashiCorp and Terraform.

I'm wondering, do you think things like cloud, the DB as a service, combined with even things like Kubernetes as a service, do you think it makes it easier for developers to manage compliance or is it a case of maybe what you hear today as developers have not had to think about it before, but in the cloud-native world, they do?

NICKI: I mean, I certainly think that cloud and the databases service type offerings help with that, because first of all, it surfaces some of the requirements. So you need to choose, "Where am I going to put my data?" And then you need to ask, "But why do I need to think about where I'm putting my data?"

So maybe not so much a developer, but certainly at the architectural level, you have to take these things into account. And I think some services are better than others, that's allowing you to do some of the things.

But I think that certainly does help. I think where it starts getting interesting, and maybe this is not so much purely for developers, but also for the people managing the developers to understand, even if your data is stored in some region where it's supposed to be, when you're testing or you need to bring data back and recreate it in order to test things out or build something, what happens in that case?

Are you still compliant or now have you brought the data somewhere else? And that's problematic. So I think developers haven't had to necessarily maybe think about some of these things, which I think in one way they need to, but in another way, it's also down to some of the folks a little bit higher up to make sure that they also don't land up in problems with some of these things.

DANIEL: I guess some of that can be built into the platform as well, Nicki, right? I was thinking, as you're talking there, in terms of when you have that Cloud Foundry or Heroku-like experience, I remember some of the paths that I've worked on, even internal ones, have those guidelines built into them. It wouldn't let me download data or if it did, it was obfuscated in some way.

NICKI: Yes. And those are, I think the type of processes and things that need to get built into platforms to ensure that the data doesn't land in the wrong place and things like that. So you're absolutely right that having a platform makes that certainly easier to be able to control some of that.

But again, it comes down to, do you even have those guardrails in the platform? So they need to be there in the first place for people to be able to take advantage of them, which means somebody has to have thought about this. Because I think what happens in many cases is that people just build the platform with the infrastructure and the auto-scaling and all these things. But sometimes those aspects are not thought about upfront.

And then it becomes slightly problematic later on because, "Ooh, well, now I shouldn't have this data and well, maybe it's not quite that easy to build it in later." So I think somebody does need to have thought about that and then yes, build it into the platform and that will make things much easier.

But those type of things need to be thought up as front, and that's why I say as a community, you need to think about the data side of things, what are the requirements around that and how does that influence the building of the platform and the like?

DANIEL: That is a great insight I think, Nicki. And a final question to wrap this up is, have you got any advice based on years of consulting? How do you get clients to think about that stuff? I know you and I have definitely been brought into projects in the past that are halfway through and it's like, "Uh-oh, we need to almost go back to step zero to do the right thinking rather than just trying to soldier through and fix this thing."

Have you got any advice for folks to... Or maybe books to recommend? I'm not sure exactly what I'm looking for here, but some advice to get folks thinking about all the things like developer experience, data management, visibility, how would you guide a mid-level manager to be thinking about all these things?

NICKI: So I mean, one of the things, which I like to do is to highlight risks, actually.

DANIEL: Love it.

NICKI: So especially for example, on the data side of things. You can tell people, "Oh, you need to comply with GDPR and you've got to think about making sure that your developers are not going to get into trouble." And that makes sense just when talking about it, however, sometimes that's not enough to actually convince people to actually do things.

Whereas, if you phrase the challenge slightly differently, imagine the scenario where your data has been breached and you're in the papers. It's like, "How important is your brand, how important is it to you that you're not going to land up in this scenario?"

And many companies are very brand-sensitive and that then suddenly strikes a chord where there's going to be a risk to some of the key either brand or innovation or something like that, that can help advocate for building things in a different way or going down a different path. So I think that's one of the examples of how we would do that.

DANIEL: Yeah. I love that. That's something I definitely took away from my time at OpenCredo, like working with yourself and Tariq, and other folks. Risk is such a key thing and making it visceral. And not to scare people, but just to get them thinking about it. It's really a key thing. Isn't it? Not just think about the good things. Think about the bad things.

NICKI: Yeah. I think as with testing, people are very happy to do the green path, green path testing. But it's the, "What happens when it goes wrong," type scenarios. And I think maybe even as humans, sometimes we're not always keen to dwell on the bad things.

DANIEL: Oh, agreed.

NICKI: But just highlighting it as you say, not in a negative sense, but just in terms of protecting yourself and company is something which is very helpful to do.

DANIEL: Super, Nicki. It's been a fantastic chat. Loving the data angle. That's something we haven't covered before. And I think super interesting and super important for folks to learn more about it, so that was brilliant. And lastly, if folks want to find you on the internet, Nicki, where's best? The OpenCredo website?

NICKI: Yeah. You can grab me on Twitter (@techiewatt) I probably don't tweet as much as I used to in the past, but just reach out to me on Twitter or LinkedIn or whatever. Just reach out to me there. Not a problem.

DANIEL: I'll be sure put it out there. I'll definitely reference your article you mentioned. I'm sure I've read it, but I'm going to re-read it. I'll link it in the OpenCredo in the LinkedIn as well. But thanks once again, Nicki. Really appreciate it.

NICKI: No worries. Thank you very much, Daniel. It was great to chat.