S3 Ep3: Unlocking the Power of Istio and Coraza

00:00.89
Dave Sudia
Right? when I started I see be a video ah viv and the and border there how they Carlos Yeah ah Meho File Square a being way. So I'm trying to learn for him to support him. Um.

00:10.38
Jose Carlos Chavez
Um, much just go says not.

00:17.11
Jose Carlos Chavez
Have a firsta super no para it the podcast but Cakelo Almostmo in spiner.

00:20.32
Dave Sudia
See um.

00:23.89
Dave Sudia
Grass. Yes, yeah I can't do the whole thing but welcome to living on the edge. Ah the podcast about cloud native technologies and I have with me Jose Carlos who is from Tetrate and ah from the oasp Carraza project. And we're going to be talking a lot about security and istio and getting into all these things. So um, yeah, first Jose Carlos if you don't mind ah just give everyone an introduction to who you are in your background.

00:53.25
Jose Carlos Chavez
Sure so I am Husi Carlos chavez as sta mentioned I work at the the trade as a software engineer. So my my background is mainly that I have been in in software for around 12 1112 years the last seven years mainly focus on let's say the cloud or or actually distributed systems. Um, and then I work in an observability for 5 years more or less like.

01:20.71
Dave Sudia
Sure, Yeah, okay.

01:32.29
Jose Carlos Chavez
But in a time when observability was like the early stages like when all this crazy stuff that we see now were like theoretical. Um and the losses. Yeah I mean most of the change happened in the bills now right.

01:33.42
Dave Sudia
Yeah I.

01:39.92
Dave Sudia
Has been a lot of change in the last in the last few years

01:48.33
Dave Sudia
Yeah, yeah.

01:51.49
Jose Carlos Chavez
But lot of lot of changes and lots of new features and that back in the time where like I remember talked a lot of sampling and now somethingpling is something meaningless because of everything so like 100% collection and all that. But yeah anyways, um, and then.

02:05.96
Dave Sudia
Yeah.

02:11.70
Jose Carlos Chavez
I move into security incidentally actually because I I move it into a company which was observed security on top of observability and so I joined it because of my observability background that ended up mainly working on security features and.

02:16.32
Dave Sudia
Okay.

02:27.83
Jose Carlos Chavez
And then yeah I move it into the tray. So this last three years I was mainly working on security.

02:30.87
Dave Sudia
Cool. So yeah I think you know I I only just learned about the os karrazza project recently. Ah, we built it into our um, we we used karraa and built it into a waff in our ah Api Gateway Edge Stack ah that was on top of emissary ingress and. It's really cool I mean you know I started I have had to learn. But as I was demoing it I had to learn sacklang and and kind of get into the the meat of it and so I just wonder you know what's been your involvement in karraa and ah, what are the things that you think are most exciting about it.

03:05.56
Jose Carlos Chavez
Yeah, so as part of one of the features that we were delivering with the trade in our in order to trade service bridge product. 1 of them was waff like the webpification firewall. So waft of what out there were mainly a.

03:18.93
Dave Sudia
Um, yeah.

03:25.32
Jose Carlos Chavez
And proptory waps like a WWaa or cloudfre waf um, that have a different integration mechanism what we were looking for was something that you could integrate not only at the ingress level but also inside the network.

03:26.90
Dave Sudia
Um, yeah.

03:39.85
Dave Sudia
Um, sure.

03:42.40
Jose Carlos Chavez
To be able to deliver the the full zero trust experience and this is how like we we got involved with into koassa we we became contributors then we ended up colleting the the project and what what we saw in Koassa and was that.

03:47.56
Dave Sudia
Okay.

03:53.93
Dave Sudia
Okay.

04:02.11
Jose Carlos Chavez
Mainly like the the facto standard for for waf for open source waf um, some years ago. What's more security More security is is in sunsetting and then other attempts to do him more security like waff were basically depending on.

04:17.99
Dave Sudia
Ah, yeah, yeah.

04:21.91
Jose Carlos Chavez
Different components of more security or following more security and and what we were looking for was something completely different right? something with a different focus focus on the Cloud Um portability was important because what we are looking for is like you can roll out. Um.

04:26.68
Dave Sudia
Sure.

04:38.63
Dave Sudia
Okay.

04:40.36
Jose Carlos Chavez
Organization-wide policies and in your in your systems so we ended up in Koraa which is which was kind of in in in a similar mindset about Theco for Mo security and and started with a own heart. So We also shape it. With this zero trust and distributed system mindset we for us from day zero. One of the most important things were performance in hyropl because we always had his mission that they could runrasa everywhere. No only at the and the English level. So if you were putting this waff in in the critical part. It should be.

05:00.78
Dave Sudia
Um, yeah.

05:17.30
Dave Sudia
Ah, so so you're integrated into the mesh as well. Yeah, okay, cool right on.

05:17.30
Jose Carlos Chavez
Super fast and also with lot of concurrency. Yeah yeah, exactly so this we got involved with and then we started add in more features. The main goal was to pour these into webassend or to land.

05:34.50
Dave Sudia
Okay.

05:35.50
Jose Carlos Chavez
And callra into webassembly so we could use it as well as a wasn't pluging and course wasn't filter in envoy let's say and and and then because of the trade has a lot of work in webassembly and we also maintain the runtime which is was was 0 we also got also a good synergy on that and ended up with ah with a really cool project which was kuasa proxy was and that you can run a nomo istio kong now and that that was mainly how we involved. This and and and this is how we we ended up in in karata we ended up doing a a really good product or let's say have a really good hold outcome and now it's getting a lot of traction because people is looking for a waft that can be that is cloud first.

06:24.41
Dave Sudia
Urban. Yeah.

06:32.58
Jose Carlos Chavez
Is open source that is portable and there are efforts to Port Kodasa into traffic into like a month ago it was released also koraa for Ais and so yeah, it's getting a lot of Hype.

06:46.57
Dave Sudia
Okay, and it's kind of becoming the standard which is I have to imagine that's just really fulfilling ah to have worked on something that becomes the thing that everybody uses you know.

06:51.44
Jose Carlos Chavez
Now.

06:59.24
Jose Carlos Chavez
Um, yeah, also we just got the the production flag shift from a wasp in in that or wasp itself recommends using Korasa in production which was also a good milestone.

07:03.88
Dave Sudia
And okay, yeah. Awesome! So what? What do you think is the driver Beyond just being Cloud Native like what are the features that you think are creating the situation where everyone's like yep, that's the one where that's the one we're going to shift to and adopt like.

07:19.24
Jose Carlos Chavez
Um, so.

07:24.54
Jose Carlos Chavez
Um, yeah, so so basically the fact that you can distribute Koassa inside your your network withinless integration not needed to and to modify and the application code because then you you make the problem of the of the.

07:41.89
Jose Carlos Chavez
Application developer but saying like okay we will roll out Koassa in a sidecar and then we we deliver the policies we delivered like at organization wide level and you don't need to do this? Um, let's say the this inclusion on every single application that is what.

07:45.53
Dave Sudia
Um, sure.

08:00.87
Jose Carlos Chavez
But for us was the most valuable. Let's a feature in when deploying this in the in the cloud. But also the fact that you could define this at other constructs like not only um at the sidecars but ingress but services and all that now that they. The getway api from Kubernetes is also raised in like and the fact that we can see that how these can match in all these near constructs is the tell us that we we took the right decision also as I said the the performance and and the the.

08:24.53
Dave Sudia
Yep.

08:39.97
Jose Carlos Chavez
Ability to to deal with high concurency was also important for us because yeah, if you if you aim to run these on every single worldload or or a considerable subset of all the workloads like you need to wear on D performance because otherwise it's going to be and.

08:42.11
Dave Sudia
Yeah, because that's.

08:58.96
Jose Carlos Chavez
It's going to be a like penalty in Latency ah meow.

09:02.32
Dave Sudia
Yeah, no, 1 ne's going to introduce a bottleneck into their network just just for the security. Yeah, so that's a familiar story to me as you said, um, you know oh the thing that really triggered for me is saying and you don't need to change the app code. Ah, and so I wonder. You know I mean obviously this was probably this was an intentional design choice. But I wonder. Ah, if that was intentional design choice if the people working on this coming from the observability world right? as you as you did ah if that influenced it because the thing it reminds me the most of is the open telemetry project right? and. And so as a s and end user going through that journey over the last six years you know I remember just just generally I remember going like wow this envoy thing is cool I'll learn to deploy it. Oh this is a huge pan like I remember I wrote my first envoy api implementation right? as they switch to the v 2 and then I just finished writing that when like istio first came out right and at one point I did this talk called if you can wait six months you should ah because it was sort of like there's so much cool stuff right on the horizon that just don't if you can if you can hold off on implementing just do because there's there you know people like you are building amazing things. Um, and and so you know even within the last year and a half ah like you know I remember trying to convince four years ago everyone to ah the open tracing is coming out wait. Nope now. It's open telemetry right? Um, but then but now you have the open telemetry operator where you just annotate.

10:36.58
Dave Sudia
Something and it injects open telemetry and you know into into as many runtimes as it can right? I think it's node python Java ah, and z sharp um, but but yeah I mean it's one. You know that that was the that's been the theme over the last five years right is like don't make developers. Do this stuff.

10:55.25
Jose Carlos Chavez
Yeah.

10:55.93
Dave Sudia
Is if we can if we can take it out of app code if we put it in a sidecar if we can just remove that concern. Um, and and so my last position I was of Cto at a little nonprofit and it would like the promise was there right? like my developers did not have to learn any of that stuff because like no, we just deploy it and it does all of it. Right? Um, you deploy the new Relic operator and it goes out there and it does its thing and you know Yeah, um so yeah I mean ah, how much do you think that experience and observability influenced that design.

11:30.79
Jose Carlos Chavez
Yeah, so it's mentioned it's funny what you mentioned so just for context I am the original outer of the open tracing library for php and one of the contributors for the for the goal 1 Um, and then when I joined this company that I told you.

11:38.53
Dave Sudia
Okay, well I monitored.

11:48.33
Jose Carlos Chavez
Um, doing this work and Observ security and top of observability. Um, as part of my work I was delivering an agent which was based on sipkin go open to themetrical like a by the time we had a.

11:49.12
Dave Sudia
Ah, yeah.

12:05.41
Jose Carlos Chavez
Ah, breaking change into this in open telemetry every day so it was kind of hard to maintain. But also one of the cool things that we were doing because we were saying like now that we are already in the in the in the application now that we are already bedded today in today's the component.

12:06.55
Dave Sudia
Um, yeah.

12:21.14
Dave Sudia
Yeah, all of it.

12:24.70
Jose Carlos Chavez
Why don't We also run security things like blocking like analyzing not only observability and and that idea was brilliant because we said like okay when we deliver this instrumentation. We also be able to do blocking and and um like tread analysis.

12:39.99
Dave Sudia
Sure.

12:42.93
Jose Carlos Chavez
Right in the in the incoming traffic but then with a myriad amount of things that were out there in the customer's deployment like the the like the crazy number of types. But of of ways people build applications I Remember we were like ended up writing different kind of um instrumentation for different ways of building an application like it was honestly a pain.

13:01.27
Dave Sudia
Ah.

13:13.86
Dave Sudia
When you get into someone's code. You're essentially writing On-prem software. Yeah yeah.

13:16.75
Jose Carlos Chavez
Because yeah, and and we were like yeah we we need to implement this because that customer that is very important and I of disclosing the name is running or building their software like this and you cannot tell them like oh yeah, the the problem here is that your.

13:25.80
Dave Sudia
Yeah, he.

13:34.73
Jose Carlos Chavez
Doing this but you shouldn't be doing this right? You cannot correct the way people are the way customers build their software and we we ended up writing all kind of integrations to fulfill customer needs and then when I move it into the trade.

13:45.35
Dave Sudia
And I met earlier a spot did 2

13:51.32
Jose Carlos Chavez
And I remember I brought a blog post about building common libraries for inside organizations right? to so you can have everything standardized it and then this process of a plugin instrumentation. Observability security will be and easier and cheaper. Um.

13:56.61
Dave Sudia
People. Yeah.

14:06.72
Dave Sudia
Sure.

14:10.68
Jose Carlos Chavez
Back in the days out to instrumentation wasn't a thing. There was some work from datadog inside open delimit to deliver day out to instrumentation Java agent. But just that right goallan was not in the roadmap a python was as well something at that time.

14:19.90
Dave Sudia
Fired. Yeah.

14:27.53
Jose Carlos Chavez
So then when I moving into the trade and started working on service Mesh I was like Wow we can do do everything in the sidecar. Why are we fighting with all this crap when we can do everything in the sidecar of course having sidecars has other complexities but like the idea of rolling out everything just like this just but adding an antation or.

14:34.87
Dave Sudia
Yeah, yeah, sure. Yeah.

14:46.94
Jose Carlos Chavez
Or applying a manifest that was brilliant and and I think ever since I work in on observability and my aim was like you could easily introduce these kind of features without with 0 over overhead or minimum overhead because I I was always obsessed with.

14:50.70
Dave Sudia
Um, because I have 3

15:01.11
Dave Sudia
Yeah, yeah.

15:06.75
Jose Carlos Chavez
The concept the the observability no the observer effect right? And which means that by observing the the phenomenon or the experiment you change the outcome right.

15:08.80
Dave Sudia
Sure.

15:15.28
Dave Sudia
Yeah.

15:19.58
Jose Carlos Chavez
For example, it happens as well in software right? when you introduce an observability middle where you're basically introducing overhead or latency and by that time the numbers are not necessarily what you will or what what will be without the the instrumentation.

15:20.88
Dave Sudia
Um, yeah.

15:27.23
Dave Sudia
Yep.

15:31.74
Dave Sudia
Yeah, without that overhead.

15:36.71
Jose Carlos Chavez
Yeah, so I was always obsessive with that like how can we roll out these without needing to to change the code and and also without the need of people to to always be applying these policies all the time but we just roll them out and that. This is how there was a perfect match with what we were trying to achieve with Qra and proxy Wasson because we want to do exactly that like roll out this waff without user involvement or not without just involvement but without the user to go into the internal sub fabrication just put it in front.

16:13.90
Dave Sudia
Sure and let it do its thing. Yeah, um so I think on that note like what are some things that you would consider best practices I think my my personal experience with any kind of security.

16:15.40
Jose Carlos Chavez
And we allow the policies.

16:30.50
Dave Sudia
Feature and you know this ranges from adding a complex waff in front of something all the way down to just installing helmet into your express app ah is that you immediately break a bunch of things because most people have not designed their application to be secure enough to not break. When you do the when when you introduce a thing that actually blocks all the bad stuff. Ah so so yeah what? what if people were going to go implement Karraza you know in whatever product they use as it's being adopted at all these things. What are what are your some of your recommendations.

17:05.84
Jose Carlos Chavez
So first of all I would say like it's important to be mindful what you're introducing like there's this example where you say like okay I put this I put a waff in front of this application or inside this application and then I roll out different kind of of. Is that my or might not be used so you you first need to identify what are the risks in your application is my application connecting to a database then yeah I probably need to to protect myself from injection is my application receiving receiving but's say internal traffic or.

17:27.51
Dave Sudia
Sure yeah.

17:42.60
Jose Carlos Chavez
Synchronous traffic then I probably need to inspect the thetp request is my application running on Ph B then I probably need to enable the the phpus. So it's not just putting a blind choass and say like I will put everything but because I will feel more safe or more secure because that's.

17:49.50
Dave Sudia
Um, and I don't know I'm going to talk about.

17:57.47
Dave Sudia
Like ah true.

18:00.28
Jose Carlos Chavez
That's not going to work. You're only adding latency. It's important to identify the risks and risk involve. What are the the possible things that go wrong and what is the likeliness to happen right? And what is the impact as well. So I would say these kind of things you have.

18:08.66
Dave Sudia
Sure.

18:17.98
Jose Carlos Chavez
You should be aware on before putting any kind of security measure in front of your application. Once you have that clear then you kind of start rolling out and applications. But and the fact that you will um, put in another piece. Or another. Let's say layer in front of your application mean that different notcons could happen that wouldn't happen in your application in Banila So resiliency in your network is also something that is important. There is actually.

18:44.15
Dave Sudia
More about sorry sure really.

18:52.44
Jose Carlos Chavez
This topic called server security resiliency which talks about how your application work react in unknown in centers right? because the Knownna centers are kind of you already hud in but the Knownna centers and in in that sense him.

18:53.38
Dave Sudia
Okay.

19:00.80
Dave Sudia
Okay.

19:11.18
Jose Carlos Chavez
Whenever you put security measurements in your in your um, ah security measures in your in your network or in your system you should be mindful about. Okay, what are the the reciliency patterns that I'm also following because these things could fail and if they were failing in in. X number of ways now it will fail in X multiply it by n number of ways. So I should be prepared for that.

19:31.73
Dave Sudia
Okay, so what does that look like in practice like if I was going to and go do 0 security resiliency does that mean at you know, just because this is that's a new term for me. So does that mean adding.

19:46.85
Jose Carlos Chavez
Is.

19:49.45
Dave Sudia
Redundancies into my security system because that then implies adding extra extra latency as you brought up right? like what? what is Ah, what are the what are the practices look like if I'm trying to do that.

20:01.62
Jose Carlos Chavez
Right? So so basically what you want is you should be or your replication should be able to manage retries first typical resiliency patterns on on networking. Um you should be able to to handle retries. You should be able to handle and.

20:16.32
Dave Sudia
Yeah, sure.

20:18.47
Jose Carlos Chavez
Nonhay path and statusus scope right? Whenever if my application is giving me ah a 4 tree or a fioff tree. Um I shouldn't crash or I shouldn't like I should be able to mitigate that in because you don't know when the attacker is in right.

20:28.75
Dave Sudia
Sure here right.

20:35.45
Jose Carlos Chavez
But you don't want to also like and take down all the system just because one of them corend is fail because that could be my application protecting itself from an attack and I don't want to turn down everything so being able to operate in degraded mode having ways to.

20:43.73
Dave Sudia
Sure right.

20:54.16
Jose Carlos Chavez
To retries and having ways to ah yeah, operate thegraded mold one one once one of my dependencies and fails having a cash front having a way to invalidate the cash in front as well. That's also important.

21:06.32
Dave Sudia
Yeah.

21:09.51
Jose Carlos Chavez
And also and and one of the things that are very underrated. What is probably the one of the most important things is to be able to get feedback from the monitoring right from the outed logs from the Logs analyze the metrics um find um.

21:19.49
Dave Sudia
Sure.

21:26.65
Jose Carlos Chavez
Anomalies or detect anomalies him look at the threshold holes like your application is usually trying to tell you something when they fail and you should be able to to listen to that and take measures according right? Some people just. Plug the waff and don't care about whether the added logs being produced and not analyzing the outd log to see like okay what is happening. Maybe my application is just trying to protect itself but then and it's a neveranging situation because then I don't do anything special and then just let it go and say like okay.

21:44.47
Dave Sudia
Hair. Yeah.

22:02.11
Jose Carlos Chavez
I'm fine for 5 or 3 But then there are different sorts of attacks that could happen. So um, why don't you block that ip why don't you analyze whether the the user agents that are happening. Why don't you take all these step forward to only prevent because um.

22:10.77
Dave Sudia
Yeah.

22:20.75
Jose Carlos Chavez
1 of the the things that happened in cybersec security is that um people tend to think that a good metric is for example, number of attacks rejected or numbers of of injections caught. But that's not important I don't care how how do you measure whether that number is.

22:27.31
Dave Sudia
Yeah.

22:39.90
Jose Carlos Chavez
High or low or or good or bad right? What you care is okay I have from this ip I have this number of consistent and file 3 or bouncing and then at some point it stopped what happened did that attacker go into the network.

22:51.24
Dave Sudia
Right.

22:56.70
Jose Carlos Chavez
Like this is the important thing now how many of ah how many of them I rejected like exactly exactly So so the metrics these metrics are are meaningful I would say and in general monitoring is underrated in terms of security bodies.

22:58.79
Dave Sudia
You can block 95% of them but the other 5% took all your customer data. Yeah.

23:14.19
Dave Sudia
So that's that's another interesting thing like connection for me is from observability to security then is I I think I'd love to get your take on this is that people who come from the observability world are used to visualizing things right to seeing to seeing the the Network map.

23:15.60
Jose Carlos Chavez
Really crucial. It's fundamental.

23:29.14
Jose Carlos Chavez
Yeah.

23:33.56
Dave Sudia
Right? to to as you you know like to seeing the graph suddenly drop and I remember the the number 1 security thing that has ever stuck with me was at kubecon San Diego 2019 Ian Coldwater did a keynote and Ian was it was all based around the untitled goose game. And Ian was saying I remember that this piece is that ah like white hats think in checklists and black hats think in maps and that's probably not quite a direct quote but it it was this idea that you know the the people who are in charge of compliance. Just we install a waff check right? You know.

24:05.83
Jose Carlos Chavez
To stick to this.

24:12.69
Dave Sudia
And and the reason that Ian was recommending ah the entitled goose game was this idea that's like you get into a place and then you kind of go where can I get from here. Um, and and you saying that just now was sort of that it was like why did this ip stop stop pinging right? Just sort of this and ah. Yeah, and and you're you're talking about. Okay I saw this graph drop off I saw you know the the network map change right? I see the the the anomalistic behavior. Um I think there it was just yeah, what? What do you think of just this idea that like. Visualizing is really important to security.

24:53.40
Jose Carlos Chavez
Yeah, because in the end and people doing security. Ah, as of now 2023 is people right? and and you need to so you need to see to be able to see and understand what is happening there is there is something I always say.

24:58.22
Dave Sudia
It sure.

25:09.94
Jose Carlos Chavez
In real life and I keep saying to my to my kid. It's like if something works don't change it and and that's true in software but not so true because things that were and vulnerable yesterday can be vulnerable today with you doing anything.

25:10.35
Dave Sudia
And the worst part of that that I worn at that.

25:24.11
Dave Sudia
Sure.

25:28.61
Jose Carlos Chavez
You you change it anything and the same path was vulnerable I mean it that wasn't vulnerable a minute ago. It is vulnerable now because yeah, exactly so um, in that sense, you cannot take compliance as ah as a.

25:33.12
Dave Sudia
Because the thing that changed is time. Yeah.

25:46.81
Jose Carlos Chavez
As a state right? It should be Ah, it's a a life thing. It's an ongoing process. You don't say I am compliant. Yeah, you can be compliant at this point of time. But next second you can be not. You can be noncompliant anymore right? So yeah, what you say about checklists? yeah.

25:49.24
Dave Sudia
Here? Yeah yeah.

25:55.90
Dave Sudia
Yeah.

26:06.73
Jose Carlos Chavez
It's it's one of the in general security like thinking of security as a property. It is not the right mindset the the right or the mindset that works as of now given the the number of threadads and the number of Cvs and. And the numbers of data rich is that you should consider security of something dynamic a process like not I'm security I'm secure now but like what security measures I'm taking now what are my risks at this point of time like what what is the likeness of this to happen.

26:28.96
ambassadorlabs
Oh you both froze.

26:37.68
Dave Sudia
Yeah.

26:44.41
Jose Carlos Chavez
There was a time where for example with wafts what you were saying like yeah I need more security I will just buy another waff and put it in front and and forget about it because yeah I have the waff right? What could go wrong and that's not anymore and reasonable because. Now you can have attacks from the inside from the outside right? you could have you could be already Compromisero it even though you both a waff right? That's not going to change the fact that someone needs already injur your network. Maybe so this is why I also and got a lot.

27:02.53
Dave Sudia
Yeah, yeah. Yeah, yeah, yeah.

27:20.75
Dave Sudia
Yeah.

27:20.76
Jose Carlos Chavez
Got cowed with a zero trust concept right? when you don't you'll want to have zero trust doesn't mean like you don't trust anyone 0 trust means that you have 0 implicit trust right? You don't trust anyone by default and you you need to very you need to make the the conscious decision of trust in something and then.

27:28.26
Dave Sudia
Right.

27:39.93
Dave Sudia
Yeah, right.

27:40.64
Jose Carlos Chavez
Having that in mind every time right? someone one was asking me the other day like in syotruus. How do you trust the sidecar. How do you know that the cyclecar won't be vulnerable and makes sense right? Yeah, the sidecar you trust the cycle. Basically you can just everything Theycars gives you and.

27:49.32
Dave Sudia
True. Yeah.

27:58.21
Jose Carlos Chavez
And it could be vulnerable but you make the choice of trusting so you keep an eye on what are the vulnerabilities on Sidecar right? But it's a conscious decision. Yeah exactly Yeah, exactly. So.

28:02.76
Dave Sudia
Um, well because you have to trust something at some point yeah to to verify a public key. You have to trust the private key. Yeah yeah.

28:15.47
Jose Carlos Chavez
Yeah, this is I would say and explicitness is the key concept here right? Everything you do everything? you trust should be explicit otherwise you will end up in blind spots and then that would be 10 years of course

28:32.56
Dave Sudia
Yeah I like that a lot. That's that I think that's a take on that I hadn't heard before that I I really appreciate. Um, the other thing that was kind of connecting for me as you were saying that is so I'm not ah, this is now a personal opinion. Ah I'm not particularly bullish on. Ai insecurity. Ah because every time I've tried to implement there was ah Austin who's now at Honeycomb did this great talk a while ago kind of saying like the issue with this is that you know what Ai is really good at seeing is ah. Highly repeatable patterns right? like you have to train it over and over again with the same thing and I used to work for company that did machine learning for visualization and you know we had to be like yep, that's a Pepsi can like 17000 times right to get it to recognize Pepsi can yeah and ah and.

29:09.68
Jose Carlos Chavez
Is.

29:22.62
Jose Carlos Chavez
Um, in different positions right.

29:26.51
Dave Sudia
And a security event is well an an operations event right? An incident regardless of whether it's security related or not is almost always an anomaly and so I think you know the systems are kind of good at saying like this number is an outlier but but terrible about deciding what to do about it.

29:41.48
Jose Carlos Chavez
Yeah, it's.

29:41.91
Dave Sudia
Right? Like when they try to set up like sell workflows that are Ai driven I'm kind of like I don't believe that. Ah, and ah, yeah, and so anyways, it's just one of those things what you were saying you know about again about kind of seeing those anomalies and um. And it being a constant like it not being a moment in time right? You have to or you you may be Compli. You may be compliant in this moment of time but you always have to track it and follow it and and stay up to date. Ah.

30:17.80
Dave Sudia
Yeah, that that just kind of stood out to me again as one of those things of it's like it's It's a very human job security right? because you have to intuit to a certain extent and and and use your knowledge of the system to try to find those edges and where the edge cases could be yeah. Ah.

30:22.70
Jose Carlos Chavez
My.

30:33.60
Dave Sudia
Any any thoughts on that.

30:34.84
Jose Carlos Chavez
Yeah, so there is this very interesting and drawing well not drawing with the image about the second world war the second war war the planes yeah that's called the the survival bias because.

30:45.96
Dave Sudia
Um, with the plane. Yeah yeah.

30:54.75
Jose Carlos Chavez
Yeah, looking at the part of the patterns. Yeah, which are like every bullet was a dot right? saying that. Okay, these are the most common ah targets for bullets and then what should we do like the the natural conclusion is I should reinforce those right.

30:58.54
Dave Sudia
Yeah.

31:07.39
Dave Sudia
Yep.

31:14.18
Jose Carlos Chavez
But then um, it was noted by someone like okay we we don't be reinforcing that because that means that the plane can still go if yeah, exactly So so they are not important but very important that what happened with those of the crushed right? and this is something like.

31:19.39
Dave Sudia
The plane made it back with those holes. Yeah.

31:27.50
Dave Sudia
Yeah, yeah.

31:33.72
Jose Carlos Chavez
People Sometimes ask like how are the waves powered by machine learning versus the the wives based on static rules like Koasa for example and this is exactly the the kind of thing you should be looking at because you need a lot of training for the model to be able to identify an old liar and and and and normallyly.

31:38.76
Dave Sudia
Um, yeah.

31:46.58
Dave Sudia
Yeah.

31:52.49
Dave Sudia
Right.

31:52.76
Jose Carlos Chavez
But sometimes you don't have that much of Data. Sometimes you just have a few attempts and then you need like to consciously understand like okay this is this could be a potential attack What happened next? What? what is entire sequence and then at some point is it disappear. Okay, it disappeared because it. It gave up it disappeared because it actually got into the system and now is is doing other kind of frequents or you should correlate look at okay for this pod then I will not only look at http traffic. But now I will look at what are the syscode that is doing because maybe it's trying to curre some other dependencies or some other.

32:26.10
Dave Sudia
Yeah, if we could train the ai on a Cd it wouldn't be a Cv we we'd know what it is yeah.

32:30.14
Jose Carlos Chavez
Pieces in the in the system. Yeah, exactly exactly exactly? Yeah, so so yeah I think it's a long way I'm ah an expert nor anist or Ai and.

32:45.29
Dave Sudia
Um, sure. Yeah.

32:48.72
Jose Carlos Chavez
But I think at this point and people working in security still relevant.

32:52.77
Dave Sudia
Yeah, yeah, um, the other thing that your your moment you know it's you're compliant at a moment in time made me think of is I Still do some volunteer work for that nonprofit and we the other day. Ah pods just stopped blowing Correctly, they just wouldn't start and ah and I think it's There's this tradeoff of. Insecurity of staying current versus pinning to a version right? So The the thing you could say this was a good practice or a bad practice. We. Ah we used Heroku build packs there and ah because they're much very small team and I made the decision early on like. I am not going to be able to write a secure container as well as Heroku can so you know when the build pack comes out. You can only run very certain commands right? Everything is very locked down within the container I think this is great and ah and so we we just we were eliminating variables for about 4 hours and finally realized that the problem was that the node build pack had updated. In some way that did not run in that cluster. Um, it was like I could run that container locally. We could run it in another cluster like there's something about our aks cluster and that is just yeah I would we still have to isolate what actually happened with ah the the thing we resolved that day was we pinned to a belt pack version before the.

33:50.26
Jose Carlos Chavez
Is just.

34:06.59
Dave Sudia
The major release but but I think that's the other thing that you know as we talk about like moments in time that you're compliant versus keeping up to date with things right? There's so much and human effort right? There's so much human effort that goes into analyzing this um, staying up to date. But sometimes the updates break things right? You know I mean you're talking just before about pushing a breaking change to the open telemetry go every day right? Um, yeah I. Ah yeah yeah, know I know yeah I mean it's it's hyperbole. Yeah yeah I Guess not taking you literally. But.

34:36.60
Jose Carlos Chavez
Ah, maybe it wasn't every day. Maybe every second second day every second day here.

34:43.97
Dave Sudia
But we've all been there right? We all know what that's like um and but I think and and again I think the time frame you're referencing we were. You're trying to figure out what it was supposed to be right? I mean that's that's not unexpected. Um, but I think the the point I'm getting to is you know like.

34:45.12
Jose Carlos Chavez
F.

34:56.20
Jose Carlos Chavez
Um.

35:01.65
Dave Sudia
How do you think about or how do you deal with that churn of trying to keep up with what you have to do to be to be secure.

35:14.30
Jose Carlos Chavez
Yeah, um I would say and insecure like unlike in life in life I Like to think that less is more insecurity less is less and more is more like the more layers you have of protection the better.

35:20.24
Dave Sudia
Yeah, true are okay.

35:31.11
Jose Carlos Chavez
Um, and no matter whether they overlap whether you believe that they are redundant insecurity redundancy is better. Um, so for example, the other day I was talking about with someone and that someone was saying like why do I need access contra. For example, if. Every application is in charge of a verifying ah a yot. So do I really need to to protect myself from from a Http request not containing a jot because yeah, if no jot I will just I will just deny and.

35:59.73
Dave Sudia
Everything requires one? Yeah sure.

36:06.49
Jose Carlos Chavez
It's not a problem but and then I was like but why if that jot library has a bug and ah and Cb is is closer than and it exploited right? Um, and and this is exactly what happens in security you have different layers of hand protection. You have mental tlas. You can have a fine grained. Ah. And access control policies. You have the waff then you have the actual authentication then you have the logic which verifies can this and subject identify. It can perform this or this action and all that so in security everything is. Everything at value right? Every every other measure that you implement out value. Of course it can bring some other problems right? As you mentioned that you could do an upgrade and then and it will and turn down your application because of. Something in the kernel or whatever and there is also security measure for that's right, if if you think about? For example, there is this concept of lift and shift when you have a legacyus application that you don't want to touch but you want to put in in the cloud and you want to to remain secure. So what do you do.

37:03.29
Dave Sudia
Um, yeah.

37:20.24
Jose Carlos Chavez
Well you put it as it is in a beerton machine and you put a waff in front. You don't touch it. You don't need to build it anymore. Maybe you don't even know how to build it. It's just a thing that runs and so so there are ways to overcome all these problems all the time just different strategies. But.

37:33.20
Dave Sudia
Yeah.

37:39.54
Jose Carlos Chavez
Whenever you think oh this effort is not worth. It's worth I can guarantee that so like maybe not yeah, maybe not the way you think it will be like maybe the things weren't the way you thought like they were in a cs.

37:46.65
Dave Sudia
Ah, da you at some point.

37:58.35
Jose Carlos Chavez
Like the promise of you just deployed this in the kernel and it will work. But it's worth is worth so so of course when you start ah with a decent or modern style. Everything is easier because you have standardized of things.

38:03.60
Dave Sudia
Yeah, yeah, on that.

38:17.50
Jose Carlos Chavez
With libraries with Frameworks and but yeah, it's always worth like and don't give up I would say that will be my my advice.

38:17.38
Dave Sudia
Sure yeah.

38:25.57
Dave Sudia
Yeah, no that yeah that reminds me again just of another you know practical story is ah we got at the et Theon Private. We had a sms verification and we ended up getting hit by this group that was doing some sms scam I Still don't even fully understand it. But. Humans are interesting. Um, so they were they were hitting us repeatedly sending text messages to as far as I understand random people. There was some some profit to be made in it and if anyone wants to reach out to me when they listen the podcast and explain it to me I'd be great I'd love to hear it. Um, anyways. So we had cloudflare in front. Ah the waff.

38:44.85
Jose Carlos Chavez
All that.

39:03.23
Dave Sudia
and and I thought this is great and so you know we rate limited the sms system and so that slowed it down for about a half an hour and then they got a botnet and started hitting us every single request came from a new ip and cloudflare has no ability or had no ability at the time to do global rate limiting. They.

39:22.44
Jose Carlos Chavez
Will hit.

39:22.95
Dave Sudia
Just had rate limiting for ip so that was ah I had edge stack the ambassador product installed but we were pretty much just using it for for the emissary ingress just for the routing capabilities right? and that was the first time I ever turned on the rate limiting feature because you could just turn on way. We had a global rate limiter and because again I was like. Why am I going to need 2 rate limiters. You know, um and that was the day that we needed 2 rate limiters. We we needed one per ip address and 1 to just be like nope you can't cost us $7000 a minute in sms fees. Ah so um, yeah, no, that's that's a great that's a great point.

39:45.71
Jose Carlos Chavez
Um, yeah, exactly. Um.

39:56.38
Jose Carlos Chavez
Yeah, and this is back to what I said about I'm not saying but machine learning basic protection is bad or not good or not all I'll just saying there's no so Silver bullet for this right? So so.

40:05.49
Dave Sudia
Um, yeah.

40:09.33
Dave Sudia
Totally yeah.

40:15.22
Jose Carlos Chavez
If I was operating this I will have one wa with and well-known standard rules like core rulesset and if I want to to double check I would have a machine learning waff and of course make it one. The source of truth because otherwise.

40:32.35
Dave Sudia
Are you? yeah.

40:32.57
Jose Carlos Chavez
A lot of latency but like yeah and as I said more is more so so it's a word to have as you mentioned like why would you have to rate limiters That's ah, a really interesting and take and it's because you don't know how how can you Be. Ta and this is part of the cyber resiliency that I was talking before like being prepared for because sometimes you can find a tre Sometimes you cannot so what do you do? I will probably just close myself right and put into into the turtle house and and not accept any any connection.

40:52.57
Dave Sudia
Yeah, um.

41:09.54
Jose Carlos Chavez
Or been able to have a way to only accept income internal traffic or there are several kind of measures you can take and but is not what is not about what measure you are going to take is that are you prepared for this.

41:13.63
Dave Sudia
Um, yeah, yeah.

41:22.75
Dave Sudia
Right? Yeah, yeah, that's a really good point. Um to shift a little bit so tetrate builds on Istio. Ah for just little background for anyone who's not familiar with istio which is unlikely at this point. Um. But Istio started as a service Mesh. It's got ingrus built in as Well. So um, so where where do you feel like Istio is as a project right now and where do you think the future is going with that.

41:55.60
Jose Carlos Chavez
So the trade is is a good contributor for east to I think in terms of organization. We're number 2 in number of contributors. Um, so ah.

41:57.74
Dave Sudia
Okay, probably after Google.

42:08.20
Jose Carlos Chavez
Basically I think now that is still was graduated in in cncf which kind of it. It wasn't part of cncf at every other conference about cncf was about istia. Although it istio wasn't part of cncf. But I think this is this is a.

42:18.32
Dave Sudia
Um, yeah.

42:24.92
Dave Sudia
I was in a couple of the governance meetings talking about that whole situation. So yeah, it was. It was an interesting path. Yeah.

42:26.40
Jose Carlos Chavez
Ah, good thing. Um in terms of synergy because yeah, um, but I think like what is you can bring now as part of cncf is's going to be like it can be more popularized it as of I think. 2 three years ago unfortunately istio while the most hype service mesh wasn't the most popular and it was linkrdy as well. But I think now and it's in one ecosystem that could eastio could even keep growing. Um.

42:51.44
Dave Sudia
True.

42:59.26
Dave Sudia
Um.

43:04.33
Jose Carlos Chavez
At the trade we we of course are in hundred percent with istio. We we keep contributing. We have actually products based on istio we have to trade these 2 subscription which is a Phipps compliance and it fis verified it to distribution that will will give you all the support you need to deploy it.

43:06.53
Dave Sudia
Sure yeah.

43:22.74
Jose Carlos Chavez
Istio in production environments. We have that the trade is still this true which is fiipps compliant and is 100% upstream istio it's available on on Amazon keys. So yeah I would say like what is coming for istio is even.

43:27.50
Dave Sudia
Okay.

43:38.43
Jose Carlos Chavez
More stories now that Eastio is also been part of cncf and consequently part of or go through the tax security landscape. It's it's there are more things coming in as well in terms of ah um.

43:43.97
Dave Sudia
Yeah, yeah, yeah.

43:55.98
Jose Carlos Chavez
Let's say in terms of of features. There are things changing as you know, um they are There are the ambient ambient mesh that is also racing we we still believe in the sidecars for the reasons I explained before like you need more control you.

44:07.85
Dave Sudia
Um, okay yeah, sure.

44:15.65
Jose Carlos Chavez
Basically and all all the like all the or all vision about security is been a like is based on the assumption that you can leverage this and come organizationwide. And and not turning this into a specifics Problem. So. So yeah I think there there are good things coming for it and and we are still investing on it for sure.

44:45.76
Dave Sudia
Cool, well anything else. You'd like to share with the audience. Ah anything you want to plug or places people can find you.

44:48.74
Jose Carlos Chavez
Um, um, yeah, um, so basically this on top of of the work we're doing at koraa Koraza getting good hype as I mentioned and you can come by a.

45:03.84
Dave Sudia
Um, yeah.

45:08.10
Jose Carlos Chavez
The Github Repository Under korasawaf.

45:08.13
Dave Sudia
And I'll also say that the documentation is excellent because I had to make extensive use of it recently.

45:14.80
Jose Carlos Chavez
Yeah, yeah, that's that's good to know for us documentation is very important and one of the cool things about documentation which is probably popular now. But for me, it was the first is that we generated documentation from the cove.

45:18.70
Dave Sudia
Um, yeah, and.

45:30.89
Dave Sudia
Um, cool. Yeah.

45:34.14
Jose Carlos Chavez
Which help us to keep it in sync right? whereas before I remember maintaining others open source projects where you maintain the code in 1 place and the documentation another place out of sy really easy. Um, where also has used as you know, um.

45:41.73
Dave Sudia
And then over here. Yep.

45:52.57
Jose Carlos Chavez
Trade is is very into webassembly like women maintain or runtime we are very active in the webassembly community and we're building this http wasn'son which is um, it' basically an avi that for using a middle where.

45:54.68
Dave Sudia
Dip.

46:02.10
Dave Sudia
Okay.

46:12.19
Jose Carlos Chavez
In your applications that will load webassembly binaries. So basically you will compile a middle work into webassembly and that you can run in istio in envoy well not yet in istio. But soon in Envoy and soon in istion. Yeah, um.

46:15.70
Dave Sudia
Um, okay yeah.

46:24.38
Dave Sudia
If it's an envoy. Yeah yeah.

46:31.90
Jose Carlos Chavez
In actually native go in traffic and so on so in dappper as well and and and this is basically 1 step forward proxy wasm and because we we like the proxy was on the spec is kind of um.

46:31.47
Dave Sudia
Okay, yeah.

46:49.33
Dave Sudia
Yeah, okay sure.

46:51.70
Jose Carlos Chavez
Suspended I would say um or hibernating and and we wanted to to provide something with more um that first was easiest easier to to to use that was decoupled from Enboy lifecycle because. 1 of the things about proxy was is that it's coupled to onboard lifecycle and then it's hard to adapt these to other and proxies or whatever and also that is blazed in fast because stdp wasm has a lot of design decisions that made it faster than envoy.

47:17.28
Dave Sudia
Um, yeah.

47:25.95
Dave Sudia
Okay.

47:29.19
Jose Carlos Chavez
Um, starting for the fact that we only have 2 faces we have the request and the response no more request heater response and request body response heater respond body. Um.

47:30.72
Dave Sudia
Sure.

47:43.16
Jose Carlos Chavez
And and yeah, we are also working on that. It's also getting some traction soon. It's gonna land into traffic which is really interesting because it's mainly a community effort. Um, and yeah, if you're interested that you can come by Htt Wasm io.

47:51.96
Dave Sudia
True.

48:00.53
Dave Sudia
Ah I'm writing that down right now I am very excited about all the wasm work because as someone who's been involved with gateways for a long time I don't like writing in Lua Ah so being able to write things in whatever I know and and get it in there.

48:02.47
Jose Carlos Chavez
We're we're doing coolest out there.

48:19.34
Dave Sudia
Is yeah yeah, yeah, no 100% um that actually so so I actually have one last question for you which is um and it may just be that I missed this part of the documentation leg. It made this may exist and I just don't know about it. But.

48:19.37
Jose Carlos Chavez
And ah does the does the pick of the eyes be right because writing in law is 1 thing. How do you test that.

48:38.26
Dave Sudia
How what's a good way to test karraa secklang rules. Ah because I think that was like 1 of the things. The biggest things I ran into was I was trying to you know, solve this issue where I was trying to fix a thing where like my ah host wasn't host name wasn't coming through correctly and so I was blocking it. You know wrong and but it was very hard. You know is a lot of like tried I was a little bit back in the stone age of like try a thing upload it refresh the page you know? Um, yeah, is there anything that I missed for that or is there anything coming.

49:10.94
Jose Carlos Chavez
Yeah, so this is very good question to close because it involves some of the topics. We talk about so there are 2 things you can do. Um first we have a playground um playground dot corassa io or you can test your.

49:18.48
Dave Sudia
Um, okay.

49:22.84
Dave Sudia
Okay.

49:30.41
Jose Carlos Chavez
Your um, your rules given a request and a response the interesting part about that playground is that it runs hundred percent in the browser. No server, no server involvement because we compile it into webassembly and then we load the into the browser.

49:39.40
Dave Sudia
Okay.

49:44.88
Dave Sudia
And just load it there here.

49:50.31
Jose Carlos Chavez
And um and this could be this could sound like yeah why you do that right way. Why is it worth? yeah because we don't have to maintain a server is less security work and but right.

49:57.70
Dave Sudia
Totally yeah, indeed less is less and more is more.

50:06.60
Jose Carlos Chavez
Before I remember when we started like how do we run the playground Now there is a server in and somewhere and I would like yeah and who is making sure that that server. Yeah, we we restart the container once a day. Okay, yeah, well now we're running it in webassembly and it's.

50:14.12
Dave Sudia
Who's going to maintain Yeah is.

50:25.58
Jose Carlos Chavez
Yeah, it's safe. Um, that's one way and I recognize the the playground could have could be more intuitive I myself have a fun experience this week with that. Um.

50:42.70
Jose Carlos Chavez
And then there is this product that I maintain which is Kora ah Http Ben which basically you can run it and you pass the directives and you use it runs already htp is bean server and you can test your rules there as well. Um, these are the the two ways.

50:43.93
Dave Sudia
Okay, all.

50:51.66
Dave Sudia
A okay cool.

50:59.64
Dave Sudia
Okay.

51:01.21
Jose Carlos Chavez
We test rules and actually Crs which is which are the maintainer of core rules that they also use a q as I did be. We been for for validating the rules for understanding the performance implications in a change of arigas and all that and.

51:13.30
Dave Sudia
Okay, and we.

51:17.87
Jose Carlos Chavez
And just one plus if you want to test your Koassa implementation or your Koassa Yeah, your cross implementation. Koraa also offers an end- to end package that we use to run against every integration to make sure that they all work consistently and.

51:25.89
Dave Sudia
Okay.

51:35.86
Jose Carlos Chavez
Yeah, we we like to test things. So.

51:38.90
Dave Sudia
I'm going to check all 3 of those things out now because yeah, we ah are the way that we distribute it primarily is we have a Pci 6 6 compliant rule set that ships basically on top of the core rule set and that's kind of the default if you just.

51:49.27
Jose Carlos Chavez
A.

51:51.10
Dave Sudia
Pull our default rule set down but then of course you can just load in a config map but there's a bunch of ways to like load in your own rule sets and so I was trying to do my own overrides on top of that and got into the weeds. Um, so I'm definitely going to go go check that out now. Well it was.

52:04.24
Jose Carlos Chavez
Um, yeah, awesome.

52:07.95
Dave Sudia
Ah, very great pleasure speaking with you Jose Carlos thank you so much for coming on and ahresquez porttolo and ah maybe we'll give me another 3 years and we will do the whole thing in spanish.

52:17.72
Jose Carlos Chavez
Yeah, thank you so much for inviting me. It was really fun to talk about these topics and and you're a great host so see the ra handdodo of forque elspan wastan defeat it Soin Capital Neabla alamo Ken deal and poittoma lenton non than rapian naus. Yeah.

52:24.26
Dave Sudia
Thank you.

52:32.14
Dave Sudia
You see must rap de Montia Opo All right? Thanks! thanks.

52:35.20
Jose Carlos Chavez
As I the material must cause us. Um. Okay, thank you so much. Dave it was awesome.