Vibe Coding vs. the CISO

Aaron Pritz (00:51.714)

All right. So this is a little bit of a simulation. guys, so Chris, you are in the simulation as well as are in for real life, the CISO of our company. Michael, you're doing a lot of tech development, played with AI. Todd, you're AI product advisor and you're getting a lot of client questions about AI, which all of that is absolutely true. So what I want to do is...

Todd (01:03.669)

Thanks

Aaron Pritz (01:20.498)

Todd and I were in a conversation with a customer last week and he was talking about the CEO is just like hardcore into vibe coding and he's having to give data access to, to them to kind of try to contain it, try to enable him to do what he's doing. What he's doing is really cool. But he's trying to kind of figure out how, how do you enable the innovation, but also control the

just the out of control, like what if he uses client data? What if he pinholes access to things that he shouldn't? So it's a new landscape, new control stuff. So what I'm going to do here is I emulated what he's doing in our own environment. And I'mgoing to show, I'm going to do a quick demo of what I did because it's interesting. It's cool. But I also want to have a little bit of an impromptu discussion on what our client should be doing to not necessarily prevent this.

that get our arms around the fact that AI is leapfrogging on a weekly or monthly basis and executives and employees are all out there trying stuff and cyber in many cases is not at the table. So does that make sense?

Chris (02:36.862)

Yeah.

Aaron Pritz (02:37.826)

All right. So, and I'm going to be a little provocative here for, I did do this in a very controlled fashion, but I'm going to overplay what I did just for effect. So I took all of our third-party risk management intellectual property and I uploaded it into Claude code. From Claude code, third-party risk is a...

Tough topic, there's GRC tools that we've tried, it had really painful. So within the confines of a few hours over the weekend and a couple of nights this weekend, I iterated probably total cumulative of 10 hours to take all of our IP and develop our entire process in a fully functioning SaaS app that's running locally right now, but could easily be replatformed and deployed.

in the cloud or on Azure or whatnot. So just to kind of talk about the tool for a little bit and kind of show how it aligns to our process. Right now we're looking at the analyst queue, kind of showing the actions. We've got the overall platform, some really cool metrics of high tier, open findings, average daily cycle time, all coming from Michael's metrics packages that he's put in place before. Some really cool summary like risk.

by domains, know, general, obviously that's a bungling of things together, but data protection, access control. From a risk summary standpoint, can drill a little bit more detail. I can see that we're only remediating 12 % of our findings and I can see more specific things on risk domain and get into details with reveal risk recommendations, which we could tune.

on the bigger themes so we could actually action enterprise level decisions based upon the themes we're getting out of third party. With our process, obviously, tiering things by t-shirt size is important. And also, we were able to add AI risk analysis on the vendors itself so you could kind of get a early look before you even get into the assessment itself and then confirm from an analyst standpoint, you know, do we believe that that would change, you know, that

Aaron Pritz (05:00.138)

individual risk and override described things like that. And obviously we can see the list of vendors. This is all hypothetical data. You can see here we have the Addix art bar and grill. And I only chose art because there's an Addix art out there. I wanted a real URL that I could scan. And then we can see we've sent a

you know, an assessment to them, a rapid triage, so we could get back to that back and then be able to have analyst analysis on the backend, as well as the risk scores and then specific contacts that would come up. This is where I onboarded that vendor before this call. And then questionnaires, obviously we can load in. I loaded in a few samples of questionnaires we have from our frameworks library, you know, the pharma assessment, the

rapid triage and then this was an evidence checklist. Interestingly enough, I also took the Natara redacted data to do the additional enhanced due diligence to put additional actions that we could put in place for specific vendors. And then monitoring wise, obviously you can see some alerts. It was able to, let me go down here, settings.

yeah, DDQ is an answers library and a full AI capability to match questions to answers banks, which I think we struggled with with some of the external tools, but at least some basic capability to log in incoming DDQ, drop in the questionnaire, and then have it matched to the current answer set that we have. And then obviously risk register, you're going to roll all those findings up, get some specific reports.

that you can push to Excel or I guess PDF PowerPoint. And then also custom framework building. If we aren't going to use a standard assessment, we can create a custom framework itself to be able to use that for that. So anyway, that's the quick demo not being an SCE just to kind of say, you know, I haven't touched code in 20 some years.

Aaron Pritz (07:22.56)

It was able to fully build out the database schema, I think over 10,000 lines of code to support it and do it all based upon business requirements and vibe coding and specific accelerators that we could load in. So pause there. And by the way, Ivery carefully chose specific accelerators that were out of date and not that recent and or not something that every other company would do. But let's assume I'm not a cybersecurity CEO.

And I don't necessarily understand the risks of putting this stuff in there. Let's assume I dumped our entire SharePoint library in. So let's start with Chris first. As the CISO, what controls would you want to put in place if you knew I and three other people within their jobs were going to be doing this without a lot of IT or current acceptable use policy? Obviously it doesn't.

contemplate something this advanced. What's your thoughts and really our clients are facing this as well. Like how do we help them get in front of kind of some of this freight train that's moving whether we're on the train or not.

Chris (08:37.743)

Yeah, is that a on my head? Did you hear that bell? Okay.

Aaron Pritz (08:42.228)

I think it was an alert on my.

Chris (08:45.169)

Okay. Holy moly. All right. We're not crazy yet. All right. So here, so let me take a second. I, and this is aimed at us in general, everyone using AI, like as cyber practitioners, I'm not quite sure why this is so confusing for everyone to kind of get their arms around. When we think about we've done this over and over again with technologies that come, have come up, right?

Aaron Pritz (08:46.806)

Hahaha

Chris (09:16.779)

Email, internet, cloud, you name it, over and over again. And this one is obviously a little different, but it's very fast paced. It's moving way faster than we can all imagine and leaps and bounds, like you said. But the same concepts apply that we've done for 30 years. It's make sure that we're taking the things that we know, the things that we've done, the risks that we've managed and apply them to a new technology. So how to do this? answer is yes, CEO, go to town, have fun. However,

What would we do on any other platform? We'd say, make sure you have your third party risk assessment done. Make sure the data is being properly controlled. Make sure the vulnerabilities are being managed. Make sure you're managing access in an environment, right? And I know these aren't a CEO waking up in the morning going, I'm going to go do this. But that's happened. That's happened with Dropbox and Box, the whole file sharing thing we went through many years ago that everybody was using. It took us a while to get our arms around. But ultimately, what do we do? We applied the same concepts.

in a different way, more advanced, more complicated as we continue to advance technologically. But I think same thing here, guys. How are going to do that? The answer is yes, go and do it with these controls and applying the controls that we've done over and over again.

Aaron Pritz (10:28.65)

And how do you feel on blocking? Right? Cause like when I, when I was on the corporate side a decade ago, cloud storage came out, we had some insider threat. The legal reaction was we'll block it all. We'll pinhole access to the stuff that are approved tools. But then we saw the problem squishing around. People were going to home personal devices to get their work done in other ways or use their preferred, you know, cloud. Yeah.

Chris (10:52.297)

There, yeah, there has to be some control there. We can't go, well, use what you want, right? Because there's a couple of things. You're managing cyber risk, but then you're also managing the optics of risk or the culture of risk, right? Saying, if a company takes a stance, go use anything you want, have fun. Where's the responsibility for that risk lie? The company, right? If you're sitting there going, well, here's our approved. You can't use these things and something happens.

That positions the company very differently in that risk conversation when opposing counsel is going, why did this happen? Or how did you let this happen? Well, we were like, was a free fall right now. That person violated the policy and the technical controls and ended up doing X different conversation. Right? So I do think, I do think there is the need for reasonable blocking. I do, however, like anything, you can't say don't use box or drop box and not provide the employee base of something else. Cause that's silliness. Cause that's exactly what you said.

Aaron Pritz (11:35.042)

Great point.

Chris (11:51.391)

people are just going to go around it and figure it out. I'm going use mega upload to share with my vendor, right? So ultimately looking at this, it's you can't do this, but we're offering this and it's just as viable as the other ones. And then over time, as we continue to advance in AI world, opening more and more capabilities and abilities that we get our ability to manage risk in the technology.

Aaron Pritz (12:13.238)

Yeah. Michael, you've been a proponent of, of Vibe coding and product development and alignment of the process stuff. Obviously you, similar to me, practitioner in cyber know some of the precautions and dangers to avoid as you, as you were leaning into it and approaching it. Like as a, as a, as a leader, you know, in the company, like, what did you think about? How did you self-regulate maybe some of the things you could have done, but didn't want to, didn't think it was the right thing to do. What was your.

was your thought pattern as you were exploring into this stuff?

Michael (12:47.546)

Yeah. So like when I did the, like with assessor kind of the, the first round with it, um, that was very, very heavily on taking, taking what we have and building from scratch to, uh, to kind of mirror some of it and then expand. So I wasn't taking anything of ours and putting it into another tool. Uh, I was pulling it up and saying, I like the outline. I like the structure. I want to use this as a foundation, but I'm building it on my own. Um, to.

You know, so it takes a few extra hours upfront to kind of build that. But then all the additional functionality. So kind of some of the more recent side projects taking sanitizing all of those things like on my own. And then then I'm comfortable running those those documents and things through through the platforms and stuff like that. But I mean, some of the key things I looked at is like.

from a security perspective. I know the one you just showed, you're running it locally. However, if somebody got a hold of your device locally is now local to them also. So there's one issue. The other thing is if you were to take the next step and say, well, I'm already running it locally on my own, I'm gonna share it with the team and you just go and post it somewhere else. Are you posting in a hosting place that is safe, secure, vetted? Have you run any like...

security testing against it, not like full blown pen tests, but have you done any security checks to see kind of what the access control and stuff like that is? So those are things where I've kept it very much internal, sanitized at my own, or again, looked at it, built it from scratch, and then expanded upon with minimizing what I share with some of those tools.

Aaron Pritz (14:32.972)

You know, for people in marketing or ops outside of IT, outside of cyber that are jumping into this, how many, without guidance, Chris, to your point, without, without any kind of structure governance, a happy path, here's the job aid to do it the right way. How many people outside of IT and cyber do we think are thinking like Michael is?

Michael (14:55.458)

not many, and I actually just saw three posts on LinkedIn within yesterday alone. Two of them were legal firms where junior and senior attorneys are just dumping everything, client documents, client meeting transcripts, everything, no sanitizing because no one told them that they couldn't. And to them, I'm saving 40 hours a week because I can dump it in there and I've not been told I can't do it.

So it's making my job easier. So literally three cases that I saw, I read them yesterday, but they all happened within the last two weeks. So yeah, I would say not, not many.

Aaron Pritz (15:35.51)

The thing about it is like a lot of us want to enable the technology. There are CISOs that don't, but trying to hold it back or say, hey, we're not going to get into this yet. It will find a way. The Jurassic Park quote, like life will find a way. And it's the same thing here. People are going to find a way to innovate. If you're telling them they can't, will find that way. Todd, from an identity standpoint, we were chatting in the office earlier like

Michael (15:48.59)

Right.

Aaron Pritz (16:03.18)

This kind of expands the aperture of identity and managing, you know, different forms of bots and agents and things like that. Like, how do you think companies are faring in that early battle?

You're on mute. You're the guy.

Todd (16:19.996)

I'm the guy. Well, I was trying to manage that echo. I'm always the guy. I'm not sure the identity teams are truly ready for this. And if you work through how some of these code, these vibe coding tools are working and how they're setting it up, one of the first things you need is access to that data. And it's easy to start with, me the documents and just let me throw it in here and let me read it in. That's one of the first places you start.

Chris (16:30.623)

Okay.

Todd (16:45.43)

But the second place you start is, let me connect to my database, let me bypass the reporting tools. And IT teams over the years have been very good at going, I'm going to control sensitive credential, I'm going to put it someplace secure, and then I'm going to present that report to you that filters that data to you in the right way, in the right place, and I keep that sensitive credential safe. Now this is pivoted to where you're going to have a lot of people that are not developers, who are not practiced IT individuals going, I need that sensitive credential.

I'm gonna put it on my laptop. It isn't gonna be encrypted. It's gonna be in a plain text file. And worse, that credential likely does not have MFA. It is probably completely open. It's gonna bypass those rules and that is gonna explode and increase. And you're gonna have info stealers that are grab those. And those accounts, by the way, are also monitored less. It's the nature of security that that's why you try to put those things and hide them away.

So you may see security teams come back and say, okay, you can do vibe coding, but we've got to create an environment for you to do it securely. And that doesn't include your laptop. That might be a place to start. Because some of the new technologies and methods to protect those identities are a little bit different than what most are used to. They're new products. And they're not geared to your average person. They're not geared to your finance person going, just give me access to the SAP data and let me start running.

I'm going to bypass Excel, I'm going to bypass SAP, I'm going to create my own reports here on the own. I think that's a challenge area that we're going to have to work through. And I think security teams are going to have to not only lean into new tools, they're going to have to lean into new audiences that they typically haven't communicated to. They're going to have to talk about development practices to non-developers. And that's going to be an interesting convergence to happen. mean, can you imagine talking about a credential vault?

Not a password vault, but a credential or an API vault to somebody in finance. Not to diminish finance, but I'm going to guess using developer tools is not their forte of practice.

Aaron Pritz (18:51.638)

Yeah. Michael has his hand up and all as I, as we pass the mic to him, I'll say when I started this project in cloud code, it first asked me like three times for a GitHub account, which I had a corporate one, which wasn't directly within GitHub. So it was really pushing me to set up what would have been a personal GitHub account, which would have hosted all the code. Whether I set it to be public or private or whatnot would have been up to my knowledge of whether I should do that or not. So to your point of like.

Once you go online at it, you it'll help you get online real quick. It'll help you set up third party app files. It's really helpful with that. Like probably saved me hours. but I think it's a slippery slope without the right guidance. Over to you,

Todd (19:33.408)

you

Michael (19:34.616)

Yeah, so that was one of the two things I was going to say was the GitHub connection for sure, especially for people who aren't coders, probably don't even know what GitHub is or how to handle it, like how to even think about security within it. So that's definitely one and it is extremely helpful and it'll do 98 % of it for you and create walkthrough documents for the other 2%. Right. The other thing is, and when you built this app,

When you went to go use it, even for testing or demo, did you create a login page to where you had to log in to test what you were doing?

Aaron Pritz (20:10.05)

I but that's because I knew that I wanted security in it. If I just wanted to do straight up development and not be burdened with that stuff, that would be an issue.

Michael (20:23.514)

Right, exactly. So how many people even inside security, to be honest, know security is, doctors are the worst patients, right? So when people are creating these tools, I'm creating this tool to save me time. Why would I cause myself more effort by making myself log into a tool that I'm making to save time? So it's just, it's another one of those easy things of maybe it starts out, well, I'm the only one using it, so I don't need to log in.

but then, hey, Aaron, I want you to check this out. No, there's no login, it's just us, it's fine. And then it does start to scale. So then it becomes hard to add some of those security things once people have been using it.

Aaron Pritz (21:04.566)

Yeah. Well, one last question for Charismatically Curious and CISO Chris. Wow, that was a four word play on alliteration. Are you going to now block cloud code and kill my fun? what would be the next And what are the next steps to protect our own self or protect me from my mayhem? I'll say mayhem.

Michael (21:19.802)

I did it while we were talking.

Chris (21:30.323)

Good question. I don't know if I could protect you from that, Aaron. Let's see here. Now, I think we're in little bit of a different situation. You understand the risk. So I think if you didn't, and we were similar size company, I would try to listen to what you're trying to do and get our arms around it, to put the controls in on the fly while you're doing your thing, or say, hey,

Aaron Pritz (21:33.898)

Ha ha ha.

Chris (21:58.527)

And we know you're on a full head of steam here, but pump the brakes for a second. Let's organize some stuff. Make sure we're at least doing some of the basics here for you. And then maybe get some agreements in place with Claude, get the enterprise kind of agreement in place, and at least get something there so we're not just using the publicly available version and there's no security controls in place. So something, not everything.

Aaron Pritz (22:24.076)

Good point. And I did get coaching before I went to set up like a POC environment and turn off the training on our data and things like that. But there's more steps, you know, as we would progress that we will and we will need, we will need in actuality as well as we would advise for our clients as their CEOs are probably doing this and not just CEOs, but leaders and employees of all levels are, you know, experimenting, right? It's, fun. Like one of our CIO clients said he spent

six hours on a holiday weekend, trying to set up an environment to help a CEO. Not because he had to, but he was like, I'm actually having fun with this. that when there's FOMO or fun, actually, Michael, the only reason I started VibeCoding, cause I saw that you were doing it. I'm like, well, shit, I gotta, I gotta get caught up here. This seems like something I'm missing out on.

Michael (23:16.1)

I think one of the other things, the GitHub thing is an easy way that security and IT can start to bring other people into the fold and encourage it, right? Because there's a lot of things that finance would come up with that we would never consider being a need or something like that. So I think that it would be a miss if IT and security just shuts everything down. Whereas...

Chris (23:25.887)

No.

Michael (23:42.196)

If it's, we already have an enterprise GitHub or a company repo that we use, bring them in. You can segment them off to a separate section so they can access other production code, but you can bring them in, hook them up, say, hey, absolutely, dump your code in here. It's at least more secure. Start with that. Encourage it, but then at the same time, you're also encouraging and enforcing some of those basic security controls.

Chris (24:08.805)

And I think like.

Aaron Pritz (24:08.93)

Well, life will find a way whether we are protecting it or not. I had another CISO that we work with that, you I was sharing last at a breakfast coffee meeting across CISOs that I was playing with Claude Code and he was like, oh yeah, I am too. And I said, how are you doing it? He was like, well, I've got an old Mac MacBook, you know, company device and I'm doing it all there. You know, so again, he was thinking about ways to not do it on his core laptop, but it was a company device. But anyway, I think it's more important for us.

Michael (24:14.394)

K.

Aaron Pritz (24:37.974)

for us as practitioners, if we don't know what these tools are, if we don't know what the potential is, we don't know what we're protecting.

Chris (24:44.767)

Yeah, and I think there's a little bit about kind of front-ending the conversation with business need and what should be done versus kind of open playing field, right? Because I think that's one of the first things you do to manage risk. If you say, well, everyone at every level of the company can go and tinker. Okay. Well, that's very broad. But number one, that's introducing more cyberists than necessary because everyone's doing something. And then

On the other hand, potentially waste of time for a company, right? Maybe there is some front ending of this with a business process to develop business cases to allow people to focus on business supporting initiatives or code to rather than just kind of have an idea one morning. Now, when you get to the leadership team level, CEO, COO going, I'm going to go try something that's different. That's a white glove VIP experience. They're going to go and do that and support them. But having the entire accounting team going, huh, I got an idea.

and go to town typing, does that even support the mission? Does that support the business, right? So wasted time and that's the first place you could probably reduce some of the cyber risks rather than having 30 people go tinker on a weekend. Maybe it's five people tinkering because it's an approved mission supporting idea. they have that scaffolding around them to go, yeah, go and give that a try and come back. Let's see how it works. So just some thoughts there, kind of riffing on what Michael said.

Aaron Pritz (26:11.07)

Well, Michael and Todd, for letting me steal the 10 minutes that turned into full 30 minutes of your one-on-one. And Bronwyn, I stole the first 15 minutes of our one-on-one, but I think this was a fun conversation. Michael, hopefully it wasn't as scary and not fun as what you envisioned it to be when I dropped the link into your invite this morning.

Michael (26:31.723)

It was definitely not as bad as the ominous drop made it seem like, so that's good.

Chris (26:37.875)

Yeah, was intellectually stimulating for me for the record we're still recording it.