Data Security And AI

  |  Compiler Team   Segurança

Compiler • • Data Security And AI | Compiler

Data Security And AI | Compiler

About the episode

The relationship between data and AI is...complicated. AI is built on data. It often needs more. A wealth of data can make AI strong. But it can also be a weakness. 

Clarence Clayton, Director of Global Privacy + AI Risk and Compliance at Red Hat, helps us understand the increasingly complex interplay between data and AI—because the flow of information isn't a one-way street.

Compiler team Red Hat original show

Inscrever-se

Subscribe here:

Listen on Apple Podcasts Listen on Spotify Subscribe via RSS Feed

Transcrição

We have to remember, like the internet is forever, right? If I, if I go back to, you know, my daughter's not a teenager anymore. But when she was a teenager, like social media, I'm like, you have to be aware that anything you put out there will live forever and people will find it. And now with AI, like if you think about like, you know, like hunting for these sorts of information, like you're looking into the background of a person, you're you're thinking about hiring or whatnot, and maybe those things that you did before that you're maybe a little bit embarrassed of, you thought you got rid of you didn't. because the internet is forever can be more easily found. Exactly. It's basically anyone can hire a team of private investigators to go find stuff now. For cheap. Kind of what it feels like. Yeah. And for really really cheap. Comparatively. Comparatively. At the very least for sure. This is Compiler, an original podcast from Red Hat. I'm your host, Emily Bock, a Senior Product Manager at Red Hat. And I'm Vincent Danen, Red Hat's Vice President of Product Security. On this show, we go beyond the buzzwords and jargon and simplify tech topics. And in this episode, we take a look at the intersection between AI and data security. In our last episode, we talked about data as a target, why it's valuable, what can be done with it, and how to protect it. What we didn't talk as much about is how AI changes everything about the data side of product security. So let's start with some basics. Vincent, would you say data is a big deal for AI? I absolutely would. I mean, without data. AI is useless, right? I mean, it's a very basic foundation of what AI does is it processes information. So data is hugely important. Absolutely. No. It's nothing without the data that makes it up. We have a little clip from Clarence Clayton, who leads a team of AI risk and compliance professionals here at Red Hat, sharing how adding AI to the mix has catapulted data from just a target to a vector of attack. What is happening now amid AI and many regards, especially when you're talking about like, you know, LLMs things of that nature is really about, you know, what you put into it to see what you're going to get back out. And so when you think about, you know, prompting and if you're prompting the information or prompting the AI technology in particular with a certain type of data, then you're going to see, well, what is it going to tell me? Is there a world where I inject some, you know, data into a prompt and it hallucinates or it returns responses that I didn't expect that it would, but maybe, you know, give me some sort of additional insight that I didn't or wouldn't have otherwise had. So it creates a new opportunity in many regards, for people like me to try to get even more creative with you know, how we manage risk and compliance and, and for the engineers and the really smart people that are building out these technologies to really do their due diligence on understanding how that data, is being, you know, utilized and making sure that, you know, we have, you know, robust tests in place and robust validation procedures in place to make sure that, you know, the AI is operating as intended. So data is absolutely now, you know, becoming even more of, an attack mechanism as opposed to the end result of what people are looking for. So we know AI runs on data. What did Clarence mean about injecting data into a prompt? Well, he's basically talking about prompt injection. So it's when an attacker crafts a malicious prompt, in order to manipulate that model's output. So think about things like trying to trick a model into revealing sensitive information, or generating false content or executing commands outside its intended purpose. Gotcha. So I kind of see it as like, when you've got internet forms and things, you put names in it, and there's all those people that'll load Doom into, like, a name field. It's kind of like the equivalent of that for AI. Is that about right? Yeah, something like that. Like, we're looking at it in ways of like, how can I trick it into telling me something that maybe it wasn't designed to tell me? Gotcha. It's like AI con-arting. Yeah, yeah. You're trying to con the AI into telling you something that it maybe it knows it shouldn't tell you. Or maybe it doesn't know it shouldn't tell you. And it's like, hey, I'm helpful, but I'll tell you stuff. Gotcha, gotcha. No, that makes sense. So when it comes to getting creative with managing risk and compliance in that kind of situation, what is... what does that mean? Where are some things we could do there? Well, you have to start thinking about the ways that a model, can expose that information that you don't want it to. So, think about something like a database, right? It's fairly simple. You have standard access controls. But with an LLM, it might be more challenging if it's multi-purpose. So think about, like, how do you compartmentalize parts of an LLM. Mhm. And this is where things like guardrails come into play. Right. Or potentially smaller, using smaller multiple models for very specific purposes that those then have appropriate access control. So you can kind of separate things out. Yeah i gotcha. I think it's a little bit like we have to be as creative with trying to foresee the kinds of things that might happen there as they are at exploiting them. Oh, 100%. I mean, if you think about it, in the security industry, I mean, we've had like 50 years, 40 years of thinking about different ways of attacks and, responding to different sorts of new novel attacks as they come out. So we've, you know, when it comes to traditional software, we have this pretty good corpus of, of information and prior incidents to go, well, that was bad. Let's think about it a different way or come up with a different solution. AI is greenfield. It's all new. So I mean, we're trying to anticipate some of these things, but the way that we think about it is traditional software development, not model development or not as much yet. Absolutely. Like we talked a little bit about in previous episodes around like the democratization of product security, and we all have a part in playing that role. And I think that the same is true of AI. It's so greenfield that there are no bad ideas yet. Well, I mean, there probably are some bad, I mean, but they're all worth considering, right? Like, we should be thinking about these things and not assuming that one way is the right way. I mean, we've already seen it even in just, you know, this year alone, the amount of change and pivots of the way that AI is being used or how we're looking at it, it's changing so rapidly. So we have to hold on to our ideas loosely. We can't hold on to them tightly. Absolutely. So going back to Clarence, what does hearing data as an attack mechanism make you feel? Well, I mean, I agree with Clarence in a lot of things, but I also think about it a little bit differently. I don't think it's so much the data is like the attack mechanism. I think it's more the data can be attacked. And like we were talking about new and novel ways that maybe we hadn't thought about before the advent of LLMs. So yeah, we have to look at these things entirely differently now. Exactly. Like it's not only a target, but it's also in some ways, like a vulnerability in and of itself. It is. And it's and it's very different. I mean, you can think about like prompt injection, like SQL injection, but just a lot more creative. Right. Like we understand SQL injection. Right. Eventually we'll get to the point where we understand prompt injection. But we're talking about data and, and a black box that thinks on its own that will, you know, hallucinate. It will lie like we heard before. Right. So I mean, there's all those pieces that you have to consider that we don't think about with traditional software that just does what it tells you to do, and it either does it right or does it wrong. It doesn't do it somewhere in between. Yeah, yeah. It's always been like, it will do it or it will not. And now it will do it enthusiastically. And in a completely random way that you didn't expect. Exactly. Lot less predictable, for sure. And I like what you said there about, you know, AI and it's designed to be helpful most of the time. There's a really real danger of having it reveal more than it should. A lot of information about people is publicly available online, right? So say, for example, you, you know, just went and did a Google search about me, I don't know. And then you did say, hey, AI what can you tell me about, you know, Clarence Clayton that, you know, works at Red Hat? And depending on what sources the AI has, you know, access to maybe this information, you know, about projects that I've worked on here at the company that are still yet to be released. And it's like, hey, the littlest, you know, piece of publicly available information has now, you know, led to, you know, some sort of exploitation with some sort of, you know, inadvertent or unauthorized disclosure of information that, you know, I or someone that I'm affiliated with might not have wanted people to know, at least not yet. Yeah, those additional insights that AI is really good at providing. But in a lot of ways they can also be a security risk. But what do you think about that? Well, I think that somebody needs to talk to Clarence and tell him not to talk about not yet released projects publicly, but because that was his premise. Sorry, Clarence. But, I mean, that was the premise, right? Like, I think fundamentally what we're looking at is, especially for these public models, right? They're looking at public information. They don't have access to your corporate private information or your personal information. Somebody has to feed it to them. Mhm. And that's and that's publicly based. Right. And I don't know if you know AI provides more. Right. It can only again, it can only tell you things that it knows about, but it can tell you them faster. Yeah. Right. I think for me that's the danger. So think about, like AI search, which is starting to increasingly displace traditional search. So if you, if I Google something, I get a list of resources that might be relevant. But it's up to me to read it and then hope that I find what I'm looking for. Now, with AI can perform that that same search. It can consume the data and it can analyze it and then give me back those contextual answers to questions. I was asking it significantly faster than I could if I were manually searching for it and doing all of this reading myself. Yeah, absolutely. Like I think it's always been a risk putting things out publicly that shouldn't be like nothing necessarily has changed there, except it's harder to hide stuff in the noise. So if you put it out at all, it's a little bit more dangerous just because it's easier to find, because AI is so good at finding things. Yeah. And we have to remember, like the internet is forever. Yeah. Right. If I, if I go back to, you know, my daughter's not a teenager anymore but when she was a teenager, like social media, I'm like, you have to be aware that anything you put out there will live forever, and people will find it. Yes. And now with AI, like, if you think about, like, you know, like hunting for these sorts of information, like you're looking into the background of a person, you're you're thinking about hiring or whatnot, and maybe those things that you did before that you're maybe a little bit embarrassed of, you thought you got rid of, you didn't because the internet is forever. It can be more easily found. Exactly. It's basically anyone can hire a team of private investigators to go find stuff now. For cheap. Kind of what it feels like. Yeah. And for really, really cheap. Comparatively. Comparatively. At the very least for sure. What about de-anonymization of data? Does AI pose a significant risk there? I mean maybe it could certainly draw connections faster than a human could. And I think that the challenge here is that it might not be right. Yeah. So if you make decisions based on that data. So I don't know, for example, if you were trying to, you know, find your long lost brother. You might end up reaching out to the wrong person but be convinced that it was the right one, because all this information that the AI provided you. Right. So, I think in that way like for de-anonymizing maybe. Yeah. Just because it can draw connections faster. I could see that. And I also see it being kind of risky on both ends. Like it's risky to try and make connections that way because it can be wrong and confidently so. But it's also risky if there is a connection out there between one piece of data and another. Like there's the example of expired credit cards. If you can tie your name to financial information, might be able to find something like that easier. So yes. Risk but also risk if you're trying to use it for evil. Oh yeah. I mean, again, there's that nobody puts out information like I certainly don't put my name in credit card in public places expecting people to find it. So like what... the data that's being fed into the AI still matters. Right. And this means data protection becomes more important. So if you're thinking about like those overall security risks, making sure that data doesn't leak in a place where the AI, we'll call it the AI can find it and consume it and regurgitate it... Probably a good idea. Absolutely. So AI obviously is changing things. It's making it easier to go through reams of data and undo some of the protections we have in place, some of that hiding in the noise kind of stuff. But that's not the end of the story. How else can the bad guys team use AI models to their advantage? So if I'm interacting with AI technology that's meant to provide information about, a knowledge base, and then I go in and say, here are my configuration settings, here's this crown jewel, you know, coding base that you know is going to, you know, be the next... solve the next problem, you know, for our company. Can you critique that? It's like, that is not at all what this system was meant to consume. But you've now injected, you know, some information into an AI, you know, system that could be, you know, sensitive in nature. And now you're entrusting the owner, the producer of that technology to, you know, not, you know, use it for any other purpose, to maybe just delete it or like, does, you know, send that to trash like that. I can't do anything with that. And you're also hoping that that environment doesn't somehow get exploited in and say, oh, let me pull all the logs of the things that, you know, people have, you know, done to interact with this system lately. It's like, oh, wow, look at this interesting code base that somebody had, you know, injected in there, perhaps unnecessarily. So the more that you can be prescriptive about this is what the system, you know, is meant to do. You know, tell me, you know, what the issue is. And then one of two things will happen. You know, you interact with and say, I'm not designed to answer a question like that. And I deleted your information or sure, tell me more. So it's almost like walking them through step by step. How to make sure that, you know, they're interacting with the system in the way that we intend to get them the outcome, you know, that they want. Now, that's a big topic because data is so important for AI. But it's not just the data it's trained on, it's the data you give it when you're interacting with it too. Is there a way to tell if the data you're injecting into a prompt gets deleted or stays isolated? I mean you really need to understand that the way a model might use the data that you provide it. All right. So if it's just using the information you provide in the prompt, say to train itself, then you've just added to that corpus of data. Not all LLMs do that. I will make a statement, I mean, this is true for social media and many other things if you're not paying for that service, you are the product. Exactly. Right. I mean, keep that thing in mind. But a lot of these models have very specific ways of training, or these model makers have very specific ways of training these models. And I'm pretty sure that they don't want it polluted with random noise that, you know, the end users are providing it. And so the model maker has better control of its data. The one thing that Clarence did note there that I think is really, really important to know that even if you're not using that data to train the model, that information is being kept there. Mhm. Right. But potentially collected there to see how people interact with that model. Yeah. Right. So the model maker might see what that intended use is. And they might decide, you know, based on a lot of people asking for cookie recipes, they might be like, oh, we need to train it on more cookie recipes, because that seems to be what people are interested in. So they're using that data for something; it might not be in the model, but now it's like not just trusting the model itself. It's trusting that model maker as well. Yeah. I kind of see it like one-way glass, almost like interacting with an AI feels like a really personal conversation that's, you know, happening in an isolated space, but you don't necessarily know who's behind the window. And from a security aspect, I assume it's best to always assume that there's someone there. And you know what? When you say that it made me think about something I read a few weeks ago. That one of the most popular uses for these AI chat bots right now is as a therapist. Yeah, exactly. That's what I was thinking of. Yeah. And so you just think about what are people telling these things to get their, their quote unquote therapy from like, that feels like a lot of very personal information that you're providing to it that some guy might be reading as you're going through like, oh, what was prompted today? And they're learning all these things about you and whether they can tie to you as a person or not, I don't know. But that information is there and somebody has access to it in the same way. I think we talked about before, video cameras and people watching feeds from in your house while they're eating lunch because they're bored. Yeah, exactly. No. And that's kind of how I try to think about it, because it's in that kind of chat world. It feels private. But not only might they have access to your data. There's also the question of selling that data after the fact to someone else. Trust is important when you're using an LLM as well. I would say. I would say. I agree. So is there a world then where that injected data can infect the AI model as a whole? I mean, probably, but you know, there is there a world? Probably. But again, it depends on the model maker and whether they're using, those prompts, from users to train that data or not. And I mean, personally, I think that's a terrible idea. Yeah. You go from a curated known good. I'll say good in quotes, set of data to like, like the Wild West of whatever crazy things people are prompting AI bots with. Like, I don't think that's good value. Yeah, I think that's a note for then like the people making LLMs, maybe even more so is that it's so dependent on data that garbage in, garbage out is kind of how it's going to go. And if you're giving up a lot of control and then the quality of the end product, if you're allowing prompts to feed it as well. Yeah. Terrible idea. I mean, I just thought actually of a bot that say answer things for I mean, we're Red Hat so we'll say answer things on Red Hat Enterprise Linux. And if I sit there and I'm a competitor I'm like, I know I'm going to go talk to that thing that answers questions about RHEL and ask it questions about FreeBSD. Exactly. And then it does it start giving answers about FreeBSD to people who are actually legitimately asking questions about RHEL. That would be terrible. Exactly. I know, I know we're, you know, in the product security season of Compiler, so we might be a little bit biased, but it does seem like a really bad idea. Yeah. Well, we're totally biased, of course. Of course. But, kind of along those lines, is there a way to set up guardrails to prevent users from misusing the system? Basically, yes. I don't know if it's easy to do. Right. But this is where model makers really need to be observant and adaptable. No, no different than any other, I'll say, security responder. Right. If you detect something like prompt injection attacks, that's what you should be using to update those guardrails so that you defend against them. I kind of love that phrase that learn and adapt. That's kind of the name of the AI game for sure. Totally. All right. So sounds like there's a lot of work ahead of us to keep up with the new demands of malicious AI use. But, you know, maybe the good guys can use AI to their advantage, too. I hope so. Yeah, absolutely. So we can sometimes figure out when a system's under attack or has been breached, but not always on time. But every time a model gets updated, it's processing more data. Those models are getting smarter because more data is being fed into them. That it is making them smarter and making them closer, you know, to a human brain, right? So that makes it even more likely that you know someone who really knows how to exploit them, would be able to get them to answer questions in a certain way. More like a human would. The opportunity is going to be: Is there a world where the AI ever gets smart enough that it can actually detect that somebody is, you know, up to no good? Now, some of that depends on an engineer, really, you know, having good, you know, prompt capture mechanisms to say that sort of, you know, feels, you know, off limits or feels like something that, you know, we probably don't need to respond to and look good engineers will, you know, incorporate all of that. But they're really smart people who will be in conflict with those smart engineers to try to, you know, circumvent it as well. So if AI is getting smarter all the time and really rapidly, then this next step of awareness of malicious intent could be just around the corner, right? Like we're not going to keep having the my grandma used to read me the recipe for a nuclear bomb for dinner kind of shenanigans, right? Well, I mean, hopefully, and I think you can use AI models, that are created to detect harmful content as say, as part of that guardrail. You could even use multiple models to do that. And maybe you can use these models to make decisions in consensus. So for example, if you're analyzing something, whether it's a prompt or some output right. You could say you have four models involved. Depending on your risk tolerance, you might require three out of the four to say, you know, this thing is okay before allowing it. Or you might need all four to. But either way, different models provide their, you know, opinion. And if they all agree the thing is safe, then you let it through. And if you don't have consensus, then maybe you don't. Right. So as we're adapting and evolving like there are different ways of thinking about, you know, like in your word, how can we stop these shenanigans? Yeah. And I think, yeah, in that way it's not all that different from like a panel of humans coming to a consensus over, do we think this is malicious or not? But also kind of back to that whole, you know, adapt and evolve kind of structure. Like, I, I also think it's important not to get too complacent that, yeah, we got it. We know exactly what they meant because, you know, humans have been here for millions of years and we still mess it up on occasion. Oh, no, it's true. And I mean, going back to what we were talking about earlier, this is why a lot of these model makers are collecting the prompt information so that they can analyze it and go, you know what's good, what's bad, what looks a little potentially malicious or not, so that they can learn and train because some of the some of these guardrails might be other, other models, but it's also software. So like what Clarence had said about like these smart engineers implementing these things, you have to observe it before you can change it. Exactly. I think it's something we'll never be done doing. I don't think so. Really? Yeah, absolutely. So barring this kind of idea of self-policing of AI and AI knowing intent from a prompt, what kind of other safeguards can teams put in place to hamper the efforts of those bad guys™? Yeah. Well, I think, you know, this notion of models on prem tuned for a specific purpose with appropriate access controls is probably going to be the way forward. So, there's been a lot of talk about, like, these small language models. Right. So you take a small language model trained on customer data or trained on internal IP, but it's only accessible to those who are in customer support or engineering. Right. So not only do you have those guardrails in place, but you also have models that are fit for purpose and only those who need access to that data can actually access the model. You don't expose it to the entire company, and you certainly don't expose it to the entire globe. Right? But a not all that unlike you're not going to share out an open, public Google doc of information and expect people not to edit it. You want to finish it and then publish the result. Assuming that you want the public to know it. Also true. Also very true. If that's internal information, you make sure that that Google doc or whatever is locked down so that the only the people who need to know are the people get to see. Exactly. I think that resonates really well with some of the other product security topics we've talked about already, like it's not that different in a lot of ways from what things we've done before. There's always been new threats and there's always been ways to try to counteract them. And we have got some pretty good strategies for coming up with new things. And it's just keeping in mind the AI dimension of it on top. Yeah, I mean, the fundamentals are the same, it's the tactical approach to solve the problem that may be different. Exactly, exactly. That's the core truths still hold. Don't want people to have stuff they shouldn't have. So you should make sure they can't. Correct. Really boiling it down there. Yes, yes. Good job Emily. Anyway there you go. Simplifying tech topics. That's what we do. So I think we covered a lot today. I want to kind of summarize some of those main points and then get your last thoughts before we wrap up here. So number one, we know data is at the core of AI. That can also make data a vector for corrupting AI models. Number two, AI can be used to glean insights from mountains of data and can reveal information that should otherwise be secret. And three, we're going to need to use AI to defend against AI attacks. Yeah, that last one I think is the most important, right? Like, people shouldn't be scared of AI because it can do some naughty things. There are benefits to AI that we shouldn't overlook because I mean, guess what? If we're... I mean, I'll use these terms loosely. If we're the good guys and we're not using AI and the bad guys have no scruples at all and they are using it to their advantage, then we're the ones at the disadvantage. So we have to use it, use it responsibly and figure out ways, I think you mentioned earlier, but like the speed of detecting attacks, right, an AI is going to detect those things much faster than a human is. So why wouldn't we use something like that? Exactly. AI, when it comes down to it, is a tool and it can... it doesn't have a morality of its own. It depends on how you use it. So we can use it to defend ourselves against people who would attack with it as well. We shouldn't be too afraid of that. It comes down to education I think. Totally, and just accepting the responsibility. Exactly. And that, you know, democratization. I think we all have a part to play in that, too. It's not just leaving it to the teams that do AI. We need those creative ideas from everywhere that we can get since it's so new, we might be coming up with some ideas we wouldn't otherwise have. Just maybe in closing there, if you think about, like, what you just said. You don't look for software engineers to solve social engineering attacks. Mhm. And we're basically looking at this is social engineering of AI in a lot of ways. Right. Software engineers can help implement those defenses. But are they the ones who are going to be able to detect the linguistics, I'll say, of what that social engineering looks like. I don't know. Absolutely. I think you nailed it. So now you've heard our thoughts. Now it's time to add yours to the conversation. Hit us up on social media at Red Hat and use the #compilerpodcast. And I think that'll do it for this episode of Compiler. And this episode is written by Johan Philippine. And a big thank you again to our guest, Clarence Clayton. Compiler is produced by the team at Red Hat with technical support from Dialect. And if you like today's episode, don't keep it to yourself. Follow the show. Write the show. Leave a review, or share it with somebody else that you know might take advantage of it. And we'll see you next time.

More about segurança

A segurança dos sistemas modernos depende da capacidade de adaptação aos ataques mais recentes e da integração à infraestrutura e aos ciclos de vida dos produtos.

Blog post

Resposta da Red Hat para vulnerabilidades do OpenPrinting CUPS: CVE-2024-47076, CVE-2024-47175, CVE-2024-47176 e CVE-2024-47177

26 de setembro de 2024
Linux, Segurança
Blog post

Relatório: O Estado da Segurança do Kubernetes em 2024

26 de junho de 2024
Segurança

About the show

Compiler

Do you want to stay on top of tech, but find you’re short on time? Compiler presents perspectives, topics, and insights from the industry—free from jargon and judgment. We want to discover where technology is headed beyond the headlines, and create a place for new IT professionals to learn, grow, and thrive. If you are enjoying the show, let us know, and use #CompilerPodcast to share our episodes.