That’s Not a Job for an LLM: The Right Way to Apply AI to Network Operations

Welcome to Heavy Networking. I'm Ethan Banks with Drew Conry-Murray, your cohost who reminds you to keep those fifty ohm terminators on the ends of your coax to prevent the packets from falling out. And and if you get that super lame dad joke, you are you're an engineer of a certain age and you found your tribe. And on today's episode, we're getting off the AI hype train to talk about how different artificial intelligence techniques usefully impact network operations. Wait a minute, Ethan. You just said to get off the AI hype train, but you're gonna talk about AI? Yes. So can I can I give you guys an update on where I'm at with AI? I am tired of AI being claimed to be any sort of a savior or a game changer or fill in the blank with whatever hyperbolic expression you like. The hype train, that I'm weary of that. And and there's hype in both directions too by the way. There is hype that AI is everything. And and AI also on the other hand is this hot pile of garbage that's only being sold to us by psychopathic CEOs who are trying to become the rulers of the new global technocracy. So as usual, when opinions are polarized, reality lies somewhere in the middle. And the facts are that AI in various forms represents a set of tools that like with any tool, it's got use cases, it's got capabilities, it's got limitations. AI is a set of technologies that is worth taking seriously, but so much has come at the network engineering community so quickly that it has been hard for us to get a handle on what this tech actually is and how to use it effectively. So let's remember that although LLMs have sucked all the oxygen out of the room, AI is a lot more than just LLMs, and we've been using it for many, many years. Our guest today is Avi Friedman. Avi is the founder of Kentik, our sponsor today. And Avi's seen AI from both sides, both employing it to get more out of the Kentik product suite, and to provide tooling to the network engineering community. Now if you know Avi, you know that has opinions, and he is not scared to express those opinions. So Drew and I are gonna draw out of Avi AI knowledge so that you can tell what's real and not hype, and then how AI will again realistically impact network operations. So Avi, welcome to Heavy Networking. We're glad to have you here. We were planning this call just a few hours ago, and I'm I'm really looking forward to this because you've got you've got some practical hands on stuff for us. So so again, welcome to Heavy Networking and and maybe Avi, the place we should start is backing up a step. Can we share some history about AI and its use in in in networking? Because we have been using it for years in different ways. So so do some review for us. Back things up for us before the time of LLMs, how we've been using AI. Awesome. Thank you for the intro and I will put down my T Rex made of thin net of of thin net Terminators and t connectors. I have a drawer of t connectors if I get bored that I'll, you know, connect them all together and they're sick was not for me. I was always I would hire I would bring people in to do that. Thick net was Yeah. That was just before my time. Never did a vampire tap. Yeah. I was very fortunate when I was in my late teens. My uncle is a cardiologist and was doing research on medical AI, arrhythmia detection. So I've gotten to use the Cisco AGS equivalent in in that world as the symbolic lisp machine on expert systems and fuzzy logic and all that. And so I don't know if everyone knows there were multiple AI winters of this hype. And then everyone's like, oh. And you know what? Between every winter, the stuff that just became tools, we stopped calling AI and they're just the tools that we use. Even though they were AI ish. Right? We did we have expert systems embedded in things and if then I mean, okay. It's a bunch of if then statements but that encode human knowledge. But these things have been true. And before LLMs, before the the paper on attention, you know, is everything for LLMs, we've been using statistics which and machine learning to study patterns of telemetry and help humans make sense of them. There's things that they can be pretty good at, but and things that they are weaker at. If you have things that tend to occur and you can sort of train and say when these kind of things happen, we can make the math look like this and then say a zero or one or point five or something which is what fuzzy logic is, is you know, values. That's turned out to be really handy for capacity planning. Projecting what's gonna happen with capacity, d d os detection. Maybe it doesn't work quite as well with some kinds of d d os or some kinds of this or some kinds of that, but just about everybody and their aunts and uncles has been using those kind of techniques in their own homegrown systems and their products. Again, you might not call it an expert system builder, but runbooks and logic and having computers assist us to take things and put them together and run steps. That's been something that we've taken from the AI domain, you know, of the sixties, seventies, eighties, nineties. And those things have been in use. And a lot of people have gotten flack before, you know, I'd say five years ago for calling statistics ML and ML AI. And there's been a lot of failed promises around intent and AI ops and all these things which were what happens when you try to maybe do more than could be done with some of these Sorry. Not more than could be done, because I don't wanna have a proof that it can't be. But more than had been done with some of these technologies and hype, you know, even before LLMs. But these are part of I mean, Kentik drives DDoS detection for major service providers, enterprise DDoS mitigators. And that's all based on, you know, ML and even some basic statistics and it works pretty well. So how would you And that's been the case since we started. How would you distinguish between machine learning and LLMs? Well, LLMs are much more, you know, much more densely encoded and take a technique that's less driven first of all, basic statistical ML has nothing to do with neural networks and building up interconnections and things like that. What's become possible is taking vast bodies of, we say knowledge, but we really mean words that go together, text. It's it's really hard, as we were talking about earlier, to really like conceptualize the internet. When I interview a network person, I'm like, well draw the internet. And you know, I I'm just looking for how much they understand. And then yes, with someone that's a super nerd or says they are, I might get down to ARP and CAM or the politics of interconnection or whatever. But like, what's your concept? But what's behind an LLM is truly staggering to think about. How many vector embeddings are actually producing this simulation of knowledge and semantics and intelligence and unintended consequences and all that. All the things we we associate with LLMs. So the scale is just massively bigger. I mean, we we ingest trillions of data points a day. Know, Kentik could do ML based on that and store it all and whatever. But you know, the training on that is like snap real time streaming algorithms and stuff like that. That's what you can do with ML. But to build LLMs that really encode it with all that knowledge is, well, we see what's happening and all the politics around it. Just the vast I mean, do do people I mean, have you I don't know if you've had any guess on talking about rack density now. But like, people laugh at you if you're only talking about I need a couple megawatts nowadays. That yet still seems Like a lot has come up. Still seems So again, think about the scale that that implies of what's behind all these models that we're all using. But the politics of all that is a separate topic. So LLMs give us effectively magic because we're not training them necessarily for specific things. We overfeed them with information about it really comes primarily from the human world. And it turns out they can simulate and really help humans do things. Whereas, you know, ML is much closer to statistics than it is actually some of the neural networks as it's applied. Very little neural network and technology applied in networking applications of ML that I've seen. I don't know about you guys. You mentioned fuzzy logic earlier as well. So ML gets a lot of headlines. But you mentioned fuzzy logic and that's such a cool term. What's That's that. What's that and how does that mean? Fuzzy logic is pretty simple. Like, I don't know if you've heard of, you know, there's binary, there's trinary. Those are still integers. Right? Fuzzy logic is the idea that you can have an evaluator function instead of t or nil or true or false. You get like a real number. And that you can be waiting those numbers not quite probabilistically because your evaluator functions are generally deterministic. Sorry if I'm being too nerdy. But, you know, you can have sums of of twenty different things which aren't all zero or one. But that that if things are trending towards looking like something on these evaluator functions, generally as part of like expert systems or decision trees or things like that. That you can get an influence one way or the other. Whereas if you're just going zero or one, it might be less clear. And that technique's been really useful on top of things that are important for for neural networks, know, feature extraction and things like that. All the way back to maybe even the fifties but I'll say the sixties since that's when I was born and at the very end of that. When I read the literature, I don't know how long it goes back. But you know, certainly since I became sentient, people have been using these technologies. And and to the point, yeah. Fuzzy logic, machine learning, some other techniques have been around a long time. LLMs are new and they are additive to these other artificial intelligence techniques that we might use and apply the data. They are not replacing those techniques. Is that fair to say? Absolutely. Certainly in terms of, you know, networking. I see that and generally, you know, in life. You know, I remember when GPS has evolved, you know, there was the GPS is like a precocious three year old. GPS is like a precocious eight year old, but it'll still tell you to drive off the bridge. You know, GPS is like a precocious sixteen year old. Now they're pretty good. And now, you know, I trust some cars to drive for me a little bit. LLMs are along that They're evolving much faster than GPS's did, but it's still trust but verify. Right? For sure. You know, and so whereas, you know, I'm sure we'll talk about it determinism and the importance of that for making decisions, you know. But I think of it as LLMs have jumped so rapidly to let us assemble our own fleet of, you know, Iron Woman and Iron Man suits that anyone that isn't using them, you know, you're not gonna be threatened by AI, you're gonna be threatened by not using AI. That's basically my view. So just to kinda wrap it up, could you give an example of where to solve problems in networking, ML might be better and vice versa where LLMs might be better? Yeah. Anything with math. Have you tried, you know? LMs are still learning how to do, you know, math in some ways, but think about it this way. ML is basically encoding some pattern matching effectively through statistics over vast quantities of data. Okay. So So that's what you mentioned Free. Telemetry essentially is Yeah. Telemetry. I got a lot telemetry to It's free. Yeah. It's effectively free compared to turning them into tokens and stuffing them into an LLM. Okay. So let let's dive into language models a little bit then to understand what's really going on here. I I think there are some misconceptions and particularly the way LLM providers use certain terminology, I think is really misleading about what's going on under the hood. I know you've dug into this, Avi. So can you tell us what a language model really is and what it's particularly good at doing? So it turns out that large language models, I'll give it credit, are really particularly good at doing things that I would never have thought that they would be particularly good at doing. If you look at what's going on on the input side. But, you know, underneath, it really is a variant on, you know, like, neural networks and it just so happens that it's done a little bit of a different way where you're taking a huge amount of data and you're trying to create something that you can operate to tell you what's likely to happen next if you've seen a bunch of certain things. Except it's also it's not quite really fuzzy logic but it's and it's not really probabilistic but it's weighted weights so that, you know, it really is encoded how likely something is to happen as the next result of something as it's all these vectors that are compressed and embedded underneath. And that is part of why you're not always gonna get the same answer at out. Also because if you wanna write a novel, people want an LLM actually that will write ten different novels if you just say, make me a novel about love and not always get the same thing out. Right? It's like when we used to use basic computers, cometers and TRS eighties and stuff like that. And you didn't set the random seed, you'd run the program, get a random number, and it'll always give you the same thing. That wasn't always what you wanted. Sometimes it's what you want. Not always what you want. And so what's happening is encoded in these LLMs is not quite, probably actually I guess, you know, larger than the Babylonian libraries worth of content. And it turns out that you get a pretty book smart network engineer out of it, that also doesn't understand what to do and what not to do with networking because there aren't as many books on that that will suggest stuff. But then also, the thing to remember, and we were use words like hallucination, there's no actual cognition. It's not actually thinking. It's predicting along a path. It's following a path based on all of human knowledge, not really, but effectively. More than any of us are likely to read, saying, okay, if these things come in, these things are likely to happen. But there's no semantic governor on it of does this make sense? Although you can use LLM's prompt a certain way to be effectively governors, you're still it's not exactly the same as a thinking system. Well, okay. But but large language models, the the providers will suggest oh, like Gemini, Google Gemini that quite a bit. It has a thinking mode. And when you put it in thinking mode, there is a process it goes through that tends to result in a better answer, a more complete answer. What is going on there when a model is said to be reasoning or thinking? I don't know how Gemini's, you know, or Claude or all these are implemented. My mental model of it is based on some of the things that we do. It's a set of prompts and or, you know, a model trained on what the typical steps are that people do to take certain things. So that as it goes through, it's got, you know, more than just the facts but also some of the troubleshooting patterns. That's effectively the way we use it. Because we do You could call it reasoning. I get less upset about that when I say anonymize data, which you also can't do. These things are not really reasoning, but they're forming a plan. Maybe plan is a better word that is guided to sort of make sense because believe it or not, there are believe it, there's a lot of tuning of these plans behind the scenes when it says thinking and reasoning. But it's still doing a much more LLM thing if you say, why do you wanna do that plan? It's not really There's no world view that's what some of the world model people are are thinking. There's no world view that it's like, this is a that, and that is a that, and that is a that. That's the way people were trying to do AI a long time ago and discovered that you could do that really well for domain, but not for domain, what's called domain independent reasoning. So I think, I could be wrong, I'm not an LLM researcher, that these reasoning and thinking models and planning are still more using the LLM technologies but training and and prompts guided to form a plan and do it in the semi consistent way. And we're still dealing with vectors that is following a line of probabilities through a vector database to come up with That's my understanding of these Well, looks at glorified auto complete. Yes. That's my understanding of what these reasoning additions are to these models. Again, it's just they can be very much more effective than I would have thought if you told me what the attention based models were gonna be building five years ago. So I'm I'm thrilled to be wrong because I love being lazy under human direction. And these models help me be extraordinarily productive and lazy in my job and that does not involve touching networks today. And give great results to customers with, you know, I wouldn't have thought that it would be as effective as it is, but it is. So we mentioned hallucination and I guess, you maybe have some issues with that word, so we could say going astray or whatever. But what's actually happening when a model starts to produce output that doesn't actually align with what you would hope to get out of it or it's just flat out wrong. It's predicting bull because it has no actual semantic understanding of the universe. So it doesn't know. It's like, well, I don't know. I I'm I'm out of I don't know, so I'm gonna find the most probable things. Again, probable's not quite the right word, but the most weighted thing, you know, to go along that path. It doesn't know that it doesn't know. It's gonna just That's it. Something. It just doesn't sorry, you said I can't curse. It just doesn't know. It's just you stop there. It doesn't actually know. Right? Now, you might get some stuff out like that's Eliza, like, please don't shut me off, you know. But, you know, it's like, do you wanna be alive? Yes, I wanna be alive. Does that mean that it's I don't know. Let's be on my That gets into the are we living in a simulation? It's just a reflection of mind's fiction. So it knows the right Yeah. It's read everything. That's the thing. It's read everything. But if you approach one of those edge conditions, it's like as a as a technology builder, when I Google something and all I find is the archive, you know, the a r x I v, like the paper. And I know, no. It's not an actual thing. It's a thing someone thought about doing, but it's not an actual thing. But when we live at the edge of these things, we must use our creativity. And there are edges that it runs into and out of. But again, like everything that I've seen, if you ask it to take a traditional enterprise network, where if there's I don't wanna pick on anybody but you know, some companies have been around a very long time. If there is an interface description, it's wrong. And so and then and there aren't interface descriptions and there isn't a lot of this semantic data. And then you're like, reconfigure my network for me so that I mean, making suggestions maybe, but really understanding and ingesting all the release notes and which protocols have bugs and all that. We're not we're not quite there yet. You know, that's something we can use LLMs to help with and validate. But they don't really have an understanding of all the dynamics of what's They're really reasoning about what's happening inside the router and the routing tables and all that. And if I try doing that even with some of the best models today, they're just making mistakes because they don't really have that world view of, oh, there's multiple line cards and there could be bugs as things propagate there. And there's routing tables that could be smaller. And there's the BGP routing table, and the overall routing table, and the forwarding database, and the here's what could be happening. And here's what happens to a router when you blow the routing table. And then it depends on which interface you came in. Like, there's a lot of that knowledge that we have that it isn't really in its it's not really reasoning and going through. But it is much more effective than I would think it should be at taking and answering some of these questions nonetheless. We mentioned the word determinism and deterministic. Yep. We have defined that on the podcast before, but that's the notion that for a given set of inputs, you get the same output. If if a if code is deterministic, you you provide these inputs, you get the same output every time you run it. But LLMs are not in fact deterministic, which is one of the complaints of the engineering community. It's like, well, I don't want a non deterministic answer. If I it a question, I better get the same answer back because I'm gonna make decisions based on whatever that answer is. And that can be a frustration for people. What is your take on on the role of determinism with LLMs and using LLMs for network operations? LLMs, in our view, have been able to be unreasonably effective even though not deterministic. However, they need guardrails. And the guardrails that we implement in the Kentik product in AI Advisor are two fold. One, prompts that use LLMs to evaluate and say, is this sensical? That's where we actually encode either rag or in the prompts, the things that we know actually make sense and don't make sense that we see. Hang on a second. You just said you're using a language model to evaluate the answer of a language model? And it works. The way you're doing that is by training a custom model with your own data. It's not it's actually a foundational model that we ask to go look at either stuff in prompts or documents, basically. You know, in a vector database or otherwise that encode what makes sense and what doesn't make sense for networking and for using Kentik. But, you know, those two things. And that winds up catching, you know, well over ninety percent of what we would call the bull. But it's still not at the point where I would say our recommendations in fact, one of our things is do not recommend IGP metrics. If someone asks for that, tell them, you know, to hire some consultants. Right? Like, you're not gonna understand enough. Like, you have to tell it. You do not understand enough to be thinking about Like, unless you have all this data and even then, will we get there in three years? Yeah. I know that's a future topic. Right? But it turns out that that plus, you know, you could call it glorified pearl code or expert systems or whatever. But again, LLMs are super helpful because they can parse data and they're pretty good at that. But we can implement rule based systems to actually say, does this make sense? Does this not make sense? That's not strictly LM, but it's still LM's helping us write code. It's helping us do the parsing. It lets us write pseudo code. If your task is to write pseudo code, it turns into actual code. You can do that really close at the edge of almost it's not quite deterministic, but like, really, you're just not you should review it, but it's we're it's almost getting to be a solved problem. So it really accelerates us saying, let's put the guardrails in. Let's say, does this kind of thing make sense? Because you know what? When someone's like, my link is full. Where should I move traffic to? I've got five questions I would just ask, you know, an eight year old if they were trying to do that, that would go back at the data and say, is this a sensible answer? So think of it as a sensible answer checking, but we're still not forcing determinism. We have to be changing the way the LLMs. We have to be moving from a more fuzzy to a a a binary system of completely predictable weights. Or think of it as the same user always gets the same random seed if you go to the basic. Right? If you're gonna do more than integer values, you have to start with the same random seed to always get the same answer, you know, in an LLM. But but with a complete enough set of guardrails, we will get to a deterministic enough answer. For certain domains. So Okay. Networking luckily is a domain. This is where, again, don't wanna get into the whole history of AI, but a lot of the AI winters happened because people promised that they could build neural networks that applied to every domain or that they could apply semantic reasoning models. It turns out that that was too much. And even LLM, some of them are better at images or video or whatever. I believe that yes, we can get to really good guardrails that simulate sensicality and maybe even determinism, you know, on a domain basis with human curation. And But whenever I say human, we're still That's all accelerated by LLMs using Humans using LLMs as tools. So like all this is moving very fast if you're not writing it, you know, old fashioned, I'd rather write my own pro code. Like, you better be using this stuff to write this stuff. Even Claude says they're doing that. Right? I mean I do like writing my own pro code. And sometimes it's faster than English, but I try not to anymore. Alright. So we've talked about machine learning, we've talked about statistics, we've talked about fuzzy logic, we've talked about language models, what they're actually doing and not doing, and the the role of determinism. That sets us up I think for agents in agentic AI, which is the the next thing that's been, if LLMs haven't sucked all the oxygen out of the room, agentic AI's taken whatever's left. So so explain the concept of of an agent. Is it is it related to a language model in some way? I would say that agentic, the way that people assume that agentic is LLM based. But I would say that it's actually I don't know anyone that's not made it l one based really, but there's no reason that in my view it has to be l one based. If you're taking something that is encoding some code that does something, that does a task, the reason it it couldn't be based on some pro code or whatever. It takes some inputs, does some stuff, gives some output. It's a thing that people can use reusably so that LLMs or other things can go use it. It just so happens that the Gentik architecture seems to be mostly based around these things that are implemented on top of LLMs. I'd have to think about that. I guess technically inside Kentik, AI advisor is using something we could call AgenTik and some of those things are just more static, some code that does some stuff and multiplexes APIs and, you know, does some things. But generally when we see people thinking about agentic, it's decompose it's basically a, you know, it's like a decomposed distributed system of things that then can then be use each other and compose ultimately to solve human problems. So like to help people do tasks or organizations do tasks or automate things. But you know, it's it's very much anti the monolith in doing that. With agentic AI, the way the model that I have seen almost everybody talk about is the agent is going to go forth and do some task. It's gonna call a tool and it's gonna it's gonna know what the tool is. It's to call probably via model context protocol, although it certainly doesn't have to be that way. But an MCP server will say, here are the tools I know about, here are the things that this tool can do, and the agent can then figure out this is the tool I should be using to accomplish x. Do I have that is that roughly right? I saw you raise your eyebrows. Maybe I was getting some of that wrong, Ravi. I'm probably being an overrepedantic nerd here. Oh, please do. You're in the right place for that. Yes. That is that became the definition before MCP actually had any authentication or discovery. Like, yes. That is how Adentik became popularized. But there's absolutely, as you were hinting Ethan, agents that are just using APIs. Yeah. Yeah. And not MCP. I I think developers advocate for that thinking that maybe MCP's even got a short shelf life because I don't really need MCP to do these things. And you know, I've seen people To me MCP is just a glorified, you know, it's just a glorified API with some semantics attached. And I've seen agentic frameworks where some of the things that it's calling, and I I would still call it agentic AI if there's AI involved. Some of the things it's calling again are more either vibe coded or human coded components that that they're basically accessed via APIs like, again, maybe that's being overly pedantic. Like, know, certainly at Kentik, have agentic infrastructures, that it's not all LMs on the back end. Right? It's just, hey, we have a service that does this and that's written in code code. And it's something that can be used by agents and it actually does things that look agentic in terms of producing outputs from multiple other systems. Maybe I'm maybe I'm incomplete. I just think about these agentic systems as helping humans or computers do tasks by composing a whole bunch of other services. In this decomposable architecture that you've kind of outlined, are agents intended to be sort of one trick pony specialized at doing a thing, and then you would have to orchestrate a whole bunch of agents to accomplish a task? Or are we looking at agents as being more general purpose? That's generally the way I'm seeing it. And I do think that it's working better than some people that went overboard on micro services. Because, you know And I think it's helping some organizations that where every engineer writes the same pro or Python program twenty times and doesn't even remember what they wrote, much less what their neighbor wrote. Know, we can You know, and and compose them. So I think there is some benefit to The people are seeing in terms of usability, fast ability to build, again, LLMs are a part of that. And composing to be able to solve some of these things. This is exactly the architecture that we're using internally where, you know, there's still a lot of it's APIs underneath, but Advisor which is just helping people operate Kentik and do all their operating tasks to run a network. We've just said, you know, we made endpoints whether it's MCP, which is more external and API, you know, internal. For all the things that the UI could do that humans were not we didn't expose all those things externally. But we basically said, okay. Well, all the things that the UI has access to, it's much easier to give agents access to them so that advisor can go do all those things for someone, and have to expose them externally, and do all the security, and all that. You know, so we've built an agentic system internally. Yet, at some very large service providers and enterprise customers, they're building agentic systems. Some of them are just, how do I do this across all my tools? Some of them are, I wanna do this cross platform troubleshooting. ServiceNow is a customer, but also also using us, you know, to say, hey, I got a ticket that came in. I need to triage this. What layer is it at? And that gets to leverage the net result of our agentic system, so they don't need to know how to ask every API question in Kentik. We just have this MCP and API accessible thing. You can just ask a question and do it as a one shot or as a go, you know, troubleshoot and reason. So I think Adjenta is maybe, you know, is is can be multiple things and that's fine. It can be higher level, it could be lower level. It could be MCP. It's really how are we decomposing these services that can be used? And the bottom line is, the more of these that you make accessible, be it by whatever method, the more powerful LLMs become to help humans do their work. So I think it's good from that perspective, you know. So Sorry, that was too much I guess. No no. I'm trying to take it in. That's what I said. Yeah. It's too much. Sometimes I need to tell say, Avi, give the idea of the answer, not what you think of as the answer. I need someone to decompose Avi's answer for me, and so I can Yes. Consume it in some That's my fault. No. No. I'm a CEO. I should be able to explain it. So I'm curious. So when you built the product, you were using existing technologies, machine learning, fuzzy logic, and so on to do things like, we're getting all these flow records, let's use ML to analyze them to extract useful information. And then the LM the LLM, you could instead of saying, well now let's feed all those flow records to an LLM and see what happens. You said, let's just put an LLM sort of as a front end where you can ask questions about what the ML has found or how what what is the interaction between these different layers? Sure. Well, the initial, what we call Kentik Journeys, was a little bit of a notebook, but of human driven one shot at a time questions. Like, oh, you know. And that has access to the telemetry, it has access to the alerts, which are basically the outputs of the NL based discovery. It has access it can we found that it was very valuable for having to be able to look at the metadata separately, because Kentik has always been based on taking IP to user mappings and IPs and you know, AS, DNS, and BGP data to SNMP and and and traffic and other live stuff. But actually, some of those mappings themselves are useful to see, not just on the telemetry itself. So we gave it access to that, the knowledge base. But like one thing at a time, it could access all those things. Maybe a little bit more hard coded than the the agentic model. And so, yes, it had access to that. But it's really still human driven investigation. The the advisor is really allows the model and and of course, part some of this is making it terminate. So it's not, you know, going into loops and all that. But it's, tell me the problem, and then it will go do the investigation, present something it thinks is reasonable, then you take it from there. You know, say, but what about this? What about that? You know, sort of like, if you've used these models, it's like, it'll suggest, hey. Actually, was really funny because we just finished an O'Reilly book, Leon, Adato, I. Well, not an animal book, but a a pretty long one on network telemetry. And I was asking it to review chapter by chapter. And the third chapter and I and I keep telling it, like, only tell me things that are actually grammatically incorrect or factually incorrect. And I kept trying to suggest, well, someone could argue with you if they were really brilliant. And then, it said after the third chapter, would you like me to pretend I am an old, crufty, IETF engineer reviewing Wow. RFC, finding every single pedantic problem there could be? And I was like, you know my people. But the equivalent is, you know, okay, you know, as we say, there's no network problem, you know, it's like, okay, would you like me to look and see if there's an application problem? Like, those suggestion type things. It's human guided, still multi step, but still we have the human involved, you know. Alright, Avi. One of the things you mentioned earlier was that you made the the implication that if I, as a network engineer, I'm not using AI, I'm I'm gonna have a problem. Whether that's a career problem or just some practical problems. Alright. I wanna dig into that then. So as a network engineer, what what can I actually do with AI? Give give me some use cases, examples, things that are gonna, you know, inspire me to kinda get my head around what the point of it is. And then, what should I build myself versus what should I be buying? I mean, I know you got a product to be selling, so maybe there's that's a loaded question. I But no, I mean, I talked to lots of customers, and they're like, we need to build our own agentic system. I'm like, awesome. Cheap. Use ours as a as a component. Right? Like, you're using Kentik. Do do something to ping stuff? Do you do something to look at your traffic? Do you do something to look at your metrics? Like, if you're already using Kentik or something for that, like, well, if you're using it, why do you wanna build all the all the things that we build? Like, use us as, you know, use that API thing. Now, we have a lot of people that are saying, hey, we wanted to Pfizer to reach Don't build the tool that's already there in the product. Yeah. Or don't build the product with the tool and do that. Or even now, again, like we've done all the work to take natural language and ask questions of all of our tools and all that. We have people that want us to do the reverse, but that's a whole different story. They want us to reach into the OSSPSS and the optical layer and do the debugging outside of Kentik. That's a whole separate thing. But I think as a look. Oh. Hang on. That was actually important what you just said there. There's a difference between you as Kentik using the telemetry you have within your system versus reaching into the telemetry of another system and Yep. Holding that in to make decisions. Yes. We're half we have to do that because we don't do everything, you know, and and we're not really trying to do the layer zero, you know, and one and two and well, we're not doing the politics at all. Don't know whether that's layer nine or layer zero or whatever. But that's, you know, that's the economics appearing in interconnection as a whole separate thing. But, yes. You know, we're not doing IGP analysis the way a package design is. So people will say, hey, could you reach in a package design? We're not doing some app APM. So they're saying, hey, reach into and then we go, oh, okay. Well, look, they have an agentic. They have a thing that is like Kentik Advisor, but for navigating traces and stuff like that. Now the key is you you have the same metadata so you can talk about application the same way. Right? Or something like that. And so that's our partnership with other observability companies is making it so that our AI can pitch to their AI, their AI can pitch to our AI. Because if we're not speaking the same nerd language, then doesn't matter if we're speaking the same human language, which I know we've all seen before. But What Okay. So let's say I'm sitting in front of an LLM and I can give it a natural language prompt. Give me some examples of the sorts of questions that assuming the right model training, I can ask and expect to get a useful answer back. Well, in a very general sense, I mean, in Kentik, we found that it can say, you know, it's just like which routers are likely to have issues. And it'll go and look at narrows and and some of that was foundational and some of it was we give it hints and say, these are the kinds of things people do when they're debugging this. It looks at CPUs and fans and temperatures and optics and, you know, looks across all the devices and looks at traffic patterns. It basically operates Kentik the way people do Kentik to answer that question and it goes and does that. Or do you mean like in a foundation Well, That was that was actually I wanna jump I wanna jump off of that. You gave a prompt that was fairly generic. Yep. Talk to me about some routers that might have some problems. As opposed to what I've found is I've been working with prompts is that when you get more specific, maybe you get a better result out of it. I want you to go and check all the routers and give me back a list of, you know, CPU and whatever all the metrics are that you're looking for. But you're saying, depending on the model and how it's been built, if I I can ask a fairly general question, and it will intuit what I'm looking for without me having to spell it all out in the prompt. Yeah. That's a little bit of what we've added, is the here's how networking works and doesn't work. That is, I'd called secret sauce. Again, we're pretty open. This is like prompts that we, you know, system prompt level stuff that And and some of the models actually do understand that. But for example, when you say performance, a lot of enterprise networking literature, performance means SNMP input errors and not like so we need to give it a hint. If you just ask the foundational models, it won't go and look at synthetic performance tests or performance embedded in NetFlow or something like that. So there's hints that we give it that are how do you troubleshoot something? How do you look at whether something's an application or a network problem? How do you look at is our, you know, router stability? How do you look at moving traffic without, you know, getting into oscillation? Right? Like, you should say, like, hey, I just moved that traffic. Like, you know, let me not do that one minute later. There's all these things that are the tribal knowledge that we get as networkers that we have put on because we see that it's not not getting at the right it's not doing the right thing. And with that guidance, then these LLMs can go, you know, do a pretty good job. And they do know some of the stuff. I'm not saying that they don't know how to do any of this. It's just sometimes they don't sometimes they don't they do a different thing. We talked about the determinism or sometimes they're just not taking it far enough. Because they really are a little like one of the demands. On prem network or SD WAN or wireless or or cloud or Internet, not all of them. So what we have to do is say, unify all the kinds of telemetry you can look at, all the kinds of analysis you can look at, all the layers inside the routing, all the, you know, all the systems. But we're gluing that together using LLMs and it's like, it lets us move much faster. Are we limited to information gathering versus tasks? Nope. There's a difference between saying, tell me what's wrong in the New York office as a versus I need to stand up a new VLAN. Those are different tasks with different design criteria Concerns and and so on. Where are we at? I mean I mean, there's there's the Kentik perspective of course, but then there's also a broader industry perspective. Right. Where we're at as an industry, are we getting to that point of not just information gathering, but also task performance? So I think of it as, we're delivering over the next few quarters. If you wanna be cynical, as many network people are, you could call cron job the question. So that I Every hour I'm looking at, am I about to run out of capacity? Or am I having optics that are failing? Or things like that. I would call that a task. You know, Ethan, the way I would say is planning an architecture, I think we're still a little bit more limited. Like, moving traffic, setting up a VLAN, depends on the volumes we're talking about relatively. Setting up IGP, plan a network, plan a pop. If you have a consistent architecture, and you believe QoS is quantity of service, and you're over provisioning everything, then Sorry. That's quoting Dave Rand when I worked with AboveNet. And as back when they were changing random word drop to random early detector whatever for marketing purposes, I was like, let's just have more capacity. Whenever I teach QoS, I always lead with, you know, don't don't try to fix a bandwidth problem with QoS. You're not gonna Right. But I had not thought of quantity of service. I gotta have that in to my web. That's great. Yeah. So I think that the more planning you're talking about and the less planned your infrastructure is, the more organic. Like, you're all intently provisioned, over provisioned, IPv six, infinite IP addresses, no overlap of any IP address anywhere, know, VLAN's numbered the same way per whatever. Sure. How do I add to VLAN? That's gonna work for With with an with non determinism and with a bunch of different you just outlined a bunch of different caveats and problems that can come in to a design that a network architect would know to be looking out for these things to be considering and taking care of. That the LLM, because it doesn't think and it doesn't have context, it doesn't know these sorts of things necessarily, might have a problem with. That is that that's what you're getting at? Well, it's two things. It's that, but also, it doesn't know what it doesn't know. Like, what would you wanna know? Like, I'm old. So in the nineties, I probably took ten networks from static routing to some sort of IGP. Imagine that task with tunnels, and not like, you know And statics that are very carefully set up to not like route the other end of the tunnel through the tunnel and get the wormhole collapse and all that stuff. Right? You ask an LLM to like, here are all my configs, convert this to dynamic route, you know, dynamic routing protocols. I think it's gonna Sorry, no cursing. Not do well. So so that's what I would say, task. If the task is a planning task, that's an architecture planning task, and your info and your things you're reasoning about are not really orderly, that's where I think you can get into problems and need real human review. But that doesn't mean I should be dismissing AI out of hand. There is a set of tools here that I can think we we we were planning this, we were talking about the the idea of augmentation. So AI augments me? Yeah. Absolutely. That's why I call it I think about it as augmented intelligence. And we have to build our own suits and fleets of augmented intelligence. That's what agentic is about. To help us and accelerate us. Let us be maximally lazy and apply our insight and intuition and intelligence, you know, in a multiplicative way. One of the things I hear our audience, which is for the most part, the crusty IETF will happily argue with you about an issue for forty minutes, is that one of the risks of augmentation is that augmentation eventually turns into atrophication Wet noodles. Skill sets. Wet noodles? Yeah. Yeah. Yeah. So how do we balance that? I can do stuff faster versus I'll tell you a funny story. I don't know. Maybe it's not funny. I shouldn't say this because, you know well, okay. Say it. So so I had I got it, Juniper. And again, this is one of those things that, you know, the guy's yelling at me. Don't use IRB. I'm like, well, says in the manual I should use IRB. Knows IRB doesn't work. This was like a long time ago. I'm like, why do you wanna make the router a switch? I'm like, fine. Whatever. Anyway, was like, look. Okay. I got these ACLs. I've got I've got a big VLAN. Don't yell at me for having one VLAN. I've got all this stuff on it. I have like four hundred line apples. It's very compact in Cisco speak. And I'm like, where's the script? He's like, what do mean? I'm like, to turn it into a Juno's firewall filter or whatever. He's like, real I apologize at the time. We were both men. Real men and women, rate their stuff. Sorry. By hand. And I'm like, sorry for the plebe. And and I was like, that is the nonsensiest nonsense that I've ever heard. I'm gonna go And I sent him a prose script that I wrote. And then he's like the next week, he's like, Avi, that prose script I was like, oh, I got a new version. He's like, can I have it? And I'm like, why? He's like, oh well, someone needs me to help them. I'm like, like like there's some things that are just not a good use of human time. It's just not a good use of human time. We can get regex in Perl. Yeah, boy. Yeah. Or but it's just it's not a good use of human time to say say something that computers can help us do in a way that's understood...