The Trust Phase: What Does it actually Take to Trust Agentic NetOps - ILOG SA
Hello, everyone. Thank you so much for joining IBM webinar. Before we get started, I'd like to go over some housekeeping items. At the bottom of your screen, you should see a Q&A box where you can start submitting your questions for our presenters today. Later today, you'll receive a link to the on-demand recording and a link to the slide deck. So without further ado, I will pass it over to Sanil. Thank you, Aditi. And welcome, everybody. Good morning, good afternoon, good evening, wherever you are. And we are really grateful for the time that you're spending here on this really interesting topic in Agentech NetOps, which is all about trust in operations. And I really want to thank Shamus for his collaboration here and his insights that he's going to bring to this webinar. Shamus, I'm going to hand it over to you for a very quick introduction, and then off you go. Thank you. Yes, I'm Seamus McGillicuddy. I'm the Vice President of Research for Enterprise Management Associates, an analyst firm that focuses on IT management. My focus is network infrastructure and operations. I conduct a lot of market research on how enterprises design, build, and manage their networks. And obviously, my research has looked at AI quite a bit over the last few years. So my portion of this presentation is going to provide you some insight into what my research is revealing about this trust phase, what it takes to actually trust agentic net ops. The first thing I want to do is just give you a sense of the engagement we're seeing in the enterprise world. And so what we see is most IT organizations now require an AI capability from their network management vendors. When I talk to network observability and automation architects, they tell me that they have about three or four incumbent vendors who are offering them AI in their solutions, trying to convince them to buy it or use it if it's not requiring additional licensing. And they're often telling me, like, I'm just trying to figure out which one is good, which one I can trust, which one is actually going to give me value. But this data shows you, this is research from my network management megatrends research, which is a broad benchmarking study of network operations practices in the enterprise space. This research published in May. And we asked them, which of the following best describes your attitude towards AI features that your network management tools offer? And 55% of them say it's actually a requirement when they're evaluating vendors to select a tool to add to their NOC or their engineering team's tool set. They want AI in there. It has to be there, either AI features or an AI roadmap at the very least. Like AI has become a signal of innovation a lot for IT organizations, I think. So 55% say it has to Be there. Almost 30% say it's helpful but not required, which means if you offer it to us, we're going to use it. Not necessarily going to make or break your chances of selling to us and delivering us value, but we see why you're offering it. We also see adoption is really in the same place. Almost 55% say we are actually using AI tools to manage our network. They are deployed today. Another 29% say they will deploy in 12 months. Now, that number might seem high to some people, but it's not surprising because a lot of vendors are just layering AI into their products these days without upcharging or adding licensing fees. They're like, oh, we've added a virtual assistant that you can query our data. You can query your product documentation. You can ask it questions like, hey, what are the most critical alerts? And maybe it's going to be able to answer that question if it's a good virtual assistant. And some of them are using agentic capabilities as well. It's penetrating the market. so people are using it do they want to go you know fully autonomous because you know a lot of vendors are talking about autonomous networking uh self-driving networking like human on the loop human in the loop you know do you need network engineers anymore can you empty out your knock and just have a bunch of agents running things um we've i ask in a lot of like i surveyed i.t organizations six to ten times a year and so i'm always asking them questions such as how much autonomy do you want to give ai if you're going to use it right so there's some recent examples i want to highlight you know ddi very specific example of applying ai to network management, DDI, DNS, DHCP, and IP address management. Everyone's got that, right? You need network identity management in the form of IP address planning and management and DNS services and DHCP services. So everyone's got that. I surveyed 300 people who are in charge of DDI architecture inside their enterprise in the spring. and I asked them, here are five increasingly advanced ways that you can apply AI to DDI technology in particular. How far does the organization want to go? You can see here, like less than 6% say fully autonomous. Like that's what they're comfortable with. Automates most or all DDI operations without human involvement, but also virtual assistants that just, you know, respond to queries and do little else. That's legacy. Only 10% say that's where they want to be. Most of them are somewhere in the middle, like agentic AI for assisted operations, something that detects, predicts, and analyzes events, recommends actions. Like, that's Where the bulk of them want to Be. Some of them Want to be partially autonomous. Some of Them want just an active virtual Assistant. But you're seeing this recognition that agentic NetOps has value. They want to go there. They're not comfortable going autonomous, but they do see the value of, like, agents sort of assisting and supplementing and augmenting their existing engineering workforces. And that's – it's becoming a common theme. Here's some other research I did. I believe I published this in September. I surveyed 251 IT professionals who were focused on network compliance, not just, you know, their network being compliant with design standards and config requirements and stuff, but regulatory compliance, aligning networks with cybersecurity standards, things like that. and i said hey for each of the following network compliance tasks with what level of ai involvement would you be comfortable and you see very few of them are rejecting ai completely no ai involvement like that is that is not happening like four years ago that number would have been higher like when i was talking to network engineers architects four years go like a lot of them were saying ai is never going to touch the network. Beginning maybe two years ago, those people disappeared or had a change of heart. And now they see the inevitability of AI because they know it's going to add value and they need to use it. But how much do they trust it? You see anywhere from 20 to 30 percent are willing to allow AI to just drive compliance, right? That's conservative. Like most of them are being conservative. AI recommends humans approve. That is where the bulk of them want to be. So trust but verify seems to be where they want to stand right now. This chart is busy, but it is interesting because it shows you that low-risk tasks, they're a little more open to autonomy with low-risk tasks, like the blue bar at the bottom, the very bottom. Like 31% of them are open to autonomous AI-driven automation of detecting compliance violations, which is probably going to be a lot of machine learning as opposed to reasoning. But they're comfortable with that because, you know, it's useful. It's not, you Know, detecting something is one thing, like changing something is another. You see the middle, the brown bar at the bottom, 19.9% say that they are willing to autonomously let AI do, to review changes that they make to the network. That is the softest autonomous preference. Like, that tells you that, you know, reviewing changes is a high-risk task. Like, you want a human reviewing those changes to make sure that the changes are good. You don't want AI to make the final judgment on that. So that is the one where they're least willing to go autonomous. So there's levels of trust depending on what AI is doing. And it's interesting to watch that evolve over time. here's some examples of people i've talked to recently about you know their feelings about autonomous net ops it's there there's openness but but but um practice pragmatism like this network tools architect someone who's in charge of the observability automation tools at a fortune 500 retailer um one of the biggest providers of food in the world uh he says i feel like closed loop can work uh we have 40 000 sites if every one of them was identical and you have pretty good rag for use cases and good prompts and everything is figured out then it could be a good candidate for full automation unfortunately that's not the case like that's even though all of our sites are supposed to be identical it's not easy for us to maintain that uniformity people make changes with cabling or add unauthorized hardware site managers don't follow standards so he's like he likes the idea of autonomy, but his network is too messy, and he doesn't trust AI to figure out the messiness. And then another network tools architect at a Fortune 500 media company that you probably have watched TV or movies or news from says, I definitely want AI to tell me what's wrong and what the potential solution is. I'm not ready for AI to go loose and fix the problem. I want it mostly supervised. I see it more as a productivity game than a system for closed-loop repair. So he trusts AI to help but not lead. So let's talk about trust in agentic net ops and AI in general. And, like, how do you develop more trust? Like, how much should you trust it? So only 31% of the companies that we surveyed earlier this year for a study I did on AI-powered network management in general, I surveyed 458 enterprise IT professionals about how they're applying AI to network management. That was all the study was about. And so these were 458 people who were engaged with that in some way. They were either using it today, piloting it, evaluating solutions. Anyone who had zero experience with AI for network management was just kicked out of the survey. And so 31% of them said that they completely trusted. it. 64%, the bulk of them were somewhat trusting, which tells you they want to trust but verify. How do they get to complete trust? That's a big open question. We also found that the stronger their trust was in AI, it correlated with more successful implementations of AI-powered and agentic network management. It correlated with network operators using AI features for day-to-day operations, as opposed to it being shelfware and break glass in case of emergency. Like, if the people in the organization trusted the AI tools, they were using it more, which leads to more success, which leads to mehr return on investment in whatever tools they've chosen. Also, if they trusted the A.I., they were more open to autonomous agentic net ops. They were more opened to the idea of letting A.A.I. drive, at least to some extent. And also, awareness of their AI tools making mistakes, returning bad insights and results, did not have a strong correlation with trust. It did not impact trust. What we saw was people who saw a lot of mistakes coming out of their A.I. tools were just as likely to trust it as people who don't see a lotof mistakes coming of it. Which leads me to believe that if an organization is good at evaluating the quality of the AI outcomes, even if they're seeing problems, they trust the AI because they're able to identify the problems and course correct. And one other thing that's kind of like a part of this dynamic, you see this quote at the bottom of my slide from a network tools director at a very large American university, research university. He said, I have a lot of faith in AI. There's always a chance of it not being perfect, but human error is always a higher risk than AI automation. And that's true. I know that at least a quarter of the downtime or outages that enterprises have today are caused by manual errors. I've seen other analyst firms claim it's a higher amount, but I've been surveying the enterprise market about that particular topic for about six years now. And consistently, they tell me that between like 23% and 27% of all their network incidents are driven by bad manual changes. Um, so I think one of the biggest pieces of the problem here, one of the biggest issues that people have, and it goes back to that, um, that network tools architect at a large retailer who told us that, um, he has 40,000 sites. And in theory, they should all be the same, but they aren't because people change cabling. They, um. Install unauthorized hardware. They make a config change without logging it properly. The root of that, what he's describing there, is a data problem. He doesn't trust the data about his network. And it's reflected here on this chart. Are you confident or doubtful that the quality of your organization's network data can support AI-driven network management in general? So 44% said they're very confident, which is actually a pretty high, in my opinion. But what you see is this dynamic 51.5% say somewhat confident. Those are the people who know there's something wrong. Like they think they're okay. Like they're thinking they're in okay. They think they have pretty good data, but they know there are gaps because data, like when I talk to network engineers about their operational data, they admit it's a mess. Like, they have good data in places, but there are problems with it. And the information on the right side of the slide kind of teases that out. The top data quality issues to agentic net ops, the things that block them from embracing the agentic net ops. A lot of it is, like, problems with observability tools. Like, 36% say they have data collection errors. 36% says they're really struggling with data storage issues. They've got some retention issues. They want to, you know, they have, they would love to be able to store all the logs they are collecting, all the raw packets they're collecting, so that AI could just dive into it, you know, but they can't, it's too expensive. 35% said they're really struggling with inconsistent data formatting across vendors, which talks to the fact that data is fragmented um you might have like multiple sources of truth about your infrastructure like uh let's say you're using like uh meraki as a source of truth but it's but but it only knows about the cisco stuff you know for the most part like you might also have like a firewall then a third-party firewall vendor in there that has its own uh cloud-based controller that's a source of truth for that um and then you might Have like a bunch of legacy hardware and you might have like uh some sites that are using juniper and it's fragmented and the data is presented to ai in different ways um also blind spots like people aren't able to monitor as much of the network as they would like that is a frequent theme like oh well i can monitor i i can't monitor every device via polling and snmp snp polls and traps and so forth because it's just too expensive I can't capture packets everywhere. That's way too expensive. So I monitor strategic parts of my network, but I don't have a full understanding of what's happening in my network. And AI can only do so much when it's dealing with the blind spots too. And then finally, siloed data conflicts that kind of overlaps with the inconsistent data formatting issue, But also the fact that, you know, you might have like three or four places where IP addressing is mapped out. You might have, like, two different repositories for config standards. Like, your service management team has a CMDB that's got a bunch of configs in it for your network, but you don't trust it because you have a Git repository for configs, you Know. They don't agree. They don' agree. They don'' agree. But what is AI supposed to do? Does AI know where to go? Is it going to always go to the right place? So how do network operators learn to trust agentic solutions? We ask them, like, in what ways can your vendors best earn trust in the AI capabilities that they provide for network management? And the number one response is transparency. Tell us about your journey. Like, vendors, let's have a conversation. Like, bring your sales engineers, bring you product development engineers into a conversation and tell us, hey, like what lessons did you learn? What mistakes did you make? Where were your breakthroughs? Like, how do you think about bringing value through AI and how, you know, like, tell us why we should trust it based on what you've done wrong and what you done right. Like, I want to know. I've seen actually a lot of vendors start to really provide much more detailed architecture, like market texture of their AI capabilities. Like, hey, here are the different layers of AI. It used to be like a black box, you Know, there's a slide, black box. Maybe like, oh, we're plugged into this. And then we have all this magic here. And then here's like where we plug in your tool, your network. Now it's like, here are like five different layers. Like here's the context layer. Here's the, you know, it's much more detailed. Data source attribution, showing which systems and logs inform the insights that AI delivered. That's another thing that most of them want. And then visualizations of how insights were found, like charts, decision trees. I don't see a lot of that in the market, but it's compelling to them. and enabling manual confirmation of AI insights with access to relevant data. So trust but verify. Like, oh, I just want to make sure this is right. Those are the ones that really resonated the most. All right, so that is my part of the presentation. I'm going to defer to Sunil now, and I'll be back at the end for a chat. But thank you for your attention. Sunil, you're muted. Thank you. Thank you, Seamus. So some really interesting numbers there, especially on people willing to trust some of the results in AI, etc. I think that's great. What I'm going to do in my section right now is, actually, I want to start off with one of the couple of numbers that Shem has used just to reinforce the context. I'm going to then introduce IBM Network Intelligence as our agentic net ops product. It senses, it reasons through governed actions, and it has always a default human in the loop. And we'll go through the particular full autonomous investigation that INI is typically doing. And then I'll also show you how do we audit it? How do we know whether it is correct? How do we knowing whether it's going to be right the next day, the following day and the next week when such complex network issues happen? And that's why, you know, really the slide talks about, um agents are capable today. Um, that's the today's situation. And how do We make it more reliable is really the next sort of horizon that we are looking at in terms of developing that trust in operations. Right. So I'm going to switch slides here. So this is a couple of numbers which I, you know, took from Seamus's slides, right? And two things stand out here for me. At least it stood out for me, you knows, when he shared the slides yesterday. And more than half the teams, you know, have AI-driven, you know, network tools deployed today, which is great. And about a third completely trust what those tools are telling them. And in my opinion, how much people trust it really depends on how complex, you know, the problem is as well, right? So just give me a sec. Okay. Yeah. So, yeah, coming back, I got distracted with my camera here. It's not working properly. And so, you know, I don't think that the gap really is about, you know, capability. Like for most, you know, network problems, the models are actually good enough, right? And the hard part is actually to getting an agent to behave in the same way, you know, every time there is a network, a complex network problem. I think that reliability is actually the big problem here. And a year ago, I would say the question was, hey, you know, can your product do this or that, etc., right? And people were wide-eyed about the magic of agenting net-ops, etc. And nowadays, I think the customers have already gotten their hands, you now, really dirty with working with agents, etc., and, you kno, they've learned a lot. And so, the buyers are now asking three things, right? Will this do it? Will whatever it is doing today do this again? Why did it decide to do it in this way? And will I know it when it's actually going wrong? And those are becoming really, really important questions. In fact, I've done hundreds of demos and POCs, et cetera. And nowadays, the first question is not about whether the model is capable or what kind of model you're using or what MCP tool you're using. It's really about these trust questions that are up front for some of our customers. And so this is also another important thing. And I think a lot of the vendors try to sell AI features, but what the customer wants is totally different. They want really by outcomes. So a NOC leader isn't measured by owning an RCA agent or a time series foundation model or how many MCP tools and A2A sessions can your system handle, et cetera. Well, that's important. Don't get me wrong, but that's not a NOc leader or a NOCs measured on. A NOC is measured on like, I want to keep my routing stable. I want to keep this 5G slice within SLA. Those are very declarative things which are very important for them. And, you know, mostly important for them are how do you restore services faster, you knows, how do you reduce, you know, services impacting events, et cetera. So on the left-hand side is, you know I see a lot of the product vendors actually, you know, talking, you know, in feature terms, but it's really important for us to make sure that we address what the customer is asking for, which is basically, you know, very, very clear outcomes. And so what I mean by an outcome is what we have, like, organically built with IBM Network Intelligence, where, you know, we have these four stages in one loop, right? We have sense, reason, govern, and act in one look. That's an outcome for us because it is not an individual tool. An individual MCP tool is actually a means to an end. What does all of those things combine together to provide an outcome? It's really, really important. So sense is where, you know, we take the metrics, the alarms, the events, the logs that come in, typically from existing observability tooling. Look, we are not going to replace any of these existing observabilty tools. You know, IBM network intelligence is actually an operational agentic persistent layer on top of observability tools. So it has a time series foundation model, a model ensemble, which groups, you know, telemetry into observations, and then I&I adds context around these observations. That's what we call as symptoms. And then I & I hands those symptoms to a bunch of agents. And the agent, as you know, is just the model plus its harness. And it works on those symptoms, not from raw telemetries. Very important to notice that we don't throw raw teleometry at the model. You know, that's probably hierarchy. You know there's a lot of problems with that. So we have to treat the data first and then, you know, throw only the symptoms and the context to the agent for it to actually do its reasoning, which brings me to the reasoning piece, which is what we call as a network level investigation. Very important. If one problem shows symptoms on many devices, you get one investigation, not one investigation per device. So a master agent then sort of directs the sub-agents, they can use skills, tools, workflows, documents through RAG, live context through the network, you know, through MCP, et cetera. And of course, engineers can actually contribute what they know to, like the tribal and the tacit knowledge that they have back to the, you know, the agent's reasoning. I think one of Shamus's last slide actually referred to that in the sense that, you know, the ability for you to contribute to the reasoning and the reasoning trajectory of the AI to correct the reasoning, reasoning trajectory of the AIs is super important to build trust. And that is actually a feature within, you know, INI itself. And then you have this govern where you have the agentic governance, every tool has approvals, any tool that changes config has approvals, credentials are scoped per component, you know, there are banned commands, tools are scoped to the task, and only those tools can run, you know, something, return something meaningful, and many, many other things as well. And of course, the action surface, which is this governed remediation, actually, INI can actually write a remediation for an incident as an automation playbook, contextually, and then validate that as a change before it can run. So tickets then can carry evidence. And, you Know, there's a check before and after every change that you do. So all of these trust signals are already sort of taken care of, you know, within INI. And so if I had to put this whole product under one page, right, this is how it would look like. And I'm not going to go into, like, all of these boxes. I think Shamus made an interesting point that earlier it was black boxes. Nowadays, you knows, AI product architectures are very clear in terms of, very transparent in terms OF what exactly they do, et cetera. That's what you're seeing here. We have the same sense, reason, act, and govern layers. and we have all of these specific functionalities which come together as one single cohesive product. So the most important things here are the data comes in from your existing tools that you already run or Kafka or Webhooks, et cetera. There is governance that is applied to everything that the agent does. And it runs in assistant mode or in autonomous mode in the cloud or on-prem, right? So I want to now show you a working demo. It's around seven minutes demo of what do I mean by an autonomous investigation? You know, where can you, like, look at all of these trust signals that I talked about in terms of human in the loop, in terms OF adding, you know, tacit or tribal knowledge back to the reasoning traces, and all of the, you now, individual AI features merging into an end-to-end outcome for our customers. I'm going to do that right now. Just give me a sec. Okay, so I'm gonna run this video now. So, you know, it has voiceover, so, you know, you can clearly see what's the sense, reason, govern, and act outcomes that INI is producing. Bank of the World is a customer of MSP International, who manages the bank's network services. Jumbo frames are being rolled out across the core. An engineer sets MTU9000 on RTR Edge 1 router's Gigabit Ethernet 3 interface, But OSPF carries the interface MTU inside its own database packets. One end now announces 9,000. The other still announces 1,500. Neither will accept the other's packets. The adjacency stops at X start. Hundreds of alarms arrive across four domains. Not one of them names the cause. No logline anywhere contains the word MTU. Traditionally, this is where a human sits down in front of a prompt, querying disparate agents by hand, piecing together breadcrumbs. IBM Network Intelligence is an autonomous, persistent operational layer that continuously ingests real-time telemetry and alerts. It is already running. Nobody prompted it. The investigation runs end-to-end on its own, and the findings arrive already assembled. A human verifies through the assistant. Let's watch that in action. Instead of waiting for human prompting, IBM Network intelligence has already started an autonomous network investigation using the data streamed to it from your network observability stack. Here, by 1 minute 4 seconds, INI correlates the first set of data, numerous threshold alerts and syslogs from various failure domains, into symptoms on RTR Edge 1 router. More incoming telemetry triggers INI to create symptoms for a neighboring device, RTR Core 2 router, due to excessive retransmissions. By 2 minutes and 19 seconds, I&I has expanded its investigation scope by including RTR-Core 2 in the ongoing investigation and continues to assess new data. Within 3 minutes and 12 seconds, INI merges a third device, RTR Core 1, where a MPLS Tunnel 100 has dropped, expanding the incident graph dynamically. Throughout this entire symptom discovery phase, the network engineer remains completely uninterrupted, blissfully unaware of the storm brewing underneath. Notice that INI's built-in time series foundation models have generated metric-based OSPF anomalies. In 5 minutes 12 seconds, INI arrived at a credible evidence-backed hypothesis, MTU mismatch between Edge 1 and Core 2. A quick look at the agentic reasoning shows that INI identifies failure domains, symptom signatures, and understands connected topologies at the control and data planes, inspects the device live states, cross-examining interface parameters and error logs, queries SEV1 and other systems of records via MCP tools, It runs a series of MCP tool calls to collect context to refute or bolster its hypotheses about the probable cause. At 7 minutes 19 seconds, INI invokes IBM Bob to dynamically author a tailored Ansible remediation playbook for the probable Cause identified and stages it directly into Ansible automation platform. Finally, it provides a summary of the full investigation, the observations, symptoms, key evidences from MCP tools, probable causes, next set of action for the network engineer, remediations, and a verification plan. Interestingly, I&I also flagged upstream integration hiccups, such as a ServiceNow API timeout. This is where human-agent collaboration enters the lifecycle. Human intervention is mandatory, not for tedious triage, but strictly for governance and verification. The engineer reviews the complete artifact package, verified root cause, live interface evidence, and concrete validation steps. Collaborating directly with the investigation, the engineer requests a re-trigger of the ServiceNow incident and change records. The user asks to also link the incident and change tickets to track the remediation when approved and applied. This contribute to investigation feature allows for human AI collaboration, where senior engineers can reorient the reasoning trajectory of INI using their own tacit or tribal knowledge, thereby improving the agentic outcomes. In this case, the engineer uses the opportunity to recreate the failed ServiceNow Incident creation attempt previously by INI, due to the API timeout issue. The INI immediately executes, generating a linked change ticket with fully articulated implementation and back-out plans. The user now reviews the incident ticket and the change ticket to verify the details. The ticket was created using a ServiceNow MCP server with specific tools for agentic collaboration. There are tools to create, search, add work notes, update incident and change tickets. The ServiceNow incident is also linked to the change-ticket by INI for easy tracking. The change ticket has details about the incident, the implementation plan, the reference to the tailored Ansible playbook to remediate the probable cause after verification. Back to the investigation. The user needs to now validate the changes proposed to bring the network services back. But in an enterprise network, you don't just fire off a change into production without verification. Looking at the investigation reasoning, we see the complete diagnosis, an MTU mismatch knocking down the OSPF adjacency and collapsing the MPLSDE tunnel. Before letting automation touch the live gear, the user wants to confirm the baseline state. In the chat window, they query the MPLSDE tunnel on RT core 1. Tunnel 100 is down because it can't find an LSP path meeting its bandwidth requirement. Next, checking the OSPs neighbors on RTR core 2 shows the adjacency to RTR edge 1 stuck in an exchange start state, constantly retransmitting packets because the MTUs don't match. Now, instead of rolling straight into execution, we validate the proposed fix in a digital twin. In the investigation notes, INI has already created an Ansible playbook which removes the MTU9000 and sets it to the default 1500 on RTR Edge's Gigabit Ethernet 3 interface. We take that validation command and run it directly against the Forward Network's Predict engine. Forward Predict is a network change verification engine that tests proposed configuration changes against a mathematically accurate digital twin of the production network before they go live. INI calls the forward networks integration over MCP to spin up a predictive change model. The prediction results speak for themselves in the forward UI. 17 regression checks evaluated. Two immediate improvements, zero regressions, and full intent verification intact. Opening the config diff confirms the surgical change. RTR Edge 1's Gigabit Ethernet 3 interface dropped from 9,000 to standard 1,500 bytes. The twin proves that restoring this link re-establishes full OSPF adjacency and brings Tunnel 100 back online safely. The user is now confident and ready to implement the proposed change. Having pre-validated the change safely in a digital twin, and having reviewed the Ansible playbook generated by INI earlier, the user engages a governed change workflow in INI to implement this change with human in the loop. With enterprise controls preserved, the change request is dispatched to an external system for senior approval. The senior engineer reviews the approval request. Once approved, Red Hat Ansible Automation Platform runs the INI authored playbook. INI calls the Ansible MCP server tools to run the job template. The user can retrieve output details and job status of the job-template execution using Ansible-MCP tools from INI. The job is run and the user verifies the run in Ansible. The run was successful, and I&I changed the configuration to harmonize the MTU to 1,500 bytes on RTR Edge 1's Gigabit Ethernet 3 interface. The network engineer validates if the network services are restored in seconds with live state MCP tools directly to the device using REST Conf or SSH. OSPF returns to full adjacency, and MPLS Tunnel 100 restores to fully operational. INI updates the probable cause of MTU mismatch on OSPF adjacency as supported by agent and closes the operational loop. By executing the entire reasoning pipeline autonomously, from continuous telemetry correlation to contextual Ansible playbook synthesis, INI dramatically accelerates time to first credible hypothesis and drives up first-pass automation adoption. This collapses engineering triage toil, maximizes the autonomous resolution rate, and shifts the human agent dynamic from repetitive manual prompting to strategic verification and policy governance. All right. That was the INI autonomous, you know, investigation demo. That's what INI is built for, to do autonomous investigations. So moving on, like what I want to do next is really, you know, to answer the question about, you know, will it do the same thing tomorrow? And if it doesn't, how would we know about it? And the rest of the presentation is all about that build of trust, you know, in INI and what are the checkpoints that we have within INI to do that. All right. So let's start with some arithmetic here, right? You see that an agent may be taking around 20 steps, or in this case, yeah, around 20 steps with each step having a 95% reliability. That actually sounds great. But then if you compound that, the whole end-to-end workflow reliability comes to around 36%. So errors do multiply very, very quickly. And you would not put a 36% reliable end- to-end AI-generated workflow anywhere close to your production network. So we know that a language model is actually non-deterministic at every step, and we don't try to change that. If you pin a model to one fixed answer, you also remove what makes it useful in a hard investigation because it has its value. And so, like, for example, it could try a second hypothesis when the first one fails. So all of those things are valuable, but we should always remember that reasoning models are non- deterministic. So we leave the model free to reason, and we make the final result repeatable with around four controls around it. So one is you constrain the inputs. Like I mentioned, the model reasons over validated observations and contextual symptoms, which INI has generated, and not from raw telemetry. So that's very important. Second one is like you ground the reasoning, like live network state through MCP that you saw, plus your own designs, your standards, and your past incidents through RAG. And, you know, I have to mention ontological grounding here through knowledge graphs, which is, I think, one of the most important, I would say, you know, topics right now in agentic net ops in terms of making and bringing more and more determinism in this totally probabilistic world that we are living, you know, as far as agentic net ops is concerned. So there's definitely huge interest across agentic right now for ontological grounding on, you know, knowledge graphs etc. And we are incubating that in INI as we speak. The next one is to govern the actions, right? So scoped tools, like you saw, you know, band commands, an engineer always approving a change. You saw that in the demo as well. The diagnosis totally stayed read-only. The remediation was anchored as a playbook. It was authored, you now, using IBM Bob, you know, as a payback. And it was pre-validated with a digital twin. These are all governed actions which increase the trust signal for our customers in order to adopt agentic net ops. And, of course, the playbook only ran when the twin showed improvements and no regressions. That's really, really, you know, important safety check that we introduce here. And then finally, of cause, we have to verify all of these outcomes, right? We checked whether the fix worked against the live network state. More importantly, we had to trace and score every run. What do I mean by that? I mean that the internal agentic investigation, every tool call is instrumented with open telemetry so that every trace and span of every agent and every MCP tool call and the end-to-end investigation is actually available for us to audit. And that's really what the customers are also asking us. Have you audited when you shipped I&I during GA? You know, what is your hallucination rate? What is your faithfulness of the answer? What is Your Resolution Honesty? You know how many, you know, tool calls, you know, are you doing and all of those things? Right. so there are four rules for for running these agents right to to answer that question earlier you know we need to change how we build agentic net ops like for example for decades software has been going through one single pipeline define build test ship the next version it works because software is deterministic it's built to be deterministic but an agent's behavior sort of moves with the data it sees the people that are using it the live network underneath and of course the idiosyncrasies and the quirks of the model. So the agent you tested three weeks ago may not actually be the agent that you're running today, even if nobody touched the code. So we have to follow these rules, like evals are your specification. You have to write down on day one, not as a QA step, but before the release, right? What do you define as a good answer? What is your grounding truth? What does a failure look like? And whether this release is better or worse than the last one second you have to measure the business outcome right the you know the accuracy the the tool calling latency and the cost of running the ai all of those things matter like the knock leader that we talked about you know asks different questions right did service impacting incidents actually go down and did network reliability improve did the knock opex come down ai is a new opex in the knock you know it there are tokens that are being used and so what is what is your cost of every investigation needs to be measured and improved all the time. Third, you know, I think we have to spend, you know, spend the last two years, at least me having spent those last two years hands-on on agenting net ops. And before that, I've worked in the NOCs. You know, I treat the agent and the model drift almost like service or network downtimes. New data, new users, a model upgrade, a new process, a prompt change, any of those things can actually change agent behavior. So you have to monitor it and alert on it just the way you monitor availability and downtimes, et cetera. And fourth one, the GA release is actually where your testing starts. So once you have instrumented like the real-time traces, production shows you how your agent behaves with real load and real questions. Production is messy. A demo looks great, but the production is messy, And every agent failure in production actually becomes the new eval test case to fix and improve on, right? So I would say, you know, we should like really measure the whole system behavior and not individual things here. So, you Know, first thing is like, you do the stepwise thing, like ask how a BGP session state is, and you should be able to call the right BGT tool. The investigation itself, did it follow the sound reasoning trajectory like you saw in the demo? The system itself, the agents, the tools, and the people together, the handoffs, the approvals, the fix, how are all of those things working together? And then over time, what changes over time? Is it the same incident the next week after a model upgrade or a prompt change or a tool change? Is itthe same answer? You need to really work on making sure all of these are absolutely reliable, right? so what does a trustworthy evaluation actually look like for us right there are four or five things we need to do here one is we have need to have golden data sets real incidents saved with the known cause and the evidence that the agent you know will will be collecting we have to have deterministic checks ai as a judge or llm as a charge is good but you also need plain code as graders with for you know yes or no answers like did it call the required tool did it call a tool that it was never supposed to be touched? Did it claim a fix without making a change? These don't need an AI as a judge. You can actually have code, which is very simple, yes or no answers. LLM as a Judge or AI judges use a second model to judge the output of the product. Does the root cause make sense? Does each claim match a tool result? These are things that we need to absolutely measure. Quality thresholds, of course, a minimum score has to pass before you release your product into into the wild right into into into customers hands and of course online evals offline evals is what i'm talking about but online evals are absolutely important like i said you have to instrument your agenting traces with open telemetry so that every live trace you know is scored as it arrives and you know you have the full test suit run you know in terms of any changes that you want to actually you know track in real time in production so really what should leaders track in agentic net ops, right? If I ran a knock, you know, these are some of the six numbers I would be asking for. Like, alarm counts and MTTR on their own actually don't show what an agent adds up, right. So capacity, incident, each analyst hands, you know, handles every shift. I think that has to improve. The time to diagnose, like you mentioned, like I mentioned in the in the demo, like from incident open to the ranked root cause with evidence, you know, what is your time to diagnosis? You know, what is the quality of this diagnosis? The share of the agent root causes that your L2 and the L3 engineers accept without any rework. Governed change. This has to be a hundred percent. Every change the agent has made has to have an audit trail and eval score and a rollback. The cost, the tokenomics or the AI economics, right? Tokens, inference, tool calls, everything should be charged to resolve a particular incident and score how much it costs, you know, to resolve this incident. And the last and the most important thing is the drift. Like you need to have regressions in your evaluations to see how much your eval scores move between releases, you know, within your product as well. So I want to be clear about one thing here, and these targets are not on, which are on the slides are just examples. You know, you have to set yours against the baseline that you measure before the pilot or, you Know, then measure it in a fixed schedule, even in production. Right. So I finally want to get to a last two or three minutes of another demo of how do we do audits and, you know, evaluations of what you saw as, you know, autonomous investigations in INI, right? So these are our own numbers, and these come from our test labs. That's the network where we actually inject faults on purpose, and this is not customer figures. But over six weeks, you know, mid-September to around October, I&I ran close to 1,900 investigations. And on an average, each investigation, you known, calls around 20 tools. The longest one was around 167 tools, to be honest, on a very complex, prolonged investigation. And about 6% of the tool calls came back with an error. And that's around 2,240 out of nearly 37,000 tool calls. So, you know, and it comes down to around 0.2 cents, 2.7 cents, you know, per audit. So we can actually do audits on all of those things completely. So I will show you that demo, you know, right now as to how we, you know, audit INI and make sure that our customers feel the trust signals that are there in INI. We are measuring the efficacies, the faithfulness, the honesty of the answer, the tool errors, the tool call so that we can continuously fix, you know, Ionize, you know, agentic framework itself. So that's what I'm going to show you now. Let me, you know, share my screen back again. It's going to be a very short demo. This is actually a live demo. So this is Ionized eval suite. By the way, this is IBM Network Intelligence. There are, this is the investigation that you saw in the demo. And this is the live investigation, which was finished in the demo that you saw. And there are other investigations as well. But this is really our evaluation suite, you know, that we are currently using to evaluate what I&I is doing, right? And we use IBM Bob as one of our AI judges to actually do a lot of these evaluations. So you will see all of those investigations that I talked about. You know, the false positive rates here are interesting. These are the false positives that INI has declared autonomously so that L2 or an L3 engineer does not need to, you know, chase the tail of a false positive. So we are declaring false positives as well. I would like to bring your attention to the category distribution here. So configuration errors, traffic anomalies, interface flaps, BGP routing instability, congestions. These are all different categories that INIs is autonomously investigating as well, right? Tool errors, the categorization progress here, you can analyze pretty much anything that you want here in terms of the categories. You will see that there are very specific investigations that have been analyzed. Let me go to the specific one here. If you go to this one, you will see that this is the one which I showed you in the demo. that's this one so you will see that the confidence is around 90 percent the RCA quality was 72 the efficiency was 80 percent and the user interest in in this particular you know investigation was 88 percent what is the you know key capability indicator key experience indicator the value realized the key business indicator how much time was saved and what does the business value you know in terms of that particular single investigation when you compare it to the per hour sort of rate in the network operation center. It also shows you, you know, the probable cause, how many tokens were used, what were the top tools that were called, and the full investigation summary, right? Exactly the whole reasoning trajectory of this particular, you now, investigation. What were the tool errors? What werethe hallucination rates? Faithfulness of the answer, et cetera. You can also then, younow, do things like go to the bob agent section in order to understand you know for all of the investigations how much money are you spending per investigation um you know what are the different categories in which you can actually score all of these investigations um you can check for drifts you can Check you know if if a particular you know investigation drifted or not um you know if the rca quality is bad you can go and understand why is it bad and things like that. And there's a separate section for it, which we call as the I&I problem section. Again, the reason I'm showing all of these things transparently is because customers expect this. This is not a black box. We are absolutely transparent about how we do audits and evaluation so that our customers feel that we are actually taking appropriate action to bring that non-determinism down and make the systems more and more reliable. So you will see here, for example, if I go to the data gaps, it'll tell you a bunch of things that, you know, happens all the time. Sure, logging as a tool has a, you now, a data problem. It's not getting its data properly. The request SSS session for, you known, trace route always has a problem. You know, IPSLA, you known, is not getting the data, although we know that there is an IPSLA probe already configured for that particular interface. Things like that are really, really important for an engineering team to go and fix it so that our system improves all the time. So that was a really quick demo of how we do agentic evaluations in real time, offline, with IBM Bob connected to all of the investigations that I&I is doing. right um let me stop sharing here and i want to you know sort of give the yeah i want sort of the uh you know thing back to um aditi here um adity we are done with the uh the sessions in terms of the slides sure son uh would you like to um go through the questions yeah absolutely start um so shamus and i are going to have a quick um q a day um regarding the content here and so shamans i have a few questions and you know if you have any questions you know to reflect on you know please please feel free to ask them but one of the one of the things that, you know, always come to mind when I have customer interactions is, like, what's the single most common reason an agent-to-KI deployment actually stalls before production? Do you have, what have you heard from all your surveys that you have been doing with enterprise customers? One key theme that I don't think we've talked about a lot here, although I think you have a good solution for it is security and compliance risk. Like it's, it often comes up as the number one barrier. One of my colleagues did research on like workload automation and applying agentic operations solutions to workload automation earlier this year and found that government's compliance to security risk was the number one thing that blocked them from engaging. And then after that, it was integration complexity. And that I saw the same thing in my research when I surveyed more than 450 enterprises about their engagement with agentic and AI-powered network management. I asked them to name their top business barriers to that engagement with installing them. And the number one response from them was also security and compliance risk. The only thing that was even close to that, other than that, was budget. So that was a big thing. And then one thing that they are finding as they deploy agents to run their network, run their infrastructure, they start to get worried about security and access control. Like, hey, what kind of role-based access do I give agentic operations? You know, that's another thing they're really kind of struggling with. Yeah. Absolutely. I mean, I completely agree. And, you know, I have the same points to make in terms of, you know, like it's rarely the model, okay? Because, you know, the model is like sufficiently good enough in terms of its intelligence, the training data that it has, it's good enough for you know um understanding uh network intelligence right but really it stalls at the security review we have the security sheriffs coming in and asking questions and for all the good reasons right the security teams like ask just one question like what exactly can this agent do to which devices on whose behalf and you know under what conditions right you know this is like really really important you know and in most deployment the honest answer is well it has a role and some tokens and that's all at the moment and so you know that actually works only for people like you mentioned our back like our back works for people you know because people you know i think can fill the gaps with judgment so an engineer won't touch a trading platform um you know connected to a core router for example at 10 in the morning um even if the role allows it to do but an agent has no such judgment right it does whatever the permission allows so as fast as it can as well and so you know as and again as often as it's asked so the the permission you know has to carry the judgment also and that comes to what we call as attribute based access control not just role-based access control where the attribute like who is the caller what is the action on whose behalf on what resource and in what context is the agent actually making you know these these are super important to be implemented upfront so that you pass security reviews you have that trust signal of security also very very important absolutely um the other question you know i would you know basically ask is um there is this phrase if you don't measure it you know you can't trust it and and so what should like buyers actually see before they you know actually believe a vendor's trust claims um this is a major issue uh because in my research i actually asked it professionals like how do you feel about your ability to evaluate ai like do you feels like um you you can actually evaluate whether the ai solutions your networking vendors provide you um actually work actually deliver value um and less than half of the people i survey uh have confidence right that they can actually evaluate this stuff um when they when they are when they do feel good about their ability to to to like evaluate what their vendors are trying to sell them uh they are usually doing four things one they're having in-depth conversations with their vendors about how those vendors built and trained their ai uh they they ask a lot of questions they they they which gets back to something i said my my my presentation around trust which is transparency vendor transparency earns trust that transparency has to happen in those conversations uh number two they're deploying like um uh they're deploying ai solutions in a proof of concept environment or or maybe just testing like turning them on temporarily and in their vendor uh their vendor like activates it and lets them play with it a little bit to see if it works that is very they have to continuously monitor those ai capabilities for accuracy observing how those ai solutions are delivering insights and making recommendations but also they want to be able to see that their feedback is impacting what ai is doing like they're like oh this this isn't useful um no i just this is not our environment is too idiosyncratic for this recommendation like the ai has to learn from them, not just be able to learn from the network. We have to learn from them. So they want that happening. They're also leaning on explainable AI features. Explainable AI, it's a whole academic discipline. And I think a lot of vendors are taking that to heart in how they present their capabilities. And then also they're working with people like me. They said that they've talked to industry analysts and read a lot of analyst research to figure out what's happening. um they don't the key point here is like don't rely on vendor demos i mean you did a couple demos here that's that's they're good um they're useful to show what you're trying to to provide in terms of value but i think most of them the ones that are um that are doing well with evaluating solutions go past uh the demo and actually dive into like those deep one-on-one conversations Absolutely. And I agree. And myself have mentioned that all of those questions which they were asking earlier in the last one year, like I'm talking about 2025, where can you do this or can you do that? And a demo can answer that question really, really well. And in 2026, the first few questions are nothing to do with the model or the agent or how many MCPs. It's all about evaluations, trust, security, right? And just to answer this, my own question, which I asked, I already answered that as part of the presentation, right, where I'm talking about how we were building software before, you know, in a very deterministic manner. And, you Know, you can't build agenting net-op systems that way anymore because you have to take care of the non-determinism that is there. And so the golden data sets that I talked about, the code checks, the quality thresholds, you Now, some of the online evaluations that we need to carry out. We need to show these to the customers and we need to make sure that they understand that, you know, we also understand the non-determinism that is there and we have to live with it because it's a feature, not a bug. And, you Know, we haveと make sure the harness around all of these things improve the overall determinism. So, yeah, absolutely. I would say the last question that I wanted to ask was, you KNOW, in one of the slides, I mentioned how vendors talk about agenting net ops. I have 1,000 tools of MCP. I have these A2A sessions that I can handle. I have this feature, that feature. More tools and more autonomy tends to increase hallucination risk. And how should a buyer think about that trade-off? I have a slightly different point of view on this, but I would like to hear yours as well. I think engineers and architects need to be paying attention to they need they need to learn how to spot mistakes uh they they need to treat ai like um a junior engineer and like check their work uh learn to spot errors bad results like and and and errors isn't necessary aren't necessarily an indicator of failure uh or quality like 46 of the people we surveyed told us that they see AI provide false or mistaken insights somewhat or very often. But that did not impact their trust in the AI because they are policing it closely. They are seeing AI recommendations that are dubious in quality, and they're making modifications to it. And they are marking it as an incorrect diagnosis and pushing AI to resolve those errors. And so, yeah, a fragmented tool set can lead to hallucination. They can treat it as a junior engineer until it starts to refine its capability. And while at the same time, you can work to improve, like, for instance, the data and the context that you're providing AI underneath to see real results there. Yeah, I mean, I've heard customers ask this question all the time because, you know, there are a plethora of MCP tools available out there, right? And you have a constraint on your context window, and you can't have thousands of tools and, you know, fill up your context windows with all of the instructions and the tools even before you have put even one single prompt into your agent TKI system. So that's really the, and so I have a slightly different view of this because, you know, the question here is not about the extra tool, but about whether that extra tool actually narrows what the agent is unsure about. If that happens, well, the tool is useful. The other way to look at it is that, is the tool just a means to an end? And what is that end outcome that you want? If you are building a tool, an MCP tool, which can give you a very specific outcome and the tool is not there just because the tool is available, I think, you know, you can make that judgment call to actually put that tool in there, right? So, you Know, that's absolutely something that we need to do because, to be honest, every tool that you add, it's a bit of a contradiction. Every tool that You add there is actually an additional data to interpret, another place for an error to come in and, You know, make the hallucination even more, and, You know just another opportunity to misread something from that particular tool output. So like, like I, like, I showed you in my demo out of the 1,839 investigations with an average of 20 tools, there was 6% of errors in tools, which means you are wasting tokens to do, you know, tool calls, you knows, do the next tool call again to correct those tool calls. Et cetera, et cetera. So, you Know, there is, there is some resources being spent and so there is a problem in when you when you add more tools but my point is that if that tool is going to reduce uh and make the ai you know less unsure about what the output is i think the tool should be useful and you know it should be you know put into into your production of course with all of the evaluations that you have to do with those tools anyway so um i don't think we have um time anymore for uh any more questions but you know we'd like to absolutely take questions from the audience as well um i would like to thank you shamus for engaging here um you know in this uh webinar in this really germane webinar in terms of the topic of trust in agentic network operations to bring your insights and your survey results which I think are very, you know, insightful. And, you know, thank you. Thank you for that. And I would say goodbye to you and the audience for listening through this entire session in terms of the two presentations and and the demos. Thanks for having me.
- Company
- ILOG SA
- Keywords
- ILOG SA