newsfilter
← Latest News
Earnings Call Transcripts·

Real-Time Event Streaming with IBM Confluent Platform for Z and LinuxONE: Performance Validation at Enterprise Scale - ILOG SA

Welcome, everyone, to the IBM Technical Webinar today. Today we present real-time event streaming with IBM Confluent Platform for Z and Linux One, performance validation at enterprise scale. We remind you that the video that you're going to watch today or the webinar you're going to see today is also available for on-demand viewing for up to one year at the same link. And you may also share with colleagues or others who may benefit from the information that you are learning today. Lastly, there is a chat feature, and we encourage you to submit questions to our speakers, and they will be answered towards the end of the presentation. And with that, let's get going. Thanks, everyone. So, hi, welcome to this webinar. Thanks for joining us today in this really interesting discussion. So we'll be talking about how Z can be the producer, consumer of real-time signals and how these signals turn into business events. So that's today's story and I am portfolio leader for IBM Z software product management integration and we have Greg with us. Greg, would you like to introduce yourself? Yes. Hello, everyone. My name is Gregory Sitchuga, and I'm product director for global client performance benchmarks for IBM Z ecosystem. Thank you, Greg. So let's quickly talk about the, I don't know, once again, yeah. So, let's quickly talk about the agenda. So yeah, so we are going to talk about what are the opportunities today What are the challenges, different challenges that we face? Why are we talking about Confluent on Z in the context of these opportunities and challenges? Performance benchmark activities that we have done with Confluent On Z, because we know that's very important for our Z customers. Then a demo around new capabilities. How do you get started with the product? And then a Q&A. Yeah. So we also have some polls for you. This is a really quick poll, and that would help us understand your architecture better, your scenarios better. So the first one is about what are your current Kafka integration patterns? Probably it will be live for maybe 10 seconds or 20 seconds. So whatever integration patterns you have, like whether you are using MQ to do eventing or you're using Kix or IMS way of transaction processing like Java SDK, etc. You're using database-related eventing like change data capture, etc., any sort of API, REST proxy, or any other way of stream processing. So you can even add anything in comments if that's if your options are not here. Okay, let me move to next slide. So the second question we had for you, poll question is, and hope this keeps it interesting for you that what are typically your sources and sync for your Kafka environment. So MQ, DB2, IMSDB, any of the VSAM datasets, sequential files, X transaction server, IAMSTM, or IAMSDB, ZDIH, ZUSSMF, Connect, or batch jobs. Okay. Hope you already got a chance to. So the last question here is about what is your Kafka processing platform? So we know there are different kind of Kafka processing platforms. There is Confluent, Event Streams, Red Hat AMQ streams, Cloud Era, Amazon MSK, OSK, or any other that you're using, open source Kafka. Okay. Okay. So hope that gives you enough time as well. So I hope we have got some poll results um okay so uh let's talk about how the technology is changing right so ai um agent agentic ai isn't a future trend uh it's in this year's plan and on the left you see is the opportunity gartner expects a third of the enterprise software apps to include agentic ai by 2028 80 of the common customer service issues will be resolved by agents and service operating costs of being able to use agentic AI will drop by 30% tentatively by 2029. So this all shows that there is going to be a massive shift to use agentech AI. But on the right is the catch. Three in four organizations do plan to deploy agentic within two years, yet 91% don't fully understand what their AI depends on. So there is a gap between how fast AI is arriving and how little control there is over what it runs on today. If agents are going to act, the data that they act on has to be trusted, governed, and checkable back to where it came from. For most of the companies, that means who are using IBM Z, this is where the transactions happen and the data that these agents operate on has to be real coming in from these transactions. And that's what our story today is about. So, yeah. So let me just share. Sorry, one second. Yeah. So next slide is the kind of highlights the problem, right? That AI agents do not act on yesterday's summary. They act on what's true right now. And this particular balance or this customer information, all this information needs to be current. And your core system already knows all about it, right? The moment it happens. A Kix transaction commits, a DB2 row changes, and the business fact exists. The problem is that those systems never announce that or say about that. They record what happened, but they do not announce it to anyone. So everything downstream, like fraud engines, personalization, and now AI agents, they wait for an extract, maybe a batch job or an overnight copy or a replicated copy. And by the time this data lands, this moment has already passed. So I would like to take a quick poll here that for in your organization, right? How fresh is the IBM Z data that your analytics or your AI sees today? Is it seconds delayed? Is that minutes later? Is It hours later? Or is it next day? Okay, so hopefully, we have got the responses. I can't see it immediately. But hopefully, I'll be able to share in some time. So yeah. So now, when once you have this data, right, you might think that, hey, from IBM Z side, I've already sent the data, I don't care how it's being used. But what your organization or you are trying to do with that data, you're actually trying to do various things, right? So before we talk about the technology, that how do we achieve it, we talk about the use cases. Like when we talk to different customers, banks, payments providers, and insurance, typically these are the common cases. Stop fraud in the moment. So not a report next morning that tells you that money left. Act before it moves in the context of transaction. Next best action, respond to a customer while they are still in the app or on the phone, not with an email or message or text message days later or, you know, hours later. A live customer view. So one current picture across every channel. So every branch, app, and different headquarters can see the same truth. And the fourth is about the AI agents you can trust. This is where your market is heading. An agent is only as good as the context it gets. So if it's acting on yesterday's data or data with no clear source or very less context, you can't let it make decision. So we do have a lot of approaches today, right? There are things that are being implemented, which are being done. And if that's the goal, it should be easily doable, but it's not. So when we have talked to a lot of customers, they find a lot of challenges, and that's why it's very hard to do this whole thing today. So imagine a transaction that happens on Z at 9 a.m. Their raw data gets extracted overnight, and a bad job transforms it, and the inside shows up next morning. So that's a full day between something happened and anyone being able to act on it, right? So there are different reasons because of which this keeps happening. The first one is the stale changes. The moment has already passed by the time the data landed. The fraud has happened and the customer has left the app. The second is point-to-point connections. So every time a new consumer or new producer is required, there is a lot of bespoke integration between the connectivity points. And there are different pipelines to maintain, different ad hoc pieces in those pipelines. And that result in fragmentation of the governance, of the lineage, of the trust, right? And the third one is no context. So things like just sending the row level change. So this one matters, right? So if you are just changing your, if you're just streaming your role level changes, your DB2 changes, and they don't tell you what caused those changes. So why this particular payment was made, what triggered it. So someone downstream has to work out what that role change meant. And that often requires a lot of, again, ad hoc work, bespoke connectivity, bespoke context creation, and sometimes it might be too late to take action on it. So this is where we talk about what next then, right? So we have these challenges. We know that the question is, if Z, what if Z didn't just record your changes, but announced it, right, in business terms? So in business term, your event could be that a claim is adjudicated or a customer payment was declined because of a fraud score. Right. So if Z announced what just happened in business terms, the moment it happened, how would things change then? okay so this is where our answer lies and it's not just a product I would say it's the full package right and the full package is about having confluent capabilities it's innovation running on z so running on linux on z. So if you see this we are talking about confluent premium package, which establishes business event refinery architecture. So the first layer is the IBM Z and Linux one that brings in the capabilities that IBM Z has, things like isolation, hardware cryptography that you already trust, performance that comes with the platform. So that's the core layer. The second part is, or the second layer is confluent core capabilities, which are already well established in market outside of Z and on Z as well since March GA. So we are talking of the brokers here, the connect capability, schema registry, cluster linking, etc. These are the same skills and capabilities that your distributed teams already use. The third layer, and which has been GA'd this November, is about the native connectivity for different Z subsystems. And it's not about just native connectivity. It's in the context of confluent capabilities, right? So you can establish lineage on those connectors. So your subsystems like DB2, ZDIH, Kix transaction server, alongside MQ, which was already available, are all these available now with the native connectivity. The layer four is about, which is going to be previewed in first quarter, is about the business event refinery architecture. So how do you create the events, correlate those events, figure out the business meaning out of those fragmented signals which are running in these different subsystems. And also things like Kafka SDK, which are going to make your overall Z native connectivity very powerful. Layer five is about native ZOS support. So we do have roadmaps for native ZOs connectors and things like ZOS brokers, ZOS connectors, etc., which is also planned. And then layer six is about the governed AI and agents running on all this ecosystem right so this is what establishes your business event refinery architecture and the reason it works is because you are able to transmit events as they happen at the transactional scale you're able to leverage the performance of z you'reable to leverage the um the transactional capabilities of z so these events will be captured at commit with no COBOL changes, no KXJVM overhead. And then they will be converted into business events, right? So from fragmented signals, we are talking of the business events here. And they'll be able to capture the lineage and the schema contracts and things like Avro support from COBOL copybook, which is like native ZOS capabilities, right. So this is where our roadmap is taking this whole platform to, right? So, yeah. So some of these things are part of the roadmap, and I must highlight that roadmaps could be subject to change, but right now this is based on the customer feedback. This is what we are planning for. Okay. Yeah. So this isthe next level deep down of how this all fits together. a second layer of business event refinery. So this is IBM Z is where the signals start, right? So your Kik subsystems, DB2 for ZOS, SMF logs, IMS, et cetera. All these are generating different signals. And your native Kafka connectors capture them without rewriting need for your application rewrites. And these applications are in touch. The service levels don't change. for the teams that they want to produce events directly from these applications. There are also capabilities like Kafka, SDK, et cetera. So these are the initial source producers and consumers from Z. Then there is the refinery part in the middle, which is the key and most important part. It captures the signals with transactional consistency because transactions are captured at commit and very low latency because everything is now co-located with your Z subsystem. It correlates them across the systems, maps the intent, and builds the context using standard tools like SMT, simple message transform, and other capabilities of the confluent. Then it also validates schemas and encrypts sensitive fields before anything leaves Z, right? You can deploy these capabilities on ZCX, Linux on Z or OpenShift, etc. So the processing layer stays close to the data. And then we talk about the middle piece, right? So that's where your confluent capabilities run. So you are not just shipping your raw signals out here, you're actually processing your signals to events that matter on the Confluent backbone within the same Z architecture. Confluent Z provides the same proven architecture as rest of your enterprise users with the lineage, governance, and rule-based security. Then we have the applications on the right that consume these finished events. Your AI agents, your ML with real-time context, analytics, automation, decisioning, etc. Right. So we are not talking about new event integration backbone here. We are we are using the same well-established streaming platform with lots of innovation that Z brings to the table and Z joins this whole event architecture as the first class producer and consumer. so going to the next one so now that we've established that uh it's really um useful to run for some of the use cases it's very useful to run these event backbone on z and the business event refinery on z itself there are other benefits that i haven't talked about yet uh so things like first of all your event backbone is very efficient and secure when you are running it on Z. Your events and systems of record maintain the Z security parameter. You are able to process your events with very low latency. And some of these things is what we'll discuss in the performance benchmark that we'll cover in the later part. You have very few moving parts. So because of the consolidation, you have Kafka brokers sprawl reduction. And you also have much lesser physical cores to manage underneath your Kafka real estate. So that becomes very easy for you. And then there is egress you don't pay. So everything happens on Z, the processing happens on z at the light speed. And in the fraction of second, like millisecond, you rather get the finished event, right? So instead of making every single event go outside of Z, then constructing things downstream, then taking the action on stale data, you are saving a lot on egress cost. Then there is trust and residency. So you have built-in hardware, cryptography, you have performance that never trades off because of security, right? And then there is no rewrites, application rewriters because of this architecture. So that's really important. So at this point, I want to call out Greg because he has spearheaded the performance activity that we have tried on Linux on Z. And he'll talk about what we have done, what we are planning to do further. But the main goal of this activity is to see that what do we achieve when we run things closer to the system of record? What are the sizing evidences? How do we ensure that we are spending much less resources? and where our limits are. So, Greg, over to you. Thank you, Anu. So, we performed two benchmarks so far. Actually, multiple tests, but if I want to group them, there will be benchmark one and benchmark two. The first one is basically comparison of Confluent Kafka performance on three x86 servers versus Confluent Kafka performance on single node OpenShift. So basically a single LPAR. The architecture is very similar. In both cases, we used open messaging benchmark and we had basically Kafka cluster containing three controllers and various number of brokers in case of x86 we have we started with as i said three physical servers because that's the minimal number of x 86 servers for high availability confluent kafka installation we use the same flash system 9500 ibm flash system 900 for both tests with block CSI driver, which basically delivers, based on our experience, the best performance results. On Linux one, as I mentioned, we use single LPAR. And since we didn't know how Confluent Kafka would behave from performance point of view, We tested with different number of processor cores, integrated facility for Linux cores. And with that, let me just show you very quickly the parameters that we used, again, as close to apples to apples as possible with emphasis on looking for the minimal minimal linux one capacity that would deliver the same performance as three x86 servers so 40 topics 100 petitions for all tests one producer per topic one consumer per topic the same message size, the same replication factor, and performance target was from 50 to 750 megabytes per second, and the test that delivered end-to-end latency less than 20 milliseconds considered to be positive. As you can imagine, because of differences in platform characteristics, we made a few observations from the very start. For example, and I'm not sharing all the observations, just a few key observations. The number of vCPUs, when we try to increase it with the number of brokers, actually deliver decreased performance because at some point, at some level of over-commitment of actual physical IFLs, the performance went down. So the conclusion is pretty simple. You need to take into account the fact that Linux 1 was designed for vertical scale and not horizontal scaling, and you can't use the same performance optimization that you would use on x86 servers. Some other parameters like linger parameter on OMB side, producer side, we needed to have different one much lower than on x86 because linux one was processing faster also since we had single lpar the number of storage adapters and network adapters assigned to confluent kafka was basically much lower on each x86 or you had the same number as that LPAR so from the very beginning as I said we tried to see if you can minimize capacity on Linux one installation and deliver the same results or close results so this table shows the summary of results again each test was performed multiple times this is the average of what we got for each throughput target and you can see from this that we got pretty close by latency within one or two milliseconds some of the tests we were achieving better than x86 servers delivered on others it was the opposite but as one of my friends said who would sweat about one or two milliseconds, right? So close enough is good enough. Another observation that we made was that usually we could deliver the same performance results on Linux 1 with few number of brokers. The next benchmark was done with a different purpose. We wanted to scale performance to much higher throughputs. And Anu already showed you quite a few use cases very briefly for large enterprises and where Confluent Platform and Confluent Kafka fit into those use cases. So this is just a very simple example based on our interactions with multiple clients. Again, it's simplified. So what it shows is that customers make different financial transactions, like at point of sale, online and mobile transactions. also go to IBM Z, which is very typical for large banks and financial institutions. Then at the end of the day, there is batch processing for settlement of those transactions and cancellations, etc. Then the results of that batch performed by IBM Z go through some ETL process. It doesn't mean that everybody will use Abinitio. There are other ETL products, but it's just a sample. Then whatever ETL creates has to be fed to operational data stores and downstream applications, and that's where Confluent Kafka can help. So, in our next benchmark, we just scaled to three LPARs. Our installation of Kafka, we used open messaging benchmark. And in this particular case, our target was, again, based on our experience with client situations. So we increased the throughput target to 2.5 gigabytes, and typically our customers do use compression, so we use compression. We changed some of the parameters based on customer inputs. So number of topics, message size was changed, number of producers changed as well. And in this test, we assumed once again, based on customer situation, that our P95 end-to-end latency is still 20 milliseconds, but we increased total consumer rate two times because number of consumers is twice the number of producers. Each of these runs on this table represents results of multiple tests, at least 10 tests for each parameter. And we found that with nine brokers and with these throughput targets under 20 milliseconds of latency, so run E gave us the best results. The next one was similar, but in collaboration with Confluent Kafka development team to match some of their tests that they performed on public cloud. We wanted to have the same publishing and consumer rate, 2.5 gigabytes per second. And as you can see, we got better latency than what was achieved by Confluent Kafka team using a cloud platform and a number of IFLs versus vCPUs we still showed that we can achieve the same results with fewer processor cores. Next, we will continue doing additional tests. In the nearest couple of weeks, we'll start using 100 gigabit integrated network adapters that were added to IBM Z17 Linux 1.5, so it's a recent addition to our platform. We also plan to start testing containerized Confluent Kafka. So the previous tests were, while we were using OpenShift, we decided to start with virtualized deployment of Confluent Kafka components. That's why OCP virtualization, which is also something that many of our customers are trying to do. So that's why we chose this deployment pattern. Please ask questions, we'll try to answer or reach out after the webinar. Back to you, Anna. Thank you, Greg. That was really good. Thank you so much. So I would just summarize what Greg mentioned in the details. details, right? So we have run a lot of tests, lot of scenarios, as you can see. And three things came out to be very consistent on the ZLinux architecture. The first one is that your P99 latency, which is 99% of the messages, right, was 6x to 8x lower as compared to open source Kafka. So 99% of the messages were delivered end-to-end. And this is being measured at 1 GB per second. So yeah. So this was much faster, right? So while 95 was comparable, P99 was way better. So that way, ZLinux was processing much more messages at the same latency as compared to x86, right? The second was if the latencies kept same and constant. So for the same comparison, we saw 30% fewer brokers required. And that happened once this scale started to increase. So while this might not be the case at 50 Mbps or 100 MbPs, wherein everything ran with three brokers, as you start to see the scale grow to 750 MBPS, et cetera, the number of brokers reduce. So that also kind of compares with how ZLinux scales. So ZLINux was able to scale vertically at higher workloads very efficiently, right, without sweating. So the second is if you are looking at physical cores comparison, so there was definitely at least 6x to 8x better core ratio. So the number of IFLs used in comparison to physical cores was much lesser. So overall, with these characteristics, based on your choice of optimization, you could choose your resource optimization, your latency optimization, or your number of brokers optimization. You have choice to run on ZLinux, right, with the advantage. Okay. So I must also state that these results were for the stated workloads that we ran in our benchmark labs on our systems. The benchmark being run was open messaging, as Greg mentioned, and this is not a general performance claim, so it could be different for the customer workloads as well, right, based on the different parameters. Okay. So now at this point, so we have proven the established need, right? So we have talked about what Confluent for Z is. We have talked about what the capabilities of the latest product are, how business event refinery helps. And the major thing that we talked about was the opportunity and challenges that we want to act in real-time. I want to show a demo here about catching the fraud in the moment. Before we launch the demo, I want you to quickly talk through what you'll see in the demo. I'll show two different demos. One is in the real- time how the user is experiencing different things, and behind the scenes what was done. As a system programmer or as an application developer. So you'll see a card payment, which will hit in Kix, and you will see the Kix emit an event that a $10 payment was made, right? And the Kafka will stream that event, and then Flink will process it. But in a complex scam scenario, when you are seeing that immediately after, let's say, $10, or a very small amount, you suddenly see a huge amount, $1,000 or $10,000. That's a potential scam. So Kix, when the second transaction happens and it goes via Kafka and then Flink detects this pattern that this is matching their complex event processing pattern, it immediately alerts and the kicks receives that alert and then event is ordered for any further payments are blocked so I think can we see the video getting launched yeah so you Thank you. the fraud in the real time itself. So I wanted to quickly check the second video, which I want to show behind the scenes, right? So yeah, so let's launch that. And okay. you you you you you Within the context of the transaction in real time, within milliseconds. So this was primarily what we wanted to showcase as the demo and post that, I think, let me just quickly share what, okay. So yeah, so now this was the different, you can think of different use cases, right? But primarily our fraud alert use case was very simple one because it was about getting the event from one subsystem and getting it processed within the Flink on ZOS itself, on ZLinux itself and getting the pattern, right. But the next part of the story is how do you really figure out other emergent events? So things like which are happening in Z, which, yeah. So things which are like are happening in Z. Let me just be on the slides. One second, I think something is changing the slides, okay. So think of your scenarios where the event is happening in different subsystems. It could be in KICS, it could be IMS, it could be DB2, and the context needs to be captured from all these separately, right? So post that, you will have to do a lot of ad hoc work to do things at your level to create the context, build the context and figure out some pattern out of that context, which is actionable in real time. And that's where the business event refinery comes into the picture, right. So if you look at the different real time business decisions, we would love to hear from you that what are the different challenges that you are facing today in terms of making those business decisions from fragmented signals? Is it about working on stale data? Or is it about the data that might not be commit consistent? Is about not having a full picture of what a business event constitutes, right? So you could say that, hey, I'm just sending raw data. I don't know what that data is being used for in the organization. Or is it about the governance around the data lineage? So you can't track the data. Your downstream applications cannot track the data, they cannot have the compliance around it, and that's why you are not dealing with the data pipelines that way, right? The next challenge which we have seen is the security around the data transfer or granular access control, et cetera, right? Then there could be technology gaps that, hey, we want to do all these things. We want to create a business event, but we don't have enough domain context or domain context lies in the Z, but the skills lie outside, right, so that's a technology gap and skill gap, and that's where AI can be used very efficiently, and that's what our business event refinery is about. So keeping or making sure that you don't have to face all these challenges. Rather, they are taken care of by the business event refinery architecture. So we would love to talk to you further. So the next steps for you would be that discover your use cases around this business event refinery infrastructure. You can connect with us. We can help you around that. We can do a short workshop to map your events, consumers, et cetera. We can doing a pilot or POC with you, wherein you can approve one high-value use case end-to-end on Z architecture. And then you can scale with us. So you can, after the proof of POT or POCs works for you, you can actually do the performance benchmark with the team here that Greg leads. they do customer benchmarks as well and you can work together with us to see that scales for you right or you can add new use cases for the scale as well so there are a lot of things you can do but i think let's start with the discovery so we are happy to uh talk further uh discuss further and connect with you uh further okay so uh let me see i have put uh last two minutes uh 10 minutes for Q&A. If we have any questions, we are happy to answer. Otherwise, you can reach out to your different IBM reps team or you can also reach out on our emails and connect with us. okay so i see one question and um it's about other than fraud what are the most common scenarios that clients are looking at to use confluent on ibmz okay this is a very good question so uh yeah so i would say a lot of scenarios uh so some of those are things like trade finance applications, which are very low latency, and customers need to make sure that these applications execute sub millisecond for the trades to happen. That's a common use case. The other common use cases are where your source and source or consumers are very close to Z. So let's say your sources are ZOS sources, DB2, Kix, and your destination is like, let's say, MongoDB or PostgreSQL, which has a proven architecture on Z Linux. So this becomes a very good reference architecture for running the whole event backbone on Z rather than streaming everything out and then using it outside, right? The third scenario, which is becoming very popular, is where your critical eventing happens on Z and Linux 1. And then through cluster linking, which ist called a hybrid broker architecture, you can send some of the events to your cloud or x86 platforms. And the fourth one is your DR scenarios or your HA scenarios. So instead of having your DR or HA in a separate cloud or separate x86 data center, a lot of customers are choosing ZLinux to be the primary or HA DR deployment site and then the other data centers as their secondary sites. so yeah a lot of different in terms of so this this was I would say deployment related fraud and crime prevention customer analytics things like where new customers need to be added to the system that requires a lot eventing so these are some of the use cases ledger data management so those are others so there are some other use cases as well where a lotof customers are using Okay. So we do have a lot of other events happening, right? So you can join some of these events like tech exchange or we have other webinars upcoming. So you can join those as well. Um, for the confluent capabilities. Okay, so, yeah. So I see some other questions and looks like they have been answered. So this is this is good. Yeah. Okay. I think that's good. So, we are happy to connect further if required. Do you want to wrap up, Michelle? Sure. Thank you so much. I hope everyone received some useful information today and will engage with us in future webinar programming. Again, thank you for your time. The link will be available for a full year, so you can go back, tune in, review anything that you missed, or feel free to share the link with your colleagues. Have a great day, everybody.

Company
ILOG SA
Keywords
ILOG SA