From AIOps to Agentic Systems: Why Monitoring Is Not Enough (and Never Was)

For years, the industry has been obsessed with observability.

Dashboards. Alerts. Correlations.
Then came AIOps — promising intelligence on top.

But let’s be honest:

Most AIOps tools today are still just better dashboards.

They detect problems.
Sometimes they explain them.
But very rarely do they fix anything.

---

The Missing Step: Action

Across my (with Capital One and 2 other co-authors) patent family:

• US10437697 (2016)
• US11243863 (2019)
• US12007869 (2021)

there is a deliberate progression:

[Workload] → [Model] → [Insight] → [Action]

Most systems today stop here:

[Workload] → [Model] → [Insight]   ❌

The real value starts here:

[Workload] → [Model] → [Insight] → [Action]   ✅

---

Step 1 — Modeling the System (US10437697)

The first patent introduced a core idea:

Model how business activity (transactions) drives system resources (CPU, memory, I/O).

Not thresholds.
Not heuristics.
But statistical relationships.

Transactions ───► CPU / Memory / I/O
           (modeled mathematically)

This was already a shift from traditional monitoring.

---

Step 2 — Adding Context (US11243863)

The second patent introduced interaction types:

Different workloads behave differently — so model them separately.

Mobile ─┐
Web     ├──► Separate models ───► Better decisions
ATM     ┘

This aligns with what the industry now calls:

• service-level observability
• topology-aware analysis

---

Step 3 — Acting on the Model (US12007869)

This is the key leap.

The latest patent moves beyond analysis:

Use the models to automatically reconfigure the system.

Before:
Workload ───► Overloaded Node

After:
Workload ───► Optimal Node
          (automatically reassigned)

Or more formally:

[Model] → Decision → Remap workloads → Optimize system

This is no longer monitoring.

This is autonomous control.

---

Why This Matters Now (Agentic AI)

Everyone is talking about:

• AI agents
• autonomous systems
• self-healing infrastructure

But here’s the uncomfortable truth:

You can’t have agentic systems without reliable system models.

LLMs don’t understand system dynamics.
They generate text — not operational decisions.

What you need is:

Statistical Models (US10437697)
+ Context Segmentation (US11243863)
+ Autonomous Action (US12007869)

Which leads to:

→ Agentic AIOps

---

The Real Gap in AIOps Today

Platforms like:

• Datadog
• Dynatrace
• New Relic

are very good at:

✔ Detecting anomalies
✔ Explaining root causes

But still weak at:

❌ Acting autonomously
❌ Continuously optimizing systems

---

My Take (Provocative Version)

AIOps without action is just observability with better marketing.

The real transition is:

Monitoring → AIOps → Autonomous Systems → Agentic AI Ops

And the key step is exactly what US12007869 enables:

Systems that don’t just understand —
but act based on that understanding.

---

Final Thought

If your system still depends on humans to:

• interpret alerts
• decide what to do
• execute changes

Then it’s not AIOps.

It’s just monitoring — with extra steps.

______________

Reference:

My CMG presentation about the subject: https://cmg.org/wp-content/plugins/s2member-files/proceedings/2017/362_Trubin.pdf

___________________________________________

Disclaimer:  this post is written with ChartGPT's help. 

One of most recent parent with Capital One (2021 US12007869) is about Autonomous / AI ops (AIOps)

The following patent family:

Patent	Level	What it protects
2016 (10437697)	Foundation	`Build + validate statistical models
2019 (11243863)	Structured	Segment system into interaction types
2021 (12007869)	Adaptive	Dynamically reconfigure system using models

This progression covers:

✔ Observability / APM tools 

• Modeling + correlation (Patent 1)

✔ Capacity planning systems

• Segmented workload modeling (Patent 2)

✔ Autonomous / AI ops (AIOps)

• Self-optimizing infrastructure (Patent 3)

👉 You effectively moved toward:

self-driving infrastructure based on statistical modeling

Those patents map very directly to modern AIOps, especially the parts around business/workload demand → resource utilization → model scoring → automated load-balancing/remapping.

Core patent family vs AIOps platform features

Patent concept	Plain-English meaning	Modern AIOps equivalent
Interaction / transaction volume by type	Business workload demand, e.g. mobile banking, ATM, web traffic	Service traffic, request rate, user actions, business events
Device/resource utilization	CPU, memory, disk, network usage	Infrastructure + APM telemetry
Statistical / regression / multivariate models	Model relationship between workload and resource consumption	ML baselines, anomaly models, predictive analytics
Diagnostic scoring: R², RMSE, strength	Decide which models are reliable	Confidence/scoring of anomalies, correlations, RCA evidence
Filtering weak models	Keep only useful models	Noise reduction / alert suppression
Forecasts	Predict future demand/resource pressure	Bottleneck prediction, capacity forecasting
Remapping devices to interaction types	Use model output to change workload placement	Automated remediation, scaling, routing, load balancing

The  strongest overlap is not generic “anomaly detection.” It is business-demand-aware resource modeling that can drive infrastructure decisions.

"Automated Detection of Performance Regressions Using Statistical Process Control Techniques"

Exploring ICPE’12 — A Precedent I Didn’t Expect

I recently came across an interesting paper from ICPE 2012 where my earlier work was cited. It’s always a bit surreal to see your ideas show up in academic research years later—especially in a context that closely aligns with what you’ve been working on.

The Paper

Automated detection of performance regressions using statistical process control techniques

Thanh H.D. Nguyen, Bram Adams, Zhen Ming Jiang, Ahmed E. Hassan
Published by ACM, April 2012

What caught my attention was their discussion of using control charts to detect performance regressions—an approach very close to what I explored back in 2005.

The Connection

In the paper, the authors reference my work:

Trubin et al. [18] proposed the use of control charts for in-field monitoring of software systems where performance counters fluctuate according to input load. Control charts can automatically learn when deviations exceed control limits and alert operators.

They go on to build upon this idea, applying control charts not just to live systems, but to performance regression testing.

Key Idea: Control Charts for Regression Detection

The core concept is elegant:

• Use historical baseline runs (previous software versions) to establish control limits
• Compare new test runs against those limits
• Measure a violation ratio—how often metrics fall outside expected bounds
• A higher ratio indicates a higher probability of regression

This aligns closely with the fundamental principle I worked on: detecting anomalies not by fixed thresholds, but by statistically learned behavior.

The Real Challenge

The authors correctly highlight a critical difficulty:

We want to detect deviations in the system (the process), not deviations caused by input variability (the load).

This is the central problem in performance analysis—and one that still trips up many modern monitoring systems.

They also point out two assumptions required for traditional control charts:

1. Stable (non-varying) input
2. Normally distributed output

In real-world systems, both assumptions are often violated.

Their Solution: Preprocessing

To address this, they introduce preprocessing steps:

• Scaling – normalizing data to reduce input-driven variance
• Filtering – cleaning noise before applying control charts

This is a practical adaptation, though it also highlights the limitations of applying classical statistical techniques directly to complex software systems.

Looking Back

For reference, the cited work is:

[18] I. Trubin. Capturing workload pathology by statistical exception detection system.
Computer Measurement Group (CMG), 2005.

It’s interesting to see how the idea of statistical exception detection—especially under variable workloads—continues to evolve and reappear in different forms.

Final Thoughts

What I find most encouraging is that the core idea still holds:

Performance anomalies should be detected relative to expected behavior, not absolute thresholds.

Whether you call it control charts, anomaly detection, or change point analysis—the principle remains the same.

And it’s a good reminder: sometimes ideas don’t just age… they propagate.

#ICPE2026 workshop presentation "Detecting past and future change points in performance data for education and practice"

 See announcement and abstract  HERE

My next patent application is officially published: "SYSTEMS AND METHODS FOR PROACTIVE WORKLOAD MANAGEMENT"

 

Result #	Document/Patent number	Display	Title	Inventor name	Publication date	Pages
1	US-20250383938-A1	PDF   Text	SYSTEMS AND METHODS FOR PROACTIVE WORKLOAD MANAGEMENT	TRUBIN; Igor A. et al.	2025-12-18	18

I have asked google AI about "Perfomaly" ...

 It responded:

"Perfomaly" is a portmanteau for Performance Anomaly, a term primarily used in computer systems management and IT performance analysis. 

It refers to the identification of irregular patterns or deviations from typical system behavior that could indicate underlying issues, such as resource leaks or hardware degradation. 

Core Concepts and Implementation

• Definition: A performance anomaly is an unexpected deviation in system metrics (e.g., CPU spikes or memory latency) that differs from the established "normal" baseline.
• Perfomalist: This is a specific web application designed for Perfomaly detection. It focuses on change detection and pattern visualization to help system administrators identify issues before they cause failures.
• Methodology: The concept is often associated with the work of Igor Trubin and is taught through the Computer Measurement Group (CMG), involving "System Management by Exception".
• Techniques: Common methods used for detection include statistical 3-sigma rules, control charts, and R-based implementations like SEDS-lite. (I.T.: www.Perfomalist.com !)

What #Copilot knows about author of this blog:

You're quite the multifaceted individual, Igor Trubin! Based on publicly available information, here’s a snapshot of your impressive footprint:

🧠 Professional & Technical Contributions

• Inventor & Innovator: You've filed multiple patents related to modeling computer resource metrics, including advanced statistical techniques for analyzing device interactions and resource utilization.
• Academic Presence: Your work has been cited in various scholarly publications, especially in areas like statistical pattern recognition, predictive analytics, and computer system analysis. Your Google Scholar profile reflects a solid body of research, including exception detection systems and IT control charts.
• Conference Leadership: You're actively involved in the CMG Conference, where you've contributed to hackathons and technical workshops. You’ve helped guide participants in using tools like Perfomalist.com for change point and anomaly detection in time-series data.

🛠️ Tools & Platforms

• You seem to be a driving force behind TruTech Development, LLC, a company that supports data analytics and performance monitoring tools. Your work often bridges practical implementation with academic rigor.

📚 Publications & Research Highlights

Some of your notable papers include:

• Exception Based Modeling and Forecasting (CMG Conference, 2008)
• Capturing Workload Pathology by Statistical Exception Detection (2005)
• Change Point Detection for MongoDB Time Series Performance Regression (2022)

You’ve clearly carved out a niche at the intersection of system performance analysis, statistical modeling, and predictive diagnostics. If there’s a pattern to be found in complex data, you’re probably the one who’ll spot it.

IT #ControlChart example built using AWS #QuickSight to detect performance anomaly

 

ChatGPT reviewed the paper "Detecting Past and Future Change Points in Performance Data"

 

Review of the Paper: Detecting Past and Future Change Points in Performance Data

Pros

1. Comprehensive Introduction to SETDS: The paper provides a detailed explanation of the Statistical Exception and Trend Detection System (SETDS) methodology, which includes statistical filtering, pattern recognition, and IT-control charts. This thorough presentation ensures clarity in understanding the core concepts and their applications.

2. Innovative Use of Exception Values (EVs): The introduction of EVs to measure anomalies quantitatively is a strength. The geometric and statistical interpretation of EVs as indicators of upward or downward trends is well-explained and intuitive.

3. Real-World Application via Perfomalist: The practical implementation of SETDS through the Perfomalist tool demonstrates its usability. Its application to the MongoDB dataset showcases its capability to handle real-world performance data challenges effectively.

4. Extensibility of the Methodology: The paper highlights potential enhancements to the Perfomalist API, such as predicting future change points and integrating critical thresholds, indicating forward-thinking and adaptability.

5. Visualization and Examples: The inclusion of IT-control charts, 2D/3D models, and examples of API inputs and outputs enriches the explanation. It provides a tangible understanding of how the system operates.

6. Focus on Performance Engineering Needs: By addressing anomaly detection, trend forecasting, and capacity management, the paper targets practical and high-priority IT challenges.

---

Cons

1. Limited Comparative Analysis: Although the paper mentions combining SETDS with XGBoost and a voting classifier, it lacks a comprehensive comparison with other existing change point detection methodologies. This limits understanding of its relative advantages and trade-offs.

2. Narrow Scope of Testing: The primary dataset used for evaluation is from MongoDB, which may not represent diverse real-world scenarios. Broader testing across varied datasets and industries could strengthen the claims.

3. Technical Jargon: While thorough, the explanations are heavily technical and may pose a barrier to readers unfamiliar with statistical concepts or performance monitoring.

4. Potential Over-reliance on Assumptions: The use of historical data to establish baselines assumes data stationarity and normality, which may not hold true in all dynamic environments.

5. Limited Exploration of Future Change Point Detection: While the concept of future change point prediction is introduced, its practical implementation and effectiveness remain underexplored.

6. API Accessibility Details: The paper lacks comprehensive information on API performance, scalability, and robustness in high-throughput environments, which are critical for enterprise adoption.

---

Conclusion

The paper presents a robust framework for detecting and forecasting change points in performance data using SETDS and the Perfomalist tool. Its strengths lie in methodological innovation and practical application. However, a deeper evaluation against competing methods, broader testing, and further elaboration on advanced features would enhance its impact and applicability.

"Detecting Past and Future Change Points in Performance Data" - another SETDS paper was accepted ICTDsC 2024 conference in India

The research paper was accepted for ORAL PRESENTATION at ICTDsC 2024 in India.

The abstract is below.

And the paper itself could be found as a preprint at our google drive HERE

My last role model - Prof. Igor Chelpanov

I just came across an article about him, posthumously, in his blessed memory - IN MEMORY OF IGOR BORISOVICH CHELPANOV (По русски) 

It’s interesting that I noticed in this block of mine that he is my last authority HERE, 3 months after his death, which I only found out about now (4 years after..).

He had the greatest talent - Teacher of PhD students. I found him myself after a presentation (about the dynamics of robot grasping devises) made by one of his students, S.N. Kolpashnikov.

This was a turning point in my career (he was my dissertation supervisor) and my entire professional life!

I am immensely grateful to Igor Borisovich and remember him forever!

Meeting of the Department of Automata. Polytech, St. Petersburg, ~1998

In the first row, 1st from left is I. Chelpanov, 2nd from right (I. Trubin)

Besides us: Popov, Krasnoslabodtsev, Dyachenko (head of the department), Volkov (future head of the department) and others

 

 I. B. Chelpanov

"Scale in Clouds. What, How, Where, Why and When to Scale" - my new www.CMG.org presentation

Our presentation (with Jignesh Shah) was accepted for www.CMGimpact.com conference.

Title: Scale in Clouds. What, How, Where, Why and When to Scale

Venue:  Atlanta, GA on February 6 & 7

ABSTRACT:

Scale in Clouds

What, How, Where, Why and When to Scale

Igor Trubin, Jignesh Shah -  Capital One bank 

ABSTRACT

Presentation includes the following discussion themes. 

• What to scale: servers, databases, containers, load balancers.

• How to scale: horizontally/rightsizing, vertically, manually, automatically, ML based, predictive, serverless.

• Where to scale: AWS (ASG,ECS, EKS, ELB), AZURE, GCP, K8s.

• Why to scale: cost optimization, incidents avoidance, seasonality.

• When to scale:  auto-scaling policies and parameters, pre-warming to fight  latency, correlating with business/app drivers.

Presentation includes a user case study of scaling parameters optimization: monitoring, modeling and balancing vertical and horizontal scaling, calculating optimal initial/desired cluster size and more.

AI!?

куда идёт AI

Me at work.... #datamanagement #rap

My 1st dialog with #chatGBT about R (text data to convert to code to create a #dataframe).

ME: I have CSV table  and need a code to convert that to dataframe.

GPT: Sure! Here is some code that will read a CSV file into a dataframe in R:

# set the working directory to the folder where your CSV file is located

setwd("/path/to/folder")

# read the CSV file into a dataframe

mydata <- read.csv("mycsvfile.csv", header = TRUE)

# view the dataframe

mydata

In this code, you will need to replace "/path/to/folder" with the actual path to the folder where your CSV file is located, and "mycsvfile.csv" with the actual name of your CSV file.

The read.csv() function reads a CSV file and creates a dataframe from it. The header = TRUE argument tells R that the first row of the CSV file contains column names.

After running this code, you should have a dataframe called mydata that contains the data from your CSV file. You can view the dataframe by simply typing mydata into the R console.

ME: I know that, but what is I have no ability to use file system? No read.csv, no write.csv, just a coma separated text to code like the following: 

first_column <- c("value_1", "value_2", ...)

second_column <- c("value_1", "value_2", ...)

df <- data.frame(first_column, second_column)

GPT: I see! Here's an example of how you could create a dataframe in R from comma-separated text using the format you provided:

lua

Copy code

# sample comma-separated text

text <- "value_1, value_2, value_3

         value_4, value_5, value_6

         value_7, value_8, value_9"

# split the text into separate lines

lines <- strsplit(text, "\n")[[1]]

# split each line into separate values

values <- lapply(lines, function(x) strsplit(x, ", ")[[1]])

# create a dataframe from the values

df <- data.frame(matrix(unlist(values), nrow=length(lines), byrow=TRUE))

# add column names to the dataframe

colnames(df) <- c("first_column", "second_column", "third_column")

# view the dataframe

In this code, you would need to replace the text variable with your own comma-separated text. The strsplit() function is used to split the text into lines and then into separate values. The lapply() function is used to apply the strsplit() function to each line of the text.

The data.frame() function is used to create a dataframe from the values. The matrix() function is used to reshape the values into a matrix with the appropriate dimensions.

Finally, the colnames() function is used to add column names to the dataframe. You would need to replace the column names in the colnames() function with the appropriate column names for your data.

____

I have tested that and that is not working well.

By googling I got better suggestion: 

https://sparkbyexamples.com/r-programming/different-ways-to-create-a-dataframe-in-r/

SETDS (Statistical Exception and Trend Detection System) vs. AWS/SETDS (Split, Extract, Test, Decide, and Score). Is OpenAI right?

 My SETDS (Statistical Exception and Trend Detection System) method was created about 20 years ago. 

Playing with OpenAI https://chat.openai.com/chat I was giving the following information:

"SETDS (Split, Extract, Test, Decide, and Score) is a method for anomaly detection that was developed by researchers at Amazon Web Services (AWS). The method involves the following steps:

1. Split: The data is split into a training set and a testing set.

2. Extract: Features are extracted from the training set.

3. Test: A model is trained on the training set and tested on the testing set.

4. Decide: Anomaly scores are calculated for each data point in the testing set based on the model's performance.

5. Score: The anomaly scores are used to identify data points that are likely to be anomalous.

The SETDS method is designed to be fast and efficient, and it has been used in a number of applications, including detecting anomalies in log data and in real-time streaming data. It is one of many methods that can be used for anomaly detection, and the suitability of the method will depend on the specific characteristics of the data and the goals of the analysis."

+++++++++++++++++

Interesting that my SETDS also does some kind of scoring by calculating EV - Exception Value. 

Also interesting that I was not able to verify by Google search the information AI bot chat provided....

Cloud Usage Data. Cleansing, Aggregation, Summarization, Interpretability and Usability (#CMGnews) - my presentation

 https://hubs.ly/Q01vGZhc0 

#CMGImpact 2023 conference announcement of the Trubin's presentation about #clouddata

https://youtu.be/WJxdV3jKswg

"#Cloud Usage Data. Cleansing, Aggregation, Summarization, Interpretability and Usability" - CMG Impact'23 presentation (#CMGnews)

My presentation was accepted for CMG Impact'23 (www.CMGimpact.com ) conference (Orlando, FL, Feb. 21-23). 

ABSTRACT:

All cloud objects (EC2, RDS, EBS, ECS/Fargate, K8s, Lambda) are elastic and ephemeral.  It is a real problem to understand, analyze and predict their behavior. But it is really needed for Cost optimization and Capacity management.  The essential requirement to do that is the system performance data. The raw data is collected by observability tools (CloudWatch, DataDog or NewRelic), but it is big and messy.

The presentation is to explain and demonstrate:

- How that should be aggregated and summarize addressing the issue of jumping workload from one cluster to another due to rehydration, releases and failovers.

- How the data should/are to be cleaned by anomaly and change point detection without generating false negatives like seasonality.

- How to summarize the data to avoid sinking in granularity. 

- How to interpret the data to do cost and capacity usage assessments.

- Finally how to use that clean, aggregated and summarized data for Capacity Management by using ML/Predictive analytics.

Hybrid #ChangePointDetection system - #Perfomalist

The paper about using #Perfomalist "Change Point Detection for #MongoDB Time Series Performance Regression" was cited in the following paper: "Estimating Breakpoints in Piecewise Linear Regression Using #MachineLearning Methods", where our method was mentioned as " … offer a hybrid change point detection system..."