By Igor Trubin
Can Normal Functioning Be Determined Experimentally?
In my recent work on the Area of Normal Functioning (ANF), I have focused primarily on statistical methods for identifying and tracking the boundaries of normal system behavior.
However, revisiting my earlier research in industrial robotics led me to another approach: ANF boundaries can also be determined experimentally.
This is not a new engineering practice. Engineers have been testing operational limits of technical systems for decades. What may be new is interpreting these experiments as a systematic way of constructing and maintaining the Area of Normal Functioning.
Even more interestingly, combining experimental testing with statistical monitoring creates an opportunity for continuously evolving ANF models.
1. The Original Idea: Experimental ANF in Industrial Robotics
During my early research with industrial robots, I investigated how a robotic gripper could reliably grasp an object even when the object's actual position differed from its expected position.
The traditional approach assumed that an object should be positioned accurately enough for successful grasping.
But what happens if it is displaced?
I investigated this question experimentally using a real industrial robot.
Starting from the ideal object position, I manually introduced positional deviations and attempted to grasp the object. By progressively increasing these deviations in different directions, I could identify the region within which the grasping operation remained successful.
Beyond certain limits, grasping failed.
The resulting region represented what I now describe as an Area of Normal Functioning for the robot-object interaction.
Importantly, the boundaries were not obtained by analyzing historical operational statistics. They were established through controlled experiments.
The approach could also accommodate several dimensions, including positional displacement, orientation, and other relevant conditions.
This was an early example of constructing an ANF directly from a system's observed capabilities.
2. The Same Principle Applies to Computer Systems
Now consider a modern computing application.
Its performance depends on multiple factors:
Workload intensity and transaction arrival rates
Number of concurrent users
CPU, memory, storage, and network resources
Application and infrastructure configuration
Response-time and throughput requirements
For a particular system configuration, there is a range of workloads within which the application functions acceptably.
As the workload increases, the system may eventually violate response-time requirements, experience unacceptable error rates, or become unstable.
This is exactly what conventional load and stress testing investigates.
A load test can establish the workload range within which the application meets its operational objectives. A stress test can explore the limits beyond that range and reveal failure modes.
Why not interpret these experiments as constructing the Area of Normal Functioning?
For example, imagine an application with a response-time objective of less than 500 milliseconds at a specified percentile.
A series of load tests might establish that this objective is consistently met for workloads between 100 and 1,200 transactions per second.
Beyond that range, performance becomes unacceptable under the tested conditions.
The experimentally established workload interval can then be represented as:
ANF = [100, 1200] transactions/second
This interval is illustrative, and its validity depends on the test configuration, workload mix, performance objectives, and measurement uncertainty.
It is important to recognize that this example describes ANF in the workload dimension. The corresponding acceptable region in response-time, resource-utilization, and other dimensions may have a different shape.
In general, ANF is better understood as a multidimensional region than as a single interval.
3. Rebuilding ANF After Application Releases and Infrastructure Changes
Modern IT organizations routinely perform performance testing after significant changes, including:
Application releases
Hardware upgrades
Infrastructure migrations
Database configuration changes
Cloud instance resizing
Major architecture modifications
These tests usually answer familiar questions:
Did performance improve or deteriorate? Can the application handle the expected workload? Did the change introduce a regression?
But the same test results could answer a broader question:
How has the Area of Normal Functioning changed?
Suppose an application upgrade increases its sustainable transaction rate from 1,200 to 1,600 transactions per second while preserving the required response time.
Conventionally, we report a performance improvement.
From the ANF perspective, we can describe this as an expansion of the operating region.
Alternatively, an inefficient application release might reduce sustainable throughput from 1,200 to 900 transactions per second.
This is not merely a performance regression. It is a contraction of ANF.
Such changes could be tracked across application versions, hardware generations, or infrastructure configurations.
The resulting history would describe how the system's operating capabilities evolve over time.
4. What Happens Between Performance Tests?
Experimental determination of ANF is valuable, but it has a practical limitation.
We cannot continuously stress every production system to its operational limits.
Even when load tests are performed regularly, system behavior may change between experiments.
Workload composition may evolve. Infrastructure may experience degradation. Resource contention may increase. Seasonal demand patterns may emerge.
This is where my Statistical Exception and Trend Detection System (SETDS) becomes particularly relevant.
SETDS uses statistical analysis to identify exceptions, trends, and changes in system behavior.
Instead of experimentally pushing the system toward failure, SETDS observes its actual operating behavior and evaluates how that behavior changes.
For a measured performance metric X, the statistically estimated ANF can be represented by time-dependent boundaries:
ANF_X(t) = [L_X(t), U_X(t)]
Here L and U represent the lower and upper boundaries of the normal operating region estimated from historical observations.
These boundaries can change as the system evolves.
The key idea is to combine two complementary approaches:
Experimental ANF construction: Establish or validate acceptable operating limits using controlled performance testing.
Statistical ANF monitoring: Use SETDS to estimate normal operating patterns, detect exceptions, identify changes, and forecast the evolution of those patterns between experiments.
An important distinction must be maintained: a statistically normal operating region is not necessarily the complete region of acceptable operation.
For instance, an application may normally operate at 30–50% CPU utilization while its experimentally validated capacity allows substantially higher utilization without violating performance requirements.
Comparing these two regions could reveal unused capacity, emerging risks, and changes in operational headroom.
5. From Periodic Testing to Continuous ANF Reconstruction
There is another interesting possibility.
Some modern systems undergo automated performance testing continuously, particularly as part of software development and continuous integration processes.
My colleagues and I explored a related approach in our ICPE 2022 paper:
"Change Point Detection for MongoDB Time Series Performance Regression."
The research investigated performance measurements from automated tests triggered by changes to the MongoDB software repository.
We developed and evaluated a hybrid change-point detection approach that included the Perfomalist method and machine-learning techniques.
The objective was to identify significant performance changes without relying solely on manual analysis or fixed thresholds.
The reported change-point detection accuracy was 83%.
At the time, we described the problem primarily in terms of identifying performance regressions.
But now I see an opportunity to reinterpret and extend this work within the ANF framework.
Every automated performance test contributes evidence about a system's behavior under particular conditions.
When tests cover a sufficiently broad and controlled range of workloads, their results can contribute to reconstructing the ANF boundaries.
Meanwhile, change-point detection can identify moments when the previously established ANF model may no longer be valid.
For example:
A new software commit triggers automated performance tests.
Performance metrics are collected and compared with historical results.
SETDS or another change-point detection method identifies a significant behavioral change.
The system evaluates whether the change affects ANF boundaries.
If sufficient experimental evidence exists, the ANF model is updated; otherwise, additional targeted testing is initiated.
Thus, continuous performance testing could evolve from a mechanism for detecting regressions into a mechanism for continuous ANF reconstruction.
A single regression or change point would not necessarily establish a new boundary. It would provide evidence that a boundary may have changed and could trigger further validation.
This distinction is essential for making the approach scientifically sound.
6. ANF as a Multidimensional, Evolving Model
Another lesson from my robotics research is that normal functioning is rarely one-dimensional.
For robotic grasping, the successful operating region may depend on displacement along multiple axes, orientation, and grasping conditions.
For computer systems, the region may depend on workload, response time, resource availability, application configuration, and many other factors.
We can represent this more generally as:
ANF(t) = {x : the system satisfies defined functioning criteria under conditions x at time t}
The vector x describes the relevant operating conditions and system measurements.
Experimental testing samples this multidimensional space and helps identify the boundaries separating acceptable and unacceptable functioning.
Statistical monitoring examines system behavior within that space over time.
The combined approach makes it possible to investigate not only whether a metric is abnormal, but also:
Whether the ANF region is expanding or contracting
Whether its center is moving
Whether its shape is changing
Whether previously normal behavior is approaching a validated failure boundary
Whether a structural change requires a new experimental assessment
In earlier work, I explored multidimensional performance models using a data-cube representation, with different projections producing control charts and other operational views.
The ANF concept offers a way to connect that work with experimentally established operating regions and their evolution.
7. Toward a Closed-Loop ANF Management Process
Combining experimental and statistical approaches suggests a practical ANF management lifecycle.
Step 1 — Construct: Establish operating boundaries through controlled experiments, including load, stress, and other relevant tests.
Step 2 — Monitor: Use SETDS to observe actual operational behavior and estimate statistical ANF boundaries.
Step 3 — Detect: Identify exceptions, trends, and change points that may indicate changes in the system's functioning.
Step 4 — Predict: Forecast whether system behavior is approaching a statistical or experimentally validated boundary.
Step 5 — Validate and Rebuild: Perform new experiments when necessary to verify or reconstruct ANF boundaries following significant changes.
This creates a feedback loop in which experimental testing and statistical monitoring reinforce one another.
The same principle may apply well beyond IT systems, including manufacturing, robotics, transportation, and other adaptive technical systems.
8. Conclusion: A Different Interpretation of Familiar Engineering Practices
The central idea is straightforward.
Performance testing determines what a system can do. Statistical monitoring describes what the system normally does and how that behavior changes. ANF provides a framework for connecting the two.
The individual methods are not new.
Robotic experiments, load testing, statistical process control, and change-point detection have long histories.
The proposed contribution is to integrate them into a common model of normal functioning, where the operating region itself becomes an object of measurement, analysis, forecasting, and reconstruction.
Looking back at my early industrial robotics experiments, I find it interesting that the same fundamental question remains relevant decades later:
How far can a system deviate from its expected operating conditions while still functioning successfully?
Today, with automated performance testing, statistical monitoring, and advanced analytics, we have the opportunity to answer that question continuously.
That may be an important next step in developing the Area of Normal Functioning framework.
References
Trubin, I. — 1986. Issledovanie protsessov zakhvatyvaniya obektov manipulirovaniya promyshlennym robotom [Research of Industrial Robot Grasping Processes]. Ph.D. dissertation (in Russian).
Leznik, M., Iqbal, M. S., Trubin, I., Lochner, A., Jamshidi, P., and Bauer, A. (2022). Change Point Detection for MongoDB Time Series Performance Regression. Companion of the ACM/SPEC International Conference on Performance Engineering (ICPE '22), pp. 45–48. https://doi.org/10.1145/3491204.3527488
Trubin, I. — Exception Based Modeling and Forecasting. CMG, 2008.
Perfomalist — Statistical exception, trend, and change-point analysis. https://perfomalist.com
No comments:
Post a Comment