Can AI Systems Have an Area of Normal Functioning?

A synthetic TTFT experiment using dynamic ANF boundaries

As I continue developing the concept of the Area of Normal Functioning (ANF), I have been looking for new types of systems where the idea could be useful.

Artificial Intelligence - and particularly Large Language Model (LLM) services - provides an interesting case.

An AI system is still a system performing work. It receives a workload, consumes computational resources, produces results, and has measurable performance characteristics. That immediately raises a question:

Can we define an Area of Normal Functioning for an AI service?

I believe we can. 

Measuring the Performance of an LLM

Consider a user sending a request to an LLM. One particularly useful performance metric is Time to First Token (TTFT) - the time between submitting a request and receiving the beginning of the response.

For request i, we can write:

TTFT_i = t_{first token} - t_request

TTFT is affected by many factors, including workload, prompt size, concurrent requests, model characteristics, and available computing resources. Therefore, it should not be expected to remain constant.

Instead of asking whether TTFT is below some fixed threshold, ANF suggests another question: What range of TTFT values represents normal functioning of this particular AI system under its current operating conditions?

A Synthetic Experiment

To explore this idea, I generated a synthetic dataset representing 600 sequential requests to an LLM service. The simulated workload includes prompt and output token counts, changing levels of concurrent requests, an initially stable operating period, gradually increasing workload, a temporary overload period, and recovery into a somewhat different operating regime.

The TTFT itself was modeled conceptually as:

TTFT_i = f(P_i, C_i) + ε_i

where Pᵢ is prompt size, Cᵢ is concurrent workload, and εᵢ represents normal random variability.

The objective was not to reproduce any particular commercial AI system. The dataset is deliberately synthetic so that the behavior of the system is known and the ANF concept can be demonstrated clearly.

Building the AI Area of Normal Functioning

For this initial experiment I used moving percentiles to describe the current operating region. For a rolling window of recent observations, the center line is:

CL(t) = P₅₀(TTFT)

The lower and upper boundaries are:

LCL(t) = P₀₅(TTFT)

UCL(t) = P₉₅(TTFT)

Thus, the Area of Normal Functioning is:

ANF(t) = [P₀₅(t), P₉₅(t)]

The important point is that these boundaries are not fixed. They move as the behavior of the system changes.

Figure 1. Synthetic LLM service TTFT with a dynamic Area of Normal Functioning (P05-P95).

In the figure, the actual series represents TTFT; P50 describes the center of normal functioning; P05 and P95 form the lower and upper ANF boundaries; and the shaded region represents the Area of Normal Functioning.

The result is similar in spirit to the IT-Control Charts I have used for many years in computer performance analysis, but here the monitored object is an AI service.

Something Interesting Happens

During the first part of the experiment, TTFT fluctuates around a relatively stable level. As workload increases, however, both TTFT and its ANF begin moving upward. Eventually the simulated service enters an overloaded state. Its TTFT becomes substantially higher, but after enough observations the ANF itself adapts to this new operating regime.

An individual observation above the current upper boundary can be represented as:

TTFT_t > UCL(t)

Such an observation may be an exception. Persistent movement of the entire ANF, however, can indicate something different: a change in the functioning of the system itself.

Exception  ->  Trend  ->  Change Point  ->  New ANF

From Detection to Prediction

The idea becomes even more interesting when trend analysis is added. Suppose the current TTFT trend can be approximated locally by:

TTFT_pred(t) = a + bt

where b > 0 indicates increasing response latency.

If the current upper ANF boundary is UCL, the estimated crossing time is:

t_cross = (UCL - a) / b

The corresponding time remaining before the forecasted ANF boundary crossing is:

TimeToANF = t_cross - t_now

This changes the question from “Has AI performance become abnormal?” to the more useful proactive question: “How long do we have before AI performance is expected to leave its current Area of Normal Functioning?”

This connects ANF directly with proactive performance and capacity management.

Beyond TTFT

TTFT is only the first example. An AI service has many possible performance dimensions. A conceptual multidimensional formulation is:

ANF_AI = f(TTFT, TPOT, Tokens/sec, QueueTime, TokenCount, Errors, ResourceUsage, ...)

Ultimately, an AI system may have a multidimensional Area of Normal Functioning rather than an ANF based on only one metric.

Even more interestingly, these dimensions are related. A TTFT that is unusual for a small prompt may be perfectly reasonable for a much larger request or under substantially greater concurrency. A more advanced formulation could therefore be conditional:

ANF = P(Performance | Workload, Model, Configuration)

In other words: What is normal performance for this AI system under these particular operating conditions? That is a much more interesting problem than simply comparing every request with a fixed threshold.

Why I Find This Interesting

ANF originated in my earlier work with technical systems. I am now exploring whether the same fundamental idea can describe normal functioning, adaptation, and change across very different kinds of systems.

AI creates an especially interesting bridge. It is clearly a technical computing system, making it suitable for traditional performance engineering. At the same time, modern AI increasingly interacts with people and becomes part of larger human-technical systems.

For now, this experiment is intentionally simple: one AI performance metric, synthetic data, and one-dimensional ANF analysis.

The logical next step is to repeat the experiment using real measurements from an LLM service. If the approach continues to work, ANF may provide a unified way to describe not only whether an AI system is functioning normally, but also how its normal state changes - and when it is about to leave it.

Note: This blog example uses synthetic data as a proof of concept. A future academic version should validate the method using measured LLM-serving telemetry and compare ANF-based detection with fixed SLOs and other adaptive baselines.