SAS code to build Control Chart

As I have already shared at the "The Master of MASF" post there was the CMG presentation (where BTW my work and this blog were mentioned) that illustrates how to use SAS to build MASF Control Charts.

The two parts of that presentation are available at the www.MXG.com along with link to SAS code that could be used to build control charts:

Automatic Daily Monitoring of Continuous Processes Theory and Practice Frank Bereznay and MP Welch (UKCMG2011) (SWCMG2011)
Filename	Size (bytes)	Date Posted
ADMCP Part 1 PDF	1,178 KB	May 17, 2011
ADMCP Part 2 PDF	8,719 KB	May 17, 2011
Coding Sample admcp_sample.sas	4KB	May 17, 2011
HTMLBLUE SG 9.3 Style Preview (Make 9.2 look like 9.3)	31KB	May 17, 2011

Not a MASF Based Statistical Techniques (Entropy-based) for Anomaly Detection in Data Centers (and Clouds)

The following papers published on Mendeley criticizes the MASF Gaussian assumption and offer other methods (Tukey and Relative Entropy) to detect anomalies statistically. (BTW I tried to use the entropy analysis to capture performance anomalies - check my other post)

1. Statistical techniques for online anomaly detection in data centers
by Chengwei Wang, Krishnamurthy Viswanathan, Lakshminarayan Choudur, Vanish Talwar, Wade Satterfield, Karsten Schwan
  
Abstract

Online anomaly detection is an important step in data center management, requiring light-weight techniques that provide sufficient accuracy for subsequent diagnosis and management actions. This paper presents statistical techniques based on the Tukey and Relative Entropy statistics, and applies them to data collected from a production environment and to data captured from a testbed for multi-tier web applications running on server class machines. The proposed techniques are lightweight and improve over standard Gaussian assumptions in terms of performance.

2. Online detection of utility cloud anomalies using metric distributions

by Chengwei Wang Chengwei Wang, V Talwar, K Schwan, P Ranganathan

Abstract

The online detection of anomalies is a vital element of operations in data centers and in utility clouds like Amazon EC2. Given ever-increasing data center sizes coupled with the complexities of systems software, applications, and workload patterns, such anomaly detection must operate automatically, at runtime, and without the need for prior knowledge about normal or anomalous behaviors. Further, detection should function for different levels of abstraction like hardware and software, and for the multiple metrics used in cloud computing systems. This paper proposes EbAT - Entropy-based Anomaly Testing - offering novel methods that detect anomalies by analyzing for arbitrary metrics their distributions rather than individual metric thresholds. Entropy is used as a measurement that captures the degree of dispersal or concentration of such distributions, aggregating raw metric data across the cloud stack to form entropy time series. For scalability, such time series can then be combined hierarchically and across multiple cloud subsystems. Experimental results on utility cloud scenarios demonstrate the viability of the approach. EbAT outperforms threshold-based methods with on average 57.4% improvement in accuracy of anomaly detection and also does better by 59.3% on average in false alarm rate with a `near-optimum' threshold-based method.

3. EbAT : Online Methods for Detecting Utility Cloud Anomalies

Chengwei Wang in Middleware (2009)

4. Performance Metric Selection for Autonomic Anomaly Detection on Cloud Computing Systems

Song Fu in 2011 IEEE Global Telecommunications Conference GLOBECOM 2011 (2011)

 

5. Mining anomalies using traffic feature distributions

Anukool Lakhina, Mark Crovella, Christophe Diot in ACM SIGCOMM Computer Communication Review (2005)

6. Krishnamurthy Viswanathan, Lakshminarayan Choudur, Vanish Talwar et al. (2012) Ranking Anomalies in Data Centers, 1-8. In NOMS. 

7. Greg Eisenhauer, Matthew Wolf, Chengwei Wang (2010) Monalytics : Online Monitoring and Analytics for Managing Large Scale Data Centers. In ICAC.

8. Fast Anomaly Detection for Large Data Centers

Ang Li Ang Li, Lin Gu Lin Gu, Kuai Xu Kuai Xu in 2010 IEEE Global Telecommunications Conference GLOBECOM 2010 (2010)

9.Online Reactive Anomaly Detection over Stream Data

Yan Fu Yan Fu, Jun-Lin Zhou Jun-Lin Zhou, Yue Wu Yue Wu in 2008 International Conference on Apperceiving Computing and Intelligence Analysis (2008)

 

10.Semantic anomaly detection in online data sources

O Raz, P Koopman, M Shaw in Proceedings of the 24th International Conference on Software Engineering ICSE 2002 (2002)

11.Statistical anomaly detection via httpd data analysis

Daniel Q Naiman in Computational Statistics & Data Analysis (2004)

12.A comparative study of real-valued negative selection to statistical anomaly detection techniques

T Stibor, J Timmis, C Eckert in Comparative and General Pharmacology (2005)

 

 

 

 

 

MASF Control Charts Against DB2 Performance Data

I have done that before... I used for that my own variation of MASF Control Chart called "IT- Control Chart". You can see the example in my older post: Power of Control Charts and IT-Chart Concept (Part 1) :

 But not only me do that! I have found the following paper in MeasureIt :

Capacity Planning has an Important Role in Assisting Senior IT Management April, 2007, writen by Rick Isom.

The paper has a good MASF reference, the list of DB2 performance metrics that are good to be analyzed by MASF Control Charts and a few examples of Control Charts in form of 24-hour profile. One example is below (Linked to picture form the original paper published on Internet):

BTW, the actual data curve is hourly aggregated data for the particular month (October) to compare with historical base-line. Similar approach was taken in the exercise I published in the following post: Adrian Heald: A simple control chart using Captell Version 6

Adrian Heald: A simple control chart using Captell Version 6

At the CMG'11 I have met Adrian and asked him to show me how his reporting tool “Captell” (www.reportingservices.com) can be used to build MASF Control Charts. Below is his response.Check also the comment to this post with my feedback.
_____________________________________________________________________________
Introduction


A control chart uses data from a specified period to derive average and upper and lower control values. For this example we are using some CPU utilization data from a UNIX machine collected over the 4 month period January through April 2011 and delivered in a CSV file. The baseline period is January, from which we calculate average values and standard deviations for each clock hour. We can then plot our control chart and compare successive month’s average with the control to see a clear picture of change.

For more information and sample reports see www.reportingservices.com
or contact
Adrian Heald
on +61 (0)411 238 755
adrian@reportingservices.com


Step 1 - Import the CPU utilization data.


The following dialog shows the table definition selecting the “Delimited text file” source type. Specify a name and folder and choose the source type.

Here we see the text file definition, all that is required is the filename and a specification of the date time format.

Step 2 – Import and view the data

This Window shows the main Captell dialog with the task importing the data

And here a view of the imported data; during the import of the data Captell automatically determines correct data types.

Step 3 – Create a query to calculate the base line

This query calculates the average and average +/- 2 standard deviations for data from January.

The query output.

Step 4 – Create a query to summarise single months data

This query calculates the average CPU for each hour throughout the month selected by the Captell parameter ‘Data\Month’.

The query output:

Step 5 – Create a chart to combine the two queries

This chart shows the baseline average CPU utilisation and upper control limit along with the average values from the current month. Captell’s ability to plot data from different sources, in this case the baseline data and the data from the new month makes reporting quite easy. The blue line with the square symbols shows the average hourly data for March, well within the control limit and all hourly values below the baseline average.

Step 6 – Change the parameter to compare a different month

Here we can see the parameter changed to April and the resultant chart. The blue line with the square symbols shows the average hourly data for April, mostly above the upper control limit and all but one hour above the January mean, indicating a substantial increase in utilization.

(Posted with the Adrian's Heald permission)

Automatic Daily Monitoring of Continuous Processes in Theory and Practice: My CMG'11 Trip Report; Part 3

As I already announced in my following posting: CMG'11 Abstract Report shows my virtual presence  another great MASF paper was published on CMG'11 conference: 

"Automatic Daily Monitoring of Continuous Processes in Theory and Practice" written and presented by Frank Bereznay & MP Welch.

I have attended the session and here are my comments:

1. Difference from MASF and SPC was stressed. "MASF is a framework and not a detailed statistical method".

2. "... key assumption, our workload is repeatable is some fashion over time. The concept of a repeatable workload is fundamental to any sort of detection testing and needs to be validated before making any investment of time and software into developing a detection system..." That is true!

3. The weekly 168-hour profile was admitted as the best one for MASF analysis:  

- the picture from commented paper

I am glad they did that as I moved from the 24-hour profile to this one long ago.  See my 2006 paper and here is the IT-Control Chart from that:

So they suggested to have 168 separate (for each hour) group of data (separated reference sets) that exactly technique I had been using since 2006. They stressed, that you need to have at least 5 month of historical data to build that weekly profile adaptive filtering policy. And if you do not have this luxury they describe the way to reduce the number of groups, for instance by separating shifts.

At this point I would slightly disagree. To have hourly summarized 6 month historical data is not a problem anymore in the modern capacity planning processe, especially in Mainframes (they used that platform for demonstration)

4.  They published some simple SAS code fragments. I have never did that! But I have started publishing R-codes and SQL scripts as they are more popular (and open sourced) programming systems.

5. They reproduced my favorite IT-Control Chart, but against daily data:

- the picture from commented paper

That is similar with my very early attempt to build a IT-Control chart in the same my 2006 paper:

- that is my 1st Control Chart builder!

But I believe the 168 hourly control chart (I call that IT-Control Chart) is better; in spite it is a bit busy ... See another example below:

 6. Some techniques for reduction of false positives were discussed.

I glad they mentioned my way to do that by using EV meta-metric:

"One technique for reducing false positives is to measure the area between under the exception (one of Truben’s techniques) to determine the extent of the deviation. In this case, this exception would not likely warrant review and is common when using the Hourly stigmatization of this data." (I believe they misspelled my name. it is Trubin - not Truben...)

Anyway, they did extensive referencing of some of my papers and even mentioned this blog and I greatly appropriate that!

All in all it is very good paper and presentation! 

 

Power of Control Charts and IT-Chart Concept (Part 1)

This is the video presentation about Control Charts. It is based on my workshop I have already run a few times. It shows how to read and use Control Charts for reporting and analyzing IT systems performance (e.g. servers, applications) . My original IT-(Control) Chart concept within SEDS (Statistical Exception Detection System) is also presented.

The Part 2 will be about "How to build" control chart using R, SAS, BIRT and just 

If anybody interested I would be happy to conduct this workshop again remotely via Internet or in person. Just put a request or just a comment here.


UPDATE: See the version of this presentation with the Russian narration:

Jonathan Gladstone: Threshold Management Diagram

Jonathan Gladstone has worked with a team to implement pro-active Mainframe CPU usage monitoring, basing his design partly on presentations and conversations with Igor Trubin (currently of IBM) and Boris Ginis (of BMC Software).

His system does not generate any alerts on this basis, but it’s a good place to go to

• find out what’s been running hot (or cool) at the system level, and/or
• figure out why at the service class level.

It compares each interval (in this case every 10 minutes) of the most recent day’s utilization (by system and by service class) with the average for a given hour on a given day of the week over the past six weeks. Each interval is compared to the set of the last 36 values in a similar timeframe. If more than one interval in an hour is higher than the 98^th percentile for its hour & day, the hour is marked yellow; if more than four intervals are high the hour is marked red. If more than one interval is lower than the 2^nd percentile for its hour & day, the hour is marked blue. Anything in between (i.e. anything that falls within roughly x-bar±2SD) is green.

Here’s the main “CPU Overview” page from his system:

The thumbnails give an idea of what’s going on – green is within normal range. Let’s look at the Sunday, Jan. 23^rd (just because all the colours are there). Clicking on any thumbnail shows that day close up:

Without going into details about what runs in which systems, we can see that they’re listed in reverse alpha order and, of course, anyone who’s looking at this knows which system is which. The user can see that a lot of systems were running well below their normal utilization on this particular Sunday. That’s mostly because of some special testing: our developers were asked to stay off the systems if they could. To see more detail let’s choose SCA6, which has all of the colours. If we click anywhere on the bar for SCA6, this next level of detail is shown:

  

(Igor Trubin: That is very similar to IT-control charts: see  http://itrubin.blogspot.com/2009/05/seds-charts-at-scmg.html , but in the older version of 24-hour format; I prefer now to use 7x24 weekly profile shown here:  https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjizmXcXEYkyoRxutIr4xS2dj83eQeVurIX3NQhrWlge4a6BYZeXM1JU9mCvdwiwvZEtMX-m4Q93BC98g9vhEKm_vSaJguMlE7RmVyU7ih_RViHc9FmjjscNzDPgsLtYglQ_SxCgERsii4/s1600-h/Untitled.jpg) 

That chart shows the system’s total utilization (from SMF70s) for individual 10-minute intervals (green area) compared to the average, high (98%ile) and low (2%ile) values for each hour based on the last six weeks. We see why some hours are marked red, yellow or blue instead of green according to the rules above. Clicking  anywhere on the green area gets a long page full of control charts that show the same information for each defined service class within that system (from SMF72s).

Among them the following, BATH_A6, is high-priority batch. Clearly it was driving some of the yellow and red flags for this system in the 2-3 and 5-7h windows:

(This post is published here with Jonathan’s Gladstone permission. He retains all publication rights and copyright for this material)

System Management by Exception

Greetings!

To keep the discussion about how to Manage computer Systems by Exception (e.g. by  using SPC, APC, MASF, 6-SIGMA, SETDS and other techniques), I run this blog and also publish/present white papers at the www.CMG.org.  Please take a look at the following set of CMG papers related to Statistical Exception Detection System (SEDS or SETDS):

2017 -  The Model Factory - Correlating Server and Database Utilization with Customer Activity"

2016 - Is your Capacity available? 

2012 - SEDS-Lite:  Using Open Source Tools (R, BIRT and MySQL) to Report and Analyze Performance Data 

2012 - AIX frame and LPAR level Capacity Planning. User Case for Online Banking Application

2010 - IT-Control Chart

2009 - CMG Workshop Power of Control Charts: Howto Read, How to Build, How to Use 

2008 - Exception Based Modeling and Forecasting

2007- System Management by Exception, Part Final

2006 - System Management by Exception, Part 6

2005 - Capturing Workload Pathology by Statistical Exception Detection System 

2004 - Mainframe Global and Workload Level Statistical Exception Detection System Based on MASF

2003 - Disk Subsystem Capacity Management Based on Business Drivers I/O Performance Metrics and MASF

2002 - Global and Application Levels Exception Detection System, Based on MASF Technique

2001 - Exception Detection System, Based on the Statistical Process Control Concept