SETDS Methodology

2022 UPDATE

Some of the SETDS features are implemented into www.Perfomalist.com tool, which is described in the following post: https://www.trutechdev.com/2021/12/ and last release notes are HERE . The detailed Perfomalist CPD method is explained in this blog: https://www.trub.in/2020/08/cpd-change-points-detection-is-planed.html

_________________________________________________________________________________

Preparing my upcoming CMG'12 presentation about SEDS-lite I try to formulate what SEDS or extended version of that - SETDS actually is.

SE(T)DS is Statistical Exception (and Trend) Detection System.  It is not an application. But could be implemented by developing one. And I have done that several times (using SAS, COGNOS, BIRT, R and other programming/reporting systems). But developing SETDS-like reports/apps is just a beginning.  The most important part of SETDS is how to use that for Systems Capacity Management and how to build that in the Service Management processes. The set of my CMG papers I wrote since 2001 (list is in the very 1st post of this blog) describes that in details.

By the way it is not absolutely necessary to develop the SETDS application because starting from BMC PP and visualizer (now it is Capacity Optimizer, Perceiver and  Proactive Net) a lot of performance tools have SETDS-like features and this blog has several posts analyzing them (e.g. see Gartner's Magic Quadrant).

A Capacity Manager just need to know how to use the home made or vendor based  SETDS-like tools features efficiently and SETDS is the method. 

So bottom line is:

SETDS is the methodology of using statistical filtering, pattern recognition, active base-lining, dynamic vs. static thresholds,  IT-Control Charts, Exception Value (EV) based reporting/smart alerting and EV based  change points/trends detection to do Systems Capacity Management including Capacity Planning and Performance Engineering.

What value SETDS could bring to a company? I will formulate that later during and after my  CMG'12 presentations on which you are welcome to attend (www.CMG.org)!

(2018 UPDATE: Note , SEDS is the unsupervised SPC/MASF ML based Anomaly Detection method)

CPD perfomalist example:

SEDS-Lite: Using Open Source Tools (R, BIRT, MySQL) to Report and Analyze Performance Data - my new CMG'12 paper

20202 UPDATE: The SEDS-Lite web app is about to be released!
_________________________________________________________
I wrote this paper with some help from Shadi G. (from Dublin, also IBMer).
The paper is based on my blog postings:
SEDS-Lite Presentation at Southern CMG Meeting in the SAS Institute
SEDS-Lite Introduction
How To Build IT-Control Chart - Use the Excel Pivot Table!
BIRT based Control Chart

BIRT Data Cubes Usage for Performance Data Analysis

Building IT-Control Chart by BIRT against Data from the MySQL Database

EV-Control Chart

UCL=LCL : How many standard deviations do we use for Control Charting? Use ZERO!

HERE IS THE VIDEO PRESENTATION
Below is the abstract:
Statistical Exception Detection (SEDS) is one of the variations of learning behavior based performance analysis methodology developed, implemented and published by Author. This paper took main SEDS tools – IT-Control Chart and Exceptions (Anomalies) Detector - and showed how that could be built by Open Source type of BI tools, such as R, BIRT and MySQL or just by spreadsheet. The paper includes source codes, tool screen-shots and report input/output examples to allow reader building/developing a light version of SEDS.
-------------------------
The presentation of this paper is scheduled on December 5th, 2012 Wednesday, 2:45:00 PM - 3:45:00 PM in Las Vegas, Nevada
-------------------------

THAT IS MY SECOND CMG'12 PAPER. THE FIRST ONE ANNOUNCED HERE:

AIX frame and LPAR level Capacity Planning. User Case for Online Banking Application

SEDS-Lite Presentation at Southern CMG Meeting in the SAS Institute

Last Friday I have made my presentation which was announced here: SEDS-Lite: Using Open Source Tools (R, BIRT and MySQL) to Report and Analyze Performance Data. That was presented at the Southern CMG Meeting in the SAS Institute, Cary, NC. The presentation slides are linked within AGENDA and also can be downloaded from HERE

I plan to write a paper based on this presentation and to submit that to this year CMG'12 conference.

Southern CMG Spring 2012 Meeting in Richmond - MXG is our Sponsor!

At SCMG we have usually two meetings each season (2 Spring and 2 Fall ones, both in Richmond and Raleigh). Last season - 2011 fall - I had my presentation; see the following post: "My Southern CMG Presentation in Richmond Is About Open Source Tools for Capacity Management ". Presentation slides are published here: slides

This spring I have the similar but updated presentation in our Raleigh SCMG meeting: "SEDS-Lite: Using Open Source Tools (R, BIRT and MySQL) to Report and Analyze Performance Data" 

So this time I am not presenting in Richmond but I found very good sponsor for that meeting - Merrill Consultants (http://www.mxg.com). Barry Merrill himself responded on my invitation and now we have a great opportunity to see and listen the legendary Capacity Management inventor!

Please consider attending our Richmond VA SCMG meeting on May 11, 2012: 
 
http://regions.cmg.org/regions/scmg/spring_12/richmond/meeting.htm

SEDS-Lite: Using Open Source Tools (R, BIRT and MySQL) to Report and Analyze Performance Data

My presentation with this name has been scheduled for the next Southern CMG meeting  at SAS Institute:

SCMG Meeting Raleigh
May 04, 2012

You are welcome to attend!

EV-Control Chart

I have introduced the EV meta-metric in 2001 as a measure of anomaly severity. EV stands for Exception Value and more explanation about that idea could be found here:  The Exception Value Concept to Measure Magnitude of Systems Behavior Anomalies 

Basically it is the difference (integral) between actual data and control limits. So far I have used EV data mostly to filter out real issues or for automatic hidden trend recognition. For instance, in my paper CMG’08 “Exception Based Modeling and Forecasting” I have plotted that metric using Excel to explain how it could be used for a new trend starting point recognition. Here is the picture from that paper where EV called “Extra Volume” and for the particular parent metric (CPU util.) it is named ExtraCPUtime:

The EV meta-metric first chart 

But just plotting that meta-metric and/or two their components (EV+ and EV-) over time gives a valuable picture of system behavior. If system is stable that chart should be boring showing near zero value all the time. So using that chart would be very easy (I believe even easier than in MASF Control Charts) to recognize unusual and statistically significant increase or decrease in actual data in very early stage (Early Warning!).

Here is the example of that EV-chart against the same sample data used in few previous posts:

1. Excel example: 

2.  BIRT/MySQL example as a continuation of the exercise from the previous post:

IT-Control chart vs. EV-Chart

Here is the BIRT screenshots that illustrate how that is built:

a.        A. Addition query to get EV calculated written directly in the additional BIRT Data Set object called “Data set for EV Chart”:

SQL query to calculate EV meta-metric

 SQL query to calculate EV metric from the data kept in MySQL table

B. Then additional bar-chart object is added to the report that is bind to that new “Data set for EV Chart”:

Result report is already shown here.

SEDS-Lite Introduction

In the purpose of sharing in codes some ideas of exception detection metodology I am developing the SEDS-Lite version using "R"-scripting (http://www.r-project.org/). One of the scripts (cchrt.r - see front picture of this post) has been already published on my blog and that builds control charts against CSV data: http://itrubin.blogspot.com/2009/03/power-of-control-charts.html

How exactly that works  and more R-scripts will be presented on my workshop at up-coming CMG'09 conference in Dallas (http://itrubin.blogspot.com/2009/07/my-cmg09-sunday-workshop.html ) - you are welcome to attend!

Some additional R-scripts can be found in my SCMG presentation http://itrubin.blogspot.com/2009/05/seds-charts-at-scmg.html

The trick is the SAS 9.2 can execute R scripts. You can also try another SAS-like product (http://www.teamwpc.co.uk/products/wps) which also understands R, plus there are some ways to use SAS data for R-graphing: http://www.hollandnumerics.co.uk/pdf/SAS2R2SAS_paper.pdf  or just using the "SAS and R" good book ( I have recently bought that and highly recommend): http://sas-and-r.blogspot.com/