SCMG meeting in Richmond: AIX frame and LPAR level Capacity Planning

The CMG - Southern Region  meeting is scheduled April 25, 2013.  I will be presenting again my CMG'12 paper:

AIX frame and LPAR level Capacity Planning. User Case for Online Banking Application

Here is agenda: LINK:

Registration:
http://regions.cmg.org/regions/scmg/spring_13/richmond/reg.htm

Location:  Northrop Grumman IT, VITA Project
 11751 Meadowville Lane, 2S-J3
 Chester, VA 23836

%V\ (%EV) to plot under Control Chart

The post Exception Value (EV) Metrics and Technical Analysis Indicators

introduced EV based metric (%V\) that I promised to find a good application for. And I did. That should be a needles (or bars) type of graph under the main control chart to mark hours that had anomalies (exceptions). Actually that idea was suggested by one of this blog guest poster (STEEDd: Another Implementation of The Near-Real-Time Control Charting and EV Calculating). 

See example:

And also I have already used the similar idea to build EV-Control Chart. 

See example:

IT-Control Charts: How to Read, How to Use

DEMO is below:

YouTube version of the presentation:

Exception Value (EV) Metrics and Technical Analysis Indicators

I HAD TO RE-ENTER THIS POST (from e-mail copy I got about my previous posting) AS BLOGGER HAS LOST MY ORIGINAL ONE!!!!

This is my 1st CMG’12 trip report.

The following CMG workshop: Albert Mavashev - "Ensuring the Performance of Business‐Critical Apps in the Cloud" on a top of Cloud monitoring topic has touched the subject that relates to this blog.

The author offers to apply some indicators from technical analysis (of securities trading) to detect anomalies or resource leaks against computer performance data.

The following known indicators were mentioned: Bollinger Bands and Momentum Oscillator

Apparently the author was not aware of the System performance exceptions/anomalies /changes detection methodologies that are commonly used in the IT Capacity Management, such as MASF and SETDS (SEDS), but he naturally came to idea of doing that and took what already used for similar purposes in technical analysis.

I believe that EV based approach that SETDS methodology uses is more appropriate for Computer performance data, but I am always opened to evaluate other means (and often do that in this blog postings)

So,

Bollinger Bands are similar to Upper and Lower Control Limits in SPC. More interesting is that in 2010 the %b (pronounced "percent b") indicator (derived from Stochastic oscillator) was introduced and that somewhat similar to EV-meta-metric I have introduced in 2001. It is similar but in inverse way! p% is between 0 and 100 for actual data WITHIN the band, but EV become non-0 when actual data go OUTSIDE the band.

Applying the same idea we can alslo also normalize EV metric (lets call it “%И”) to the |UCL-LCL| band:

So if the actual data is not an anomaly, %И=0;

if it is a actual data jump that is higher than UCL or lower than LCL in a half of band – it will be %И >100%.

Not sure whether it make some practical sense or not, but really interesting twist!

Let me play with this %И indicator finding the good place to use and I will report here of any good results.....

In return I would suggest to apply EV and %И indicators to technical analysis to estimate magnitude of some market unusual responses! Why not?

--
Posted By Blogger to System Management by Exception at 1/29/2013 06:00:00 PM
 (BUT LOST!!! - my comment iTrubin)

Database Space Capacity Planning: Exception Based Method

In my 2008 CMG paper "Exception based Modeling and Forecasting" I have mentioned some approach that my colleague,  coauthor and friend Ray W.  implemented to provide proactive Database Space Capacity planning:

By the way, my first and best American manager is mentioned there - Kevin M. As stated in the paper he has actually gave me the idea of the SEDS and he was the co-author of my 1st SEDS related CMG'01 paper - Exception Detection System, Based on the Statistical Process Control Concept

The Open Group Certified IT Specialist (Open CITS)

I have just got certified by The Open Group

IT Specialist Certification for Actualizing IT Solutions

I have finally got certified:

IT Capacity Council related CMG papers

- Robert E. Chaney: 1. The Capacity Performance Council, Start Yours Today! 

2. Divide and Conquer: Implementing the Capacity Performance Council in Pieces

- R. Kephart The Job You Save

- Linwood Merritt: 1. Seeing the Forest AND the Trees: Capacity Planning for a Large Number of Servers (CMG Paper)

2. A Capacity Planning Partnership with the Business;

3. Adding Business Drivers to the Capacity Planning Process  

- Igor Trubin (me) 

AIX frame and LPAR level Capacity Planning. User Case for Online Banking Application - my new CMG'12 paper

 

SETDS Methodology

2022 UPDATE

Some of the SETDS features are implemented into www.Perfomalist.com tool, which is described in the following post: https://www.trutechdev.com/2021/12/ and last release notes are HERE . The detailed Perfomalist CPD method is explained in this blog: https://www.trub.in/2020/08/cpd-change-points-detection-is-planed.html

_________________________________________________________________________________

Preparing my upcoming CMG'12 presentation about SEDS-lite I try to formulate what SEDS or extended version of that - SETDS actually is.

SE(T)DS is Statistical Exception (and Trend) Detection System.  It is not an application. But could be implemented by developing one. And I have done that several times (using SAS, COGNOS, BIRT, R and other programming/reporting systems). But developing SETDS-like reports/apps is just a beginning.  The most important part of SETDS is how to use that for Systems Capacity Management and how to build that in the Service Management processes. The set of my CMG papers I wrote since 2001 (list is in the very 1st post of this blog) describes that in details.

By the way it is not absolutely necessary to develop the SETDS application because starting from BMC PP and visualizer (now it is Capacity Optimizer, Perceiver and  Proactive Net) a lot of performance tools have SETDS-like features and this blog has several posts analyzing them (e.g. see Gartner's Magic Quadrant).

A Capacity Manager just need to know how to use the home made or vendor based  SETDS-like tools features efficiently and SETDS is the method. 

So bottom line is:

SETDS is the methodology of using statistical filtering, pattern recognition, active base-lining, dynamic vs. static thresholds,  IT-Control Charts, Exception Value (EV) based reporting/smart alerting and EV based  change points/trends detection to do Systems Capacity Management including Capacity Planning and Performance Engineering.

What value SETDS could bring to a company? I will formulate that later during and after my  CMG'12 presentations on which you are welcome to attend (www.CMG.org)!

(2018 UPDATE: Note , SEDS is the unsupervised SPC/MASF ML based Anomaly Detection method)

CPD perfomalist example:

SAS code to build Control Chart

As I have already shared at the "The Master of MASF" post there was the CMG presentation (where BTW my work and this blog were mentioned) that illustrates how to use SAS to build MASF Control Charts.

The two parts of that presentation are available at the www.MXG.com along with link to SAS code that could be used to build control charts:

Automatic Daily Monitoring of Continuous Processes Theory and Practice Frank Bereznay and MP Welch (UKCMG2011) (SWCMG2011)
Filename	Size (bytes)	Date Posted
ADMCP Part 1 PDF	1,178 KB	May 17, 2011
ADMCP Part 2 PDF	8,719 KB	May 17, 2011
Coding Sample admcp_sample.sas	4KB	May 17, 2011
HTMLBLUE SG 9.3 Style Preview (Make 9.2 look like 9.3)	31KB	May 17, 2011

Not a MASF Based Statistical Techniques (Entropy-based) for Anomaly Detection in Data Centers (and Clouds)

The following papers published on Mendeley criticizes the MASF Gaussian assumption and offer other methods (Tukey and Relative Entropy) to detect anomalies statistically. (BTW I tried to use the entropy analysis to capture performance anomalies - check my other post)

1. Statistical techniques for online anomaly detection in data centers
by Chengwei Wang, Krishnamurthy Viswanathan, Lakshminarayan Choudur, Vanish Talwar, Wade Satterfield, Karsten Schwan
  
Abstract

Online anomaly detection is an important step in data center management, requiring light-weight techniques that provide sufficient accuracy for subsequent diagnosis and management actions. This paper presents statistical techniques based on the Tukey and Relative Entropy statistics, and applies them to data collected from a production environment and to data captured from a testbed for multi-tier web applications running on server class machines. The proposed techniques are lightweight and improve over standard Gaussian assumptions in terms of performance.

2. Online detection of utility cloud anomalies using metric distributions

by Chengwei Wang Chengwei Wang, V Talwar, K Schwan, P Ranganathan

Abstract

The online detection of anomalies is a vital element of operations in data centers and in utility clouds like Amazon EC2. Given ever-increasing data center sizes coupled with the complexities of systems software, applications, and workload patterns, such anomaly detection must operate automatically, at runtime, and without the need for prior knowledge about normal or anomalous behaviors. Further, detection should function for different levels of abstraction like hardware and software, and for the multiple metrics used in cloud computing systems. This paper proposes EbAT - Entropy-based Anomaly Testing - offering novel methods that detect anomalies by analyzing for arbitrary metrics their distributions rather than individual metric thresholds. Entropy is used as a measurement that captures the degree of dispersal or concentration of such distributions, aggregating raw metric data across the cloud stack to form entropy time series. For scalability, such time series can then be combined hierarchically and across multiple cloud subsystems. Experimental results on utility cloud scenarios demonstrate the viability of the approach. EbAT outperforms threshold-based methods with on average 57.4% improvement in accuracy of anomaly detection and also does better by 59.3% on average in false alarm rate with a `near-optimum' threshold-based method.

3. EbAT : Online Methods for Detecting Utility Cloud Anomalies

Chengwei Wang in Middleware (2009)

4. Performance Metric Selection for Autonomic Anomaly Detection on Cloud Computing Systems

Song Fu in 2011 IEEE Global Telecommunications Conference GLOBECOM 2011 (2011)

 

5. Mining anomalies using traffic feature distributions

Anukool Lakhina, Mark Crovella, Christophe Diot in ACM SIGCOMM Computer Communication Review (2005)

6. Krishnamurthy Viswanathan, Lakshminarayan Choudur, Vanish Talwar et al. (2012) Ranking Anomalies in Data Centers, 1-8. In NOMS. 

7. Greg Eisenhauer, Matthew Wolf, Chengwei Wang (2010) Monalytics : Online Monitoring and Analytics for Managing Large Scale Data Centers. In ICAC.

8. Fast Anomaly Detection for Large Data Centers

Ang Li Ang Li, Lin Gu Lin Gu, Kuai Xu Kuai Xu in 2010 IEEE Global Telecommunications Conference GLOBECOM 2010 (2010)

9.Online Reactive Anomaly Detection over Stream Data

Yan Fu Yan Fu, Jun-Lin Zhou Jun-Lin Zhou, Yue Wu Yue Wu in 2008 International Conference on Apperceiving Computing and Intelligence Analysis (2008)

 

10.Semantic anomaly detection in online data sources

O Raz, P Koopman, M Shaw in Proceedings of the 24th International Conference on Software Engineering ICSE 2002 (2002)

11.Statistical anomaly detection via httpd data analysis

Daniel Q Naiman in Computational Statistics & Data Analysis (2004)

12.A comparative study of real-valued negative selection to statistical anomaly detection techniques

T Stibor, J Timmis, C Eckert in Comparative and General Pharmacology (2005)

 

 

 

 

 

MASF Control Charts Against DB2 Performance Data

I have done that before... I used for that my own variation of MASF Control Chart called "IT- Control Chart". You can see the example in my older post: Power of Control Charts and IT-Chart Concept (Part 1) :

 But not only me do that! I have found the following paper in MeasureIt :

Capacity Planning has an Important Role in Assisting Senior IT Management April, 2007, writen by Rick Isom.

The paper has a good MASF reference, the list of DB2 performance metrics that are good to be analyzed by MASF Control Charts and a few examples of Control Charts in form of 24-hour profile. One example is below (Linked to picture form the original paper published on Internet):

BTW, the actual data curve is hourly aggregated data for the particular month (October) to compare with historical base-line. Similar approach was taken in the exercise I published in the following post: Adrian Heald: A simple control chart using Captell Version 6

Theory of Anomaly Detection: Stanford University Video Lectures

That is the part of Machine Learning Lectures:   https://class.coursera.org/ml/lecture/preview/index.

XV. Anomaly Detection (Week 9)

(expanded, click to collapse)

• Problem Motivation (8 min)
• Gaussian Distribution (10 min)
• Algorithm (12 min)
• Developing and Evaluating an Anomaly Detection System (13 min)
• Anomaly Detection vs. Supervised Learning (8 min)
• Choosing What Features to Use (12 min)
• Multivariate Gaussian Distribution (Optional) (14 min)

Availability vs. Capacity

Continuing the previous posts about  "Battle between "Gut-feeling" and Engineering." ....

Engineer 2: Igor - if you would like to find further extensions of your equation, you might check out Volume 1 of "Breaking the Availability Barrier," which I co-authored and which is available on Amazon. Also, check out several papers I published in the Availability Digest in the Geek Corner (http://www.availabilitydigest.com/articles.htm). A subscription is free.

Igor Trubin: Yes, I have already looked briefly at your book and referenced it in my blog, Very good book and I plan to read that all.

UPDATE: After reading one of suggested above papers ( "Calculating Availability - Redundant Systems" Some useful rules come out of the derivation of the availability equation.) I was able to show to my client that the following more general cluster (system) availability formula proves that for three-node cluster (n=3) the one spare node (s=1) configuration could provide approximately the same cluster availability as in the case of two spare nodes (s=2) but Capacity usage could be a critical factor as seen below. 
 

 

That could be a way to save money by allocating less capacity with the same number but more reliable nodes.

Systems Availability Arena: Battle between "Gut-feeling" and Engineering. Round 3. (2 and 3 are in the previous posts)

UPDATE: the start is here

"Gut-feeler 1" • A comment on the math for availability calculations: Certainly not being a math hero, I can still understand that playing with formulas can be much fun. But we ought to take care whether the results generated by these formulas are actually producing value in real world situations.

Misleadig fomulas can be dangerous and ought to be kept in a pen and paper / chalk and blackboard environment. When those formulas can be "googled", there is the risk that some younger and less experienced person working in the role of an IT architect takes them at face value and bases real world decisons on those - or even worse, someone might write a related Wikipedia article and multiplies the damage ...

Igor's initial formula suggests that any four node cluster is hundred times more reliable than a two node cluster, and that any cluster can easily achieve 99,999% availability if you only add enough nodes. Bill's revised formula would suggest an even stronger growth in availability when adding cluster nodes ...

"Engineer 2" has already pointed out that these are moot calculations, as the formulas are just not applicable in those real world environments we are talking about. You would not construct a large building or a bridge using elegant formulas that produce over-optimistic results which are in stark contrast with real world experience. Everybody knows that after the collapse, the architect would go to jail ...

IT architects do also carry responsibility - it might be somewhat limited when desining a webshop for selling cosmetics or toys, when only the shopowner would be disappointed after investing in additional nodes and not getting the expected reliability in return. But for instance, when building 911 emergency communication systems controlling police, ambulance and fire brigade services, lives are at stake and could be lost due to system outages. Here, only the most reliable IT infrastructure is good enough - and creating false expectations by misleading formulas would be fatal.

"Engineer 3":  "In Theory, there is no difference between Theory and Practice. In Practice, there is" I don't know where I found the quote above but I like it. Nevertheless, I think one should in fact do both for any system of significant importance:

(1) Use Math, a.k.a."Theory"  to calculate the expected availability and adjust as needed to match the required availability. Ignoring Math/Theory and replacing it with only gut feeling and/or trust in vendor statements does not sound right to me. (2) Apply gut feeling; maybe better called experience or "Practice"; Combine the two and you are off to a good start IMHO...

"Engineer 1" • ...  I disagree with Gut-feeler1's bridge building analogy. My father was involved into real bridges design and construction and he always started with pretty heavy math modeling and calculation. I cannot imagine that modern bridges are built just by the "gut feeling". Again I agree with "Engineer 3"  I am for the engineering approach in clusters design that combines math modeling and applying the best practice experience. 

UPDATE: other rounds are here

Systems Availability Arena: Battle between "Gut-feeling" and Engineering. Round 2.

This is continuation of the previous post.

Gut-feeler 1 • Gut feelings aren't always bad - for instance, when working on an IT project meant to support really critical business processes, and hence with a lot of money or even lives at stake, your gut feeling might be that standard clustering just isn't good enough and you need something significantly better here.
 

 Then it comes to curiosity - is there something better around than the usual standard clustering ? Is there some other IT infrastructure, one that is fault tolerant and self-healing, providing much more reliability right out of the box than you could ever achieve using the plain vanilla stuff and the most sophisticated clustering conceiveable ?

If being that curious, chances are you will end up at NonStop.

Not being curious and just doing calculations, you just might end up with adding more nodes to your standard cluster hoping to make it more reliable - which in the real world often turns out to be a false hope ...

But don't get me wrong, I'm not at all against calculations. A very important one is on capacity, will (n - 1) nodes still support my workload, when a node went down for whatever reason ? That's often overlooked ... 

Engineer 2  Two comments, one on the Trubin law, and one on .. comments [above]. .... Adding a node of m 9s to a node of m9s adds m9s to a cluster. The overall availability of a cluster of n nodes, each with m 9s availability, is mn 9s.... For instance, a three-node cluster of nodes with 2 9s availability will be six nines...

Let f = the failure probability of a node [A=(1-f)]. If f is an even number of nines, then the failure probability of a node is f= 0.1^m, where m is the number of nines (for instance, for three nines, f=0.1^3 =.001 and A =(1-0.1^3) = 0.999). For an n-node cluster, its availability is 1-(1-A)^n = 1-[1-(1-0.1^m)^n = 1-(0.1^m)n = 1-0.1^mn.

In general, if a node has a failure probability of f, then an n-node cluster has an availability of 1-f^mn. Two nodes with availabilities of 0.95 will have an availability of 0.9975.

Of course, this assumes that the cluster fails only if all nodes fail. Generally, a cluster can withstand the failure of some nodes but not all. In this case, the above relations can be modified to accommodate this situation.

"Gut-feeler 1"'s suggestion that adding nodes does not result in this additional availability is quite correct. The above relations apply only to hardware failures (or whatever failures might be included in the nodal availability), and are accurate for those. However, once the hardware availability becomes high (say four 9s), other factors that are not node related come into play, such as software bugs, operator errors, and environmental faults (power, air conditioning). These limit the practical availability that can be achieved. In effect, after a certain point, increases in hardware availability become irrelevant to system availability.

Thanks for starting a very interesting and meaningful thread, Igor. 

See the next post for the next round. 

 

Systems Availability Arena: Battle between "Gut-feeling" and Engineering!

I have put my Cluster Availability 9's Equation post to LinkedIn Continuous Availability forum and got 19 comments, divided in about two camps: "Gut-feelers" and "Engineers". Below is the first two comments. (See next posts for other comments.) 

Gut-feeler 1:  "Nice formula, looks good in theory – but won’t hold true in the real world. The reason for this is the underlying assumption of an ideal cluster, which does not exist in the real world. When looking at some real world implementation like the Oracle RAC cluster, you will find that a simple two-node cluster configuration will typically deliver somewhere between three and four nines of availability.

Now, will adding a third node to that cluster add another 9 to the availability figure ? Will a ten node cluster really provide 99.999999999999 % availability ? Will a cluster with hundred nodes run continuously for thousands of years without any outage ?

Certainly not, and talking to system administrators running typical cluster installations will quickly reveal that large clusters are quite complex and difficult to handle, hence more prone to failure than simple two-node clusters.

Even when looking at the HP NonStop architecture – which comes pretty close to the ideal cluster – the formula would not apply. A NonStop system (which internally is indeed a cluster, each NonStop CPU resembling a node) delivers roughly five nines of availability – but there is no significant availability difference between systems eg. with four and with sixteen CPU’s (cluster nodes).

So it is not so important how many nodes you have – but it is very important what kind of cluster you have ! 

Engineer 1 • I know this particular formula is too simple for the real world, so I completely agree with your comment. But still the complexity of big clusters can be modeled by more complex math. models by e.g. adding more boxes with parallel and series type of connections. And the formula will be much more ugly but useful I believe... Plus each individual node could be decomposed on some structure to model both HW and SW parts. The approach is written in some books I mentioned in my other posts, and it is suppose to be a tool to do that, but I am not aware of any. Are you?

In my real life this type of calculation/modeling is just a starting point to get rough estimation and then using some monitoring data to get that adjusted. What I do not like is when some Architects makes some decisions of the level of cluster redundancy without ANY calculation (!), just based on their gut feelings....

(NB: Real names can be found on the actual LinkedIn forum tread)
UPDATE: see the start point here