2016 UPDATE. My paper from that year CMG conference can be found under the following link now:
SYSTEM MANAGEMENT BY EXCEPTION, PART 6
Here is the list of CMG2006 (http://www.cmg.org/) papers that discussed statistical exception detection technique:
• A Priori Evaluation of Data and Selection of Forecasting Model, Alexander Gilgur, Michael Perka MonoSphere, Inc.
LINK: http://www.daschmelzer.com/cmg2006/PDFs/038.pdf
The paper shows how important to capture “OutLaers” to produce meaningful forecasts. To do that they use some SEDS-like algorithm “Detection of Outlier Events”.
• 2006 Best Paper Award paper: Did Something Change? Using Statistical Techniques to Interpret Service and Resource Metrics. Frank M. Bereznay, Kaiser Permanente
LINK: http://cmg.org/conference/cmg2006/awards/6139.pdf
Author has some references to my papers:
“…Statistical techniques are not new to CMG. Starting in the early 1990’s there have been numerous papers addressing this subject, [Brey90], [Chu92], [Lipner92] and [Schwartz93]. This body of work seemed to cumulate with Jeff Buzen and Annie Schum’s 1995 CMG Paper introducing Multivariate Adaptive Statistical Filtering (MASF) as a new statistical technique [Buzen95]. Interest in the subject seemed to decline from that point on, with the notable exception of Igor Trubin’s set of papers on the application of MASF to many measurement and management areas [Trubin01], [Trubin02], [Trubin03], [Trubin04], and [Trubin05]. All of these papers are excellent treatments of the subject and are recommended reading…”
During discussions at this presentation some questions were asked (e.g. Sean Meidhan from BEN, who implemented some MASF ideas in there tool) about “fault positive” (fault alerts) situation sometimes generated out this technique. I had to step up and give some clarifications how SEDS handles that.
(08/2007 UPDATE:
Frank M. Bereznay have recently gave the interview to CMG MeasureIT: MeasureIT - Issue 5.08 - Getting to Know Mullen Award Winner ...
"...I also noticed that the number of papers in this area seemed to be declining since the mid to late 1990s. There were a number of papers leading up to Jeff Buzen and Annie Shum's MASF (multivariate adaptive statistical filtering) paper in 1995, and since then the trend seemed to decline with the exception of Igor Trubin's work, so I wanted to give the statistical methods some additional visibility.."
He also will be presenting at this fall SCMG meetings September 27 in Richmond and September 28 in Raleigh :
Using Statistical Techniques to Interpret Service and Resource Metrics )
• ACTIVE BASELINING IN PASSIVE DATA ENVIRONMENTS, Mike Tsykin, Fujitsu Australia, Ltd.
LINK: http://www.fujitsu.com/downloads/AU/active_baselining_in_passive_data_environments.pdf
Author in this paper uses the SPC approach for baselining. I have met with him in the previous CMG conferences discussing my SEDS technique, he picked up SPC idea and implemented that for alerting part of some Fujitsu performance tool.
• Dials for a PM Dashboard: Velocity’s Missing Twin, and Quantifying Surprise, Rich Olcott, IBM Information Technology Services
LINK: http://www.daschmelzer.com/cmg2006/PDFs/102.pdf
This paper has also some discussion how SPC should be used. E.g. how Simple Average cold be replaced by EWMA. He has references on my paper and M.Tsykin’s paper.
• My paper: SYSTEM MANAGEMENT BY EXCEPTION, PART 6, Igor Trubin, PhD
LINK: http://www.daschmelzer.com/cmg2006/PDFs/021.pdf
ABSTRACT: Statistical Exception Detection System (SEDS) has been successfully used for more than six years to automatically produce web-based exception reports against the performance data warehouse for a large, multi-platform environment. Adding some application specific metrics including middleware traffic and response times made SEDS an excellent tool for application performance management. This paper also describes how to create statistical control charts using a spreadsheet in order to capture a performance issue without using expensive tools with built-in SPC procedure.
_________________________________________________________________
CONCLUSION: There is still a big interest to MASF, SPC and SixSigma methods applied to system performance data. This year CMG'07 conference has already announced some papers related to this subject as well, including my next paper:
System Management by Exception, Part Final, Dr. Igor A. Trubin, IBM; Ray White, IBM
LINK: PDF | System Management by Exception, Part Final. - ResearchGate
ABSTRACT: Statistical Exception Detection System (SEDS) has been successfully used for more than seven years to automatically produce web-based exception reports and smart alerts against the performance data warehouse for a large, multi-platform environment. This paper starts with an overview of how SEDS uses SPC and MASF techniques and how SEDS could be used as a part of Lean/Six Sigma. Then it focuses on the memory usage exceptions that SEDS captures to proactively identify server and application performance issues.
(08/2007 UPDATE: This paper is presented also September 27 in Richmond SCMG: meting: System Management by Exception, Part Final)
SYSTEM MANAGEMENT BY EXCEPTION, PART 6
Here is the list of CMG2006 (http://www.cmg.org/) papers that discussed statistical exception detection technique:
• A Priori Evaluation of Data and Selection of Forecasting Model, Alexander Gilgur, Michael Perka MonoSphere, Inc.
LINK: http://www.daschmelzer.com/cmg2006/PDFs/038.pdf
The paper shows how important to capture “OutLaers” to produce meaningful forecasts. To do that they use some SEDS-like algorithm “Detection of Outlier Events”.
• 2006 Best Paper Award paper: Did Something Change? Using Statistical Techniques to Interpret Service and Resource Metrics. Frank M. Bereznay, Kaiser Permanente
LINK: http://cmg.org/conference/cmg2006/awards/6139.pdf
Author has some references to my papers:
“…Statistical techniques are not new to CMG. Starting in the early 1990’s there have been numerous papers addressing this subject, [Brey90], [Chu92], [Lipner92] and [Schwartz93]. This body of work seemed to cumulate with Jeff Buzen and Annie Schum’s 1995 CMG Paper introducing Multivariate Adaptive Statistical Filtering (MASF) as a new statistical technique [Buzen95]. Interest in the subject seemed to decline from that point on, with the notable exception of Igor Trubin’s set of papers on the application of MASF to many measurement and management areas [Trubin01], [Trubin02], [Trubin03], [Trubin04], and [Trubin05]. All of these papers are excellent treatments of the subject and are recommended reading…”
During discussions at this presentation some questions were asked (e.g. Sean Meidhan from BEN, who implemented some MASF ideas in there tool) about “fault positive” (fault alerts) situation sometimes generated out this technique. I had to step up and give some clarifications how SEDS handles that.
(08/2007 UPDATE:
Frank M. Bereznay have recently gave the interview to CMG MeasureIT: MeasureIT - Issue 5.08 - Getting to Know Mullen Award Winner ...
"...I also noticed that the number of papers in this area seemed to be declining since the mid to late 1990s. There were a number of papers leading up to Jeff Buzen and Annie Shum's MASF (multivariate adaptive statistical filtering) paper in 1995, and since then the trend seemed to decline with the exception of Igor Trubin's work, so I wanted to give the statistical methods some additional visibility.."
He also will be presenting at this fall SCMG meetings September 27 in Richmond and September 28 in Raleigh :
Using Statistical Techniques to Interpret Service and Resource Metrics )
• ACTIVE BASELINING IN PASSIVE DATA ENVIRONMENTS, Mike Tsykin, Fujitsu Australia, Ltd.
LINK: http://www.fujitsu.com/downloads/AU/active_baselining_in_passive_data_environments.pdf
Author in this paper uses the SPC approach for baselining. I have met with him in the previous CMG conferences discussing my SEDS technique, he picked up SPC idea and implemented that for alerting part of some Fujitsu performance tool.
• Dials for a PM Dashboard: Velocity’s Missing Twin, and Quantifying Surprise, Rich Olcott, IBM Information Technology Services
LINK: http://www.daschmelzer.com/cmg2006/PDFs/102.pdf
This paper has also some discussion how SPC should be used. E.g. how Simple Average cold be replaced by EWMA. He has references on my paper and M.Tsykin’s paper.
• My paper: SYSTEM MANAGEMENT BY EXCEPTION, PART 6, Igor Trubin, PhD
LINK: http://www.daschmelzer.com/cmg2006/PDFs/021.pdf
ABSTRACT: Statistical Exception Detection System (SEDS) has been successfully used for more than six years to automatically produce web-based exception reports against the performance data warehouse for a large, multi-platform environment. Adding some application specific metrics including middleware traffic and response times made SEDS an excellent tool for application performance management. This paper also describes how to create statistical control charts using a spreadsheet in order to capture a performance issue without using expensive tools with built-in SPC procedure.
_________________________________________________________________
CONCLUSION: There is still a big interest to MASF, SPC and SixSigma methods applied to system performance data. This year CMG'07 conference has already announced some papers related to this subject as well, including my next paper:
System Management by Exception, Part Final, Dr. Igor A. Trubin, IBM; Ray White, IBM
LINK: PDF | System Management by Exception, Part Final. - ResearchGate
ABSTRACT: Statistical Exception Detection System (SEDS) has been successfully used for more than seven years to automatically produce web-based exception reports and smart alerts against the performance data warehouse for a large, multi-platform environment. This paper starts with an overview of how SEDS uses SPC and MASF techniques and how SEDS could be used as a part of Lean/Six Sigma. Then it focuses on the memory usage exceptions that SEDS captures to proactively identify server and application performance issues.
(08/2007 UPDATE: This paper is presented also September 27 in Richmond SCMG: meting: System Management by Exception, Part Final)