Popular Post

Search This Blog

Friday, October 21, 2016

Interesting 1992 conference paper about SPC and machine learning written by Shewhart.

Interpreting statistical process control (SPC) charts using machine learning and expert system techniques

Conference Paper · June 1992

Statistical process control (SPC) charts are one of several tools
used in quality control. The SPC quality control tool has been
under-utilized due to the lack of experienced personnel able to identify
and interpret patterns within the control charts. The Special Projects
Office of the Center for Supportability and Technology Insertion (CSTI)
has developed a hybrid machine-learning and expert-system software tool
which automates the process of constructing and interpreting control
charts. The software tool draws control charts, identifies various chart
patterns, advises what each pattern means, and suggests possible
corrective actions. The application is easily modifiable for process
specific applications through simple modifications to the knowledge base
portion using any word processing software. The authors discuss control
charts, software functionality, software design, machine learning, and
the expert system

Thursday, October 20, 2016

SCMG meeting in Cary, NC

I enjoy listening and presenting!

Monday, October 10, 2016

Interesting paper about "Adaptive Anomaly Detection in Cloud"

Adaptive Anomaly Detection in Cloud using Robust and Scalable Principal Component Analysis 


This paper proposes a novel and scalable model for automatic anomaly detection on a large system such as a cloud. Anomaly detection issues early warning of unusual behavior in dynamic environments by learning system characteristic from normal operational data. Anomaly detection in large systems is difficult to detect due heterogeneity, dynamicity, scalability, hidden complexity, and time limitation. To detect anomalous activity in the cloud, we need to monitor the datacenter and collect cloud performance data. In this paper, we propose an adaptive anomaly detection mechanism which investigates principal components of performance metrics. It transforms the performance metrics into a low-rank matrix and then calculates the orthogonal distance using the Robust PCA algorithm. The proposed model updates itself recursively learning and adjusting the new threshold value in order to minimize reconstruction errors. This paper also investigates the robust principal component analysis in distributed environments using Apache Spark as the underlying framework, specifically addressing cases in which a normal operation might exhibit multiple hidden modes. The accuracy and sensitivity of the model is tested on Google data center traces and Yahoo! datasets. The model achieves an 87.24% accuracy.
MY COMMENT: By the way the paper has referenced to MASF technique which I have enhanced and have been using (check my SETDS methodology)  for years  to capture anomalies  (exceptions) and sudden short term trends against huge server farms (20,000+ servers) including private and public clouds. Note my way is much-much simpler and in spite the MASF has indeed a high rate of false positives, SETDS has the way to handle that well.

Friday, October 7, 2016

Southern CMG meeting in Cary, NC on October 20th - Final Agenda

The SCMG is proud to announce our Fall 2016 Meeting, an all-day event on October 20, 2016 

MetLife – Grace Hopper Auditorium (Bldg. 1 (MET 1), Floor 01, Room 600)
101 MetLife Way
Cary NC, 27513

ð  HURRY:  REGISTER NO LATER THAN OCT. 14!!  This covers Breakfast and Lunch.   We need to confirm the number of Registrants so that we can properly plan the catering.  We also need your Registration information in order to get a list to MetLife Security so that we will have visitor badges ready for you the morning of Oct. 20 when you arrive in the MET1 lobby.

To REGISTER, use the  registration page.  You can use a PayPal account or credit card.   Your registration payment through the PayPal button logs your registration.


8:00-8:45 ET
Registration / Breakfast
Speaker BIO
Breakfast provided with Sponsor Session:  AppDynamics
Rick Weaver

“Best Practices - Populating Big Data Repositories from DB2, IMS and VSAM”
Over the past 25 years, Rick Weaver has become a well-known mainframe expert specializing in database protection, replication, recovery and performance. Because of his vast expertise, he has authored numerous articles, whitepapers and other valuable pieces on database technologies, and frequently spoken on the subjects of database recovery and performance at conferences, symposiums and user groups.
Ann Dowling

“Capacity Management Is Still Relevant”
Ann joined MetLife in April 2016 as the Director of Capacity & Forecast Engineering.  In this role Ann will build on her extensive background working at IBM in various disciplines including capacity planning, process architecture, performance engineering, and offering management.  Her professional passion is Capacity Planning in support of the business and how it drives the applications that consume resources on the IT infrastructure.  Ann’s most rewarding work has been leading teams to consolidate toward a common ‘best practices’ approach to capacity management.  She did so for a series of consolidations of independent data centers within IBM which evolved into her role as the global Capacity Management process owner.  That work grounded Ann’s move to consulting services with external, non-outsourced customers to evaluate their capacity management capabilities, identify strengths and gaps to then build a roadmap for improvement.   The next step was working with a specific, large account to lead a team on the implementation phases of the roadmap that gave Ann a more hands-on role working directly with the engineering and operations teams and management.  She has been on the planning committee and speaker for various IBM and CMG technical conferences.  She was an instructor for IBM’s Architecting for Performance class and author of a four-part series on “Exploring Analytics to enable the Business and Service Value of Capacity Planning”.
Kyle Parrish
CMG 2015 Mullen Award Winner

“Too Big to Test: Breaking a production brokerage platform without causing financial devastation”
Kyle currently works as a Director of Technology Risk in the FI Information Security group at Fidelity Investments.  Kyle joined Fidelity in January of 2011 as a Director of Performance Architecture charged with driving end-to-end testing of the Fidelity Brokerage systems.  Prior to joining Fidelity, Kyle worked as a consultant for over 13 years, after a career in both the private sector and a university research setting.  Kyle’s roles have spanned everything from program management to performance engineering to security, across industries as varied as airlines, financial services, manufacturing, retail, pharmaceuticals, and state government.  
LUNCH provided with Sponsor Session:  Cirba
Igor Trubin

“Is Your Capacity Available?”
I started my career in 1979 as an IBM/370 system engineer. In 1986 I got my PhD. in Robotics at St. Petersburg Technical University (Russia) and then worked as a professor teaching there CAD/CAM, Robotics and Computer Science for about 12 years. I published 30 papers and made several presentations for international conferences related to the Robotics, Artificial Intelligent and Computer fields. In 1999 I moved to the US and worked at Capital One bank in Richmond as a Capacity Planner. My first CMG paper was written and presented in 2001. The next one, "Global and Application Level Exception Detection System Based on MASF Technique," won a Best Paper award at CMG 2002 and was presented again at UKCMG 2003 in Oxford, England. My CMG 2004 paper about applying MASF technique to mainframe performance data was republished in the IBM z/Series Expo. I also presented my papers in Central Europe CMG conference and in numerous US regional meetings. I continue to enhance my exception detection methodologies. After working more than 2 years as the Capacity Management team lead for IBM, I had worked for SunTrust Bank for 3 years and then got back to IBM holding for 2+ years  Sr. IT Architect position. Currently I work for Capital One bank as IT Manager for IT Capacity Management group. In 2015 I have been elected to the CMG (http://www.cmg.org) board of directors. Blog: www.Trub.in
Shawn Lundvall

“zBNA: Theory and Overview”
I started my IBM career in 2001 in Poughkeepsie in the Systems Architecture group writing the Principals of Operations. In 2005 I got the opportunity to do hardware design of the fixed point unit. In 2007 I moved to Richmond supporting clients as a Client Technical Specialist. In 2013 I joined the Washington Systems Center as a Software Engineer and am now a developer for zBNA and zPCR.
Ken Christiance

“IBM GTS Cirba Case Study “
Ken Christiance – Distinguished Engineer with 28 years’ experience in IBM.   He has been working in Strategic Outsourcing field since 1993; experience that spans service management architectures, virtualization/server management and analytics.  Ken is currently a member of the Technology, Innovation and Automation team that supports architecture and solution design for system automation, virtualization and distributed server management.  Ken is patented and published for technologies that provide usage accounting and billing, policy based automation, network design, virtualization and service management tooling
SCMG Committee Meeting
Optional:  a breakout room will be reserved for folks who would like to hold an impromptu BOF or post SCMG informal opportunity to network.