Popular Post

Wednesday, January 16, 2019

My CMG IMPACT conference presentation is scheduled 2/20/19 Wednesday at 1:30pm - "Catching Anomaly and Normality in Cloud by Neural Net and Entropy Calculation"

See details here: https://cmgimpact.com/timetable/event/catching-anomalies-in-the-cloud/ 

UPDATE: That was a successful presentation, slides and video will available later for download.
By Igor Trubin, Capital One Bank
Part 1.  The Neural Network (NN) is not a new machine learning method. About 12 years ago I was involved as a Capacity Planning resource for the project of building an infrastructure (servers) to run NN for the fraud detection application. Now NN got much more attention and popularity as a part of AI, mostly because the computing power is increased dramatically and respectively more tasks can be done by using NN.
The goal of the presentation is  to demystify the technique in some simple terms and examples to show what it actually is and how that could be used for Capacity and Demand management. That is done by developing R code to recognize typical workload pasterns, like OLTP, or others in the time series performance data daily profiles.
Part 2. It is the typical concern to detect anomalies for short living objects or for the object with very small amount of measurements. Why? Number of those objects could be thousands and thousands so it is important to separate exceptional ones with anomalies for further investigation.  That could be servers or customers that have just started being monitored or public cloud objects (EC2s, ASGs) that usually have very short lifespan. Suggested approach to detect anomalous behavior of this type of objects is  to estimate the Entropy of the each object. If the entropy is low, everything should be in order and most likely OK. If not – there is a possible disorder there or mess and someone needs to check what is going on with the object. The method is implemented in the cloud based application written on R that scans every  hour all cloud Auto Scaling Groups (ASG) to detect imbalanced ones in term of number of EC2 instances in the group. That allows to separate a couple hundreds ASGs out of hundreds thousands of them.
This entropy based method is well known and it described in details in the following www.Trub.in  blog post:
“Quantifying Imbalance in Computer Systems” which is written based on CMG’12 paper.

Tuesday, December 11, 2018

CMG IMPACT conference discount #cmgnews

All -

I have promotional code for you in case you want to attend. Just respond and I'll give you a good discount!

Thank you!

Igor Trubin 


Wednesday, November 21, 2018

Great Customer Feedback about the #CapacityManagement Service I Provide

I have just got an internal feedback which has impressed me so I have decided to share that here as a good indication of making difference by Capacity Management service I provide to my customers:

..."Igor has helped us on a couple of instances, prevent potential disasters within our platform by the historical trends he’s tracking.  We know this since there was one occasion where our server crashed when we didn't react fast enough to Igor’s email warnings..."

Wednesday, November 14, 2018

AIXCHANGE (Virtual #CMGnews Event about #AI) November 27 @ 10:00 am - 3:00 pm EST

Attend this virtual conference all about artificial intelligence!
Conference rooms and office stand-ups are all buzzing. Artificial intelligence (AI) is emerging to represent one of the largest technological shifts across countless industries in recent history.
From driving manufacturing to supporting marketing to improving customer retention to improving ITops, AI is become the most in-demand technology of IT execs and quickly rising to the top of the list for IT investment in 2019.
According to recent reports, the machine learning market alone is anticipated to grow from $1.4B in 2017 to $8.8B by 2022.
CMG wants to help you to navigate this technological shift and keep you the smartest team member at the table. On November 27th, join CMG and its partners for AIXCHANGE. The latest of CMG’s virtual conference program will feature live presentations from companies and individuals leading in the AI space.
Scheduled Sessions
  • 10:00 AM - The History and Future of AI with Bryan Krouse
  • 11:00 AM - The Machines are Talking - The Future of AI and Chat for the Enterprise with Stephen Mallik
  • 12:00 PM - A Practical Guide for Information Discovery Using Machine Learning and Visualization
  • 1:00 PM - To Be Announced!
  • 2:00 PM - Soon AI will Test Everything with Jason Arbon

Tuesday, November 13, 2018

"Implementation and Interpretation of Control Charts in R" - qcc package

I was pointed to the nice resent on-line publication: https://datascienceplus.com/implementation-and-interpretation-of-control-charts-in-r/ 

"...Control charts are used during the Control phase of DMAIC methodology. Control charts, also known as Shewhart charts or process-behavior charts, are a statistical process control tool used to determine if a manufacturing or business process is in a state of control. If analysis of the control chart indicates that the process is currently under control, then no corrections or changes to process control parameters are needed. Moreover, data from the method can be used to predict the future performance of the process. If the control chart indicates that the process is not in control, analysis of the chart can help determine the sources of variation, as this will result in degradation of process performance..."

My comment:
When I had been developing SEDS (Performance Anomaly Detection System) long ago (years ago) I looked at that package (and referenced the link to my early CMG papers, BTW) even not knowing how to write R programs (now I can!)... They might improved that, but I did not find that time the way to do MASF type of control charts. I have even dreamed to build SETDS charts (IT-Control Charts)  package on a open source way. So my approach is the same but different, please read details in my paper:

Friday, October 19, 2018

#CMGnews: The training course Performance Anomaly Detection is now free for CMG Members. Visit the Membership Benefits Page for Access.

Be a CMG member and get the following benefits:
  • Access to a repository of more than over 1,000 research papers, presentations, and white papers from industry experts for the information you need for delivering improved results.
  • Free 1 hour webinars featuring industry experts and technology demos and recorded webinar access.
  • Free newsletters that feature the latest informative CMG news.
  • Quarterly issues of the CMG Journal
  • Discounted registration for CMG International-hosted events.
  • Access to CMG imPACt conference presentations and papers.
  • Access to our members-only videos channel with event videos, interviews and more.
  • Discounted registration for CMG’s international annual imPACt conference bringing together the best in the industry for four days of research, training, workshops, networking and practice sharing.
  • Access to our international Member Directory.
  • Access to CMG's Slack channel and LinkedIn to take part in performance and capacity discussions, hot topics, evaluations, information, networking, stay up to date with company news, and more!

Wednesday, October 17, 2018

#CMGnews: Registration is opened for #IMPACT2019 conference! Check discounts!

  • IMPACT 2019 will be an action-packed, 3-day conference filled with information and collaboration. Register today for #IMPACT2019 and take advantage of $100 off your conference pass with code FOB2019:  https://cmgimpact.com/ #cmgnews 

  • Act today and take $100 off your registration fee with code “FOB2019”. To find out more about content, sessions, and activities and what makes CMG’s IMPACT the best #technology conference on the planet, click here:  https://cmgimpact.com/ #cmgnews #IMPACT2019

  • Join hundreds of #industryleaders for CMG's 44th International Conference! #IMPACT2019 promises to be an exciting conference with great learning and networking opportunities. Discount code (ASK ME!) available to save $100 off a conference pass. Act today and save!  https://cmgimpact.com/ #cmgnews 

  • IMPACT 2019 #Conference sessions will educate and enlighten, enabling attendees to take a #leadership role in their own companies’ #digitaltransformations. Register today for #IMPACT2019 and take advantage of $100 off your conference pass with “FOB2019”: https://cmgimpact.com/ #cmgnews 

Tuesday, October 16, 2018

www.CMG.org Announces Launch Of Dynamic New Brand And Communications Platform

Computer Management Group (www.CMG.org), one of the worlds most influential organizations of IT professionals committed to digital transformation initiatives and best practices, is delighted to announce the launch of its new brand and communications platform—a platform built to showcase its measurement and management of computer systems and networks from a performance and capacity ... Read More »

Wednesday, October 10, 2018

#Opmantek product has implemented my SEDS method (MASF based #AnomalyDetection) and referenced my work positively

After I have published my previous post

#Opmantek product has implemented my Statistical Exception Detection System (SEDS) - MASF based #AnomalyDetection

Opmantek CTO has reached me out, confirmed they used my methodology and provided a very positive feedback:

"I am glad you contacted Opmantek about this blog, and we have updated it to include your name and a reference to one of the key blog articles.

I have been meaning to reach out to you to let you know about what we had done, technically the product is still in Beta but it seems the marketing team have pushed forward with making it generally available.

We found your work of great value as we looked through various methodologies for trending, in the end we implemented something based on SEDS with a few changes/additions, to be honest I would have to review the code to see what the differences are.

At the moment the product opTrend is working well enough, but we need to make some refinements and enhancements before it will be the first release.

Your research and publications are of great value and highly appreciated."