Popular Post

Tuesday, August 28, 2018

Catching Anomaly and Normality in Cloud by Neural Net and Entropy Calculation - #CMGnews

- I have submitted my next CMG (https://cmgimpact.com/) presentation and waiting for the acceptance. BTW I used some past CMG and this blog posts as a material for my presentation.

E.G. You may check it out using below links:

  Quantifying Imbalance in Computer Systems

Monday, June 18, 2018

#CMGnews: #CLOUDXCHANGE virtual conference is tomorrow! Check the agenda here

CMG will be streaming live sessions from 7:00 AM to 7:00 PM EDT at www.cmg.org/cloudxchange. Check out our confirmed speakers below and share your participation with #cloudXchange2018.

·     7:00 AM EDT - The Economics of Cloud Computing - Owen Rogers, 451 Research

·     8:00 AM EDT - An Introduction to Open Source Application Performance Monitoring (APM) - Dr. Andreas Brunnert, RETIT GmbH

·     9:00 AM EDT- Business Process with Cloud Native Computing: Overcoming Challenges of Traceability in Micro Services-based Architecture - Yuri Shkuro, Uber Technologies

·     10:00 AM EDT - Customer Experience and the Journey to Digital Experience Optimization - Dan Boutin, Blue Triangle

·     10:30 AM EDT: Utilizing software defined parallel IO to enable secured, cloud work-spaces  - Andrew Caldwell, R3SOLV

·     11:00 AM EDT - Let’s Get Real About Self-Driven IT Ops - Jeff Henry, CA Technologies

·     12:00 PM EDT - Performance Unit Testing - Why and When They Work - Erik Squires

·     12:30 PM EDT - The World of Private and Hybrid Cloud - Anthony Maiello, Tier4

·     1:00 PM EDT - Exposing the Cost of Performance Hidden in the Cloud - Neil J. Gunther, Performance Dynamics

·     1:30 PM EDT - Continuous Load Testing Using Real Browsers from the Cloud - Tim Koopmans, TIRCENTIS, FLOOD IO

·     2:00 PM EDT - Measuring Client Performance & Inventing Teleportation - Edward Hunter, Netflix

·     3:00 PM EDT - How Linux Containers Work - Sasha Goldshtein, SELA Group

·     3:30 PM EDT - Business By The Numbers: Embedded Analytics for Devices, Edges and Servers - Alan Clark, Telchemy

·     4:00 PM EDT - The Machines are Talking: The Ride of Anomaly Detection - Shanti Subramanyam, Orzota, Inc.

·     4:30 PM EDT - Serverless In Production, An Experience Report - Yan Cui, DAZN

·     5:00 PM EDT - The Journey to Cloud Through the Eyes of a Chief Architect - Kim Eckert, IBM Services

·     6:00 PM EDT - Lessons from Automating Cloud Clients Installation and Testing - Wilson Mar, JetBloom

·     6:30 PM EDT - How to Effectively Implement and Manage Blockchain Performance Engineering - Zak Cole, Whiteblock

Friday, June 15, 2018

#CMGnews: The #Compuware expert speaks about #MachineLearning, #DevOps and products (#zAdviser)

That was one of the session during the National Capital Area Computer Management Group (www.CMG.org) meetup meeting "Performance Engineering for/on Mainframe and Cloud" See agenda is here: https://www.meetup.com/NCACMG/events/251111371/

This session was for members and NOVA college students.

Tuesday, June 12, 2018

#CMGnews - I am presenting at #CLOUDXCHANGE #VirtualConference the CMG Professional Development Offerings

That will be an announcement with some explanations of the on-line classes that are honored by CMG experts.

Among others that includes my class:

The short presentation is scheduled on the next Tuesday 6/19 at about 5:30. Please preregister to the virtual conference HERE  and connect to my presentation! 

To enroll go HERE or find CMG school at https://cmg1.teachable.com/

Sunday, June 10, 2018

#CMGnews in #DC- #CloudCapacity Management - fragment of the presentation

Full version is available for www.CMG.org members
NEXT MEETING ON 6/14/18 RSVP - https://www.meetup.com/NCACMG/

Thursday, May 31, 2018

Hey! If you live close to DC - We meet again to discuss Mainframe and Cloud related stuff in our NCA CMG meetup. Check the agenda!

Hey! If you live close to DC - We meet again to discuss Mainframe related stuff in our meetup: 

LINK https://www.meetup.com/NCACMG/events/251111371/

When: Thursday, June 14, 2018
10:00 AM to 3:00 PM
Add to calendar



1. "Preparing Today’s Infrastructure for Tomorrow’s Workloads", Kofi Hayford, DataCore

2. "Flash, Flash and more Flash: What’s in it for me?", John Baker from Data Kinetics

ABSTRACT: Flash. Is this the light on my camera, a plugin for my browser, or a superhero that runs really fast? If you’re in IT, flash likely refers to something else: flash memory of some sort. But even then, the term is both ubiquitous and vague. The solid-state drive in your laptop; the new memory in your z14; your new “all flash” storage array. These all boast “flash” technology that promises to make everything faster.
Flash technology is not new but reductions in price and advancements in capacity are exploding adoption. There are many compelling arguments for flash – but is it the solution to all storage bottlenecks? Take a stroll with John as we slow down the hype to explore the various implementations of flash technology and which, if any, can best speed up your datacenter. And don’t worry; I won’t be wearing a red, spandex suit.

3. "Dave’s not here: R4HA, AWLC, SCRT and other 4-letter software pricing words", John Baker from Data Kinetics

ABSTRACT: David Chase has left the building. Which leaves many of us scratching our heads about the whirlwind of available pricing options. IBM Z continues to improve efficiency and options like Country-Wide Multiplex (CMP), Mobile and Container pricing, and soft capping options, improve ROI but how does all this stuff work? The R4HA remains key but even that requires context. Its measured on each LPAR but often combined depending on what, and where, various subsystems are running. Which value, average, or total determines your software bill? And what if some LPARs are capped? This session is for everyone. Performance, Capacity, or Finance. Costs are driving more and more of our decisions. Lets start by understanding how these costs are determined.

PRESENTER: John Baker has over 25 years in the IT industry as both a customer and consultant. For the last 20 years, John has focused on mainframe performance and cost optimization. As a customer, John designed, implemented and maintained many critical projects such as WLM Goal Mode, GDPS/Data Mirroring, and merging datacenters.

Light LUNCH is provided by Compuware

4. Compuware presentations by Kelly Vogt:

- "The Practical and Wholesale Application of Strobe to Reduce Cost, Improve Scalability and Performance",

ABSTRACT: Tuning is often a reactive activity. A crisis presents and we turn out in force to stamp out the fire; then return to the myriad other things we have to do… But what if we flip that around and make it a proactive activity for fun and profit? See how a common sense approach to proactive treasure hunting in your system can save large amounts of money in your installation and improve service for your customers.

- "The Necessity of, and the Value Proposition for, Total Batch Automation with ThruPut Manager"

ABSTRACT: What is the future of your batch strategy? Is it nimble? Fluid? For all the automation brought to the mainframe, why is it that batch is still manually operated? All that tracking jobs and endless commands fiddling about with initiators and job classes. Learn why we cannot continue on like we are… and what we can do about it.

PRESENTER: Kelly Vogt joined Compuware in February as an Field Tech in support of ThruPut Manager. He has 38 years in the mainframe arena, with 24 years in systems programming and performance management. His last 14 years were spent in management leading Large Systems Engineering for Humana. He has extensive experience buying and renewing IBM and ISV hardware/software; knowledge of software license models and contracts; performance and capacity planning experience; and the day in, day out of data center operation.

5. "Performance Anomaly Detection and more at CMG Training Program" - 
https://www.cmg.org/professional-development/ Igor Trubin, IT Manager at Capital One, CMG director.

Wednesday, April 18, 2018

Performance #AnomalyDetection Online Course is Launched

What is covered:

  • Machine learning based Anomaly Detection technique
  • Classical (SPC) and MASF (For system performance data) Control Chartirting
  • Where is the Control Chart Used?
  • What are the types of Control Charts?
  • Reading, building, and interpreting Control Charts
  • Typical cases of real world issues captured by anomaly detection system (VMs, Mainframes, Middleware, E2E response and more)
  • How to build free AWS cloud server with R and build there control charts
  • Performance anomaly (Perfomaly) detection system R implementation example (SEDS-lite - open source based tool)

"Perfomaly"=Performance Anomaly:

Control Chart as a Machine Learning Tool to Detect Anomalies

Tuesday, April 17, 2018

TrendieR is a #ChangePointsDetection system based on Exception Value (EV) data analysis

The right term for SETDS methodology is Anomaly and Change (points) detection.

The second part of SETDS - trend detection recently implemented on R and got the "TrendieR" name - is actually a Change Points Detection tool.

See more details how it works in the previous post:

AnomalyDetection vs. NoveltyDetection. SETDS Method Detects and Separates both

But in short the change points are the the following equation roots (solutions)


Where t is time and EV is Exception Value (magnitude of exception calculated as the difference between actual data and baseline statistical limits UCL and LCL). 

CMG.org paper there was the attempt to describe that calculation more formally by the following formulas:

Wednesday, February 7, 2018

More about #CloudCapacityPlanning from #CMGnews

Visiting #CapitalOneCafe

What is Capacity Management?

What is Capacity Management? [Webinar Recap]: Capacity management is the practice of making sure IT resources meet business demands today and down the road—without over-provisioning. But the role of capacity management has changed as IT environments have evolved.

Tuesday, February 6, 2018

"Machine Learning for Predictive Performance Monitoring" - interesting #CMGJournal article (#CMGnews)

Tim Browning has a lot of good publications about Capacity Management in www.CMG.org  and also in this blog:

"Entropy-Based Anomaly Detection for SAP z/OS Systems"

#CMGamplify - "#DataScience Tools for Infrastructure Operational Intelligence"

"the review of cloud computing article "Optimal Density of Workload Placement"

He has just published his new paper in the CMG Journal:  
            "Machine Learning for Predictive Performance Monitoring",
which is available for CMG members

I have enjoyed reading the paper, below is the abstract:

I like especially his following very true saying: 

"...Machines don’t actually “learn” nor do statistical algorithms represent some mechanistic disembodied intelligence. However, human learning and intelligence is greatly assisted by statistical modeling in much the same way that optics technology assists vision..."

I appreciate he referenced two my CMG papers under his "Useful Related Materials" section:

- Trubin, Igor, “Exception Based Modeling and Forecasting”, CMG2008 Proceedings
- Trubin, Igor, “Capturing Workload Pathology by Statistical Exception Detection System”,
CMG2005 Proceedings.

Thursday, February 1, 2018

#AnomalyDetection vs. #NoveltyDetection. SETDS Method Detects and Separates both

Reading "Anomaly detection with Apache MXNet":

"An important distinction has to be made between anomaly detection and “novelty detection.” The latter turns up new, previously unobserved, events that still are acceptable and expected. For example, at some point in time, your credit card statements might start showing baby products, which you’ve never before purchased. Those are new observations not found in the training data, but given the normal changes in consumers’ lives, may be acceptable purchases that should not be marked as anomalies."

I figured out that my SETDS method has this Novelty Detection included as my

EV based trends detection  method (e.g. implemented in R as "TrendieR") finds recent change points in the time-serious data and then by building trend-forecast checks if the change is permanent or not. So if it is permanent the possible "novelty" is detected.  

So the 1st part of SETDS  (e.g. implemented as "SonR" on R) captures just anomalies and/or outliers, then Trend detection separates cases that indicate the possible "novelty". (something changed and stays changed and growing). Still false positive could be there though.... 

BTW there is a 3rd level of SETDS which is actually the way to correlate performance data with demand (drivers) data  to build meaningful forecasts (e.g. implemented as "Model Factory")