tag:blogger.com,1999:blog-50486430982781770702024-03-19T04:48:21.685-04:00System Management by ExceptionThis blog relates to experiences in the Systems Capacity and Availability areas, focusing on statistical filtering and pattern recognition and BI analysis and reporting techniques (SPC, APC, MASF, 6-SIGMA, SEDS/SETDS and other) Igor Trubinhttp://www.blogger.com/profile/17758940374397545163noreply@blogger.comBlogger353125tag:blogger.com,1999:blog-5048643098278177070.post-44238005780550446542023-12-15T15:08:00.009-05:002023-12-15T15:10:47.260-05:00"Scale in Clouds. What, How, Where, Why and When to Scale" - my new www.CMG.org presentation<p><span face="verdana, sans-serif" style="background-color: white; color: #222222; font-size: small;">Our presentation (with<b> Jignesh Shah</b>) was accepted for </span><a data-saferedirecturl="https://www.google.com/url?q=http://www.cmgimpact.com/&source=gmail&ust=1702752995528000&usg=AOvVaw1UqWuWA1-2tsgKjcM_vsbH" href="http://www.cmgimpact.com/" style="color: #1155cc; font-family: verdana, sans-serif; font-size: small;" target="_blank">www.CMGimpact.com</a><span face="verdana, sans-serif" style="background-color: white; color: #222222; font-size: small;"> </span><wbr style="color: #222222; font-family: verdana, sans-serif; font-size: small;"></wbr><span face="verdana, sans-serif" style="background-color: white; color: #222222; font-size: small;">conference.</span></p><div class="gmail_default" style="background-color: white; color: #222222; font-family: verdana, sans-serif; font-size: small;">Title: <strong style="color: #23496d; font-family: Arial, sans-serif; font-size: 16px;">Scale in Clouds. What, How, Where, Why and When to Scale</strong></div><div class="gmail_default" style="background-color: white; color: #222222; font-family: verdana, sans-serif; font-size: small;">Venue: <b> Atlanta, GA on <span face="Arial, sans-serif" style="color: #23496d; font-size: 15px;">February 6 & 7</span></b></div><div class="gmail_default" style="background-color: white; color: #222222; font-family: verdana, sans-serif; font-size: small;"><span face="Arial, sans-serif" style="color: #23496d; font-size: 15px;"><br /></span></div><div class="gmail_default" style="background-color: white; color: #222222; font-family: verdana, sans-serif; font-size: small;"><span face="Arial, sans-serif" style="color: #23496d; font-size: 15px;">ABSTRACT:</span></div><div class="gmail_default" style="background-color: white; color: #222222; font-family: verdana, sans-serif; font-size: small;"><p dir="ltr" style="line-height: 1.08; margin-bottom: 3pt; margin-top: 0pt; text-align: center;"><span face="Arial, sans-serif" style="background-color: transparent; color: black; font-size: 26pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"><b>Scale in Clouds</b></span></p><p dir="ltr" style="line-height: 1.08; margin-bottom: 0pt; margin-top: 0pt; text-align: center;"><span face="Calibri, sans-serif" style="background-color: transparent; color: black; font-size: 19pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">What, How, Where, Why and When to Scale</span></p><br /><p dir="ltr" style="line-height: 1.08; margin-bottom: 0pt; margin-top: 0pt;"><span face="Calibri, sans-serif" style="background-color: transparent; color: black; font-size: 17pt; font-style: italic; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">Igor Trubin, </span><span face="Roboto, sans-serif" style="color: #202124; font-size: 14.5pt; font-style: italic; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">Jignesh Shah - Capital One bank </span></p><br /><p dir="ltr" style="line-height: 1.08; margin-bottom: 0pt; margin-top: 0pt;"><span face="Calibri, sans-serif" style="background-color: transparent; color: black; font-size: 14pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">ABSTRACT</span></p><p dir="ltr" style="line-height: 1.08; margin-bottom: 0pt; margin-top: 0pt;"><span face="Calibri, sans-serif" style="background-color: transparent; color: black; font-size: 14pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">Presentation includes the following discussion themes. </span></p><ul style="margin-bottom: 0px; margin-top: 0px;"><li dir="ltr" style="background-color: transparent; color: black; font-family: Calibri, sans-serif; font-size: 14pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; margin-left: 15px; vertical-align: baseline; white-space-collapse: preserve;"><p dir="ltr" role="presentation" style="line-height: 1.08; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; font-size: 14pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline;">What </span><span style="background-color: transparent; font-size: 14pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">to scale: servers, databases, containers, load balancers.</span></p></li></ul><ul style="margin-bottom: 0px; margin-top: 0px;"><li dir="ltr" style="background-color: transparent; color: black; font-family: Calibri, sans-serif; font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; margin-left: 15px; vertical-align: baseline; white-space-collapse: preserve;"><p dir="ltr" role="presentation" style="line-height: 1.08; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; font-size: 14pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline;">How </span><span style="background-color: transparent; font-size: 14pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">to scale: horizontally/rightsizing, vertically, manually, automatically, ML based, predictive, serverless.</span></p></li><li dir="ltr" style="background-color: transparent; color: black; font-family: Calibri, sans-serif; font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; margin-left: 15px; vertical-align: baseline; white-space-collapse: preserve;"><p dir="ltr" role="presentation" style="line-height: 1.08; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; font-size: 14pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline;">Where </span><span style="background-color: transparent; font-size: 14pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">to scale: AWS (ASG,ECS, EKS, ELB), AZURE, GCP, K8s.</span></p></li><li dir="ltr" style="background-color: transparent; color: black; font-family: Calibri, sans-serif; font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; margin-left: 15px; vertical-align: baseline; white-space-collapse: preserve;"><p dir="ltr" role="presentation" style="line-height: 1.08; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; font-size: 14pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline;">Why </span><span style="background-color: transparent; font-size: 14pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">to scale: cost optimization, incidents avoidance, seasonality.</span></p></li><li dir="ltr" style="background-color: transparent; color: black; font-family: Calibri, sans-serif; font-size: 14pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; margin-left: 15px; vertical-align: baseline; white-space-collapse: preserve;"><p dir="ltr" role="presentation" style="line-height: 1.08; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; font-size: 14pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline;">When</span><span style="background-color: transparent; font-size: 14pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> to scale: auto-scaling policies and parameters, pre-warming to fight latency, correlating with business/app drivers.</span></p></li></ul><p dir="ltr" style="line-height: 1.08; margin-bottom: 0pt; margin-top: 0pt;"><span face="Calibri, sans-serif" style="background-color: transparent; color: black; font-size: 14pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">Presentation includes a user case study of scaling parameters optimization: monitoring, modeling and balancing vertical and horizontal scaling, calculating optimal initial/desired cluster size and more.</span></p><p dir="ltr" style="line-height: 1.08; margin-bottom: 0pt; margin-top: 0pt;"><span face="Calibri, sans-serif" style="background-color: transparent; color: black; font-size: 14pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"><br /></span></p><p dir="ltr" style="line-height: 1.08; margin-bottom: 0pt; margin-top: 0pt;"><span face="Calibri, sans-serif" style="background-color: transparent; color: black; font-size: 14pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"></span></p><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEj2vHuMb865yQTiIeT8pprtr1AO3vsQlLZ43nP6cyIfkrXDNkLmuBWMgxWinYYL0EUhYrThzi29tIBbdN16Dwnw6YkuCoRG3mGeujqHdwVJHJkf8KJtugQLz2g6WgtO20RnsQ1gB8F6VGc_dKVlURMjo5Vpqq4BEjpWzWDAoT9Y3_s1dUZbYG5s4R1GHoc" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="245" data-original-width="617" height="159" src="https://blogger.googleusercontent.com/img/a/AVvXsEj2vHuMb865yQTiIeT8pprtr1AO3vsQlLZ43nP6cyIfkrXDNkLmuBWMgxWinYYL0EUhYrThzi29tIBbdN16Dwnw6YkuCoRG3mGeujqHdwVJHJkf8KJtugQLz2g6WgtO20RnsQ1gB8F6VGc_dKVlURMjo5Vpqq4BEjpWzWDAoT9Y3_s1dUZbYG5s4R1GHoc=w400-h159" width="400" /></a></div></div><p></p></div>Igor Trubinhttp://www.blogger.com/profile/17758940374397545163noreply@blogger.com0tag:blogger.com,1999:blog-5048643098278177070.post-71567528558639906352023-11-21T12:39:00.001-05:002023-11-21T12:39:45.081-05:00AI!?<a href="https://youtube.com/watch?v=amwo2Da9Hpg&si=1udW2WYZKpY7ErNQ">куда идёт AI</a><div><br></div><div><br></div>Igor Trubinhttp://www.blogger.com/profile/17758940374397545163noreply@blogger.com0tag:blogger.com,1999:blog-5048643098278177070.post-80773320205819893642023-09-05T23:40:00.001-04:002023-09-05T23:40:09.157-04:00Me at work.... #datamanagement #rap<iframe width="480" height="270" src="https://youtube.com/embed/g-cWkj1JI5Y?si=GSNDPjjVSJMnc56d" frameborder="0"></iframe>Igor Trubinhttp://www.blogger.com/profile/17758940374397545163noreply@blogger.com0tag:blogger.com,1999:blog-5048643098278177070.post-69909631248264550012023-04-21T09:31:00.009-04:002023-04-21T10:01:05.428-04:00My 1st dialog with #chatGBT about R (text data to convert to code to create a #dataframe).<p><br /></p><p>ME: I have CSV table and need a code to convert that to dataframe.</p><p>GPT: Sure! Here is some code that will read a CSV file into a dataframe in R:</p><p># set the working directory to the folder where your CSV file is located</p><p>setwd("/path/to/folder")</p><p># read the CSV file into a dataframe</p><p>mydata <- read.csv("mycsvfile.csv", header = TRUE)</p><p># view the dataframe</p><p>mydata</p><p><br /></p><p>In this code, you will need to replace "/path/to/folder" with the actual path to the folder where your CSV file is located, and "mycsvfile.csv" with the actual name of your CSV file.</p><p>The read.csv() function reads a CSV file and creates a dataframe from it. The header = TRUE argument tells R that the first row of the CSV file contains column names.</p><p>After running this code, you should have a dataframe called mydata that contains the data from your CSV file. You can view the dataframe by simply typing mydata into the R console.</p><p><br /></p><p>ME: I know that, but what is I have no ability to use file system? No read.csv, no write.csv, just a coma separated text to code like the following: </p><p>first_column <- c("value_1", "value_2", ...)</p><p>second_column <- c("value_1", "value_2", ...)</p><p>df <- data.frame(first_column, second_column)</p><p><br /></p><p>GPT: I see! Here's an example of how you could create a dataframe in R from comma-separated text using the format you provided:</p><p>lua</p><p>Copy code</p><p># sample comma-separated text</p><p>text <- "value_1, value_2, value_3</p><p> value_4, value_5, value_6</p><p> value_7, value_8, value_9"</p><p><br /></p><p># split the text into separate lines</p><p>lines <- strsplit(text, "\n")[[1]]</p><p><br /></p><p># split each line into separate values</p><p>values <- lapply(lines, function(x) strsplit(x, ", ")[[1]])</p><p><br /></p><p># create a dataframe from the values</p><p>df <- data.frame(matrix(unlist(values), nrow=length(lines), byrow=TRUE))</p><p><br /></p><p># add column names to the dataframe</p><p>colnames(df) <- c("first_column", "second_column", "third_column")</p><p><br /></p><p># view the dataframe</p><p>In this code, you would need to replace the text variable with your own comma-separated text. The strsplit() function is used to split the text into lines and then into separate values. The lapply() function is used to apply the strsplit() function to each line of the text.</p><p>The data.frame() function is used to create a dataframe from the values. The matrix() function is used to reshape the values into a matrix with the appropriate dimensions.</p><p>Finally, the colnames() function is used to add column names to the dataframe. You would need to replace the column names in the colnames() function with the appropriate column names for your data.</p><p>____</p><p>I have tested that and that is not working well.</p><p>By googling I got better suggestion: </p><p><a href="https://sparkbyexamples.com/r-programming/different-ways-to-create-a-dataframe-in-r/">https://sparkbyexamples.com/r-programming/different-ways-to-create-a-dataframe-in-r/</a></p>Igor Trubinhttp://www.blogger.com/profile/17758940374397545163noreply@blogger.com0tag:blogger.com,1999:blog-5048643098278177070.post-89572258034857361242022-12-20T20:24:00.001-05:002022-12-20T20:28:11.560-05:00SETDS (Statistical Exception and Trend Detection System) vs. AWS/SETDS (Split, Extract, Test, Decide, and Score). Is OpenAI right?<p> My <b>SETDS </b>(<b><a href="https://www.researchgate.net/profile/Igor-Trubin/publication/221447683_Exception_Based_Modeling_and_Forecasting/links/541cf9e30cf241a65a15cdba/Exception-Based-Modeling-and-Forecasting.pdf" target="_blank">Statistical Exception and Trend Detection System</a></b>) method was created about 20 years ago. </p><p>Playing with OpenAI <a href="https://chat.openai.com/chat">https://chat.openai.com/chat</a> I was giving the following information:</p><p style="--tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-ring-color: rgba(59,130,246,0.5); --tw-ring-offset-color: #fff; --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-offset-width: 0px; --tw-ring-shadow: 0 0 transparent; --tw-rotate: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-scroll-snap-strictness: proximity; --tw-shadow-colored: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-skew-x: 0; --tw-skew-y: 0; --tw-translate-x: 0; --tw-translate-y: 0; background-color: #f7f7f8; border: 0px solid rgb(217, 217, 227); box-sizing: border-box; color: #374151; font-family: Söhne, ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, Ubuntu, Cantarell, "Noto Sans", sans-serif, "Helvetica Neue", Arial, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Noto Color Emoji"; font-size: 16px; margin: 0px 0px 1.25em; white-space: pre-wrap;"><b>"<i>SETDS </i></b><i>(Split, Extract, Test, Decide, and Score) is a method for anomaly detection that was developed by researchers at Amazon Web Services (AWS). The method involves the following steps:</i></p><ol style="--tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-ring-color: rgba(59,130,246,0.5); --tw-ring-offset-color: #fff; --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-offset-width: 0px; --tw-ring-shadow: 0 0 transparent; --tw-rotate: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-scroll-snap-strictness: proximity; --tw-shadow-colored: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-skew-x: 0; --tw-skew-y: 0; --tw-translate-x: 0; --tw-translate-y: 0; background-color: #f7f7f8; border: 0px solid rgb(217, 217, 227); box-sizing: border-box; color: #374151; counter-reset: item 0; display: flex; flex-direction: column; font-family: Söhne, ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, Ubuntu, Cantarell, "Noto Sans", sans-serif, "Helvetica Neue", Arial, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Noto Color Emoji"; font-size: 16px; list-style-image: initial; list-style-position: initial; margin: 1.25em 0px; padding: 0px 0px 0px 1rem; white-space: pre-wrap;"><li style="--tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-ring-color: rgba(59,130,246,0.5); --tw-ring-offset-color: #fff; --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-offset-width: 0px; --tw-ring-shadow: 0 0 transparent; --tw-rotate: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-scroll-snap-strictness: proximity; --tw-shadow-colored: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-skew-x: 0; --tw-skew-y: 0; --tw-translate-x: 0; --tw-translate-y: 0; border: 0px solid rgb(217, 217, 227); box-sizing: border-box; margin: 0px; padding-left: 0.375em;"><p style="--tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-ring-color: rgba(59,130,246,0.5); --tw-ring-offset-color: #fff; --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-offset-width: 0px; --tw-ring-shadow: 0 0 transparent; --tw-rotate: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-scroll-snap-strictness: proximity; --tw-shadow-colored: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-skew-x: 0; --tw-skew-y: 0; --tw-translate-x: 0; --tw-translate-y: 0; border: 0px solid rgb(217, 217, 227); box-sizing: border-box; margin: 0px;"><i>Split: The data is split into a training set and a testing set.</i></p></li><li style="--tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-ring-color: rgba(59,130,246,0.5); --tw-ring-offset-color: #fff; --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-offset-width: 0px; --tw-ring-shadow: 0 0 transparent; --tw-rotate: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-scroll-snap-strictness: proximity; --tw-shadow-colored: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-skew-x: 0; --tw-skew-y: 0; --tw-translate-x: 0; --tw-translate-y: 0; border: 0px solid rgb(217, 217, 227); box-sizing: border-box; margin: 0px; padding-left: 0.375em;"><p style="--tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-ring-color: rgba(59,130,246,0.5); --tw-ring-offset-color: #fff; --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-offset-width: 0px; --tw-ring-shadow: 0 0 transparent; --tw-rotate: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-scroll-snap-strictness: proximity; --tw-shadow-colored: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-skew-x: 0; --tw-skew-y: 0; --tw-translate-x: 0; --tw-translate-y: 0; border: 0px solid rgb(217, 217, 227); box-sizing: border-box; margin: 0px;"><i>Extract: Features are extracted from the training set.</i></p></li><li style="--tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-ring-color: rgba(59,130,246,0.5); --tw-ring-offset-color: #fff; --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-offset-width: 0px; --tw-ring-shadow: 0 0 transparent; --tw-rotate: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-scroll-snap-strictness: proximity; --tw-shadow-colored: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-skew-x: 0; --tw-skew-y: 0; --tw-translate-x: 0; --tw-translate-y: 0; border: 0px solid rgb(217, 217, 227); box-sizing: border-box; margin: 0px; padding-left: 0.375em;"><p style="--tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-ring-color: rgba(59,130,246,0.5); --tw-ring-offset-color: #fff; --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-offset-width: 0px; --tw-ring-shadow: 0 0 transparent; --tw-rotate: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-scroll-snap-strictness: proximity; --tw-shadow-colored: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-skew-x: 0; --tw-skew-y: 0; --tw-translate-x: 0; --tw-translate-y: 0; border: 0px solid rgb(217, 217, 227); box-sizing: border-box; margin: 0px;"><i>Test: A model is trained on the training set and tested on the testing set.</i></p></li><li style="--tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-ring-color: rgba(59,130,246,0.5); --tw-ring-offset-color: #fff; --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-offset-width: 0px; --tw-ring-shadow: 0 0 transparent; --tw-rotate: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-scroll-snap-strictness: proximity; --tw-shadow-colored: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-skew-x: 0; --tw-skew-y: 0; --tw-translate-x: 0; --tw-translate-y: 0; border: 0px solid rgb(217, 217, 227); box-sizing: border-box; margin: 0px; padding-left: 0.375em;"><p style="--tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-ring-color: rgba(59,130,246,0.5); --tw-ring-offset-color: #fff; --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-offset-width: 0px; --tw-ring-shadow: 0 0 transparent; --tw-rotate: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-scroll-snap-strictness: proximity; --tw-shadow-colored: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-skew-x: 0; --tw-skew-y: 0; --tw-translate-x: 0; --tw-translate-y: 0; border: 0px solid rgb(217, 217, 227); box-sizing: border-box; margin: 0px;"><i>Decide: Anomaly scores are calculated for each data point in the testing set based on the model's performance.</i></p></li><li style="--tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-ring-color: rgba(59,130,246,0.5); --tw-ring-offset-color: #fff; --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-offset-width: 0px; --tw-ring-shadow: 0 0 transparent; --tw-rotate: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-scroll-snap-strictness: proximity; --tw-shadow-colored: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-skew-x: 0; --tw-skew-y: 0; --tw-translate-x: 0; --tw-translate-y: 0; border: 0px solid rgb(217, 217, 227); box-sizing: border-box; margin: 0px; padding-left: 0.375em;"><p style="--tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-ring-color: rgba(59,130,246,0.5); --tw-ring-offset-color: #fff; --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-offset-width: 0px; --tw-ring-shadow: 0 0 transparent; --tw-rotate: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-scroll-snap-strictness: proximity; --tw-shadow-colored: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-skew-x: 0; --tw-skew-y: 0; --tw-translate-x: 0; --tw-translate-y: 0; border: 0px solid rgb(217, 217, 227); box-sizing: border-box; margin: 0px;"><i>Score: The anomaly scores are used to identify data points that are likely to be anomalous.</i></p></li></ol><p style="--tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-ring-color: rgba(59,130,246,0.5); --tw-ring-offset-color: #fff; --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-offset-width: 0px; --tw-ring-shadow: 0 0 transparent; --tw-rotate: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-scroll-snap-strictness: proximity; --tw-shadow-colored: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-skew-x: 0; --tw-skew-y: 0; --tw-translate-x: 0; --tw-translate-y: 0; background-color: #f7f7f8; border: 0px solid rgb(217, 217, 227); box-sizing: border-box; color: #374151; font-family: Söhne, ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, Ubuntu, Cantarell, "Noto Sans", sans-serif, "Helvetica Neue", Arial, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Noto Color Emoji"; font-size: 16px; margin: 1.25em 0px 0px; white-space: pre-wrap;"><i>The SETDS method is designed to be fast and efficient, and it has been used in a number of applications, including detecting anomalies in log data and in real-time streaming data. It is one of many methods that can be used for anomaly detection, and the suitability of the method will depend on the specific characteristics of the data and the goals of the analysis."</i></p><p>+++++++++++++++++</p><p>Interesting that my SETDS also does some kind of scoring by calculating EV - Exception Value. </p><p>Also interesting that I was not able to verify by Google search the information AI bot chat provided....</p>Igor Trubinhttp://www.blogger.com/profile/17758940374397545163noreply@blogger.com0tag:blogger.com,1999:blog-5048643098278177070.post-7770974951466862142022-12-16T11:49:00.000-05:002022-12-16T11:49:03.706-05:00Cloud Usage Data. Cleansing, Aggregation, Summarization, Interpretability and Usability (#CMGnews) - my presentation<p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://hubs.ly/Q01vGZhc0" style="margin-left: 1em; margin-right: 1em;" target="_blank"><img alt="" data-original-height="1089" data-original-width="1426" height="489" src="https://blogger.googleusercontent.com/img/a/AVvXsEiZ1u38hp0G5cfmQATz4jfZcSl1DTpdchMB_mWrNee-kqirZI2Uh3KdKQW9JWZR7V7OwvKHtdztSffSEQAK4f7clVHd-as4MVtoZ9mDMWvJLjD21goS8d2BORbpHEMpRYXDMpKewtrXhQjesZXHkkYqyOW1D5aK_NOBT9rUihpjj6sJh98KuBPs3kOe=w640-h489" width="640" /></a></div> <a href="https://hubs.ly/Q01vGZhc0">https://hubs.ly/Q01vGZhc0</a> <p></p>Igor Trubinhttp://www.blogger.com/profile/17758940374397545163noreply@blogger.com0tag:blogger.com,1999:blog-5048643098278177070.post-68319646147772811132022-12-09T15:06:00.004-05:002022-12-09T16:04:37.179-05:00#CMGImpact 2023 conference announcement of the Trubin's presentation about #clouddata<iframe frameborder="0" height="270" src="https://youtube.com/embed/Go0Y313Fr5Q" width="480"></iframe><div><a href="https://youtu.be/WJxdV3jKswg">https://youtu.be/WJxdV3jKswg</a></div>Igor Trubinhttp://www.blogger.com/profile/17758940374397545163noreply@blogger.com0tag:blogger.com,1999:blog-5048643098278177070.post-17007815886858471152022-11-28T17:37:00.000-05:002022-11-28T17:37:04.713-05:00"#Cloud Usage Data. Cleansing, Aggregation, Summarization, Interpretability and Usability" - CMG Impact'23 presentation (#CMGnews)<p>My presentation was accepted for<b> CMG Impact'23</b> (<a href="http://www.CMGimpact.com">www.CMGimpact.com </a>) conference (Orlando, FL, Feb. 21-23). </p><p>ABSTRACT:</p><p>All cloud objects (EC2, RDS, EBS, ECS/Fargate, K8s, Lambda) are elastic and ephemeral. It is a real problem to understand, analyze and predict their behavior. But it is really needed for Cost optimization and Capacity management. The essential requirement to do that is the system performance data. The raw data is collected by observability tools (CloudWatch, DataDog or NewRelic), but it is big and messy.</p><p>The presentation is to explain and demonstrate:</p><p>- How that should be aggregated and summarize addressing the issue of jumping workload from one cluster to another due to rehydration, releases and failovers.</p><p>- How the data should/are to be cleaned by anomaly and change point detection without generating false negatives like seasonality.</p><p>- How to summarize the data to avoid sinking in granularity. </p><p>- How to interpret the data to do cost and capacity usage assessments.</p><p>- Finally how to use that clean, aggregated and summarized data for Capacity Management by using ML/Predictive analytics.</p><p><br /></p><p></p><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEg16M0xIYZhimDtOFDx-vTmiIbWiUWOMiYNCUVVNRsyG54U_wFp61MEEUf9bIleRrRtyKkOlR4PEfCreZk1iynhXfK7tqQbqx4s1jgwUYyjvqLSCeA9Pk5bFdhoDgieWpaIdOSRv8UptI4IClkUa6G5rp_6wXhwjoXpqWMwsqIH4I_U0xWnE0IjEj1Z" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="528" data-original-width="987" height="342" src="https://blogger.googleusercontent.com/img/a/AVvXsEg16M0xIYZhimDtOFDx-vTmiIbWiUWOMiYNCUVVNRsyG54U_wFp61MEEUf9bIleRrRtyKkOlR4PEfCreZk1iynhXfK7tqQbqx4s1jgwUYyjvqLSCeA9Pk5bFdhoDgieWpaIdOSRv8UptI4IClkUa6G5rp_6wXhwjoXpqWMwsqIH4I_U0xWnE0IjEj1Z=w640-h342" width="640" /></a></div><br /><br /></div><br /><br /><p></p>Igor Trubinhttp://www.blogger.com/profile/17758940374397545163noreply@blogger.com0tag:blogger.com,1999:blog-5048643098278177070.post-59401508234201296722022-11-06T23:17:00.001-05:002022-11-06T23:17:46.469-05:00Hybrid #ChangePointDetection system - #Perfomalist<p><span style="color: rgba(0, 0, 0, 0.9); font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", "Fira Sans", Ubuntu, Oxygen, "Oxygen Sans", Cantarell, "Droid Sans", "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Lucida Grande", Helvetica, Arial, sans-serif; font-size: 14px;">The paper about using<a href="http://www.perfomalist.com"> </a></span><span style="color: #3778cd; font-family: -apple-system, system-ui, BlinkMacSystemFont, Segoe UI, Roboto, Helvetica Neue, Fira Sans, Ubuntu, Oxygen, Oxygen Sans, Cantarell, Droid Sans, Apple Color Emoji, Segoe UI Emoji, Segoe UI Emoji, Segoe UI Symbol, Lucida Grande, Helvetica, Arial, sans-serif;"><span style="border: var(--artdeco-reset-link-border-zero); box-sizing: inherit; font-size: 14px; font-weight: var(--font-weight-bold); line-height: inherit !important; margin: var(--artdeco-reset-base-margin-zero); overflow-wrap: normal; padding: var(--artdeco-reset-base-padding-zero); position: relative; touch-action: manipulation; vertical-align: var(--artdeco-reset-base-vertical-align-baseline);"><a href="http://www.perfomalist.com">#Perfomalist</a></span></span><span style="color: rgba(0, 0, 0, 0.9); font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", "Fira Sans", Ubuntu, Oxygen, "Oxygen Sans", Cantarell, "Droid Sans", "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Lucida Grande", Helvetica, Arial, sans-serif; font-size: 14px;"> "<a href="https://www.trutechdev.com/2022/11/Change%20Point%20Detection%20for%20MongoDB%20Time%20Series%20Performance%20Regression" style="color: #3778cd; text-decoration-line: none;">Change Point Detection for </a></span><span style="color: #444444; font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", "Fira Sans", Ubuntu, Oxygen, "Oxygen Sans", Cantarell, "Droid Sans", "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Lucida Grande", Helvetica, Arial, sans-serif; font-size: 13px;"><span style="border: var(--artdeco-reset-link-border-zero); box-sizing: inherit; font-size: 14px; font-weight: var(--font-weight-bold); line-height: inherit !important; margin: var(--artdeco-reset-base-margin-zero); overflow-wrap: normal; padding: var(--artdeco-reset-base-padding-zero); position: relative; touch-action: manipulation; vertical-align: var(--artdeco-reset-base-vertical-align-baseline);"><a href="https://www.trutechdev.com/2022/11/Change%20Point%20Detection%20for%20MongoDB%20Time%20Series%20Performance%20Regression" style="color: #3778cd; text-decoration-line: none;">#MongoDB</a></span></span><span style="color: rgba(0, 0, 0, 0.9); font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", "Fira Sans", Ubuntu, Oxygen, "Oxygen Sans", Cantarell, "Droid Sans", "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Lucida Grande", Helvetica, Arial, sans-serif; font-size: 14px;"><a href="https://www.trutechdev.com/2022/11/Change%20Point%20Detection%20for%20MongoDB%20Time%20Series%20Performance%20Regression" style="color: #3778cd; text-decoration-line: none;"> Time Series Performance Regression</a>" was cited in the following paper: "<a href="https://www.semanticscholar.org/paper/Estimating-Breakpoints-in-Piecewise-Linear-Using-Onder-De%C4%9Firmenci/f25c24020c8329100e658d885ed9e3072fa17d4e" style="color: #3778cd; text-decoration-line: none;">Estimating Breakpoints in Piecewise Linear Regression Using </a></span><span style="color: #444444; font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", "Fira Sans", Ubuntu, Oxygen, "Oxygen Sans", Cantarell, "Droid Sans", "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Lucida Grande", Helvetica, Arial, sans-serif; font-size: 13px;"><span style="border: var(--artdeco-reset-link-border-zero); box-sizing: inherit; font-size: 14px; font-weight: var(--font-weight-bold); line-height: inherit !important; margin: var(--artdeco-reset-base-margin-zero); overflow-wrap: normal; padding: var(--artdeco-reset-base-padding-zero); position: relative; touch-action: manipulation; vertical-align: var(--artdeco-reset-base-vertical-align-baseline);"><a href="https://www.semanticscholar.org/paper/Estimating-Breakpoints-in-Piecewise-Linear-Using-Onder-De%C4%9Firmenci/f25c24020c8329100e658d885ed9e3072fa17d4e" style="color: #3778cd; text-decoration-line: none;">#MachineLearning</a></span></span><span style="color: rgba(0, 0, 0, 0.9); font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", "Fira Sans", Ubuntu, Oxygen, "Oxygen Sans", Cantarell, "Droid Sans", "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Lucida Grande", Helvetica, Arial, sans-serif; font-size: 14px;"><a href="https://www.semanticscholar.org/paper/Estimating-Breakpoints-in-Piecewise-Linear-Using-Onder-De%C4%9Firmenci/f25c24020c8329100e658d885ed9e3072fa17d4e" style="color: #3778cd; text-decoration-line: none;"> Methods</a>", where our method was mentioned as " … offer a hybrid change point detection system..." </span></p><p style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13px;"><span style="color: rgba(0, 0, 0, 0.9); font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", "Fira Sans", Ubuntu, Oxygen, "Oxygen Sans", Cantarell, "Droid Sans", "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Lucida Grande", Helvetica, Arial, sans-serif; font-size: 14px;"></span></p><div class="separator" style="background-color: white; clear: both; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13px; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhDBuxxAHObPYW0uT9IvZlkpHf9H0lHoIJsdYyxVkagPGwfHeg2EjNLHIf_6KazEDRplXs62oGSfH1viL4vO-5rRmz6UP-ND_d6nR55RPrpl5rSdbQZpgFADjxHU0M9K3O8O6S9hv4RGLIalPs_dTe_5I6SSxcc7AkVH1hZH7qJq9LI1cbl5PPhsZNmqw" style="color: #3778cd; margin-left: 1em; margin-right: 1em; text-decoration-line: none;"><img alt="" data-original-height="862" data-original-width="832" height="640" src="https://blogger.googleusercontent.com/img/a/AVvXsEhDBuxxAHObPYW0uT9IvZlkpHf9H0lHoIJsdYyxVkagPGwfHeg2EjNLHIf_6KazEDRplXs62oGSfH1viL4vO-5rRmz6UP-ND_d6nR55RPrpl5rSdbQZpgFADjxHU0M9K3O8O6S9hv4RGLIalPs_dTe_5I6SSxcc7AkVH1hZH7qJq9LI1cbl5PPhsZNmqw=w619-h640" style="background: transparent; border-radius: 0px; border: 1px solid transparent; box-shadow: rgba(0, 0, 0, 0.2) 0px 0px 0px; padding: 8px; position: relative;" width="619" /></a></div>Igor Trubinhttp://www.blogger.com/profile/17758940374397545163noreply@blogger.com0tag:blogger.com,1999:blog-5048643098278177070.post-4319390027501209272022-08-23T13:48:00.008-04:002022-08-23T21:30:19.772-04:00CMG'08 Trip Report<strong>Visualization and Analysis of Performance Data using R</strong>
<em>Jim Holtman </em>
<em>
</em><u>Summary</u>:
I did not attend this, but that is about free statistical and graphical tool (“R” tool and “S” language <a goog_docs_charindex="5213" href="http://www.r-project.org/">http://www.r-project.org/</a> ). Note: there is interface to SAS dataset function in open lib: <a href="http://lib.stat.cmu.edu/S/dataset">http://lib.stat.cmu.edu/S/dataset</a>
Functions that define and manipulate S "dataset" objects. A dataset is a matrix whose columns (variables) may be of different data types. Though motivated by a need to interface to SAS, they are useful in any data analysis. There is some function that relates to SPC: <a goog_docs_charindex="5627" href="http://lib.stat.cmu.edu/S/JohnsonSystem.q">JohnsonSystem</a> (<a href="http://lib.stat.cmu.edu/S/JohnsonSystem.q">http://lib.stat.cmu.edu/S/JohnsonSystem.q</a>)
In 2004 he published CMG paper about R usage: <a goog_docs_charindex="5702" href="http://www.cmg.org/proceedings/2004/4055.pdf">The Use of R for System Performance Analysis</a> . See also <a goog_docs_charindex="5768" href="http://www.ats.ucla.edu/stat/r/library/lecture_graphing_r.htm">Lecture: Graphing in R</a> (<a href="http://www.ats.ucla.edu/stat/r/library/lecture_graphing_r.htm">http://www.ats.ucla.edu/stat/r/library/lecture_graphing_r.htm</a>) or <a goog_docs_charindex="5814" href="http://ieee.cincinnati.fuse.net/R_IEEE_V2.pdf">http://ieee.cincinnati.fuse.net/R_IEEE_V2.pdf</a>
<em>Major takeaways: </em>That might be a good SAS/Graph replacement. I also think about writing some "S" program to build SEDS type of Control charts to illustrate how that works, for instance THAT COULD BE USED for a workshop similar Mr. Holtman had done. <div>
<span style="font-family: arial;"><b><u>Automating Process Pathology Detection – Rule Engine Design Hints</u></b>
</span><em>Ron Kaminski
</em>
<span style="font-style: italic;"><u>Summary:</u></span>
This is about analytical approach to capture pathologies like run-away and memory leaks. BTW Ron referenced my papers as an example of different (statistical) approach to do the same. This is continuation of his previous work in this field: http://www.cmg.org/proceedings/2003/3027.pdf
In private conversation he actually expressed some interest to put together both approaches to see how that works from different angles... I am opened. </div><div>
<strong><u>CMG-T: Modeling and Forecasting</u></strong>
<em>Speaker: Dr. Michael A. Salsburg </em>
<em><u>Summary:</u>
</em>Just a good an overview and tutorial for queuing theory and simulation based modeling and forecasting vs. statistical modeling-forecasting way I presented in my paper.</div><div>
<a name="_Toc219026170"><strong><u>eBay - the Shape of Infrastructure to Come</u></strong></a>
<em>Speaker: Paul Strong</em>
<em><u>Summary:</u></em>
Cloud computing is a “Outsourcing 2.0”, sooner or later even banks will use that approach to use capacity on-demand from cloud instead of having own computer farm…. </div><div>
<strong><u>Exception Based Modeling and Forecasting </u></strong>
<em>Speaker: Dr. Igor A. Trubin </em>
<em></em>
<em><u>Summary:
</u></em>This is my presentation which was successful and attracted more than 60 attendees. There were a lot of questions and comments during and before this session, positive comments were received from Mark Friedman (After I had to clarify for him 3-D concept of weekly control charts... - my bad ,I was probably not very clear presenting that...) and Ron Kaminski who expressed some interest in my EV algorithm to capture recent bad trends as that solves some problems of workload pathology recognition on which he has been working recently. </div><div>
<a name="_Toc219026171"><strong><u>So You Want to Manage Your z-Series MIPS? Then Detect & Control Application Workload Variance!</u></strong></a>
<em>Speaker: John S. Van Wagenen, Caterpillar</em>
<em></em>
<em><u>Summary: </u></em>
Unfortunately I could not attend this session as I presented mine in the same time. But this paper is about SEDS-like approach to manage Mainframe capacity! And that presentation got prestigious Mullen award!
There is a similar paper written by the same author last year: <a href="http://www.cmg.org/membersonly/2007/papers/7012.pdf">Performance Monitoring Process for Out of Standard Applications</a>
<em><u>Major takeaways:</u></em>
SEDS approach is valid and our implementation on mainframe might be adjusted using this paper methodology. </div><div>
<a name="_Toc219026172"><strong><u>Predicting the Relative Performance of CPU</u></strong></a>
<em>Speaker: Debbie Sheetz</em>
<u><em>Summary:</em>
</u>I used similar approach (see my 1st CMG paper and 1st figure in my last paper) in the past and know how challenging is to apply SPEC or other benchmarks to real servers with different configurations.
<em><u>Major takeaways:</u>
</em>This paper could be helpful in соме consolidation projects. </div><div>
<a name="_Toc219026173"><strong><u>Panel: Michelson Panel - Visualization</u></strong></a>
<em>Speaker: Jeff Buzen</em>
<em><u>Summary:</u>
</em>That was interesting to see deferent ways to present data visually. During this panel discussions I realized that my weekly control charts and especially 3-D version of that are kind of unique. I have even approached Dr. Buzen, with my comments about that… </div><div>
<a name="_Toc219026177"><strong><u>Mainstream NUMA and the TCP/IP stack</u></strong></a>
<em>Speaker: Mark B. Friedman </em>
<em></em>
<u><em>Summary:</em>
</u>This is brilliant but very scary paper. Two scary points:
A. For multicore servers the speed of memory access could be unpredictable and sometimes deadly slow because of NUMA – non universal memory access. And there are no any metrics or tools to measure that!
B. High performance network (1-10 and higher Gb) cannot be fully utilized, because it might consume all CPU cycles only to process network related interrupts.
<em></em>
<em>Major takeaways</em>:
It’s OK if network interface bandwidth utilization is low. And we should be careful with using modern multicore processors (8 and more cores). </div><div>
<a name="_Toc219026179"><u><strong>Performance and Capacity Management in an Outsourced Environment</strong></u></a>
<em>Speaker: Jeff Hammond </em>
<em></em>
<em><u>Summary</u>:</em>
This is very useful information about what we could expect working with outsourced service (people) or if we got outsourced ourselves. It confirms my own experience.
<em>Action Items:</em> Be prepared just in case!</div>Igor Trubinhttp://www.blogger.com/profile/17758940374397545163noreply@blogger.com0tag:blogger.com,1999:blog-5048643098278177070.post-45852337858796653502022-03-24T13:15:00.005-04:002022-03-24T13:23:05.716-04:00Our poster presentation "SPEC Research — Introducing the #PredictiveAnalytics Working Group" is scheduled at #ICPE2022 #ICPEconf Poster & Demo (Monday - April 11, 2022, 5:15pm)<p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://icpe2022.spec.org/program_files/schedule/" style="margin-left: 1em; margin-right: 1em;" target="_blank"><img alt="" data-original-height="431" data-original-width="754" height="366" src="https://blogger.googleusercontent.com/img/a/AVvXsEjx2PPeKU8R1FDc_3Yud-jHpoHL4VKcvbNkmfNtu4tMbaBPV_wzaLX3MaG8dbz2QFIl16WAceoAM5wtVyzmIHIuUVwt_46KRAci-VkOWT1UsjREq5a96hEG8MCA_T-uaezbrN1tWkOKRlDuKh9RQ1XUvl_Lm-XtJxJAJ2IzAdoUm-CPwL531oy-vw6v=w640-h366" width="640" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><a href="https://icpe2022.spec.org/program_files/schedule/" rel="nofollow" target="_blank"><span> </span>https://icpe2022.spec.org/program_files/<span style="font-size: large;">schedule/</span></a><p></p>Igor Trubinhttp://www.blogger.com/profile/17758940374397545163noreply@blogger.com0tag:blogger.com,1999:blog-5048643098278177070.post-56883901093750856432022-03-16T11:28:00.001-04:002022-03-16T11:28:04.755-04:00I am happy to co-author 2 papers for #ICPE2022 #ICPEconf<p>Online conference program <a href="https://icpe2022.spec.org/program_files/schedule/" rel="nofollow" target="_blank">https://icpe2022.spec.org/program_files/schedule/ </a> scheduled our following presentations:</p><p><a id="poster" style="color: #f58229; font-family: "Trebuchet MS", Verdana, Helvetica, Arial, sans-serif; font-size: 24px;">Poster & Demo (Monday - April 11, 2022, 5:15pm )</a></p><p style="background-color: white; font-family: "Trebuchet MS", Verdana, Helvetica, Arial, sans-serif; line-height: 24px; text-align: justify;">André Bauer, Mark Leznik, Md Shahriar Iqbal, Daniel Seybold, Igor Trubin, Benjamin Erb, Jörg Domaschka and Pooyan Jamshidi. <b>SPEC Research — Introducing the Predictive Data Analytics Working Group</b></p><p style="background-color: white; font-family: "Trebuchet MS", Verdana, Helvetica, Arial, sans-serif; line-height: 24px; text-align: justify;"><a id="data" style="color: #f58229; font-size: 24px; text-align: left;">Data Challenge (Tuesday - April 12,, 4:15pm - 4:55pm)</a></p><p style="background-color: white; font-family: "Trebuchet MS", Verdana, Helvetica, Arial, sans-serif; line-height: 24px; text-align: justify;">Md Shahriar Iqbal, Mark Leznik, Igor Trubin, Arne Lochner, Pooyan Jamshidi and André Bauer. <b>Change Point Detection for MongoDB Time Series Performance Regression</b></p><p style="background-color: white; font-family: "Trebuchet MS", Verdana, Helvetica, Arial, sans-serif; line-height: 24px; text-align: justify;"><b></b></p><div class="separator" style="clear: both; text-align: center;"><b><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhPPRtxTclvdP0hnEmTXlKz2mojk_PLQkWb5hHciwKJFJXhs7E4DCTnDFZzpAVfOcLNAEmVxuDf7N9b18aHSp20Hx7Khg-5I0cy6pm3KsllgsyMK365O5DpGXCbEqeXkeCKaKFOoutuR3q1HzhLv-2ZVWVobWhcKns-wye87kGjq9BuI8LHEFHYOAD3" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="346" data-original-width="557" height="199" src="https://blogger.googleusercontent.com/img/a/AVvXsEhPPRtxTclvdP0hnEmTXlKz2mojk_PLQkWb5hHciwKJFJXhs7E4DCTnDFZzpAVfOcLNAEmVxuDf7N9b18aHSp20Hx7Khg-5I0cy6pm3KsllgsyMK365O5DpGXCbEqeXkeCKaKFOoutuR3q1HzhLv-2ZVWVobWhcKns-wye87kGjq9BuI8LHEFHYOAD3" width="320" /></a></b></div><b><br /><br /></b><p></p>Igor Trubinhttp://www.blogger.com/profile/17758940374397545163noreply@blogger.com0tag:blogger.com,1999:blog-5048643098278177070.post-35209635436779378472022-02-28T15:59:00.009-05:002022-03-04T11:51:42.403-05:00"Change Point Detection (#ChangeDetection) for MongoDB Time Series Performance Regression" paper for ACM/SPEC ICPE 2022 Data Challenge Track<h3 class="post-title entry-title" itemprop="name" style="background-color: #fefdfa; color: #d52a33; font-family: Georgia, Utopia, "Palatino Linotype", Palatino, serif; font-size: 22px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: normal; line-height: normal; margin: 0px; position: relative;"><span face="Arial, Helvetica, sans-serif" style="background-color: white; color: #500050; font-size: small;">The ACM/SPEC ICPE 2022 - Data Challenge Track Committee has decided to ACCEPT our article:</span></h3><p><span face="Arial, Helvetica, sans-serif" style="background-color: white; color: #500050; font-size: small;">TITLE: <b><i>Change Point Detection for MongoDB Time Series Performance Regression</i></b></span><br style="background-color: white; color: #500050; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><span face="Arial, Helvetica, sans-serif" style="background-color: white; color: #500050; font-size: small;">AUTHORS: <i>Md Shahriar Iqbal, Mark Leznik,<b> Igor Trubin,</b> Arne Lochner, Pooyan Jamshidi and André Bauer</i></span><br style="background-color: white; color: #500050; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><br style="background-color: white; color: #500050; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><br /></p><div class="separator" style="clear: both; text-align: center;"><a href="https://icpe2022.spec.org/tracks-and-submissions/data-challenge-track/" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="344" data-original-width="928" height="149" src="https://blogger.googleusercontent.com/img/a/AVvXsEjClbJ3uJnhmXYHFG_aeAjnSNvORSuFWU-dmtxWfgr9RGHAz7joIdVNGEGPsdsPKFULRrn8CrnueJxzqE9aXe1GILidSE5DPwPMUvBeyWIhhSacueQWKUvm7JSms9X1RsMqdLKPFwdxB6FX7qeKXvrPGFt8_QW_5geCprekhCwyfRiHqArO-z4LSImM=w400-h149" width="400" /></a></div><div class="separator" style="clear: both; text-align: center;"><a href="https://icpe2022.spec.org/tracks-and-submissions/data-challenge-track/">https://icpe2022.spec.org/tracks-and-submissions/data-challenge-track/</a></div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;"><div class="separator" style="clear: both;">ABSTRACT</div><div class="separator" style="clear: both; text-align: justify;">Commits to the MongoDB software repository trigger a collection</div><div class="separator" style="clear: both; text-align: justify;">of automatically run tests. Here, the identification of commits </div><div class="separator" style="clear: both; text-align: justify;">responsible for performance regressions is paramount. Previously, the</div><div class="separator" style="clear: both; text-align: justify;">process relied on manual inspection of time series graphs to identify</div><div class="separator" style="clear: both; text-align: justify;">signicant changes, later replaced with a threshold-based detection</div><div class="separator" style="clear: both; text-align: justify;">system. However, neither system was sufficient for finding changes</div><div class="separator" style="clear: both; text-align: justify;">in performance in a timely manner. This work describes our recent</div><div class="separator" style="clear: both; text-align: justify;">implementation of a change point detection system built upon the</div><div class="separator" style="clear: both; text-align: justify;"><b><a href="https://www.perfomalist.com/">Perfomalist</a> </b>approach in combination with XGBoost algorithm. The</div><div class="separator" style="clear: both; text-align: justify;">algorithm produces a list of change points representing significant</div><div class="separator" style="clear: both; text-align: justify;">changes from a given history of performance results. We are able</div><div class="separator" style="clear: both; text-align: justify;">to automatically detect change points and achieve an 83% accuracy,</div><div class="separator" style="clear: both; text-align: justify;">all while reducing the human effort in the process.</div></div><div><br /></div>More <b style="text-align: justify;"><a href="https://www.perfomalist.com/">Perfomalist</a>'s </b><b> </b>approach details can be found in this blog post:<br /><div><h3 class="post-title entry-title" itemprop="name" style="background-color: #fefdfa; color: #d52a33; font-family: Georgia, Utopia, "Palatino Linotype", Palatino, serif; font-size: 22px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: normal; line-height: normal; margin: 0px; position: relative;"><a href="https://www.trub.in/2020/08/cpd-change-points-detection-is-planed.html">CPD - Change Point Detection (#ChangeDetection) is implemented in the free web tool Perfomalist</a></h3><div><br /></div><div>The result of initial usage of Perfomalist CPD API against MongoDB data is published HERE:</div><div><h3 class="post-title entry-title" itemprop="name" style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 22px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; position: relative;"><a href="https://www.trutechdev.com/2022/03/perfomalist-changedetection-api-was.html" target="_blank">Perfomalist #ChangeDetection API was used against #MongoDB #perfomanceTesting dataset</a></h3></div><p> </p><p> </p><p><br /></p></div>Igor Trubinhttp://www.blogger.com/profile/17758940374397545163noreply@blogger.com0tag:blogger.com,1999:blog-5048643098278177070.post-86504984193348802992022-02-09T12:11:00.005-05:002022-02-10T13:00:52.354-05:00My Cloud Optimization team at #CapitalOne bank won the CMG.org #Innovation Award (#CMGNews)<p> <span face="Arial, Helvetica, sans-serif" style="background-color: white; color: #222222; font-size: small;"> </span><a data-saferedirecturl="https://www.google.com/url?q=https://urldefense.com/v3/__https://www.cmg.org/2022/02/capital-one-announced-as-winner-of-the-impact-innovation-award/__;!!FrPt2g6CO4Wadw!dMfzQCHwvVpPfy2By61IwfgYxKnu22LQoSNRHsfCfyhaobjTVXrJ0dEBB5fbx6m9-Au8$&source=gmail&ust=1644511208834000&usg=AOvVaw2w4KMuC98_dOHr2umSGYcm" href="https://urldefense.com/v3/__https://www.cmg.org/2022/02/capital-one-announced-as-winner-of-the-impact-innovation-award/__;!!FrPt2g6CO4Wadw!dMfzQCHwvVpPfy2By61IwfgYxKnu22LQoSNRHsfCfyhaobjTVXrJ0dEBB5fbx6m9-Au8$" style="background-color: white; color: #1155cc; font-family: Arial, Helvetica, sans-serif; font-size: small;" target="_blank">https://www.cmg.org/2022/<wbr></wbr>02/capital-one-announced-as-<wbr></wbr>winner-of-the-impact-<wbr></wbr>innovation-award/</a></p><div class="separator" style="clear: both; text-align: center;"><a href="https://www.cmg.org/2022/02/capital-one-announced-as-winner-of-the-impact-innovation-award/" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;" target="_blank"><img border="0" data-original-height="1000" data-original-width="1000" height="400" src="https://blogger.googleusercontent.com/img/a/AVvXsEis9xXvwmQRXeZ-FPlvmflMufQ6P3SyE1PhvmTY9usVLGhKOZIYI3squmiu65L-RgDYE1Nc8y0i1elbQ3rNBVDJq-Dd3v_UUBDaTjM6ieTAKl3eoIWBuJqkdieRBtZ6sC0xuMM9gIrcsddqfN39zHAbm7w9vFsomfgxb1k_I_TGo9HQpTh9CDoCxJiH=w400-h400" width="400" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgP5EQkp-XD7QY2Cglp5yMjiL8ezccrCR-5sMqfe0U6LuCGjnTceznH5oYajQacDFwU6zQbWB4dZ3MEK4XyWixSWAlMHTmnKVxXdWeffpBtezt52uDfg0UizyTNTKbYsLLSrlxJn46zrO0WxDQz_GJ748uygy1awtTVeYzJMQUarJ3CdaEkcTViSLWS=s1783" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1000" data-original-width="1783" height="179" src="https://blogger.googleusercontent.com/img/a/AVvXsEgP5EQkp-XD7QY2Cglp5yMjiL8ezccrCR-5sMqfe0U6LuCGjnTceznH5oYajQacDFwU6zQbWB4dZ3MEK4XyWixSWAlMHTmnKVxXdWeffpBtezt52uDfg0UizyTNTKbYsLLSrlxJn46zrO0WxDQz_GJ748uygy1awtTVeYzJMQUarJ3CdaEkcTViSLWS=s320" width="320" /></a></div><br /><p><br /></p>Igor Trubinhttp://www.blogger.com/profile/17758940374397545163noreply@blogger.com0tag:blogger.com,1999:blog-5048643098278177070.post-61836201363506125762022-02-03T10:18:00.004-05:002022-02-03T10:18:54.596-05:00My publications in RG got 5000+ reads <p><a href="https://www.researchgate.net/profile/Igor-Trubin">https://www.researchgate.net/profile/Igor-Trubin </a></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEiS75OS4x55IVSA09rlqjms2Qni5GPwltp46VEX8WS4Lnsk2UUN786MVvg1bUpuFSZkVQP1f3q0eIcHK2fylM1aE55muWGDS4fMfrvxGxPrOa9FMORqlmdQM1lSd5lDzMzbmkJ8V0V6_IBeJavDfMROwhPS2HKZnfg-S2695dlRejNKGfzZQDyCDecx=s1274" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="1274" data-original-width="712" height="640" src="https://blogger.googleusercontent.com/img/a/AVvXsEiS75OS4x55IVSA09rlqjms2Qni5GPwltp46VEX8WS4Lnsk2UUN786MVvg1bUpuFSZkVQP1f3q0eIcHK2fylM1aE55muWGDS4fMfrvxGxPrOa9FMORqlmdQM1lSd5lDzMzbmkJ8V0V6_IBeJavDfMROwhPS2HKZnfg-S2695dlRejNKGfzZQDyCDecx=w358-h640" width="358" /></a></div><br /><p><br /></p>Igor Trubinhttp://www.blogger.com/profile/17758940374397545163noreply@blogger.com0tag:blogger.com,1999:blog-5048643098278177070.post-37521695232764219542022-01-21T21:11:00.003-05:002022-01-21T21:11:36.972-05:00Panel Discussion: Roadmap for Cultivating Performance-Aware Software Engineers<p> </p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj0r6LTvfebmHgeLOgQKi_ZnKL7Xd7o8PqEawJdB3Ni3O2vNYby8TSMFwwOP48j7pkta6IjMpbadBAh_qR3uTSmUc9STPHDQZxHjY3peSLAoxOg_Qw4EhTv41g63TzrnM5F8Sz6qyGeYXI/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="579" data-original-width="1106" height="336" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj0r6LTvfebmHgeLOgQKi_ZnKL7Xd7o8PqEawJdB3Ni3O2vNYby8TSMFwwOP48j7pkta6IjMpbadBAh_qR3uTSmUc9STPHDQZxHjY3peSLAoxOg_Qw4EhTv41g63TzrnM5F8Sz6qyGeYXI/w640-h336/image.png" width="640" /></a></div><p></p>Igor Trubinhttp://www.blogger.com/profile/17758940374397545163noreply@blogger.com1tag:blogger.com,1999:blog-5048643098278177070.post-9469231312343917232022-01-21T20:32:00.004-05:002022-01-21T21:01:24.782-05:00"#CloudServers Rightsizing with #Seasonality Adjustments" - my presentation at CMG IMPACT conference (#CMGnews)<p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://cmgimpact.com/sessions-schedule/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="757" data-original-width="1096" height="442" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqrj1SIyks1xLZ0MBtkHOi2Wo_bpvH1C6VqTCoyknefX-cpASXuLGUxT62OaI1qw8sbnD5LxaIZV-Scl8Lp9aWNuf5H_cCjrP8K7KjIzlkNQ7ePxj5Qff-KgMzXrfQWPeABSpZSQkZVEM/w640-h442/image.png" width="640" /></a></div><br />Feb 4, 2022 12:15 Virtual at <a href="https://cmgimpact.com/sessions-schedule/">https://cmgimpact.com/sessions-schedule/</a><br /><br /><p></p>Igor Trubinhttp://www.blogger.com/profile/17758940374397545163noreply@blogger.com0tag:blogger.com,1999:blog-5048643098278177070.post-9275786438569459032022-01-06T10:33:00.001-05:002022-01-06T10:33:45.805-05:00"Performance Anomaly and Change Point Detection for Large-Scale System Management" - my paper published at Springer<p> </p><div class="separator" style="clear: both; text-align: center;"><a href="https://media.springernature.com/w306/springer-static/cover/book/978-981-16-6369-7.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="464" data-original-width="306" height="464" src="https://media.springernature.com/w306/springer-static/cover/book/978-981-16-6369-7.jpg" width="306" /></a></div><br /><span class="BookTitle" style="background-color: #fcfcfc; box-sizing: border-box; color: #333333; font-family: "Source Sans Pro", Helvetica, Arial, sans-serif; font-size: 1.4rem; letter-spacing: 0.017em;"><a data-track-action="Book title" data-track-label="" data-track="click" href="https://link.springer.com/book/10.1007/978-981-16-6369-7" style="background-color: initial; box-sizing: border-box; color: #004aa7;">Intelligent Sustainable Systems</a></span><span class="page-numbers-info" style="background-color: #fcfcfc; box-sizing: border-box; color: #333333; font-family: "Source Sans Pro", Helvetica, Arial, sans-serif; font-size: 1.4rem; letter-spacing: 0.017em;"> pp 403-407</span><span class="u-inline-block u-ml-4" style="background-color: #fcfcfc; box-sizing: border-box; color: #333333; display: inline-block; font-family: "Source Sans Pro", Helvetica, Arial, sans-serif; font-size: 1.4rem; letter-spacing: 0.017em; margin-left: 4px !important;">| <a data-track-action="Cite as link" data-track-label="Enumeration section" data-track="click" href="https://link.springer.com/chapter/10.1007%2F978-981-16-6369-7_36#citeas" style="background-color: initial; box-sizing: border-box; color: #004aa7;">Cite as</a></span><p></p><div class="ArticleHeader main-context" style="background-color: #fcfcfc; box-sizing: border-box; color: #333333; font-family: "Source Sans Pro", Helvetica, Arial, sans-serif; font-size: 1.4rem; letter-spacing: 0.017em; line-height: 1.4; margin-bottom: 36px; zoom: 1;"><div class="MainTitleSection" style="box-sizing: border-box; font-family: Georgia, serif; margin: 0px 0px 24px; overflow-wrap: break-word; word-break: break-word;"><h1 class="ChapterTitle" lang="en" style="box-sizing: border-box; font-size: 2.8rem; font-weight: 400; letter-spacing: 0.008em; line-height: 1.3; margin: 0px 0px 8px;">Performance Anomaly and Change Point Detection for Large-Scale System Management</h1></div><div class="authors u-clearfix authors--enhanced" data-component="SpringerLink.Authors" style="box-sizing: border-box; zoom: 1;"><ul class="u-interface u-inline-list authors__title" data-role="AuthorsNavigation" style="border-bottom: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-size: 1.4rem; letter-spacing: 0.017em; list-style: none; margin: 0px; padding: 0px; position: relative;"><li style="box-sizing: border-box; display: inline-block; letter-spacing: normal; margin: 0px; padding: 0px; vertical-align: middle;"><a class="selected" data-track-action="Authors tab" data-track-label="" data-track="click" href="https://link.springer.com/chapter/10.1007%2F978-981-16-6369-7_36#authors" style="background-color: initial; box-sizing: border-box; color: #333333; display: block; min-width: 160px; padding-bottom: 4px; padding-left: 0px; padding-right: 16px; position: relative; text-decoration-line: none;">Authors</a></li><li style="box-sizing: border-box; display: inline-block; letter-spacing: normal; margin: 0px; padding: 0px; vertical-align: middle;"><a data-track-action="Authors and affiliations tab" data-track-label="" data-track="click" href="https://link.springer.com/chapter/10.1007%2F978-981-16-6369-7_36#authorsandaffiliations" style="background-color: initial; box-sizing: border-box; color: #004aa7; display: block; min-width: 160px; padding-bottom: 4px; padding-left: 16px; padding-right: 16px; position: relative; text-decoration-line: none;">Authors and affiliations</a></li></ul><span class="marker" style="background: rgb(51, 51, 51); box-sizing: border-box; height: 2px; left: 0px; margin-top: -2px; position: absolute; transition: transform 0.3s ease 0s, -webkit-transform 0.3s ease 0s; width: 160px;"></span><div class="authors__list" data-role="AuthorsList" id="authors" style="box-sizing: border-box; overflow: hidden; transition: opacity 0.3s ease 0s;" tabindex="-1"><ul class="test-contributor-names" style="box-sizing: border-box; letter-spacing: -0.31em; list-style: none; margin: 0px; padding: 8px 0px 24px;"><li class="u-mb-2 u-pt-4 u-pb-4" itemprop="author" itemscope="" itemtype="http://schema.org/Person" style="box-sizing: border-box; display: inline-block; letter-spacing: normal; margin-bottom: 2px !important; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 4px !important; padding-left: 0px; padding-right: 0px; padding-top: 4px !important; vertical-align: middle;"><span class="authors__name" itemprop="name" style="box-sizing: border-box;">Igor Trubin</span><span class="author-information" style="box-sizing: border-box; vertical-align: text-bottom;"><span class="authors__contact" style="box-sizing: border-box;"><a data-track-action="Email author" data-track-label="" data-track="click" href="mailto:Igor.Trubin@capitalone.com" itemprop="email" style="background-color: initial; box-sizing: border-box; color: #004aa7; display: inline-block;" title="Igor.Trubin@capitalone.com"><img alt="Email author" height="24" src="https://link.springer.com/springerlink-static/images/svg/email.svg" style="border: 0px; box-sizing: border-box; height: auto; margin: 0px; max-width: 100%; vertical-align: middle; width: auto;" width="24" /></a></span></span></li></ul></div><div class="authors__affiliations" id="authorsandaffiliations" style="box-sizing: border-box; height: 0px; opacity: 0; overflow: hidden; transition: opacity 0.3s ease 0s;" tabindex="-1"><div class="authors-affiliations u-interface" style="background-image: linear-gradient(90deg, rgba(242, 242, 242, 0.4) 0px, rgba(242, 242, 242, 0.4)); box-sizing: border-box; display: table; font-size: 1.4rem; letter-spacing: 0.017em; margin: 0px 0px 24px; padding: 16px; width: 727px;"><ul class="test-contributor-names" style="border-right: 1px solid rgb(204, 204, 204); box-sizing: border-box; display: table-cell; list-style: none; margin: 0px; padding: 0px 24px 4px 0px; vertical-align: top; width: 208.5px;"><li class="u-mb-2 u-pt-4 u-pb-4" itemprop="author" itemscope="" itemtype="http://schema.org/Person" style="box-sizing: border-box; margin-bottom: 2px !important; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 4px !important; padding-left: 0px; padding-right: 0px; padding-top: 4px !important;"><span class="authors-affiliations__name" itemprop="name" style="box-sizing: border-box; display: inline; margin-right: 12px;"></span><ul class="authors-affiliations__indexes u-inline-list" data-role="AuthorsIndexes" style="box-sizing: border-box; display: inline; letter-spacing: -0.31em; list-style: none; margin: 0px; padding: 0px;"><li data-affiliation="affiliation-1" style="background-color: #cccccc; border-radius: 100px; box-sizing: border-box; display: inline-block; font-size: 1.2rem; letter-spacing: normal; line-height: 1; margin: 0px 4px 2px 0px; padding: 2px 8px; vertical-align: middle;"></li></ul><span class="author-information" style="box-sizing: border-box; display: block; margin-bottom: 12px; vertical-align: text-bottom;"><span class="author-information__contact u-icon-before" style="box-sizing: border-box; display: block; position: relative;"><a data-track-action="Email author" data-track-label="" data-track="click" href="mailto:Igor.Trubin@capitalone.com" itemprop="email" style="background-color: initial; box-sizing: border-box; color: #004aa7;" tabindex="-1" title="Igor.Trubin@capitalone.com"></a></span></span></li></ul><ol class="test-affiliations" style="border: none; box-sizing: border-box; display: table-cell; list-style: none; margin: 0px; padding: 0px 0px 0px 16px; vertical-align: top; width: 486.5px;"><li class="affiliation" data-affiliation-highlight="affiliation-1" data-test="affiliation-1" itemscope="" itemtype="http://schema.org/Organization" style="border-radius: 4px; box-sizing: border-box; letter-spacing: -0.31em; margin: 0px; padding: 4px 8px; position: relative;"><span class="affiliation__count" style="box-sizing: border-box; display: inline-block; left: 8px; letter-spacing: normal; position: absolute; top: 4px; vertical-align: top;">1.</span><span class="affiliation__item" style="box-sizing: border-box; display: inline-block; letter-spacing: normal; padding-left: 2.5em; vertical-align: top;"><span class="affiliation__name" itemprop="name" style="box-sizing: border-box;"></span><span class="affiliation__address" itemprop="address" itemscope="" itemtype="http://schema.org/PostalAddress" style="box-sizing: border-box;"><span class="affiliation__city" itemprop="addressRegion" style="box-sizing: border-box;"></span><span class="affiliation__country" itemprop="addressCountry" style="box-sizing: border-box;"></span></span></span></li></ol></div></div></div><div class="main-context__container" data-component="SpringerLink.ArticleMetrics" style="box-sizing: border-box; display: flex; margin-bottom: 24px;"><div class="main-context__column" style="border-right: 1px solid rgb(204, 204, 204); box-sizing: border-box; flex: 0 1 auto; margin-right: 16px; padding-right: 16px;"><span style="box-sizing: border-box;">Conference paper</span><div class="article-dates" style="box-sizing: border-box; line-height: 1.8;"><span class="article-dates__label" style="box-sizing: border-box; font-weight: 600;">First Online: </span><span class="article-dates__first-online" style="box-sizing: border-box;">17 December 2021</span></div></div><div class="main-context__column" style="border-right: 0px; box-sizing: border-box; flex: 0 1 auto; margin-right: 0px; padding-right: 0px; padding-top: 0px;"><ul class="article-metrics u-sansSerif" id="book-metrics" style="box-sizing: border-box; letter-spacing: -0.31em; list-style: none; margin: 0px 0px 0px -4px; padding: 0px;"><li class="article-metrics__item" style="box-sizing: border-box; display: inline-block; letter-spacing: normal; margin: 0px 4px; padding: 0px; text-align: center; vertical-align: middle;"><span class="test-metric-count article-metrics__views" style="border-radius: 50%; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #666666; display: block; height: 3em; line-height: calc(3em - 2px); margin: 0px auto; padding: 0px; position: relative; width: 3em;">1</span><span class="test-metric-name article-metrics__label" style="box-sizing: border-box; font-size: 1.2rem;">Downloads</span></li></ul></div></div><span class="vol-info" id="test-SeriesTitle" style="box-sizing: border-box; display: inline-block; margin-bottom: 0px;">Part of the <a class="gtm-book-series-link" href="https://link.springer.com/bookseries/15179" style="background-color: initial; box-sizing: border-box; color: #004aa7;">Lecture Notes in Networks and Systems</a> book series (LNNS, volume 334)</span></div><section class="Abstract" id="Abs1" lang="en" style="background-color: #fcfcfc; box-sizing: border-box; color: #333333; font-family: Georgia, serif; font-size: 17px; letter-spacing: 0.102px; padding-top: 0px;" tabindex="-1"><h2 class="Heading" style="background-color: #f2f2f2; border-top: 2px solid rgba(51, 51, 51, 0.2); box-sizing: border-box; font-size: 2.6rem; font-weight: 400; letter-spacing: 0.008em; line-height: 1.3; margin-bottom: 0px; margin-left: -17.4375px; margin-top: 0px; overflow-wrap: break-word; padding: 12px 0px 12px 17.4375px; width: 744.438px; word-break: break-word;">Abstract</h2><p class="Para" id="Par1" style="box-sizing: border-box; margin-bottom: 1.2em; margin-top: 1em; overflow-wrap: break-word; word-break: break-word;">The presentation starts with the short overview of the classical statistical process control (SPC)-based anomaly detection techniques and tools including Multivariate Adaptive Statistical Filtering (MASF); Statistical Exception and Trend Detection System (SETDS), Exception Value (EV) meta-metric-based change point detection; control charts; business driven massive prediction and methods of using them to manage large-scale systems such as on-prem servers fleet or massive clouds. Then, the presentation is focused on modern techniques of anomaly and normality detection, such as deep learning and entropy-based anomalous pattern detections.</p></section><div class="KeywordGroup" lang="en" style="background-color: #fcfcfc; box-sizing: border-box; color: #333333; font-family: Georgia, serif; font-size: 17px; letter-spacing: 0.102px; margin-bottom: 24px;"><h2 class="Heading" style="box-sizing: border-box; font-size: 2.6rem; font-weight: 400; letter-spacing: 0.008em; line-height: 1.3; margin-bottom: 0.5em; margin-top: 0.5em;">Keywords</h2><span class="Keyword" style="background-color: #f2f2f2; border-radius: 2px; box-sizing: border-box; display: inline-block; margin-bottom: 0.3em; margin-right: 0.3em; padding: 0px 0.2em;">Anomaly detection </span><span class="Keyword" style="background-color: #f2f2f2; border-radius: 2px; box-sizing: border-box; display: inline-block; margin-bottom: 0.3em; margin-right: 0.3em; padding: 0px 0.2em;">Change point detection </span><span class="Keyword" style="background-color: #f2f2f2; border-radius: 2px; box-sizing: border-box; display: inline-block; margin-bottom: 0.3em; margin-right: 0.3em; padding: 0px 0.2em;">Business driven forecast </span><span class="Keyword" style="background-color: #f2f2f2; border-radius: 2px; box-sizing: border-box; display: inline-block; margin-bottom: 0.3em; margin-right: 0.3em; padding: 0px 0.2em;">Control chart </span><span class="Keyword" style="background-color: #f2f2f2; border-radius: 2px; box-sizing: border-box; display: inline-block; margin-bottom: 0.3em; margin-right: 0.3em; padding: 0px 0.2em;">Deep Learning </span><span class="Keyword" style="background-color: #f2f2f2; border-radius: 2px; box-sizing: border-box; display: inline-block; margin-bottom: 0.3em; margin-right: 0.3em; padding: 0px 0.2em;">Entropy analysis </span></div><div class="note test-pdf-link" id="cobranding-and-download-availability-text" style="background-color: #fcfcfc; background-image: linear-gradient(90deg, rgba(242, 242, 242, 0.4) 0px, rgba(242, 242, 242, 0.4)); border-bottom: 1px solid rgb(217, 217, 217); border-top: 1px solid rgb(217, 217, 217); box-sizing: border-box; color: #333333; font-family: "Source Sans Pro", Helvetica, Arial, sans-serif; font-size: 1.4rem; letter-spacing: 0.102px; line-height: 1.5625; margin-bottom: 24px; margin-left: -17.4375px; padding: 12px 16px 12px 17.4375px; text-align: center; width: 744.438px;"><div id="chapter_no_access_banner" style="box-sizing: border-box;">This is a preview of subscription content, <a data-track-action="Preview banner - Log in" data-track-label="" data-track="click" href="https://link.springer.com/signup-login?previousUrl=https%3A%2F%2Flink.springer.com%2Fchapter%2F10.1007%252F978-981-16-6369-7_36" id="test-login-banner-link" style="background-color: initial; box-sizing: border-box; color: #004aa7;">log in</a> to check access.</div></div><section class="Section1 RenderAsSection1" id="Bib1" style="background-color: #fcfcfc; box-sizing: border-box; color: #333333; font-family: Georgia, serif; font-size: 17px; letter-spacing: 0.102px; margin-top: 1em; padding-top: 24px;" tabindex="-1"><h2 class="Heading" style="background-color: #f2f2f2; border-top: 2px solid rgba(51, 51, 51, 0.2); box-sizing: border-box; font-size: 2.6rem; font-weight: 400; letter-spacing: 0.008em; line-height: 1.3; margin-bottom: 0px; margin-left: -17.4375px; margin-top: 0px; overflow-wrap: break-word; padding: 12px 0px 12px 17.4375px; width: 744.438px; word-break: break-word;">References</h2><div class="content" style="box-sizing: border-box;"><ol class="BibliographyWrapper" style="box-sizing: border-box; list-style: none; margin: 0px; padding: 0px;"><li class="Citation" style="box-sizing: border-box; margin: 16px 0px; padding: 0px; position: relative;"><div class="CitationNumber" style="box-sizing: border-box; float: left; font-family: "Source Sans Pro", Helvetica, Arial, sans-serif; margin-right: 4px; min-width: 2em; text-align: right;">1.</div><div class="CitationContent" id="CR1" style="box-sizing: border-box; margin-left: 0px; overflow: hidden; padding-left: 0px;">Trubin, I.: Exception based modeling and forecasting. In: Proceedings of Computer Measurement Group (2008)<span class="Occurrences" style="box-sizing: border-box; display: block;"><span class="Occurrence OccurrenceGS" style="box-sizing: border-box; display: inline-block; margin-right: 16px;"><a class="google-scholar-link gtm-reference" data-reference-type="Google Scholar" href="https://scholar.google.com/scholar?q=Trubin%2C%20I.%3A%20Exception%20based%20modeling%20and%20forecasting.%20In%3A%20Proceedings%20of%20Computer%20Measurement%20Group%20%282008%29" rel="noopener" style="background-color: initial; box-sizing: border-box; color: #004aa7;" target="_blank"><span style="box-sizing: border-box;">Google Scholar</span></a></span></span></div></li><li class="Citation" style="box-sizing: border-box; margin: 16px 0px; padding: 0px; position: relative;"><div class="CitationNumber" style="box-sizing: border-box; float: left; font-family: "Source Sans Pro", Helvetica, Arial, sans-serif; margin-right: 4px; min-width: 2em; text-align: right;">2.</div><div class="CitationContent" id="CR2" style="box-sizing: border-box; margin-left: 0px; overflow: hidden; padding-left: 0px;">Jeffrey Buzen, F., Annie Shum, S.: MASF—multivariate adaptive statistical filtering. In: Proceedings of Computer Measurement Group (1995)<span class="Occurrences" style="box-sizing: border-box; display: block;"><span class="Occurrence OccurrenceGS" style="box-sizing: border-box; display: inline-block; margin-right: 16px;"><a class="google-scholar-link gtm-reference" data-reference-type="Google Scholar" href="https://scholar.google.com/scholar?q=Jeffrey%20Buzen%2C%20F.%2C%20Annie%20Shum%2C%20S.%3A%20MASF%E2%80%94multivariate%20adaptive%20statistical%20filtering.%20In%3A%20Proceedings%20of%20Computer%20Measurement%20Group%20%281995%29" rel="noopener" style="background-color: initial; box-sizing: border-box; color: #004aa7;" target="_blank"><span style="box-sizing: border-box;">Google Scholar</span></a></span></span></div></li><li class="Citation" style="box-sizing: border-box; margin: 16px 0px; padding: 0px; position: relative;"><div class="CitationNumber" style="box-sizing: border-box; float: left; font-family: "Source Sans Pro", Helvetica, Arial, sans-serif; margin-right: 4px; min-width: 2em; text-align: right;">3.</div><div class="CitationContent" id="CR3" style="box-sizing: border-box; margin-left: 0px; overflow: hidden; padding-left: 0px;">Trubin, I.: Review of IT control chart. CIS J. <span class="EmphasisTypeBold " style="box-sizing: border-box; font-weight: 600;">4</span>(11), 2079–8407 (2013)<span class="Occurrences" style="box-sizing: border-box; display: block;"><span class="Occurrence OccurrenceGS" style="box-sizing: border-box; display: inline-block; margin-right: 16px;"><a class="google-scholar-link gtm-reference" data-reference-type="Google Scholar" href="http://scholar.google.com/scholar_lookup?title=Review%20of%20IT%20control%20chart&author=I.%20Trubin&journal=CIS%20J.&volume=4&issue=11&pages=2079-8407&publication_year=2013" rel="noopener" style="background-color: initial; box-sizing: border-box; color: #004aa7;" target="_blank"><span style="box-sizing: border-box;">Google Scholar</span></a></span></span></div></li><li class="Citation" style="box-sizing: border-box; margin: 16px 0px; padding: 0px; position: relative;"><div class="CitationNumber" style="box-sizing: border-box; float: left; font-family: "Source Sans Pro", Helvetica, Arial, sans-serif; margin-right: 4px; min-width: 2em; text-align: right;">4.</div><div class="CitationContent" id="CR4" style="box-sizing: border-box; margin-left: 0px; overflow: hidden; padding-left: 0px;">Perfomalist Homepage, <span class="ExternalRef" style="box-sizing: border-box;"><a href="http://www.perfomalist.com/" rel="noopener" style="background-color: initial; box-sizing: border-box; color: #004aa7;" target="_blank"><span class="RefSource" style="box-sizing: border-box; overflow-wrap: break-word; word-break: break-word;">http://www.perfomalist.com</span></a></span>. Last accessed on 10 June 2021<span class="Occurrences" style="box-sizing: border-box; display: block;"></span></div></li><li class="Citation" style="box-sizing: border-box; margin: 16px 0px; padding: 0px; position: relative;"><div class="CitationNumber" style="box-sizing: border-box; float: left; font-family: "Source Sans Pro", Helvetica, Arial, sans-serif; margin-right: 4px; min-width: 2em; text-align: right;">5.</div><div class="CitationContent" id="CR5" style="box-sizing: border-box; margin-left: 0px; overflow: hidden; padding-left: 0px;">Trubin, I., et al.: Systems and methods for modeling computer resource metrics. US Patent 10,437,697 (2016)<span class="Occurrences" style="box-sizing: border-box; display: block;"><span class="Occurrence OccurrenceGS" style="box-sizing: border-box; display: inline-block; margin-right: 16px;"><a class="google-scholar-link gtm-reference" data-reference-type="Google Scholar" href="https://scholar.google.com/scholar?q=Trubin%2C%20I.%2C%20et%20al.%3A%20Systems%20and%20methods%20for%20modeling%20computer%20resource%20metrics.%20US%20Patent%2010%2C437%2C697%20%282016%29" rel="noopener" style="background-color: initial; box-sizing: border-box; color: #004aa7;" target="_blank"><span style="box-sizing: border-box;">Google Scholar</span></a></span></span></div></li><li class="Citation" style="box-sizing: border-box; margin: 16px 0px; padding: 0px; position: relative;"><div class="CitationNumber" style="box-sizing: border-box; float: left; font-family: "Source Sans Pro", Helvetica, Arial, sans-serif; margin-right: 4px; min-width: 2em; text-align: right;">6.</div><div class="CitationContent" id="CR6" style="box-sizing: border-box; margin-left: 0px; overflow: hidden; padding-left: 0px;">Trubin, I.: Capturing workload pathology by statistical exception detection. In: Proceedings of Computer Measurement Group (2005)<span class="Occurrences" style="box-sizing: border-box; display: block;"><span class="Occurrence OccurrenceGS" style="box-sizing: border-box; display: inline-block; margin-right: 16px;"><a class="google-scholar-link gtm-reference" data-reference-type="Google Scholar" href="https://scholar.google.com/scholar?q=Trubin%2C%20I.%3A%20Capturing%20workload%20pathology%20by%20statistical%20exception%20detection.%20In%3A%20Proceedings%20of%20Computer%20Measurement%20Group%20%282005%29" rel="noopener" style="background-color: initial; box-sizing: border-box; color: #004aa7;" target="_blank"><span style="box-sizing: border-box;">Google Scholar</span></a></span></span></div></li><li class="Citation" style="box-sizing: border-box; margin: 16px 0px; padding: 0px; position: relative;"><div class="CitationNumber" style="box-sizing: border-box; float: left; font-family: "Source Sans Pro", Helvetica, Arial, sans-serif; margin-right: 4px; min-width: 2em; text-align: right;">7.</div><div class="CitationContent" id="CR7" style="box-sizing: border-box; margin-left: 0px; overflow: hidden; padding-left: 0px;">Loboz, C.: Quantifying imbalance in computer systems. In: Proceedings of Computer Measurement Group (2011)<span class="Occurrences" style="box-sizing: border-box; display: block;"><span class="Occurrence OccurrenceGS" style="box-sizing: border-box; display: inline-block; margin-right: 16px;"><a class="google-scholar-link gtm-reference" data-reference-type="Google Scholar" href="https://scholar.google.com/scholar?q=Loboz%2C%20C.%3A%C2%A0Quantifying%20imbalance%20in%20computer%20systems.%20In%3A%20Proceedings%20of%20Computer%20Measurement%20Group%20%282011%29" rel="noopener" style="background-color: initial; box-sizing: border-box; color: #004aa7;" target="_blank"><span style="box-sizing: border-box;">Google Scholar</span></a></span></span></div></li></ol></div></section><section class="Section1 RenderAsSection1" style="background-color: #fcfcfc; box-sizing: border-box; color: #333333; font-family: Georgia, serif; font-size: 17px; letter-spacing: 0.102px; margin-top: 1em; padding-top: 24px;"></section>Igor Trubinhttp://www.blogger.com/profile/17758940374397545163noreply@blogger.com1tag:blogger.com,1999:blog-5048643098278177070.post-5938569396510751562021-12-02T13:45:00.005-05:002021-12-02T15:56:20.515-05:00Dynamics of Anomalies or Phases in a Dynamic Object Life<p>A dynamic object may have following several phases in its lifetime:</p><p>1. Initial phase to set a norm - anomalies cannot be detected as there is no baseline sample is established yet. Could be tired later as an outlier.</p><p>2. Stable period without any anomalies.</p><p>3. Unstable period when anomalies are appearing: suddenly or with gradually increasing rate.</p><p>4. Anomalies are introducing a new norm and the rate of anomalies is gradually decreasing.</p><p>5. =>2. The next stable period. </p><p>6. =>3. … and so on.</p><p>To detect those dynamic object phases one can use Anomaly and Change Point detection methods. One of them is SETDS (described in this blog), which has been implementing now as a <a href="https://www.blogger.com/blog/post/edit/5048643098278177070/593856939651075156#">www.Perfomalist.com</a> tool. </p><p>Here is an example how the <b><i>Perfomalist </i></b>(<a href="https://www.perfomalist.com/sample-upload.csv" style="font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen, Ubuntu, Cantarell, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif;" target="_blank">Download Input Data Sample</a>) test data is used to detect stable and unstable periods.</p><p>Data consists of 28 weeks. To see some dynamic and to catch when anomalies started appearing, the data was divided into 23 data sets. </p><p>- The 1st one has 4 initial weeks (initial baseline or reference/learning set) plus following week (1st "current" week). </p><p>- The 2nd one has 5 initial weeks as the next (on one week bigger) baseline and following week as the next "current" week. </p><p>- The 3rd one... the same mechanism as described above.</p><p>Then the <a href="https://www.blogger.com/blog/post/edit/5048643098278177070/593856939651075156#">www.Perfomalist.com</a> was applied 23 times (could be automated using Personalist APIs) and results were combined into the spreadsheet. </p><p>The table and daily summarized charts are below. The result shows clearly 2nd (stable) and 3rd (unstable) phases. </p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6DBO5nXKiZXDtkdAAOtT1u9ejViik7tRopQkV2BaU304H247s0bs6wc1R7-c4FdK3QewHHWTwcXFKa-3dCIO965MGZbxaYJ9faNBM9GJIEElKAd24bWdYlr7GyYKcha5OjnkVVBJkiqw/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="394" data-original-width="1021" height="123" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6DBO5nXKiZXDtkdAAOtT1u9ejViik7tRopQkV2BaU304H247s0bs6wc1R7-c4FdK3QewHHWTwcXFKa-3dCIO965MGZbxaYJ9faNBM9GJIEElKAd24bWdYlr7GyYKcha5OjnkVVBJkiqw/" width="320" /></a></div><p></p><p></p><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVtHfHz0hKUUUv0PaJMtSWXk9i-PeZFQA3epmwh7tw4HYljRWNI-S6isgpqrjGcbMQDKyO8-Ue9XhlccchonE9QVBUeRrFOd6piqP3gPQgGI0vmrNE2lavrHGhDnmBXm8F5KidAtOvXUc/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="752" data-original-width="533" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVtHfHz0hKUUUv0PaJMtSWXk9i-PeZFQA3epmwh7tw4HYljRWNI-S6isgpqrjGcbMQDKyO8-Ue9XhlccchonE9QVBUeRrFOd6piqP3gPQgGI0vmrNE2lavrHGhDnmBXm8F5KidAtOvXUc/w284-h400/image.png" width="284" /></a></div><br /><br /></div><div class="separator" style="clear: both; text-align: left;"><div class="separator" style="clear: both; text-align: center;"><br /></div>Another way to detect is CPD and to do that another <a href="https://www.trutechdev.com/2021/11/the-change-points-detection-perfomalapi.html">Perfomalist API</a> can be used.</div><div class="separator" style="clear: both; text-align: left;">Applying that to the same data the similar result is seen:</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPdPYrmYNyHQaNZ4_sdO_WPPhYjna4lljY_ckNwiqrIr9nyVE6QeJXuJEJrIBPko4Z9cFkzyTLmwuHo1yCu0mEkuSLzuXQDd1sU9c0nHwha7BxjzeOQRcrEoDufBIbsXhu1WKF-u5IvHee/s640/image.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="553" data-original-width="640" height="277" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPdPYrmYNyHQaNZ4_sdO_WPPhYjna4lljY_ckNwiqrIr9nyVE6QeJXuJEJrIBPko4Z9cFkzyTLmwuHo1yCu0mEkuSLzuXQDd1sU9c0nHwha7BxjzeOQRcrEoDufBIbsXhu1WKF-u5IvHee/s320/image.png" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: left;">The results are similar but a bit different and I know why.... </div><br /><div class="separator" style="clear: both; text-align: left;"><br /><br /></div><br /><br /></div><br /><br /><p></p>Igor Trubinhttp://www.blogger.com/profile/17758940374397545163noreply@blogger.com0tag:blogger.com,1999:blog-5048643098278177070.post-85883151638474128902021-11-23T12:16:00.003-05:002022-01-10T13:02:10.788-05:00Join me with CMG – your technology community – at #CMGIMPACT22. Use code Trubin at cmgimpact.com/ for 50% off IMPACT tickets cmgimpact.com/register/ #cmgnews #technology #InformationTechnology #ITconference #ContinuingEducation #ProfessionalDevelopment<div class="separator" style="clear: both; text-align: center;">PLEASE REGISTER: <a href="https://cmgimpact.com/register/">https://cmgimpact.com/register/</a></div><div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjdhjK4fHwx_NE4q54-EjLkP2iV6eoxim7BL0B93epKdPI8SEh1ZDS4kl54B0vaLkAdH_B0wN_OiRRtImUhbWQikoeFs2S3dVs2LZcf6SwcM0qpzRXraub8rgcCh3SrcjCh6BGrnVosQs8/s1600/1637687762399823-0.png" style="margin-left: 1em; margin-right: 1em;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjdhjK4fHwx_NE4q54-EjLkP2iV6eoxim7BL0B93epKdPI8SEh1ZDS4kl54B0vaLkAdH_B0wN_OiRRtImUhbWQikoeFs2S3dVs2LZcf6SwcM0qpzRXraub8rgcCh3SrcjCh6BGrnVosQs8/s1600/1637687762399823-0.png" width="400" />
</a>
</div><div class="separator" style="clear: both; text-align: center;"><a href="https://cmgimpact.com/cloud-servers-rightsizing-with-seasonality-adjustments/">https://cmgimpact.com/cloud-servers-rightsizing-with-seasonality-adjustments/ </a></div><div class="separator" style="clear: both; text-align: center;"><p style="background: rgb(255, 255, 255); border: 0px; box-sizing: border-box; color: #00171f; font-family: "Open Sans", sans-serif; font-size: 16px; margin: 0px; outline: 0px; padding: 0px 0px 1em; text-align: left; text-size-adjust: 100%; vertical-align: baseline;">When the cloud servers rightsizing algorithm calculates the baseline level for the current year application server’s usage, the seasonal adjustment needs to be calculated and applied by adding the additional anticipated change, which could be increasing or decreasing the capacity usage. We describe the method and illustrate it against the real data.</p><p style="background: rgb(255, 255, 255); border: 0px; box-sizing: border-box; color: #00171f; font-family: "Open Sans", sans-serif; font-size: 16px; margin: 0px; outline: 0px; padding: 0px 0px 1em; text-align: left; text-size-adjust: 100%; vertical-align: baseline;">The cloud servers rightsizing recommendation generated based on seasonality adjustments, would reflect the seasonal patterns, and prevent any potential capacity issues or reduce an excess capacity.<br style="box-sizing: border-box;" />The ability to keep multi-year historical data of 4 main subsystems of application servers’ capacity usage opens the opportunity to detect seasonality changes and estimate additional capacity needs for CPU, memory, disk I/Os, and network. A multi-subsystem approach is necessary, as very often the nature of the application could be not CPU but I/Os or Memory or Network-intensive.</p><p style="background: rgb(255, 255, 255); border: 0px; box-sizing: border-box; color: #00171f; font-family: "Open Sans", sans-serif; font-size: 16px; margin: 0px; outline: 0px; padding: 0px 0px 1em; text-align: left; text-size-adjust: 100%; vertical-align: baseline;">Applying the method daily allows downsizing correctly if the peak season passes and the available capacity should be decreased, which is a good way to achieve cost savings.</p><p style="background: rgb(255, 255, 255); border: 0px; box-sizing: border-box; color: #00171f; font-family: "Open Sans", sans-serif; font-size: 16px; margin: 0px; outline: 0px; padding: 0px 0px 1em; text-align: left; text-size-adjust: 100%; vertical-align: baseline;">In the session, the detailed seasonality adjustment method is described and illustrated against the real data. The method is based on and developed by the author’s SETDS methodology, which treats the seasonal variation as an exception (anomaly) and calculates adjustments as variations from a linear trend.</p><p style="background: rgb(255, 255, 255); border: 0px; box-sizing: border-box; color: #00171f; font-family: "Open Sans", sans-serif; font-size: 16px; margin: 0px; outline: 0px; padding: 0px; text-align: left; text-size-adjust: 100%; vertical-align: baseline;">Key Takeaways</p><ul style="background: rgb(255, 255, 255); border: 0px; box-sizing: border-box; color: #00171f; font-family: "Kulim Park", Helvetica, Arial, Lucida, sans-serif; font-size: 16px; line-height: 26px; list-style-image: initial; list-style-position: initial; margin: 0px; outline: 0px; padding: 0px 0px 23px 1em; text-align: left; text-size-adjust: 100%; vertical-align: baseline;"><li style="background: transparent; border: 0px; box-sizing: border-box; margin: 0px; outline: 0px; padding: 0px; text-size-adjust: 100%; vertical-align: baseline;">How to build seasonal adjustments into the cloud rightsizing</li><li style="background: transparent; border: 0px; box-sizing: border-box; margin: 0px; outline: 0px; padding: 0px; text-size-adjust: 100%; vertical-align: baseline;">To get familiar with cloud objects rightsizing techniques</li></ul></div>Igor Trubinhttp://www.blogger.com/profile/17758940374397545163noreply@blogger.com1tag:blogger.com,1999:blog-5048643098278177070.post-14194728454962765692021-11-22T11:29:00.005-05:002021-11-22T11:31:54.426-05:00The Change Point Detection SETDS based method is implemented as a Perfomalist API. Everybody is welcome to test!<p>How to use it explained <a href="https://www.trutechdev.com/2021/11/the-change-points-detection-perfomalapi.html">HERE</a>:</p><p><a href="https://www.trutechdev.com/2021/11/the-change-points-detection-perfomalapi.html">https://www.trutechdev.com/2021/11/the-change-points-detection-perfomalapi.html</a></p><div style="text-align: left;"><span style="font-family: inherit;">Example of the <span style="color: #444444;">step jump event detected by the API (Output got from </span><span style="color: #444444;"><span>the API call via Postman and </span><span>spreadsheet</span><span> was used to chart the result)</span></span><span style="color: #444444;">:</span></span></div><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhaY2_S9w81Cn67dK_BifBNftmBfTfyE0OmnAfmZIN17HQHfUWUHGyeKXh4_-CZN_ZnGV0kGZn9CHBAc2zLcZN9O4h55JmRw0HPje-CDyUce7rXsiOvsg2CWc7_terBr4J4g_hGvzBv4R3m/s512/image.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="481" data-original-width="512" height="602" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhaY2_S9w81Cn67dK_BifBNftmBfTfyE0OmnAfmZIN17HQHfUWUHGyeKXh4_-CZN_ZnGV0kGZn9CHBAc2zLcZN9O4h55JmRw0HPje-CDyUce7rXsiOvsg2CWc7_terBr4J4g_hGvzBv4R3m/w640-h602/image.png" width="640" /></a></div><br /><b style="color: #444444; font-size: 13px;"><br /></b><p></p>Igor Trubinhttp://www.blogger.com/profile/17758940374397545163noreply@blogger.com0tag:blogger.com,1999:blog-5048643098278177070.post-79534482182331633642021-10-30T13:28:00.003-04:002021-10-30T13:28:34.252-04:00My presentation "Cloud Servers Rightsizing with Seasonality Adjustments" has been accepted for CMG IMPACT 2022. #CMGnews<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiptbTPVWF7cuZb4sZcOwiEjtCbKybveB1weFZol86YqPpGAs44LhgyVGLrOwsIZE3iBy6WXGca8LJgjws0QLiS4e26pR7iCof7Kme8BN1r01wgCbYqLxnb6pXzAV4EKBqUuszxuWekEJ4/s1421/Screen+Shot+2021-10-30+at+1.23.19+PM.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="759" data-original-width="1421" height="342" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiptbTPVWF7cuZb4sZcOwiEjtCbKybveB1weFZol86YqPpGAs44LhgyVGLrOwsIZE3iBy6WXGca8LJgjws0QLiS4e26pR7iCof7Kme8BN1r01wgCbYqLxnb6pXzAV4EKBqUuszxuWekEJ4/w640-h342/Screen+Shot+2021-10-30+at+1.23.19+PM.png" width="640" /></a><div style="text-align: center;"><a href="http://CMGimpact.com" target="_blank"></a><a href="http://CMGimpact.com" target="_blank">www.CMGimpact.com </a></div></div>Igor Trubinhttp://www.blogger.com/profile/17758940374397545163noreply@blogger.com0tag:blogger.com,1999:blog-5048643098278177070.post-43552820812964707452021-09-21T18:07:00.006-04:002021-10-07T13:23:33.181-04:00Got my 1st #AWScertification<p> <span color="var(--color-text)" face="-apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", "Fira Sans", Ubuntu, Oxygen, "Oxygen Sans", Cantarell, "Droid Sans", "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Lucida Grande", Helvetica, Arial, sans-serif" style="font-size: var(--artdeco-reset-base-font-size-hundred-percent);">View of my verified achievement from Amazon Web Services (AWS) is <a href="https://www.credly.com/badges/c4e84069-0110-4833-80b1-44ddf3574622/linked_in">HERE</a></span></p><div class="separator" style="clear: both; text-align: center;"><span color="var(--color-text)" face="-apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", "Fira Sans", Ubuntu, Oxygen, "Oxygen Sans", Cantarell, "Droid Sans", "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Lucida Grande", Helvetica, Arial, sans-serif" style="font-size: var(--artdeco-reset-base-font-size-hundred-percent);"><a href="https://www.credly.com/badges/c4e84069-0110-4833-80b1-44ddf3574622/linked_in" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="358" data-original-width="323" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWYm7KL2QTN5_1NwZbfzGZckNkQnxyJEjgEfcM52qO7t0ABnC43ZcFQn6AOQN6eV2OdHQXeE2xY5Db1e_LQZstreqjLbkFI8JnEHQ1rE2rCHwY-ImcDZWTrUiP7annnmNhovVcw6iLiB8/" width="217" /></a></span></div><article class="feed-shared-article feed-shared-update-v2__content" style="border: var(--artdeco-reset-base-border-zero); box-sizing: inherit; color: rgba(0, 0, 0, 0.9); display: var(--artdeco-reset-base-display-block); font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", "Fira Sans", Ubuntu, Oxygen, "Oxygen Sans", Cantarell, "Droid Sans", "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Lucida Grande", Helvetica, Arial, sans-serif; font-size: 16px; margin: 8px 0px 0px; overflow: hidden; padding: var(--artdeco-reset-base-padding-zero); vertical-align: var(--artdeco-reset-base-vertical-align-baseline);"><div class="feed-shared-article--with-large-image" style="border: var(--artdeco-reset-base-border-zero); box-sizing: inherit; font-size: var(--artdeco-reset-base-font-size-hundred-percent); margin: var(--artdeco-reset-base-margin-zero); padding: var(--artdeco-reset-base-padding-zero); vertical-align: var(--artdeco-reset-base-vertical-align-baseline);"><div class="feed-shared-article__description-container" style="align-items: flex-start; border: var(--artdeco-reset-base-border-zero); box-sizing: inherit; display: flex; flex-grow: 1; font-size: var(--artdeco-reset-base-font-size-hundred-percent); justify-content: space-between; margin: var(--artdeco-reset-base-margin-zero); padding: 8px 12px; transition: background-color 0.6s ease-in-out 0s, border 0.6s ease-in-out 0s; vertical-align: var(--artdeco-reset-base-vertical-align-baseline);"><div class="flex-grow-1" style="border: var(--artdeco-reset-base-border-zero); box-sizing: inherit; flex-grow: 1; font-size: var(--artdeco-reset-base-font-size-hundred-percent); margin: var(--artdeco-reset-base-margin-zero); padding: var(--artdeco-reset-base-padding-zero); vertical-align: var(--artdeco-reset-base-vertical-align-baseline);"><div class="display-flex" style="border: var(--artdeco-reset-base-border-zero); box-sizing: inherit; display: flex; font-size: var(--artdeco-reset-base-font-size-hundred-percent); margin: var(--artdeco-reset-base-margin-zero); padding: var(--artdeco-reset-base-padding-zero); vertical-align: var(--artdeco-reset-base-vertical-align-baseline);"><div class="display-flex flex-column flex-grow-1" style="border: var(--artdeco-reset-base-border-zero); box-sizing: inherit; display: flex; flex-direction: column; flex-grow: 1; font-size: var(--artdeco-reset-base-font-size-hundred-percent); margin: var(--artdeco-reset-base-margin-zero); padding: var(--artdeco-reset-base-padding-zero); vertical-align: var(--artdeco-reset-base-vertical-align-baseline);"><div style="border: var(--artdeco-reset-base-border-zero); box-sizing: inherit; font-size: var(--artdeco-reset-base-font-size-hundred-percent); margin: var(--artdeco-reset-base-margin-zero); padding: var(--artdeco-reset-base-padding-zero); vertical-align: var(--artdeco-reset-base-vertical-align-baseline);"><h2 class="feed-shared-article__title break-words t-14 t-bold t-black" style="--artdeco-reset-typography_getfontsize: 1.4rem; --artdeco-reset-typography_getlineheight: 1.42857; -webkit-box-orient: vertical; -webkit-line-clamp: 2; background: var(--artdeco-reset-base-background-transparent); border: var(--artdeco-reset-base-border-zero); box-sizing: inherit; color: var(--color-text); display: inline; font-size: var(--artdeco-reset-typography_getFontSize); font-weight: var(--artdeco-reset-typography-font-weight-bold); line-height: 2rem; margin: var(--artdeco-reset-base-margin-zero); max-height: 4rem; outline: var(--artdeco-reset-base-outline-zero); overflow-wrap: break-word; overflow: hidden; padding: var(--artdeco-reset-base-padding-zero); text-overflow: ellipsis; vertical-align: var(--artdeco-reset-base-vertical-align-baseline); word-break: break-word;"><span dir="ltr" style="background: var(--artdeco-reset-base-background-transparent); border: var(--artdeco-reset-base-border-zero); box-sizing: inherit; font-size: var(--artdeco-reset-base-font-size-hundred-percent); line-height: inherit; margin: var(--artdeco-reset-base-margin-zero); outline: var(--artdeco-reset-base-outline-zero); padding: var(--artdeco-reset-base-padding-zero); vertical-align: var(--artdeco-reset-base-vertical-align-baseline);">AWS Certified Cloud Practitioner was issued by Amazon Web Services Training and Certification to Igor Trubin</span></h2></div></div></div></div></div></div></article>Igor Trubinhttp://www.blogger.com/profile/17758940374397545163noreply@blogger.com0tag:blogger.com,1999:blog-5048643098278177070.post-10965108720288855952021-07-30T14:23:00.006-04:002021-08-03T11:01:09.326-04:00"Performance #Anomaly and #ChangePointDetection For Large-Scale System Management" for WorldS4 2021 - my presentation slides deck is available on RG <p> I have successfully made my presentation at<a href="https://conferences.ieee.org/conferences_events/conferences/conferencedetails/51998"> WorldS4 conference</a>. Presentation deck is available <a href="https://www.researchgate.net/publication/353584064_Performance_Anomaly_and_Change_Point_Detection_For_Large-Scale_System_Management_for_WorldS4_2021">HERE</a></p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhuIYL2fZx71QxiHNlQO9myeGFJaf-zhsRf0pjJJkRz3sElcIB-xAAE95biJYvXnBlKWlMcjVcQx8m7h7blt3E7cEe0894mD8rMExDd7tnDV_xdJqazFaAmUhc7hVTMpl7Tp045MNTnmMo/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="760" data-original-width="1016" height="299" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhuIYL2fZx71QxiHNlQO9myeGFJaf-zhsRf0pjJJkRz3sElcIB-xAAE95biJYvXnBlKWlMcjVcQx8m7h7blt3E7cEe0894mD8rMExDd7tnDV_xdJqazFaAmUhc7hVTMpl7Tp045MNTnmMo/w400-h299/image.png" width="400" /></a></div><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhIKDuSQO_i9qFzmOjpGqIDUgUJ_JQDblu01Rlsp6tFHFvq4nYXUKFfeRiG_gjOux_Wk86dDnmdlnbUnglITmfNrK-H6-vKPXDfBspQfLm2zamY1U6EG5ZDmLtJew2VxJfyKwc-eNEUaM4/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="753" data-original-width="993" height="486" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhIKDuSQO_i9qFzmOjpGqIDUgUJ_JQDblu01Rlsp6tFHFvq4nYXUKFfeRiG_gjOux_Wk86dDnmdlnbUnglITmfNrK-H6-vKPXDfBspQfLm2zamY1U6EG5ZDmLtJew2VxJfyKwc-eNEUaM4/w640-h486/image.png" width="640" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEizOMBZrd1EeAa-toFEbZwxcGV8jWe4-3ZywhUUT7aloAqjKkdmJNlz3dFYD6SXY7IpVknyToytmn76nkk-HQ1Rdu3CuDL0IvLhxDQsvmBXOdAZioo8P8AARTORFxZJL4LoqV0CeghdBIg/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="1447" data-original-width="2048" height="226" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEizOMBZrd1EeAa-toFEbZwxcGV8jWe4-3ZywhUUT7aloAqjKkdmJNlz3dFYD6SXY7IpVknyToytmn76nkk-HQ1Rdu3CuDL0IvLhxDQsvmBXOdAZioo8P8AARTORFxZJL4LoqV0CeghdBIg/" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhes6mcbw8TCzcMeaf0yDv-2ywnrrID5H97V6Wug-vyS-_fBLdKZn6iRlIitnSdAmGn-0rKcB5OLoebkBkrAx6dCc7C59Y3NgPPzyUC1poeYjHC5cfJ4ZRa8b9m6kEDigSj2cAJfBMhyt4/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="797" data-original-width="1124" height="227" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhes6mcbw8TCzcMeaf0yDv-2ywnrrID5H97V6Wug-vyS-_fBLdKZn6iRlIitnSdAmGn-0rKcB5OLoebkBkrAx6dCc7C59Y3NgPPzyUC1poeYjHC5cfJ4ZRa8b9m6kEDigSj2cAJfBMhyt4/" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjeJ-9Y-NWbEjDYqjeJuOViw5ZcAvNSIAXcRnPKJi0oO0rgjOgCo7X9ba58ouPzk0UL_71USwJ4LAGZwISW5rl9lWFkzJaPvQLQFTYa8wCIGLaMvVhvJ7Bn2C_z9VaR0ZiF8gAntQHzvlE/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="624" data-original-width="879" height="227" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjeJ-9Y-NWbEjDYqjeJuOViw5ZcAvNSIAXcRnPKJi0oO0rgjOgCo7X9ba58ouPzk0UL_71USwJ4LAGZwISW5rl9lWFkzJaPvQLQFTYa8wCIGLaMvVhvJ7Bn2C_z9VaR0ZiF8gAntQHzvlE/" width="320" /></a></div><br /><br /></div><br /><br /></div><br /><br /></div><br /><br /></div><br /><br /><p></p>Igor Trubinhttp://www.blogger.com/profile/17758940374397545163noreply@blogger.com0tag:blogger.com,1999:blog-5048643098278177070.post-70420725592209415412021-07-23T11:35:00.005-04:002021-07-23T11:36:24.292-04:00I'm excited to present my paper "Performance #Anomaly and Change Point Detection for Large-Scale System Management" at 5th World Conference on Smart Trends in Systems, Security and Sustainability <p>See that in the agenda: <a href="https://sched.co/lEkJ">https://sched.co/lEkJ</a></p><p> </p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh7c7UIwZCQcFjTv9hojRIsOKHo0fuCkfdyAL5Lld1ZtuaLTf1ZBQ0y84GllH7A7miMiKD2UK-ZSGEvW5e1tpWmtb_Zd7tAwT1rn-rboBiAN70QPtIVN2Su8guWE4ub8X8F6BhsW-LwUF0/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="873" data-original-width="1170" height="478" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh7c7UIwZCQcFjTv9hojRIsOKHo0fuCkfdyAL5Lld1ZtuaLTf1ZBQ0y84GllH7A7miMiKD2UK-ZSGEvW5e1tpWmtb_Zd7tAwT1rn-rboBiAN70QPtIVN2Su8guWE4ub8X8F6BhsW-LwUF0/w640-h478/image.png" width="640" /></a></div><br /><p></p>Igor Trubinhttp://www.blogger.com/profile/17758940374397545163noreply@blogger.com0