Probability Density Distribution Graph for CPU usage analysis

The normal distribution is useful to understand the probability density of a large sample data set. There are many real world application of this theory. The example in this article shows how this can be used for analyzing the CPU usage of a particular process or individual core or overall CPU usage over a period of time. There may be occasional spikes up to 100%, but that may not necessarily an indicator of an issue for the overall system as long as the normal operating mode is utilizing less than the threshold.

A graphical representation of probability density distribution combined with cumulative distribution will help to visualize the overall usage.

Probability density function is represented as 

The above simplifies to: 

 - Standard deviation

 - Mean

This can be implemented in a programming language, I've used javascript and DyGraph chart library to create visualization of the data.

Sample data set:

var testData = [2,0,95.9,37.9,55.9,71.9,69.9,71.9,93.8,8,6,4,10,8,6,56,22,23.9,26,2,2,4,4,2,2,2,2,2,2,4,2,2,4,4,4,4,4,4,4,2,4,2,6,4,4,4,4,4,4,4,4,0,0,2,4,6,8,8,12,6,4,4,6,6,4,4,4,4,6,6,6,4,2,4,4,51.9,2]

mean:

function getMean(data) {
   var sum = 0;
   var count = data.length;  
   for (var i = 0; i < count; i++) {
      sum += parseFloat(data[i]);
   }
   return sum/count;   
}

Standard deviation

function getSDev(data) {
   var sum = 0;
   var count = data.length;
   for (var i = 0; i < count; i++) {
      sum += parseFloat(Math.pow((parseFloat(data[i]) - mean), 2));
   }  
   return Math.sqrt(sum/count);
}

Probability density distribution

function normalDistributedData(sampleData) {
   var sortedData = sampleData.sort(sortNum);
   var count = sortedData.length;
   mean = getMean(sortedData);
   sd = getSDev(sortedData);
  
   var resultSet = [];
  
   var row, x, distFromMean, zScore, px, cdfx;
   for (var i = 0; i < count; i++) {
      row = [];
      x = sortedData[i];
      distFromMean = x - mean;
      zScore = distFromMean/sd; 
      cdfx = cdfF(x, mean, sd) * 100;
      px = pF(x, sd, zScore);
   
      row.push(x);
      row.push(px);
      row.push(cdfx);

      resultSet.push(row);
   }
  
   return resultSet;
}
    

DyGraphs

Dygraph provides powerful and easy to use javascript chart libraries: http://dygraphs.com/

Graph to plot the above data in normal distribution form:

     var ndata = normalDistributedData(testData);
     g = new Dygraph(
          document.getElementById("normDistriDiv"),
          ndata,
          {
            labels: [ 'Date', 'p(x)', 'CDF(%)' ],
            series: {
              'CDF(%)': {
                axis: 'y2'
              },
              'p(x)': {
                 fillGraph: true
              }
            },
            axes: {
              y: {
                axisLabelWidth: 60
              },
              y2: {
                // set axis-related properties here
              }
            },
            ylabel: '',
            y2label: '',
            animatedZooms:true,
            //Highlight mean+/-SD
            underlayCallback: highlightSDArea
          }
      );

The resulting graph on the UI looks like the below:

References:

• https://en.wikipedia.org/wiki/Normal_distribution
• https://www.khanacademy.org