Friday, 21 February 2014

98th% values and Max values are same in graph data of Trans resp percentile grap

Problem Details:- We are getting same values in columns for 98th Percentile and 100% in graph data of Trans resp percentile graph. We ran the test with almost 20 scripts. For some of transactions values are differing, but most of the transactions values are same for 98th percentile and Maximum.
Request you to help me on this.
Kindly let me know if any other details are required.
Please keep the below people in the loop for any response that you provide.
josue.gutierrez1@ge.com
squall.yang@ge.com

Problem Solution:-  Regarding the description of your issue,
That is actually an expected behavior, because there may be many iterations of the same transaction that ended up with the same timing.
A percentile is the value of a variable below which a certain percent of observations fall. For example, the 20th percentile is the value below which 20 percent of the observations may be found. In this case, you want to look at the 98th percentile values of your transactions which is a value pretty close to 100th%. Also, from the 98th% values that are different to the maximum value, you should see a small difference between them.

Is it a totally different behavior the one that you are facing? If so, please attach screenshots from Analysis explaining this issue.

Let me know how we may proceed with our ticket.

More Information:-

Actually the results from the load test is not something that we can control.
In order to make sure that you we are analyzing the data correctly, it is always recommended to use the latest version + patch of Analysis.
In this moment, the latest version is Analysis 11 with Patch 2 Critical Update as the latest applied patch.

We do not have notice about any issues or bugs related to the statistics handling of LoadRunner Analysis.

Also, I see that, in the example you sent, you executed each transaction for 9 iterations. It will take the Maximum value because if it takes the one that is under it, it will be the 88th percentile. You can set Analysis to show you the 88th percentile, or lower and you will start getting different values than the Maximum value.
If you execute enough iterations to take the 98th percentile calculations, then you will notice different values. This will start happening from 50 iterations and so on.

Please let me know how to proceed with our ticket.

More Information:-

 

if we calculate the 98 percentile using an excel formula we found that it provides us a different values for 98 percentile and maximum values. I’m attaching an example of the calculation and you could see that we have made the calculation for transaction over and below 50 iterations and it looks like for transactions with less than 50 iterations the formula interpolates the values to get the 98th percentile, is the Analysis tool able to emulate that behavior?

Also this is the information about version and patches for LR
Analysis Version: 11.00.0.0
Installed Patches: LR_03014 for HP LoadRunner 11.00 QFE
HP LoadRunner 11.00 Patch 1
HP LoadRunner 11.00 Patch 2

More Information:-

Thanks for your responses.
However, by definition Excel performs this interpolation if k is not a multiple of 1/(n – 1). (taken from http://office.microsoft.com/en-us/excel-help/percentile-HP005209211.aspx)
And LoadRunner Analysis instead of doing this interpolation, uses the definition of Percentile and rounds the result of the position to the nearest integer and then takes the value that corresponds to that rank.
Squall was able to prove that by saying:
“For those transactions which got more than 50 iterations we will have the difference between the max and 98 percentile.”
That is because the 98th percentile will return the position of the 49th value.

I have reviewed this ticket with our Senior Engineers and they have confirmed this behavior.

Please let me know how to proceed with our ticket.

More Information:-

I get your point. However, as previously explained, LoadRunner Analysis is doing right at the time of calculating the Percentile, even for transactions with less than 50 iterations.
Statistically, to calculate the 98th percentile value this needs to be done:
1. Sort the transaction instances by their value.
2. Remove the top 2% instances.
3. The highest value left is the 98th percentile.

In case that you execute less than 50 iterations, if you try to remove the top 2% instances of the transaction times, you will not be able to remove any instances and will still get the maximum value.

The following document describes how the percentile is calculated:
http://support.openview.hp.com/selfsolve/document/KM169562

More Information:-

If it is ok for you, I can create an Enhancement Request to add the interpolation as an option at the time of calculating the Percentile values. However, this would be a feature that our R&D department will consider to implement in future versions of LoadRunner

 

1 comment: