lisa.regression.RegressionResult#

class lisa.regression.RegressionResult(testcase_id, old_count, new_count, alpha=None)[source]#

Bases: object

Compute failure-rate regression between old and new series.

The regression is checked using Fisher’s exact test.

Parameters:

  • testcase_id (str) – ID of the testcase, used for pretty-printing

  • old_count (ResultCount) – number of times the test passed and failed in the old series

  • new_count (ResultCount) – number of times the test passed and failed in the new series

  • alpha (float) – Alpha risk when carrying the statistical test

Properties

failure_delta_pc

Delta between old and new failure rate in percent.

failure_pc

Tuple of failure rate in percent for old an new series.

fix_validation_min_iter_nr

Number of iterations required to validate a fix that would “revert” a regression.

p_val

P-value of the statistical test.

sample_size

Tuple of sample sizes for old and new series.

significant

True if there is a significant difference in failure rate, False otherwise.

Methods

from_result_list()

Build a RegressionResult from two list of lisa.tests.base.Result, or objects that can be converted to bool.

get_p_val()

Compute the p-value of the statistical test, with the given alternative hypothesis.

Properties#

property RegressionResult.failure_delta_pc[source]#

Delta between old and new failure rate in percent.

property RegressionResult.failure_pc[source]#

Tuple of failure rate in percent for old an new series.

property RegressionResult.fix_validation_min_iter_nr[source]#

Number of iterations required to validate a fix that would “revert” a regression.

Assuming that the “fixed” failure rate is exactly equal to the “old” one, this gives the number of iterations after which comparing the “fixed” failure rate with the “new” failure rate will give a statistically significant result.

property RegressionResult.p_val[source]#

P-value of the statistical test.

property RegressionResult.sample_size[source]#

Tuple of sample sizes for old and new series.

property RegressionResult.significant[source]#

True if there is a significant difference in failure rate, False otherwise.

Methods#

classmethod RegressionResult.from_result_list(testcase_id, old_list, new_list, alpha=None)[source]#

Build a RegressionResult from two list of lisa.tests.base.Result, or objects that can be converted to bool.

Note

Only FAILED and PASSED results are taken into account, other results are ignored.

Parameters:
RegressionResult.get_p_val(alternative='two-sided')[source]#

Compute the p-value of the statistical test, with the given alternative hypothesis.