lisa.tests.base.TestBundleBase#

class lisa.tests.base.TestBundleBase(res_dir, plat_info)[source]#

Bases: Serializable, ExekallTaggable, ABC

A LISA test bundle.

Parameters:

res_dir (str) – Directory in which the target execution artifacts reside. This will also be used to dump any artifact generated in the test code.
plat_info (lisa.platforms.platinfo.PlatformInfo) – Various informations about the platform, that is available to all tests.

The point of a TestBundleBase is to bundle in a single object all of the required data to run some test assertion (hence the name). When inheriting from this class, you can define test methods that use this data, and return a ResultBundle.

Thanks to Serializable, instances of this class can be serialized with minimal effort. As long as some information is stored within an object’s member, it will be automagically handled.

Please refrain from monkey-patching the object in from_target(). Data required by the object to run test assertions should be exposed as __init__ parameters.

Note

All subclasses are considered as “application” code, as opposed to most of the rest of lisa which is treated as a library. This means that the classes and their API is subject to change when needs evolve, which is not always backward compatible. It’s rarely an issue since these classes are used “manually” mostly for debugging, which is a version-specific activity. Likewise, the set of tests will evolve as existing tests are replaced by more general implementations, that could be organized and named differently.

Design notes:

from_target() will collect whatever artifacts are required from a given target, and will then return a TestBundleBase. Note that a default implementation is provided out of _from_target.

from_dir() will use whatever artifacts are available in a given directory (which should have been created by an earlier call to from_target() and then to_dir()), and will then return a TestBundleBase.

VERIFY_SERIALIZATION is there to ensure the instances can serialized and deserialized without error.

res_dir parameter of __init__ must be stored as an attribute without further processing, in order to support result directory relocation.

Test methods should have a return annotation for the ResultBundle to be picked up by the test runners.

Implementation example:

from lisa.target import Target
from lisa.platforms.platinfo import PlatformInfo
from lisa.utils import ArtifactPath

class DummyTestBundle(TestBundle):

    def __init__(self, res_dir, plat_info, shell_output):
        super().__init__(res_dir, plat_info)

        self.shell_output = shell_output

    @classmethod
    def _from_target(cls, target:Target, *, res_dir:ArtifactPath) -> 'DummyTestBundle':
        output = target.execute('echo $((21+21))').split()
        return cls(res_dir, target.plat_info, output)

    def test_output(self) -> ResultBundle:
        return ResultBundle.from_bool(
            any(
                '42' in line
                for line in self.shell_output
            )
        )

Usage example:

# Creating a Bundle from a live target
bundle = TestBundle.from_target(target, plat_info=plat_info, res_dir="/my/res/dir")
# Running some test on the bundle
res_bundle = bundle.test_foo()

# Saving the bundle on the disk
bundle.to_dir("/my/res/dir")

# Reloading the bundle from the disk
bundle = TestBundle.from_dir("/my/res/dir")
# The reloaded object can be used just like the original one.
# Keep in mind that serializing/deserializing this way will have a
# similar effect than a deepcopy.
res_bundle = bundle.test_foo()

Warning

Arbitrary code can be executed while loading an instance from a YAML or Pickle file. To include untrusted data in YAML, use the !untrusted tag along with a string

Note

As a subclass of lisa.tests.base.TestBundleBase, this class is considered as “application” and its API is therefore more subject to change than other parts of lisa.

Classes

__class__

Metaclass of TestBundleBase.

Attributes

`VERIFY_SERIALIZATION`	When True, this enforces a serialization/deserialization step in `from_target()`.
`ATTRIBUTES_SERIALIZATION` ^inherited	Attributes to be treated specially during serialization.
`DEFAULT_SERIALIZATION_FMT` ^inherited	Default format used when serializing objects.
`YAML_ENCODING` ^inherited	Encoding used for YAML files.

Properties

`_children_test_bundles`
`logger` ^inherited	Convenience short-hand for `self.get_logger()`.

Methods

`_from_target()`
`_get_filepath()`
`add_undecided_filter()`	Turn any method returning a `ResultBundleBase` into a decorator that can be used as a test method filter.
`can_create_from_target()`	returns: Whether the given target can be used to create an instance of this class :rtype: bool.
`check_from_target()`	Check whether the given target can be used to create an instance of this class.
`from_dir()`	Wrapper around `lisa.utils.Serializable.from_path()`.
`from_target()`	Factory method to create a bundle using a live target.
`get_tags()`	Dictionary of tags and tag values.
`to_dir()`	See `lisa.utils.Serializable.to_path()`.
`__copy__()` ^inherited	Regular shallow copy operation, without dropping any attributes.
`__getstate__()` ^inherited	Filter the instance’s attributes upon serialization.
`__setstate__()` ^inherited
`from_path()` ^inherited	Deserialize an object from a file.
`get_logger()` ^inherited	Provides a `logging.Logger` named after `cls`.
`log_locals()` ^inherited	Debugging aid: log the local variables of the calling function.
`to_path()` ^inherited	Serialize the object to a file.
`to_yaml()` ^inherited	Return a YAML string with the serialized object.

Classes#

TestBundleBase.__class__#: alias of TestBundleMeta

Attributes#

TestBundleBase.VERIFY_SERIALIZATION = True#: When True, this enforces a serialization/deserialization step in from_target().

Note

The deserialized instance is thrown away in order to avoid using what is in effect a deepcopy of the original bundle. Using that deepcopy greatly increases the memory consumption of long running processes.

TestBundleBase.ATTRIBUTES_SERIALIZATION = {'allowed': [], 'ignored': [], 'placeholders': {}}#

Inherited attribute, see lisa.utils.Serializable.ATTRIBUTES_SERIALIZATION

Attributes to be treated specially during serialization.

TestBundleBase.DEFAULT_SERIALIZATION_FMT = 'yaml'#

Inherited attribute, see lisa.utils.Serializable.DEFAULT_SERIALIZATION_FMT

Default format used when serializing objects.

TestBundleBase.YAML_ENCODING = 'utf-8'#

Inherited attribute, see lisa.utils.Serializable.YAML_ENCODING

Encoding used for YAML files.

Properties#

property TestBundleBase._children_test_bundles[source]#

List of references to TestBundleBase instances self relies on (directly and indirectly).

This is used for some post-deserialization fixup that need to walk the whole graph of TestBundleBase.

property TestBundleBase.logger#

Inherited property, see lisa.utils.Loggable.logger

Convenience short-hand for self.get_logger().

Methods#

abstract classmethod TestBundleBase._from_target(target, *, res_dir)[source]#

Internals of the target factory method.

Note

This must be a classmethod, and all parameters except target must be keyword-only, i.e. appearing after args* or a lonely *:

@classmethod
def _from_target(cls, target, *, foo=33, bar):
    ...

classmethod TestBundleBase._get_filepath(res_dir)[source]#

Returns the path of the file containing the serialized object in res_dir folder.

TestBundleBase.add_undecided_filter()#

Turn any method returning a ResultBundleBase into a decorator that can be used as a test method filter.

The filter decorator is accessible as the undecided_filter attribute of the decorated method.

Once a test is decorated, the filter method will be run in addition to the wrapped test, and if the filter does not succeed, the ResultBundleBase result will be set to Result.UNDECIDED.

Example:

class Foo(TestBundle):
    @TestBundle.add_undecided_filter
    def test_foo(self, xxx=42, ...):
        ...

    # Alternatively, ResultBundle return annotation will
    # automatically decorate the method with TestBundleMeta
    # metaclass.
    def test_foo(self, xxx=42, ...) -> ResultBundle:
        ...

class Bar(Foo):
    # Set xxx=55 as default, but this can be overriden when
    # test_bar() is called.
    @Foo.test_foo.undecided_filter(xxx=77)
    def test_bar(self, yyy=43, ...) -> ResultBundle:
        ...

The resulting decorated method can take the union of keyword parameters:

bar = Bar()
bar.test_bar(xxx=33, yyy=55)
# Same as
bar.test_bar(33, yyy=55)
# But this fails, since only keyword arguments can be passed to the
# wrapping pre-test
bar.test_bar(33, 55)

If there is a parameter conflict, it is detected at import time and will result in a TypeError.

Note

Even if the pre-test does not succeed, the wrapped test is still executed, so that the ResultBundle metrics are updated and the artifacts still produced. This can be important in order to manually analyse results in case the pre-filter was overly conservative and marked a usable result as UNDECIDED.

classmethod TestBundleBase.can_create_from_target(target)[source]#

Returns:: Whether the given target can be used to create an instance of this class
Return type:: bool

check_from_target() is used internally, so there shouldn’t be any: need to override this.

classmethod TestBundleBase.check_from_target(target)[source]#

Check whether the given target can be used to create an instance of this class

Raises:: lisa.tests.base.ResultBundleBase with result as lisa.tests.base.Result.SKIPPED if the check fails

This method should be overriden to check your implementation requirements

classmethod TestBundleBase.from_dir(res_dir, update_res_dir=True)[source]#

Wrapper around lisa.utils.Serializable.from_path().

It uses _get_filepath() to get the name of the serialized file to reload.

classmethod TestBundleBase.from_target(target: Target, *, res_dir: ArtifactPath = None, **kwargs)[source]#

Factory method to create a bundle using a live target

Parameters:

target (lisa.target.Target) – Target to connect to.
res_dir (str or lisa.utils.ArtifactPath) – Host result directory holding artifacts.
custom_collector (lisa.trace.CollectorBase) – Custom collector that will be used as a context manager when calling the workload.

This is mostly boiler-plate code around _from_target(), which lets us introduce common functionalities for daughter classes. Unless you know what you are doing, you should not override this method, but the internal lisa.tests.base.TestBundleBase._from_target() instead.

TestBundleBase.get_tags()[source]#

Dictionary of tags and tag values.

Return type:: dict(str, object)

TestBundleBase.to_dir(res_dir)[source]#: See lisa.utils.Serializable.to_path()

TestBundleBase.__copy__()#

Inherited method, see lisa.utils.Serializable.__copy__()

Regular shallow copy operation, without dropping any attributes.

TestBundleBase.__getstate__()#

Inherited method, see lisa.utils.Serializable.__getstate__()

Filter the instance’s attributes upon serialization.

TestBundleBase.__setstate__(dct)#: Inherited method, see lisa.utils.Serializable.__setstate__()

classmethod TestBundleBase.from_path(filepath, fmt=None)#

Inherited method, see lisa.utils.Serializable.from_path()

Deserialize an object from a file.

classmethod TestBundleBase.get_logger(suffix=None)#

Inherited method, see lisa.utils.Loggable.get_logger()

Provides a logging.Logger named after cls.

classmethod TestBundleBase.log_locals(var_names=None, level='debug')#

Inherited method, see lisa.utils.Loggable.log_locals()

Debugging aid: log the local variables of the calling function.

TestBundleBase.to_path(filepath, fmt=None)#

Inherited method, see lisa.utils.Serializable.to_path()

Serialize the object to a file.

TestBundleBase.to_yaml()#

Inherited method, see lisa.utils.Serializable.to_yaml()

Return a YAML string with the serialized object.

lisa.tests.base.TestBundleBase#

Classes#

Attributes#

Properties#

Methods#

This Page