Typical LISA experiment#

This notebook shows a typical LISA-use case:

Connecting to a target
Configuring an rt-app workload
Collecting a trace while executing a workload
Displaying the trace
Analysing the trace

It can serve as a template for different kind of experiments, - you could only change the workload to execute & the trace events to collect

[1]:

import logging
from lisa.utils import setup_logging
setup_logging()

2025-02-05 13:09:45,223 INFO    : root         : Using LISA logging configuration: /home/dourai01/Work/projects/lisa/logging.conf

Target configuration#

Target communication is abstracted away under a Target class. We’re going to create an instance of it and that’ll let us run whatever experiment we want on a given target.

Relevant documentation:

Target: https://tooling.sites.arm.com/lisa/latest/target.html#lisa.target.Target
TargetConf: https://tooling.sites.arm.com/lisa/latest/sections/api/generated/lisa.target.TargetConf.html

[2]:

from lisa.target import Target, TargetConf

[3]:

# target = Target(
#     kind='linux',
#     name='myhikey960',
#     host='192.168.0.1',
#     username='root',
#     password='root',
# )
# Uses settings from target_conf.yml
target = Target.from_default_conf()

2025-02-05 13:09:52,578 WARNING : lisa.target.Target : No platform information could be found: Key "('platform-info',)" needs to appear at the top level
2025-02-05 13:09:52,582 INFO    : lisa.target.Target : Target configuration:
├ devlib:
    ├ excluded-modules from default (list): []
    ├ max-async from user (int): 5
    └ platform:
        └ class from default (str): devlib.platform.Platform
├ host from user (str): 127.0.0.1
├ kernel:
    ├ modules:
        ├ build-env from user (str): host
        ├ make-variables from user (dict): {'V': 1}
        └ overlay-backend from user (str): overlayfs
    └ src from user (str): /home/dourai01/Work/projects/linux-master
├ kind from user (str): linux
├ lazy-platinfo from user (bool): True
├ name from default (str): <noname>
├ password from user (str): <password>
├ port from user (int): 8022
├ strict-host-check from user (bool): True
├ tools from default (list): []
├ username from user (str): root
└ wait-boot:
    ├ enable from default (bool): True
    └ timeout from default (int): 10
2025-02-05 13:10:11,579 INFO    : lisa.target.Target : Connected to target <noname>
2025-02-05 13:10:11,881 INFO    : lisa.target.Target : Effective platform information:
├ abi from target (FilteredDeferredValue): <lazy value of abi>
├ cpu-capacities:
    ├ orig from target (FilteredDeferredValue): <lazy value of orig>
    ├ writeable from target (FilteredDeferredValue): <lazy value of writeable>
    └ rtapp from target(platform-info/rtapp/calib),target(platform-info/cpu-capacities/orig),target(platform-info/cpu-capacities/writeable) (str): <depends on lazy keys: platform-info/rtapp/calib, platform-info/cpu-capacities/orig, platform-info/cpu-capacities/writeable>
├ cpus-count from target (FilteredDeferredValue): <lazy value of cpus-count>
├ freq-domains from target (FilteredDeferredValue): <lazy value of freq-domains>
├ freqs from target (FilteredDeferredValue): <lazy value of freqs>
├ kernel:
    ├ config from target (FilteredDeferredValue): <kernel config>
    ├ symbols-address from target (FilteredDeferredValue): <symbols address>
    └ version from target (FilteredDeferredValue): <lazy value of version>
├ name from target-conf (str): <noname>
├ nrg-model from target (FilteredDeferredValue): <lazy value of nrg-model>
├ numa-nodes-count from target (FilteredDeferredValue): <lazy value of numa-nodes-count>
├ os from target (FilteredDeferredValue): <lazy value of os>
├ rtapp:
    └ calib from target (FilteredDeferredValue): <lazy value of FilteredDeferredValue.__init__.<locals>._callback>
└ capacity-classes from target(platform-info/cpu-capacities/orig) (str): <depends on lazy keys: platform-info/cpu-capacities/orig>
2025-02-05 13:10:13,266 INFO    : lisa._kmod._KernelBuildEnv : Toolchain detected: CC=clang, CROSS_COMPILE=aarch64-linux-gnu-, LLVM=1, ARCH=arm64

Setting up an rt-app workload#

rt-app is very convenient for scheduler experiments, and the majority of the tests within LISA rely on it. Here we’re going to create a somewhat useless workload just to show off the API.

Relevant documentation:

rt-app: scheduler-tools/rt-app
rt-app LISA class: https://tooling.sites.arm.com/lisa/latest/sections/api/generated/lisa.wlgen.rta.RTA.html
RTAPhase class: https://tooling.sites.arm.com/lisa/latest/sections/api/generated/lisa.wlgen.rta.RTAPhase.html
PeriodicWload class: https://tooling.sites.arm.com/lisa/latest/sections/api/generated/lisa.wlgen.rta.PeriodicWload.html

[4]:

from lisa.wlgen.rta import RTA, RTAPhase, PeriodicWload

[5]:

rtapp_profile = {
    f'tsk{cpu}_{i}': RTAPhase(
        prop_wload=PeriodicWload(
            duty_cycle_pct=20,
            period=16e-3,
            duration=1,
        )
    )
    for cpu in range(target.number_of_cpus)
    for i in range(3)
}

rt-app needs some calibration information (20% duty cycle isn’t the same amount of work on all platforms!). It can be manually specified like so:

[6]:

def provide_calibration(calibration):
    target.plat_info["rtapp"].add_src("user", {"calib" : calibration})

[9]:

# Uncomment if you want to use this
# provide_calibration({0: 307, 1: 302, 2: 302, 3: 302, 4: 155, 5: 155, 6: 155, 7: 155})

However, it is automatically collected when first creating an rt-app workload if it is not specified, so you can forego the above step and let the calibration happen on-demand:

[10]:

wload = RTA.from_profile(target, rtapp_profile, name='experiment_workload')

2025-02-05 13:10:44,572 INFO    : lisa.target.Target : Creating result directory: /home/dourai01/Work/projects/lisa/results/Target-<noname>-20250205_131044.570241
2025-02-05 13:10:44,578 INFO    : lisa.target.Target : Creating result directory: /home/dourai01/Work/projects/lisa/results/Target-<noname>-20250205_131044.578229
2025-02-05 13:10:44,582 INFO    : lisa.target.Target : Creating result directory: /home/dourai01/Work/projects/lisa/results/Target-<noname>-20250205_131044.582048
2025-02-05 13:10:44,586 INFO    : lisa.target.Target : Creating result directory: /home/dourai01/Work/projects/lisa/results/Target-<noname>-20250205_131044.582048/RTA-experiment_workload-20250205_131044.586347

Running the workload#

[11]:

import os
from lisa.trace import FtraceCollector

We need to specify the trace events we want to record. We could list what’s available like so:

[12]:

available_events = target.execute("cat /sys/kernel/debug/tracing/available_events").splitlines()

# That's gonna be a pretty big list, let's focus on the scheduler events
sched_events = [
    event
    for event in available_events
    if (
        event.startswith("sched:") or
        event.startswith("task:")
    )
]
print(sched_events)

['task:task_newtask', 'task:task_rename', 'sched:sched_kthread_stop', 'sched:sched_kthread_stop_ret', 'sched:sched_kthread_work_queue_work', 'sched:sched_kthread_work_execute_start', 'sched:sched_kthread_work_execute_end', 'sched:sched_waking', 'sched:sched_wakeup', 'sched:sched_wakeup_new', 'sched:sched_switch', 'sched:sched_migrate_task', 'sched:sched_process_free', 'sched:sched_process_exit', 'sched:sched_wait_task', 'sched:sched_process_wait', 'sched:sched_process_fork', 'sched:sched_process_exec', 'sched:sched_prepare_exec', 'sched:sched_stat_wait', 'sched:sched_stat_sleep', 'sched:sched_stat_iowait', 'sched:sched_stat_blocked', 'sched:sched_stat_runtime', 'sched:sched_pi_setprio', 'sched:sched_move_numa', 'sched:sched_stick_numa', 'sched:sched_swap_numa', 'sched:sched_skip_vma_numa', 'sched:sched_wake_idle_without_ipi']

Let’s just collect the base events required to plot task scheduling:

[13]:

events = [
    'sched_switch',
    'sched_wakeup',
    'sched_wakeup_new',
    'task_rename',
]

And now we can actually record traces while running our workload:

[14]:

trace_path = os.path.join(wload.res_dir, "trace.dat")
ftrace_coll = FtraceCollector(target, events=events, buffer_size=10240, output_path=trace_path)

with wload, ftrace_coll:
    wload.run()

2025-02-05 13:11:01,183 INFO    : lisa._kmod._KernelBuildEnv : Toolchain detected: CC=clang, CROSS_COMPILE=aarch64-linux-gnu-, LLVM=1, ARCH=arm64
2025-02-05 13:11:01,375 INFO    : lisa._kmod._KernelBuildEnv : Toolchain detected: CC=clang, CROSS_COMPILE=aarch64-linux-gnu-, LLVM=1, ARCH=arm64
2025-02-05 13:11:15,765 INFO    : sched        : Scheduler sched_domain procfs entries found
2025-02-05 13:11:15,766 INFO    : sched        : Detected kernel compiled with SCHED_DEBUG=y
2025-02-05 13:11:15,767 INFO    : sched        : CPU capacity sysfs entries found
2025-02-05 13:11:20,433 INFO    : lisa.wlgen.rta.RTA : Created workload's run target directory: /root/devlib-target/lisa/wlgen/20250205_131044_3b90c73fb83a4959bb7f2bb0d65dff75
2025-02-05 13:11:22,861 WARNING : lisa.wlgen.rta.RTA : CPU capacities will not be updated on this platform
2025-02-05 13:11:22,864 INFO    : lisa.wlgen.rta.RTA : CPU capacities according to rt-app workload: {0: 1024, 1: 1024, 2: 1024, 3: 1024}
2025-02-05 13:11:46,405 INFO    : lisa.wlgen.rta.RTA : Execution start: rt-app  /root/devlib-target/lisa/wlgen/20250205_131044_3b90c73fb83a4959bb7f2bb0d65dff75/experiment_workload.json 2>&1
2025-02-05 13:12:19,128 INFO    : lisa.wlgen.rta.RTA : Wiping target run directory: /root/devlib-target/lisa/wlgen/20250205_131044_3b90c73fb83a4959bb7f2bb0d65dff75

Loading up the trace#

We have a Trace class that lets us easily access trace events. It can also do some post-processing to provide different kinds of analysis.

[15]:

from lisa.trace import Trace

We also save some platform information (number of CPUs, available frequencies, kernel version…) that comes in handy for doing some analysis:

[16]:

print(target.plat_info)

├ abi from target (str): arm64
├ cpu-capacities:
    ├ orig from target (dict): {0: 1024, 1: 1024, 2: 1024, 3: 1024}
    ├ writeable from target (bool): False
    └ rtapp from user(platform-info/rtapp/calib),target(platform-info/cpu-capacities/orig),target(platform-info/cpu-capacities/writeable) (dict): {0: 1024, 1: 1024, 2: 1024, 3: 1024}
├ cpus-count from target (int): 4
├ freq-domains from target (FilteredDeferredValue): <lazy value of freq-domains>
├ freqs from target (FilteredDeferredValue): <lazy value of freqs>
├ kernel:
    ├ config from target (TypedKernelConfig): <kernel config>
    ├ symbols-address from target (FilteredDeferredValue): <symbols address>
    └ version from target (KernelVersion): 6.13.0-00918-g95ec54a420b8 6 SMP PREEMPT Fri Jan 24 22:39:48 GMT 2025
├ name from target-conf (str): <noname>
├ nrg-model from target (FilteredDeferredValue): <lazy value of nrg-model>
├ numa-nodes-count from target (int): 1
├ os from target (FilteredDeferredValue): <lazy value of os>
├ rtapp:
    └ calib from user (dict): {0: 383, 1: 342, 2: 350, 3: 387}
└ capacity-classes from target(platform-info/cpu-capacities/orig) (list): [[0, 1, 2, 3]]

You can pass the platform info directly from the Target, but it’s a good idea to save it on the disk so that you can re-run whatever analysis code you want several months down the line after the platform was lost in a tragic fire. It’s why we save this information somewhere instead of polling the target when we want to use them - we can run analysis code offline. Here we show how to save to/restore this platform information from the disk.

[17]:

plat_info_path = os.path.join(wload.res_dir, "platinfo.yaml")
target.plat_info.to_yaml_map(plat_info_path)

2025-02-05 13:12:24,094 INFO    : lisa.energy_model.EnergyModel.from_target : Attempting to load EM using LinuxEnergyModel
2025-02-05 13:12:25,551 ERROR   : LinuxTarget  : Module "cpufreq" failed to install on target: Module "cpufreq" is not supported by the target
2025-02-05 13:12:25,554 INFO    : lisa.target.Target : Loading target devlib module cpufreq
2025-02-05 13:12:25,993 ERROR   : LinuxTarget  : Module "cpufreq" failed to install on target: Module "cpufreq" is not supported by the target
2025-02-05 13:12:26,540 ERROR   : LinuxTarget  : Module "cpufreq" failed to install on target: Module "cpufreq" is not supported by the target
2025-02-05 13:12:26,543 INFO    : lisa.target.Target : Loading target devlib module cpufreq
2025-02-05 13:12:27,073 ERROR   : LinuxTarget  : Module "cpufreq" failed to install on target: Module "cpufreq" is not supported by the target
2025-02-05 13:12:27,093 INFO    : lisa.platforms.platinfo.PlatformInfo : Attempting to read kallsyms from target

[18]:

from lisa.platforms.platinfo import PlatformInfo

[19]:

plat_info = PlatformInfo.from_yaml_map(plat_info_path)

trace_path = os.path.join(wload.res_dir, 'trace.dat')
trace = Trace(trace_path, plat_info=plat_info)

Looking at the trace#

Kernelshark can be opened from the notebook:

[20]:

# trace.show()

Analysing the trace#

Relevant documentation: https://tooling.sites.arm.com/lisa/latest/trace_analysis.html

DataFrame libraries#

LISA supports two dataframe (table) libraries:

polars: https://docs.pola.rs/api/python/stable/reference/index.html
pandas: https://pandas.pydata.org/docs/index.html

Polars is a more modern alternative to pandas and most of the internal machinery of LISA has been moved to polars. At this point, pandas is available for backward compatibility and some internal code still has not been converted, but eventually there will not be any direct dependencies on pandas anymore. Since most dataframe-producing APIs are related to the Trace class, the switch between the two libraries can be achieved at that level:

[21]:

import polars as pl

# This creates a view of the trace that will provide polars.LazyFrame dataframes.
# It is also possible to create the trace object for polars directly with
# Trace(..., df_fmt='polars-lazyframe'). The result is the same.
trace = trace.get_view(df_fmt='polars-lazyframe')

By default, Trace will use the timestamps collected in the trace. While useful to correlate to other system’s aspects (e.g. dmesg log), this is ideal when comparing multiple runs of the same workload, or even a single run if the absolute timestamp does not matter.

[22]:

# This can be achieved in Trace directly with Trace(..., normalize_time=True)
trace = trace.get_view(normalize_time=True)

Reading trace events#

[23]:

df = trace.df_event("sched_switch")
df.collect()

[23]:

shape: (2_351, 11)

Time	__cpu	__pid	prev_comm	prev_pid	prev_prio	prev_state	next_comm	next_pid	next_prio	__comm
duration[ns]	u32	i32	cat	i32	i32	i64	cat	i32	i32	cat
207936ns	0	0	"swapper/0"	0	120	0	"sshd"	4850	120	"<idle>"
500784ns	3	4986	"trace-cmd"	4986	120	32	"swapper/3"	0	120	null
3699136ns	2	0	"swapper/2"	0	120	0	"rcu_preempt"	16	120	"<idle>"
3783872ns	0	4850	"sshd"	4850	120	0	"migration/0"	19	0	"sshd"
4034304ns	2	16	"rcu_preempt"	16	120	128	"swapper/2"	0	120	"rcu_preempt"
…	…	…	…	…	…	…	…	…	…	…
4s 417871968ns	0	4059	"kworker/u16:4"	4059	120	128	"swapper/0"	0	120	"kworker/u16:4"
4s 424197680ns	1	0	"swapper/1"	0	120	0	"rcu_preempt"	16	120	"<idle>"
4s 424765088ns	1	16	"rcu_preempt"	16	120	128	"swapper/1"	0	120	"rcu_preempt"
4s 428291344ns	1	0	"swapper/1"	0	120	0	"rcu_preempt"	16	120	"<idle>"
4s 428672432ns	1	16	"rcu_preempt"	16	120	128	"swapper/1"	0	120	"rcu_preempt"

[24]:

task = df.filter(
    pl.col('__comm')
        # All string fields in ftrace events are loaded as Categorical in LISA due to their repetitive nature.
        .cast(pl.String)
        .str.starts_with("tsk")
).select(pl.col('__comm').first()).collect().item()
task

[24]:

'tsk0_0-0'

The standard DataFrame operations are available, so you can filter/slice it however you wish:

[26]:

df.filter(pl.col('next_comm') == task).collect()

[26]:

shape: (96, 11)

Time	__cpu	__pid	prev_comm	prev_pid	prev_prio	prev_state	next_comm	next_pid	next_prio	__comm
duration[ns]	u32	i32	cat	i32	i32	i64	cat	i32	i32	cat
1s 330732µs	0	4999	"tsk3_0-9"	4999	120	1	"tsk0_0-0"	4990	120	"tsk3_0-9"
1s 387599856ns	0	4996	"tsk2_0-6"	4996	120	0	"tsk0_0-0"	4990	120	"tsk2_0-6"
1s 411637296ns	0	4999	"tsk3_0-9"	4999	120	0	"tsk0_0-0"	4990	120	"tsk3_0-9"
1s 427541936ns	0	4992	"tsk0_2-2"	4992	120	0	"tsk0_0-0"	4990	120	"tsk0_2-2"
1s 443112992ns	0	4999	"tsk3_0-9"	4999	120	0	"tsk0_0-0"	4990	120	"tsk3_0-9"
…	…	…	…	…	…	…	…	…	…	…
2s 428980128ns	0	4868	"sshd"	4868	120	1	"tsk0_0-0"	4990	120	"sshd"
2s 435000064ns	0	4997	"tsk2_1-7"	4997	120	0	"tsk0_0-0"	4990	120	"tsk2_1-7"
2s 443807664ns	0	0	"swapper/0"	0	120	0	"tsk0_0-0"	4990	120	"<idle>"
2s 447198400ns	0	4997	"tsk2_1-7"	4997	120	1	"tsk0_0-0"	4990	120	"tsk2_1-7"
2s 459987824ns	0	0	"swapper/0"	0	120	0	"tsk0_0-0"	4990	120	"<idle>"

Using the trace analysis#

Example dataframes#

LISA ships a number of namespaced trace analysis methods. They can all be called on a trace with trace.ana.<analysis name>.<method name>(). They fall mostly into two categories:

Method starting with df_: returns a pandas DataFrame
Method starting with plot_: returns a holoviews element ready to be displayed

[27]:

trace.ana.tasks.df_tasks_runtime().collect()

[27]:

shape: (40, 3)

pid	comm	runtime
i32	cat	f64
5002	"sh"	0.109627
4990	"tsk0_0-0"	0.331726
4868	"sshd"	0.344764
27	"migration/2"	0.026698
5005	"trace-cmd"	0.351035
…	…	…
4998	"tsk2_2-8"	0.27628
4870	"sshd"	0.288097
4995	"tsk1_2-5"	0.331213
4986	"trace-cmd"	0.0
3662	"kworker/3:0"	0.002586

[28]:

df = trace.ana.tasks.df_task_states(task, stringify=True)
df.collect()

[28]:

shape: (252, 9)

Time	target_cpu	cpu	curr_state	next_state	delta	duration_delta	curr_state_str	next_state_str
duration[ns]	i32	u32	i64	i64	f64	duration[ns]	str	str
1s 240709136ns	-1	0	1	512	0.0883236	88323600ns	"S"	"R"
1s 329032736ns	0	3	512	8192	0.001699	1699264ns	"R"	"R"
1s 330732µs	-1	0	8192	1	0.001847	1847488ns	"R"	"S"
1s 332579488ns	-1	0	1	512	0.048222	48222496ns	"S"	"R"
1s 380801984ns	0	0	512	8192	0.006798	6797872ns	"R"	"R"
…	…	…	…	…	…	…	…	…
2s 447198400ns	-1	0	8192	1	0.00662	6619728ns	"R"	"S"
2s 453818128ns	-1	0	1	512	0.006069	6068768ns	"S"	"R"
2s 459886896ns	0	0	512	8192	0.000101	100928ns	"R"	"R"
2s 459987824ns	-1	0	8192	16	0.001535	1534592ns	"R"	"X"
2s 461522416ns	-1	0	16	-1	null	null	"X"	null

The trace.ana object can be used to set default values to analysis methods. Simply calling it with keyword arguments will set default values, which can later be overridden when the method is called if necessary.

This avoids repetition of fixed parameters such as ŧask, tasks, cpu etc. Just be careful as some methods might take more parameters than you expect: some task-related methods also accept a cpu parameter to restrict to a given CPU, so it might be a good idea to have a proxy object for all CPU-related calls and another one for task-related calls.

[30]:

ana = trace.ana(task=task)

df = ana.tasks.df_task_states(stringify=True)

# Default values can be overridden by calling it again
ana2 = ana(task='trace-cmd')
# And overridden again when calling the method
df = ana2.tasks.df_task_states(task=task, stringify=True)

[31]:

from lisa.analysis.tasks import TaskState

[32]:

# df[df.curr_state == TaskState.TASK_ACTIVE][1:1.2]

df.filter(pl.col('curr_state') == TaskState.TASK_ACTIVE).collect()

[32]:

shape: (96, 9)

Time	target_cpu	cpu	curr_state	next_state	delta	duration_delta	curr_state_str	next_state_str
duration[ns]	i32	u32	i64	i64	f64	duration[ns]	str	str
1s 330732µs	-1	0	8192	1	0.001847	1847488ns	"R"	"S"
1s 387599856ns	-1	0	8192	0	0.004855	4854832ns	"R"	"R"
1s 411637296ns	-1	0	8192	0	0.004184	4184384ns	"R"	"R"
1s 427541936ns	-1	0	8192	0	0.004365	4364944ns	"R"	"R"
1s 443112992ns	-1	0	8192	0	0.004048	4048112ns	"R"	"R"
…	…	…	…	…	…	…	…	…
2s 428980128ns	-1	0	8192	0	0.001849	1848960ns	"R"	"R"
2s 435000064ns	-1	0	8192	1	0.001505	1504816ns	"R"	"S"
2s 443807664ns	-1	0	8192	0	0.003066	3066496ns	"R"	"R"
2s 447198400ns	-1	0	8192	1	0.00662	6619728ns	"R"	"S"
2s 459987824ns	-1	0	8192	16	0.001535	1534592ns	"R"	"X"

Example plots#

[33]:

import holoviews as hv
# Before rendering any plot, a plotting backend for holoviews has to be chosen.
# Bokeh provides much better interactive (and HTML) plots than matplotlib.

# THIS MUST BE DONE AFTER ALL IMPORTS.
# Otherwise there might be issues that lead to
# not displaying plots until hv.extension() is called again.

hv.extension('bokeh')

[38]:

trace.ana.tasks.plot_task_total_residency(task)

[38]:

[36]:

trace.ana.tasks.plot_tasks_wakeups_heatmap(bins=200)

[36]:

[37]:

trace.ana.tasks.plot_tasks_forks_heatmap(bins=200)

[37]:

[ ]:

[ ]: