In this subsection, we study different available tracing tools. Tracing tools provide different techniques to aggregate data from the target system. Moreover, some tracing tools implement basic analyses to send to the visualization component.
Table 2.1 Available Kernel Tracepoints and their type
Types Family Events
Virtual Machine
Processor
kvm_set_irq, kvm_ioapic_set_irq,kvm_msi_set_irq, kvm_ack_irq, kvm_mmio,kvm_fpu, kvm_entry, kvm_exit,kvm_hypercall, kvm_hv_hypercall, kvm_pi, kvm_cpuid , kvm_apic,kvm_inj_*,
kvm_apic_accept_irq, kvm_msr,kvm_pic_set_irq, kvm_apic_ipi„, kvm_eoi, kvm_pv_eoi,kvm_nested_*, kvm_emulate_insn„kvm_write_tsc_offset,
kvm_update_master_clock,kvm_track_ts , kvm_apic_accept_irq
Memory kvm_age_page, kvm_try_async_get_page, kvm_async_*, kvm_page_fault, kvm_mmu_*
Host
OS module_*, printk_console, random_*, rcu_utilization, regulator_*, gpio_*, sched_*, signal_*, workqueue_* Processor power_*, irq_*, timer_*
Memory kmem_*, mm_*
Disk block_*, jbd2_*, scsi_*
Network napi_poll, net_*, skb_*, sock_*, rpc_*, udp_fail_queue_rcv_skb Graphics asoc_snd_*, v4l2
2.4.2.1 LTTng
The Linux Trace Toolkit next generation (LTTng) is designed to offer very low overhead kernel and user-space tracing [34]. It implements a very fast lock-free, wait-free read-copy- update (RCU) buffer to store data and eventually copy it on disk. A very low overhead, which remains almost constant when the number of parallel cores increases, because of per- core buffers, makes LTTng the default tool for tracing real-time application in Linux. LTTng supports both static and dynamic tracing. It can add dynamic tracepoints with Kprobe. In addition, static tracepoints can be added in the source code of the kernel, as well as in user-space programs with UST. LTTng also provides hooks to the TRACE_EVENT macro for kernel events. LTTng exports the events in the Common Trace Format (CTF) to be written on disk [35].
Figure 2.5 shows the different components in LTTng and how they interact with application and the Linux kernel. The most important component is the session daemon. It is responsible for managing and controlling the other components. The LTTng kernel modules include: a set of probes to be attached to Linux kernel tracepoints and to the entry and exit of system calls. The Ring buffer module is an implementation of a ring buffer where the consumer daemon reads events. The LTTng Consumer daemon shares the ring buffer between user-space and
kernel-space to collect kernel trace data and copy it to disk. In [36], the authors provide a map between kernel tracepoints and their system activity on the host. The extended version of mapping between kernel tracepoints and their types, for both host and kernel, is depicted in Table 2.1
Figure 2.5 LTTng components and their tracing path [3]
LTTng is currently the fastest kernel/user-space tracing toolkit for Linux [37]. The major factor for such improvement is its lock-free implementation of the LTTng ring buffer.
2.4.2.2 Perf
Perf is a monitoring tool inside the Linux kernel mostly used for sampling and profiling. Perf can access and gather data from the hardware Performance Monitoring Unit (PMU), like cache misses in different levels, miss branch predictions, TLB hits, and so on. Perf can hook to TRACE_EVENT macros in the kernel and save events at run-time in the perf ring-buffer.
2.4.2.3 Ftrace
Ftrace is an older tracer included in the Linux kernel that allows function tracing, system calls tracing, dynamic instrumentation and so on. It provide function graphs that show the entry and exit of all functions at the kernel level. It supports Kprobe for dynamic instrumentation and provides hooks to the TRACE_EVENT infrastructure. It keeps the events in memory and does not write them to disk automatically. In Ftrace, the size of a payload is limited to the size of a page. The ring buffer is implemented as a linked list and a buffer page can be read once it is full.
2.4.2.4 SystemTap
SystemTap is a tracing tool which allows inserting tracepoints dynamically, as well as col- lecting events from tracepoints defined using the TRACE_EVENT macro [38]. It is similar to Ktap, the tracing code is written as a script. The scripts are written in the SystemTap language, highly similar to the C language. These scripts are converted into C code, and then are inserted into the Linux kernel as a module. Kprobe is used to insert dynamically tracepoints into the Linux kernel. This approach, similar to Dtrace on Solaris [39], is very flexible. However, SystemTap has serious performance issues especially for collecting trace events [37]. Furthermore, its dynamic instrumentation is trap based, which adds numerous context switches.
2.4.2.5 eBPF
The newest entrant in the long list of available tracers is the Extended Berkeley Packet Filter (eBPF). BPF, which was in older Linux kernels, has been enhanced and became eBPF in the Linux 4.x series kernels. eBPF allows to do much more than just packet filtering in BPF. It enables the user to write any program and to insert it into any location in the kernel using Kprobe. It can also attach probes to available static tracepoints in the Linux kernel. eBPF scripts are compiled at runtime and executed on the small BPF VM. eBPF can now be used as a tracer, much like SystemTap or Dtrace, since it can aggregate events, analyze them, and store them in a trace. It uses the perf buffer to save events in memory and can store them as CTF events on disk. eBPF is a great tool for aggregation and live monitoring. Although an elaborate, feature-rich and easy to use tool, eBPF is still under development and does not offer the same performance, maturity and features as LTTng. Figure 2.6 depicts eBPF tc hooks, that first compile the program with LLVM and then inject it into the kernel using the bpf syscall. After the injection into the kernel, the kernel verifier checks the script for
tc LLVM/clang cls_bpf Userspace Kernel C source code bpf_prog.c ELF-compiled BPF bpf_prog.o Network stack tc ingress tc egress
Net device Net device
bpf() syscall
JIT
Packets
User program
Maps
Figure 2.6 eBPF architecture for tc interface [4]
safety, attaches it to the traffic controller interface and runs it in a small VM.
2.4.2.6 Performance and Evaluation
In this project, we used LTTng since it is a system-wide and very fast tracer [37] [40]. It also provides unified kernel and user-space tracing, which is very useful in our project. LTTng can gather KVM tracepoints (Kernel tracepoints) as well as Qemu tracepoints (User-space tracepoints).