Status and Desiderata for Syscall Tracing and Virtualization Support

The Linux kernel offers several mechanisms for system call tracing and virtualization, such as ptrace, ptrace with PTRACE_SYSEMU, seccomp with SECCOMP_RET_TRACE, seccomp-unotify, and prctl with PR_SET_SYSCALL_USER_DISPATCH. Each of these methods has unique application domains, strengths, and limitations in terms of complexity and usability, while sharing core capabilities. Although primarily used for building powerful debugging tools (e.g., gdb, strace), these mechanisms can also be leveraged to construct syscall-based virtual machines, achieving varied performance levels and encountering specific challenges.

A distinctive feature of syscall-based virtual machines is their ability to selectively emulate system calls. For example, one might choose to emulate open(2) but not socket(2), thereby providing the guest environment with a tailored "view" of the underlying operating system.

This seminar will explore the current state of syscall tracing and virtualization techniques through practical demonstrations, examine their inherent limitations, and propose potential improvements to enhance usability and performance. Specific focus will be given to the challenges of virtualizing the poll(2) and select(2) syscall families, which are particularly intricate when managing a mixed environment of virtualized and real file descriptors.