In this article, I’ll describe how to hunt for rootkits in linux, Rootkits are extremely advanced pieces of code, not any one can write it, However there’s a lot of Proof of concept code demonstrating rootkit techniques and how to build one from scratch, In the article I will present a technique based on instructions in some system calls, which can be used to detect rootkits.
I’ll explain techniques not only how to detect rootkits but also how to conduct a forensics and hunt down malware, I’ll focus only to cover the basics at first and understand the nature of the behavior. For those interested in learning about kernel rootkits, I have a separate post titled “Writing a Simple Rootkit for Linux” and “The linux kernel modules programming” which I recommend reading before delving into this topic.
Once a rootkit gets kernel-level access, it can lie about everything. Your ps command? It might not show that malicious process. Your ls? Those hidden files won’t appear. Even your network monitoring tools can be fooled.
In this guide, I’ll walk you through the techniques I use to hunt rk. We’ll start with understanding how they work, then dive into detection methods that actually work. I’ll explain everything as we go.
How Kernel Rootkits Work
Kernel runs in what we call “ring 0” the most privileged execution level. Everything else, including your applications and most system tools, runs in “ring 3” with limited privileges.
Ring 0: Kernel space (God mode - can do anything)
Ring 1: Device drivers (rarely used on x86)
Ring 2: Device drivers (rarely used on x86)
Ring 3: User space (your apps, limited privileges)
When a rootkit gets into ring 0, it’s game over for AV detection methods, most of those run in, yep, ring 3. At that point, the rootkit can modify kernel data structures, hook system calls, and basically rewrite the rules for how your system behaves.
Shitt get interesting when rootkits target the system call table, think of it as the kernel’s phone book that maps system calls to their actual functions. When you run ls, it eventually calls sys_getdents() to list directory contents. A rk can intercept this call and filter out files it wants to hide.
syscall_table = (void *)kallsyms_lookup_name("sys_call_table");
original_getdents = (void *)syscall_table[__NR_getdents64];
syscall_table[__NR_getdents64] = &malicious_getdents;
What’s happening here?
First, the rootkit hunts down the system call table using kallsyms_lookup_name() Once it finds it, the rootkit politely saves the original getdents64 function pointer (because it’s not completely rude), then swaps it out with its own version.
Now every time any program tries to list dir contents, the rootkit’s function runs first. It can call the original function, filter out any files it wants to hide, and return the “cleaned” results. Sneaky, right?
Modern kernels tho have made this much harder. Take SMEP and SMAP for instance, which basically tell the kernel “hey, don’t execute any code that comes from user space when you’re running in kernel mode.” Then there’s KASLR, which shuffles around where everything lives in memory each time you boot, making it a pain in the ass to find the syscall table. And don’t get me started on Control Flow Integrity, it’s like having a bouncer that checks if function calls are going where they’re supposed to go.
Though rootkits have adapted by using techniques like Return-Oriented Programming (ROP) or finding legitimate kernel functions to abuse, See : Linux Rootkits Part 1: Introduction and Workflow
Timing-Based Analysis
Now, how do we catch these things? One approach I’ve used is timing analysis, though it’s not perfect. The idea is simple if a rk is filtering system call results, it should take longer than normal operations.
Let’s say you’re reading /etc/passwd versus /proc/net/tcp. In a clean system, both should take roughly the same time. But if a rootkit is hiding network connections, the /proc/net/tcp read might take longer due to the filtering overhead.
start_time = get_timestamp();
result = sys_read("/etc/passwd", buffer, size);
normal_time = get_timestamp() - start_time;
start_time = get_timestamp();
result = sys_read("/proc/net/tcp", buffer, size);
suspicious_time = get_timestamp() - start_time;
if (suspicious_time > normal_time * THRESHOLD) {
}
This method has serious limitations. The Linux kernel is complex, and execution times vary based on countless factors, CPU load, I/O wait, memory pressure, different code paths. You’ll get false positives a LOT.
Tools like rkhunter and chkrootkit use more sophisticated versions of this approach, but they still struggle with false positives. They’re good as a first pass, but don’t rely on them alone.
Memory Forensics with Volatility
When you suspect a rootkit infection, memory analysis can reveal what’s actually running versus what the compromised kernel is telling you. Let me walk you through a investigation I did using Volatility. The beauty of memory forensics is that it analyzes the raw memory dump, bypassing any kernel-level lies.
This plugin is used to provide a full process listing of the system. Its output is approximately the same as would be obtained running the ps -aux command via a terminal, The resulting output shown Everything in this list of processes appeared to be normal, with the exception of one process, specifically the very last process in the list. Its name of F00 is not a known standard Linux
$ python3 vol.py --info | grep linux_
Next we use linux_pslist This plugin prints the list of active processes starting from the init_task symbol and walking the task_struct->tasks linked list. It does not display the swapper process. If the DTB column is blank, the item is likely a kernel thread. Result the same numbers of processes were found using this plugin as with the previous plugin. Again,
the only process that did not appear to belong was F00, Next is dump the files for further analysis we utilize linux_lsof plugin which mimics the lsof command on a live system. It prints the list of open file descriptors and their paths for each running process
Pid FD Path
-------- -------- ----
1 0 /dev/null
1 1 /dev/null
1 2 /dev/null
After we list the files and detect what we looking for by tracking down the suspicious process it’s time prints details of process memory, including heaps, stacks, and shared libraries linux_proc_maps This very powerful plugin can be used to learn important information about the underlying system as a whole
0x8050000-0x8051000 r-x 2777 /usr/F_00/F_00
0x8051000-0x8052000 rw- 4096 2777 /usr/F_00/F_00
0xb75d7000-0xb75d8000 rw- 0 0
What this plugin reveals about this process is the actual location of the files associated with
suspicious process F_00 specifically its actual location, /usr/ also it is revealed by its permission of r-x. Interestingly, this process only relies on two libraries, whereas most system processes rely on many additional libraries, Finally it’s time time dump this process directly from the memory image, to do that let’s call linux_find_file This plugin can be used to not only dump pre-identified files from the memory image (using information obtained from other plugins) but it can also list all filesystem objects with an open handle in memory we can simply dump any target file with the argument -F for example:
$ python3 vol.py ... linux_find_file -F “/usr/F_00/F_00”
Inode Number Inode
-------------------- ---------------
0161170 0xf
$ python3 vol.py ... linux_find_file -i 0xf -O mal.elf
The “Inode” represents the location in memory where this specific “inode” is stored. With this information, the file “mal.elf” was generated and dumped from the memory image. The next step is to verify the hash of the file and check if there is a match in any malware database or perform further analysis on the binary.
The methods I’ve shown you are foundational, but the rootkit game has evolved. Intel’s Control-flow Enforcement Technology (CET) can detect ROP-based rootkits by monitoring unexpected control flow changes. AMD has similar features with their Control Flow Integrity.
Also, you can use eBPF programs to monitor kernel behavior from a privileged but isolated context:
SEC("kprobe/kallsyms_lookup_name")
int monitor_symbol_lookup(struct pt_regs *ctx) {
char symbol_name[64];
bpf_probe_read_str(symbol_name, sizeof(symbol_name), (void *)PT_REGS_PARM1(ctx));
if (strstr(symbol_name, "sys_call_table")) {
bpf_trace_printk("Suspicious symbol lookup: %s", symbol_name);
}
return 0;
}
Based on my experience, writing a few of them and yeah, hunting these things too, if you even suspect you’ve been hit by a rootkit or some kind of malware, the very first thing you should do is grab a memory image. Do it before touching anything else, especially before rebooting. I can’t tell you how many investigations have been completely trashed because someone panicked and hit the reset button. Once that memory’s gone, so is a huge chunk of your evidence.
When it comes time to do forensics, boot from a clean, trusted external medium. Never trust a potentially compromised system to inspect itself that’s asking the fox if the henhouse is secure.
Now, maybe it’s a full-blown kernel-level rootkit, maybe it’s just some sketchy userland malware. Doesn’t matter if you’re even a little paranoid, it’s worth digging in. Start with a proper memory dump and run it through Volatility those tools can surface unlinked modules, hidden processes, and shady syscall table hooks that won’t show up in /proc. From there, do a syscall table integrity check. Compare it against a clean baseline and look for anything hijacking entries like execve, getdents, or read.
If you’re feeling surgical, trace the system with ftrace follow the execution path of suspect syscalls and see if anything’s rerouting under the hood. And don’t forget to scan /sys/module/ and /dev/kmem directly some rootkits unlink from /proc/modules, but they usually slip up somewhere. When you’re up against something this deep, trust nothing the system tells you at face value. Look beneath it.