A Beginner's Guide to Process Inspection and Troubleshooting on Linux

Understanding processes forms the foundation for many critical security tasks on Linux systems. Whether conducting threat hunting, incident response, or malicious file analysis, you need visibility into process activity to understand what is happening across a system.

When suspicious events occur or malware infiltrates a server, processes provide the clues to uncover the source. By inspecting process relationships and changes, a security analyst can reveal the impact of threats and track down the root cause even if other artifacts like files have been deleted.

This beginner’s guide aims to equip you with two powerful commands for Linux process troubleshooting - ps and pstree. With mastery of these tools, you will gain the ability to:

Let's dive into process inspection and troubleshooting and discovery!

Using ps to View Processes

The ps command in Linux provides comprehensive visibility into the processes running locally or system-wide at any point in time.

At a basic level, ps displays details like process ID, owner, CPU usage, executable path, and more about the processes running on the current terminal session. But its real power comes from a range of options that allow tailored inspection of all process activity - not just the small subset you can observe.

$ ps
  PID TTY          TIME CMD
 9835 pts/0    00:00:00 ps

$ ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND 
root           1  0.0  0.0  19356

Common options like aux, -f, and -e give you different lenses to peer into process execution:

Stacked together, ps auxef can expose all processes, users, and arguments to detect anomalies.

Beyond details, ps can also visualize hierarchical process trees with the --forest argument. Trees highlight the ancestors and descendants within a process chain - invaluable context during incident investigation.

$ ps -ef --forest

With crafty manipulation of ps, you can illuminate all facets of process activity to pinpoint threats.

Digging into Processes with pstree

While ps acts as a flashlight into process details, pstree focuses its firepower on mapping relationships between processes for enhanced detection and hunting.

At its core, pstree visually wires together processes into a tree based on which processes spawned child processes. This immediate context delivers insight into the chain of events tied to any suspicious process:

pstree

init─┬─apache2───9*[apache]
     ├─sshd─sshd─bash─pstree
     ├─xorg───{xorg}

Augmented with parameters like -p for PIDs and -u for users, you can precisely track owners and pivot between processes. The -g flag surfaces groupings to reason about connections between process clusters.

Just like adversaries, pstree allows you to follow threads of execution through relationship graphs. This facilitates revealing signs ofattack tactics as well as cleanup remaining after threats.

With ps + pstree, you have a toolkit to dissect process state and behavioral connections for battling threats. Now let's see them in action together!

pstree -g -p -a

Putting It All Together

Imagine during server monitoring you come across a Python process consuming 50%+ CPU constantly over the past few days - highly abnormal. After isolation, you still find the process running with its high workload.

Time to go process hunting! Your goal is to determine the purpose of the process, its origin, and if it poses a threat.

Visually mapping program relationships with pstree -p reveals the parent PID of the Python process belongs to an sshd foreground worker. Expected so far.

But further inspection of Python's command line arguments with ps -fep <PID> uncovers a reference to an unusual script /tmp/export.py.

Exporting data from /tmp is suspicious and may indicate exfiltration activity. But what access does this script have?

Using pstree -ugp exposes the Python process running as root! Privilege escalation provides the answer - a critical piece of the puzzle.

With the context provided by creatively leveraging ps + pstree, you successfully tracked down the threat vector and overall impact of the crypto mining malware disguised as an innocent system process.

Conclusion

Process analysis forms the foundation of Linux security operations. With ps and pstree now in your toolkit, you have the power to illuminate process state and relationships to enhance detection, incident investigation, and threat hunting.

Some key points to take away:

ps - Inspect process details like users, CPU usage, full command lines

pstree - Map hierarchical relationships between processes

Together - Leverage process context to identify, investigate, and hunt threats

I challenge you to test your skills on your own Linux systems! Experiment with combining ps and pstree to deep dive into how processes interconnect. The more hands-on practice, the faster you will unlock these tools' immense power. Happy hunting!

We hope this post served as a good source of information to know basic process inspection and troubleshooting on Linux. Visit our website, thesecmaster.com, and social media pages on Facebook, LinkedIn, Twitter, Telegram, Tumblr, & Medium and subscribe to receive updates like this.