Well strace, tcpdump, and lsof are indeed great utilities that should be part of every sysadmin's toolset, and that is precisely the reason why you will love sysdig, a powerful open source tool for system-level exploration and troubleshooting, introduced by its creators as "strace + tcpdump + lsof + awesome sauce with a little Lua cherry on top." Humor aside, one of the great features of sysdig resides in its ability not only to analyze the "live" state of a Linux system, but also to save the state in a dump file for offline inspection. What's more, you can customize sysdig's behavior or even enhance its capabilities by using built-in (or writing your own) small scripts called chisels. Individual chisels are used to analyze sysdig-captured event streams in various script-specific fashions.
In this tutorial we'll explore the installation and basic usage of sysdig to perform system monitoring and troubleshooting on Linux.
Installing Sysdig
For this tutorial, we will choose to use the automatic installation process described in the official website for the sake of simplicity, brevity, and distribution agnosticity. In the automatic process, the installation script automatically detects the operating system and installs all the necessary dependencies.
Run the following command as root to install sysdig from the official apt/yum repository:
# curl -s https://s3.amazonaws.com/download.draios.com/stable/install-sysdig | bash
Once the installation is complete, we can invoke sysdig as follows to get a feel for it:
# sysdig
Our screen will be immediately filled with all that is going on in our system, not allowing us to do much more with that information. For that reason, we will run:
# sysdig -cl | less
to see a list of available chisels.
The following categories are available by default, each of which is populated by multiple built-in chisels.
CPU Usage
Errors
I/O
Logs
Misc
Net
Performance
Security
System State
To display information (including detailed command-line usage) on a particular chisel, run:
# sysdig -cl [chisel_name]
For example, we can check information about spy_port chisel under "Net" category by running:
# sysdig -i spy_port
Chisels can be combined with filters (which can be applied to both live data or a trace file) to obtain more useful output.
Filters follow a "class.field" structure. For example:
fd.cip: client IP address.
evt.dir: event direction can be either '>' for enter events or '<' for exit events.
The complete filter list can be displayed with:
# sysdig -l
In the rest of the tutorial, I will demonstrate several use cases of sysdig.
Sysdig Example: Troubleshooting Server Performance
Suppose your server is experiencing performance issues (e.g., unresponsiveness or significant delays in responding). You can use the bottlenecks chisel to display a list of the 10 slowest systems calls at the moment.
Use the following command to check up on a live server in real time. The "-c" flag followed by a chisel name tells sysdig to run the specified chisel.
# sysdig -c bottlenecks
Alternatively, you can conduct a server performance analysis offline. In that case, you can save a complete sysdig trace to a file, and run the bottlenecks chisel against the trace as follows.
First, save a sysdig trace (use Ctrl+c to stop the collection):
# sysdig -w trace.scap
Once the trace is collected, you can check the slowest systems calls that were performed during the capture interval by running:
# sysdig -r trace.scap -c bottlenecks
You want to pay attention fo columns #2, #3, and #4, which indicate execution time, process name, and PID, respectively.
Sysdig Example: Monitoring Interactive User Activities
Suppose you as a sysadmin want to monitor interactive user activities in a system (e.g., what command a user typed from the command line, and what directories the user went to). That is when spy_user chisel comes in handy.
Let's first collect a sysdig trace with a couple of extra options.
# sysdig -s 4096 -z -w /mnt/sysdig/$(hostname).scap.gz
"-s 4096" tells sysdig to capture up to 4096 bytes of each event.
"-z" (used with "-w") enables compression for a trace file.
"-w " saves sysdig traces to a specified file.
In the above, we customize the name of the compressed trace file on a per-host basis. Remember that you can interrupt the execution of sysdig at any moment by pressing Ctrl + c.
Once we've collected a reasonable amount of data, we can view interactive activities of every user in a system by running:
# sysdig -r /mnt/sysdig/debian.scap.gz -c spy_users
The first column in the above output indicates the PID of the process associated with a given user's activity.
What if you want to target a specific user, and monitor the user's activities only? You can filter the results of the spy_users chisel by username:
# sysdig -r /mnt/sysdig/debian.scap.gz -c spy_users "user.name=johndoe"
Sysdig Example: Monitoring File I/O
We can customize the output format of sysdig traces with "-p" flag, and indicate desired fields (e.g., user name, process name, and file or socket name) enclosed inside double quotes. In this example, we will create a trace file that will only contain writing events in home directories (which we can inspect later with "sysdig -r writetrace.scap.gz").
# sysdig -p "%user.name %proc.name %fd.name" "evt.type=write and fd.name contains /home/" -z -w writetrace.scap.gz
Sysdig Example: Monitoring Network I/O
As part of server troubleshooting, you may want to snoop on network traffic, which is typically done withtcpdump. With sysdig, traffic sniffing can be done as easily, but in more user friendly fashions.
For example, you can inspect data (in ASCII) that has been exchanged with a particular IP address, served by a particular process (e.g., apache2):
# sysdig -s 4096 -A -c echo_fds fd.cip=192.168.0.100 -r /mnt/sysdig/debian.scap.gz proc.name=apache2
If you want to monitor raw data transfer (in binary) instead, replace "-A" with "-X":
# sysdig -s 4096 -X -c echo_fds fd.cip=192.168.0.100 -r /mnt/sysdig/debian.scap.gz proc.name=apache2
Thank you :)
