a linux trace/probe tool.

簡介

SystemTap是我目前所知的最強大的核心除錯工具，有些傢伙甚至說它無所不能：）

(1) 發展歷程

Debuted in 2005 in Red Hat Enterprise Linux 4 Update 2 as a technology preview.

After four years in development, System 1.0 was released in 2009.

As of 2011 SystemTap runs fully supported in all Linux distributions.

(2) 官方介紹

SystemTap provides free software(GPL) infrastructure to simplify the gathering of information about the

running Linux system. This assists diagnosis of a performance or functional problem. SystemTap eliminates

the need for the developer to go through the tedious and disruptive instrument, recompile, install, and reboot

sequence that may be otherwise required to collect data.

SystemTap provides a simple command line interface and scripting language for writing instrumentation for

a live running kernel plus user-space application. We are publishing samples, as well as enlarging the internal

"tapset" script library to aid reuse and abstraction.

Among other tracing/probing tools, SystemTap is the tool of choice for complex tasks that may require live analysis,

programmable on-line response, and whole-system symbolic access. SystemTap can also handle simple tracing

jobs.

Current project members include Red Hat, IBM, Hitachi, and Oracle.

(3) 獲取原始碼

git clone git://sourceware.org/git/systemtap.git

安裝

(1) Ubuntu發行版

1. 安裝systemtap包

apt-get install systemtap

2. 安裝依賴包

gcc：C語言編譯器

elfutils：提供分析除錯資訊的庫函式

linux-headers-generic：編譯核心模組所需的核心標頭檔案以及模組配置資訊

3. 安裝核心除錯資訊(kernel-debuginfo)

kernel-debuginfo提供了除錯核心所需的符號表，如果沒有安裝的話SystemTap的威力就會大打折扣，

只能提供kprobes系列的功能。

下載對應的核心版本，我的是linux-image-3.11.0-12-generic-dbgsym_3.11.0-12.19_amd64.ddeb

下載後安裝：dpkg -i linux-image-3.11.0-12-generic-dbgsym_3.11.0-12.19_amd64.ddeb

4. 驗證

stap -ve 'probe kernel.function("do_fork") { print("hello world\n") exit() }'

如果沒有提示錯誤，就是安裝成功了。

(2) CentOS/RedHat發行版

使用yum安裝下列rpm包即可：

systemtap：SystemTap包

gcc：C語言編譯器

elfutils：提供庫函式來分析除錯資訊

kernel-devel：編譯核心模組所需的核心標頭檔案及模組配置資訊

kernel-debuginfo：提供所需的核心除錯資訊來定位核心函式和變數的位置

使用

一些例子SystemTap的簡單例子。

(1) stap

通常直接使用stap執行用SystemTap語法編寫的指令碼即可。

stap - systemtap script translator/driver

stap test.stp // .stp字尾的檔案是用SystemTap語法編寫的指令碼

指令碼主要元素：probe point + probe handler

stap [options] FILE // Run script in file 

stap [options] -e SCRIPT // Run given script.

stap [options] -l PROBE // List matching probes.

stap [options] -L PROBE // List matching probes and local variables.

常用選項

-h：幫助

-g：guru模式，嵌入式C程式碼需要

-m：指定編譯成的模組名稱

 -v：add verbosity to all passes

-k：不刪除臨時目錄

-p NUM：stop after pass NUM 1-5, instead of 5 (parse, elaborate, translate, compile, run)

-b：bulk (percpu file) mode, 使用RelayFS將資料從核心空間傳輸到使用者空間

-o FILE：輸出到指定檔案，而不是stdout

-c CMD：start the probes, run CMD, and exit when it finishes

stap是SystemTap的前端，當出現以下情況時退出：

1. The user interrupts the script with a CTRL-C.

2. The script executes the exit() function.

3. The script encounters a sufficient number of soft errors.

4. The monitored command started with the stap program's -c option exits.

(2) staprun

如果我們的輸入不是.stp指令碼，而是一個用stap生成的模組，那麼就用staprun來執行。

staprun - systemtap runtime

staprun [OPTIONS] MODULE [MODULE-OPTIONS]

staprun的作用：

The staprun program is the back-end of the Systemtap tool. It expects a kernel module produced by

the front-end stap tool.

Splitting the systemtap tool into a front-end and a back-end allows a user to compile a systemtap script

on a development machine that has the kernel debugging information (need to compile the script) and

then transfer the resulting kernel module to a production machine that doesn't have any development

tools or kernel debugging information installed.

staprun is a part of the SystemTap package, dedicated to module loading and unloading and kernel-to-user

data transfer.

常用選項

-o FILE：Send output to FILE. 

-D：Run in background. This requires '-o' option.

(3) 監測核心函式

一個簡單指令碼，每當核心函式do_fork()被呼叫時，顯示呼叫它的程序名、程序ID、函式引數。

global proc_counter

probe begin {
	print("Started monitoring creation of new processes...Press ^C to terminate\n")
	printf("%-25s %-10s %-s\n", "Process Name", "Process ID", "Clone Flags")
}

probe kernel.function("do_fork") {
	proc_counter++
	printf("%-25s %-10d 0x%-x\n", execname(), pid(), $clone_flags)
}

probe end {
	printf("\n%d processes forked during the observed period\n", proc_counter)
}

(4) 監測系統呼叫

一個簡單指令碼，顯示4秒內open系統呼叫的資訊：呼叫程序名、程序ID、函式引數。

probe syscall.open
{
	printf("%s(%d) open(%s)\n", execname(), pid(), argstr)
}

probe timer.ms(4000) # after 4 seconds
{
	exit()
}

(5) 監測原始檔中所有函式入口和出口

括號內的探測點描述包含三個部分：

function name part：函式名

@file name part：檔名

function line part：所在行號

例如：

probe kernel.function("*@net/socket.c") {}
probe kernel.function("*@net/socket.c").return {}

這裡指定函式名為任意（用*表示），指定檔名為net/socket.c，探測函式的入口和返回。

還可以用“：行號”來指定行號。

(6) 查詢匹配的核心函式和變數

查詢名字中包含nit的核心函式：

stap -l 'kernel.function("*nit*")'

查詢名字中包含nit的核心函式和變數：

stap -L 'kernel.function("*nit*")'

(7) 自帶的用例集

/root/systemtap/testsuite/systemtap.examples/，包含了許多用例指令碼。

主要有幾個方面：

network、io、interrupt、locks、memory、process、virtualization等

(8) 監控所有程序的收發包情況

global recv, xmit

probe begin {
	printf("Starting network capture...Press ^C to terminate\n")
}

probe netdev.receive {
	recv[dev_name, pid(), execname()] <<< length
}

probe netdev.transmit {
	xmit[dev_name, pid(), execname()] <<< length
}

probe end {
	printf("\nCapture terminated\n\n")
	printf("%-5s %-15s %-10s %-10s %-10s\n", 
		"If", "Process", "Pid", "RcvPktCnt", "XmtPktCnt")
	
	foreach([dev, pid, name] in recv) {
		recvcnt = @count(recv[dev, pid, name])
		xmtcnt =  @count(xmit[dev, pid, name])
		printf("%-5s %-15s %-10d %-10d %-10d\n", dev, name, pid, recvcnt, xmtcnt)
	}
}

(9) Systemtap usage stories and interesting demos

官網提供的很多例子。