systemTap
采集工具
bcc
分析工具
sysstat
监控工具
netstat
网络监控工具
分类
- tools/argdist: Display function parameter values as a histogram or frequency count. Examples.
- tools/bashreadline: Print entered bash commands system wide. Examples.
- tools/bpflist: Display processes with active BPF programs and maps. Examples.
- tools/capable: Trace security capability checks. Examples.
- tools/compactsnoop: Trace compact zone events with PID and latency. Examples.
- tools/criticalstat: Trace and report long atomic critical sections in the kernel. Examples
- tools/deadlock: Detect potential deadlocks on a running process. Examples.
- tools/drsnoop: Trace direct reclaim events with PID and latency. Examples.
- tools/funccount: Count kernel function calls. Examples.
- tools/inject: Targeted error injection with call chain and predicates Examples.
- tools/klockstat: Traces kernel mutex lock events and display locks statistics. Examples.
- tools/opensnoop: Trace open() syscalls. Examples.
- tools/readahead: Show performance of read-ahead cache Examples.
- tools/reset-trace: Reset the state of tracing. Maintenance tool only. Examples.
- tools/stackcount: Count kernel function calls and their stack traces. Examples.
- tools/syncsnoop: Trace sync() syscall. Examples.
- tools/threadsnoop: List new thread creation. Examples.
- tools/tplist: Display kernel tracepoints or USDT probes and their formats. Examples.
- tools/trace: Trace arbitrary functions, with filters. Examples.
- tools/ttysnoop: Watch live output from a tty or pts device. Examples.
- tools/ucalls: Summarize method calls or Linux syscalls in high-level languages. Examples.
- tools/uflow: Print a method flow graph in high-level languages. Examples.
- tools/ugc: Trace garbage collection events in high-level languages. Examples.
- tools/uobjnew: Summarize object allocation events by object type and number of bytes allocated. Examples.
- tools/ustat: Collect events such as GCs, thread creations, object allocations, exceptions and more in high-level languages. Examples.
- tools/uthreads: Trace thread creation events in Java and raw pthreads. Examples.
Memory and Process Tools
- tools/execsnoop: Trace new processes via exec() syscalls. Examples.
- tools/exitsnoop: Trace process termination (exit and fatal signals). Examples.
- tools/killsnoop: Trace signals issued by the kill() syscall. Examples.
- tools/kvmexit: Display the exit_reason and its statistics of each vm exit. Examples.
- tools/memleak: Display outstanding memory allocations to find memory leaks. Examples.
- tools/oomkill: Trace the out-of-memory (OOM) killer. Examples.
- tools/pidpersec: Count new processes (via fork). Examples.
- tools/rdmaucma: Trace RDMA Userspace Connection Manager Access events. Examples.
- tools/shmsnoop: Trace System V shared memory syscalls. Examples.
- tools/slabratetop: Kernel SLAB/SLUB memory cache allocation rate top. Examples.
Performance and Time Tools
- tools/dbslower: Trace MySQL/PostgreSQL queries slower than a threshold. Examples.
- tools/dbstat: Summarize MySQL/PostgreSQL query latency as a histogram. Examples.
- tools/funcinterval: Time interval between the same function as a histogram. Examples.
- tools/funclatency: Time functions and show their latency distribution. Examples.
- tools/funcslower: Trace slow kernel or user function calls. Examples.
- tools/hardirqs: Measure hard IRQ (hard interrupt) event time. Examples.
- tools/mysqld_qslower: Trace MySQL server queries slower than a threshold. Examples.
- tools/ppchcalls: Summarize ppc hcall counts and latencies. Examples.
- tools/softirqs: Measure soft IRQ (soft interrupt) event time. Examples.
- tools/syscount: Summarize syscall counts and latencies. Examples.
CPU and Scheduler Tools
- tools/cpudist: Summarize on- and off-CPU time per task as a histogram. Examples
- tools/cpuunclaimed: Sample CPU run queues and calculate unclaimed idle CPU. Examples
- tools/llcstat: Summarize CPU cache references and misses by process. Examples.
- tools/offcputime: Summarize off-CPU time by kernel stack trace. Examples.
- tools/offwaketime: Summarize blocked time by kernel off-CPU stack and waker stack. Examples.
- tools/profile: Profile CPU usage by sampling stack traces at a timed interval. Examples.
- tools/runqlat: Run queue (scheduler) latency as a histogram. Examples.
- tools/runqlen: Run queue length as a histogram. Examples.
- tools/runqslower: Trace long process scheduling delays. Examples.
- tools/wakeuptime: Summarize sleep to wakeup time by waker kernel stack. Examples.
- tools/wqlat: Summarize work waiting latency on workqueue. Examples.
Network and Sockets Tools
- tools/gethostlatency: Show latency for getaddrinfo/gethostbyname[2] calls. Examples.
- tools/bindsnoop: Trace IPv4 and IPv6 bind() system calls (bind()). Examples.
- tools/netqtop tools/netqtop.c: Trace and display packets distribution on NIC queues. Examples.
- tools/sofdsnoop: Trace FDs passed through unix sockets. Examples.
- tools/solisten: Trace TCP socket listen. Examples.
- tools/sslsniff: Sniff OpenSSL written and readed data. Examples.
- tools/tcpaccept: Trace TCP passive connections (accept()). Examples.
- tools/tcpconnect: Trace TCP active connections (connect()). Examples.
- tools/tcpconnlat: Trace TCP active connection latency (connect()). Examples.
- tools/tcpdrop: Trace kernel-based TCP packet drops with details. Examples.
- tools/tcplife: Trace TCP sessions and summarize lifespan. Examples.
- tools/tcpretrans: Trace TCP retransmits and TLPs. Examples.
- tools/tcprtt: Trace TCP round trip time. Examples.
- tools/tcpstates: Trace TCP session state changes with durations. Examples.
- tools/tcpsubnet: Summarize and aggregate TCP send by subnet. Examples.
- tools/tcpsynbl: Show TCP SYN backlog. Examples.
- tools/tcptop: Summarize TCP send/recv throughput by host. Top for TCP. Examples.
- tools/tcptracer: Trace TCP established connections (connect(), accept(), close()). Examples.
- tools/tcpcong: Trace TCP socket congestion control status duration. Examples.
Storage and Filesystems Tools
- tools/bitesize: Show per process I/O size histogram. Examples.
- tools/cachestat: Trace page cache hit/miss ratio. Examples.
- tools/cachetop: Trace page cache hit/miss ratio by processes. Examples.
- tools/dcsnoop: Trace directory entry cache (dcache) lookups. Examples.
- tools/dcstat: Directory entry cache (dcache) stats. Examples.
- tools/biolatency: Summarize block device I/O latency as a histogram. Examples.
- tools/biotop: Top for disks: Summarize block device I/O by process. Examples.
- tools/biopattern: Identify random/sequential disk access patterns. Examples.
- tools/biosnoop: Trace block device I/O with PID and latency. Examples.
- tools/dirtop: File reads and writes by directory. Top for directories. Examples.
- tools/filelife: Trace the lifespan of short-lived files. Examples.
- tools/filegone: Trace why file gone (deleted or renamed). Examples.
- tools/fileslower: Trace slow synchronous file reads and writes. Examples.
- tools/filetop: File reads and writes by filename and process. Top for files. Examples.
- tools/mdflush: Trace md flush events. Examples.
- tools/mountsnoop: Trace mount and umount syscalls system-wide. Examples.
- tools/virtiostat: Show VIRTIO device IO statistics. Examples.
Filesystems Tools
- tools/btrfsdist: Summarize btrfs operation latency distribution as a histogram. Examples.
- tools/btrfsslower: Trace slow btrfs operations. Examples.
- tools/ext4dist: Summarize ext4 operation latency distribution as a histogram. Examples.
- tools/ext4slower: Trace slow ext4 operations. Examples.
- tools/nfsslower: Trace slow NFS operations. Examples.
- tools/nfsdist: Summarize NFS operation latency distribution as a histogram. Examples.
- tools/vfscount: Count VFS calls. Examples.
- tools/vfsstat: Count some VFS calls, with column output. Examples.
- tools/xfsdist: Summarize XFS operation latency distribution as a histogram. Examples.
- tools/xfsslower: Trace slow XFS operations. Examples.
- tools/zfsdist: Summarize ZFS operation latency distribution as a histogram. Examples.
- tools/zfsslower: Trace slow ZFS operations. Examples.
Networking
Examples:
- examples/networking/distributed_bridge/: Distributed bridge example.
- examples/networking/http_filter/: Simple HTTP filter example.
- examples/networking/simple_tc.py: Simple traffic control example.
- examples/networking/simulation.py: Simulation helper.
- examples/networking/neighbor_sharing/tc_neighbor_sharing.py examples/networking/neighbor_sharing/tc_neighbor_sharing.c: Per-IP classification and rate limiting.
- examples/networking/tunnel_monitor/: Efficiently monitor traffic flows.
- examples/networking/vlan_learning/vlan_learning.py examples/vlan_learning.c: Demux Ethernet traffic into worker veth+namespaces.
BPF Introspection
Tools that help to introspect BPF programs.
- introspection/bps.c: List all BPF programs loaded into the kernel. 'ps' for BPF programs. Examples.
CPU 常用相关
uptime
观看负载情况，一般后端开发人员不怎么用。1,5,15 分钟负载状态。
top
监控系统中最大消耗的进程。一般开发人员观看内存和cpu使用情况的。关注虚拟内存和cpu比。
vmstat
监控cpu整体使用情况
[root@lt coder]# vmstat -w 1
procs -----------------------memory---------------------- ---swap-- -----io---- -system-- --------cpu--------
r[线程数] b swpd free buff cache si so bi bo in cs[切换上下文] us[用户态占比] sy[系统态占比] id wa[等待io] st
1 0 0 7180468 2108 549120 0 0 46 39 70 80 1 0 99 0 0
0 0 0 7180444 2108 549120 0 0 0 0 83 118 0 0 100 0 0
0 0 0 7180212 2108 549120 0 0 0 10 273 484 1 1 98 0 0
mpstat
具体显示每个cpu情况
mpstat -P ALL 1
Linux 3.10.0-1160.92.1.el7.x86_64 (lt) 05/06/2024 _x86_64_ (2 CPU)
06:01:16 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
06:01:17 PM all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
06:01:17 PM 0 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00
06:01:17 PM 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
pidstat
pidstat 1
execsnoop
perf
内存相关
内存泄漏 valgrand
