# Linux Static Performance Tools Cheatsheet ## Table of Contents 1. [System Architecture Overview](#system-architecture-overview) 2. [Performance Analysis Tools by Layer](#performance-analysis-tools-by-layer) 3. [Hardware Layer Tools](#hardware-layer-tools) 4. [Operating System Layer Tools](#operating-system-layer-tools) 5. [Application Layer Tools](#application-layer-tools) 6. [Detailed Tool Reference](#detailed-tool-reference) ## System Architecture Overview The Linux system architecture consists of multiple layers, each with specific performance monitoring and analysis tools. Here's a detailed breakdown of each layer and its components: ### Hardware Layer - CPUs - DRAM - I/O Controllers - Network Controllers - Storage Devices ### Operating System Layer Components 1. Device Drivers 2. Block Device 3. Network Device 4. Virtual Memory 5. File Systems 6. TCP/UDP 7. IP Stack 8. System Call Interface 9. Scheduler 10. Clock Source ### Application Layer - Applications - System Libraries - VFS (Virtual File System) - Sockets ## Performance Analysis Tools by Layer ### Hardware Level Tools #### CPU Analysis Tools ```bash # CPU information and statistics /proc/cpuinfo # Detailed CPU information cpuid # CPU identification and features lscpu # CPU architecture information cpu-x # CPU identification tool # CPU Performance Monitoring numact1 # NUMA topology and statistics lstopo # Hardware topology viewer dmidecode # DMI table decoder ``` #### Memory Analysis ```bash # Memory statistics and information free -m # Memory usage overview vmstat # Virtual memory statistics dmidecode # Memory hardware information # Memory Performance Tools lstopo # Memory topology visualization numact1 # NUMA memory statistics ``` ### Operating System Level Tools #### File System Tools ```bash # Disk and Filesystem Analysis df -h # Filesystem usage lsblk # Block device information blockdev # Block device operations smartctl # SMART disk monitoring fdisk -l # Disk partition information lsusb # USB device information # Advanced Storage Analysis mdadm # RAID management and monitoring dmesg # Storage device messages ``` #### Network Analysis Tools ```bash # Basic Network Tools ip route # IP routing table tc # Traffic control ethtool # Network interface settings iwconfig # Wireless interface configuration lldptool # Link Layer Discovery Protocol tool # Network Statistics netstat # Network statistics ss # Socket statistics ip # IP configuration and statistics ``` #### System Call Interface Tools ```bash # System Call Monitoring strace # System call tracer ltrace # Library call tracer syscall # System call information sysctl # Kernel parameter configuration ``` #### Scheduler Analysis ```bash # Scheduler Tools schedtool # Process scheduler configuration chrt # Real-time scheduler priorities taskset # CPU affinity configuration ``` ### Application Level Tools #### Process Analysis ```bash # Process Monitoring top # Process activity monitoring htop # Interactive process viewer ps # Process status pstree # Process hierarchy view ``` #### System Library Analysis ```bash # Library Tools ldd # Shared library dependencies ldconfig # Library path configuration ld # Link editor for object files ``` ## Detailed Tool Reference ### System Information Tools #### sysctl ```bash # View all system parameters sysctl -a # View specific parameter sysctl kernel.hostname # Modify parameter sysctl -w kernel.hostname=newname # Load settings from file sysctl -p /etc/sysctl.conf ``` #### dmesg ```bash # View kernel messages dmesg # Clear kernel ring buffer dmesg -c # Show human-readable timestamps dmesg -T # Follow new messages dmesg -w ``` ### Storage Performance Tools #### smartctl ```bash # Check disk health smartctl -a /dev/sda # Run short self-test smartctl -t short /dev/sda # Show test results smartctl -l selftest /dev/sda # Enable SMART smartctl -s on /dev/sda ``` #### iostat ```bash # Basic I/O statistics iostat # Extended statistics iostat -x # Device utilization iostat -d # Updates every 2 seconds iostat 2 ``` ### Network Performance Tools #### ethtool ```bash # Show interface settings ethtool eth0 # Show interface statistics ethtool -S eth0 # Set interface speed ethtool -s eth0 speed 1000 # Check link status ethtool -i eth0 ``` #### tc (Traffic Control) ```bash # Show queueing discipline tc qdisc show # Add rate limiting tc qdisc add dev eth0 root tbf rate 1mbit burst 32kbit latency 400ms # Show filter rules tc filter show dev eth0 # Delete all rules tc qdisc del dev eth0 root ``` ### Process Management Tools #### schedtool ```bash # Show process scheduler info schedtool -p 1234 # Set SCHED_FIFO policy schedtool -F -p 50 1234 # Set CPU affinity schedtool -a 0-3 1234 # Show all scheduler policies schedtool -h ``` #### numact1 ```bash # Show NUMA topology numactl --hardware # Run command on specific node numactl --membind=0 command # Show NUMA statistics numactl --show # Set memory policy numactl --preferred=0 command ``` ### Virtual Memory Analysis #### vmstat ```bash # Memory statistics vmstat # Detailed memory info vmstat -s # Active/inactive memory vmstat -a # Disk statistics vmstat -d ``` ### Advanced Analysis Tools #### SystemTap ```bash # Create simple probe stap -e 'probe syscall.open { println(execname()) }' # Monitor file operations stap -e 'probe vfs.read { printf("%s read %d bytes\n", execname(), count) }' # Track process creation stap -e 'probe process.begin { printf("%s started\n", execname()) }' ``` #### perf ```bash # CPU performance counters perf stat command # System-wide sampling perf record -a sleep 10 # Show recorded data perf report # Live performance monitoring perf top ``` ### Hardware Management #### dmidecode ```bash # Show all hardware info dmidecode # Show memory info dmidecode -t memory # Show CPU info dmidecode -t processor # Show system info dmidecode -t system ``` ### Volume Management Tools #### MegaCli ```bash # Show RAID configuration MegaCli -CfgDsply -aALL # Check battery status MegaCli -AdpBbuCmd -aALL # Show drive information MegaCli -PDList -aALL # Check rebuild status MegaCli -PDRbld -ShowProg -physdrv[252:2] -aALL ``` ### Performance Monitoring Best Practices 1. **Regular Monitoring** - Set up baseline measurements - Monitor trends over time - Create alerts for anomalies 2. **Resource Usage** - Track CPU utilization - Monitor memory consumption - Watch I/O patterns - Analyze network traffic 3. **System Health** - Check system logs - Monitor hardware status - Track error rates - Verify service status 4. **Performance Optimization** - Identify bottlenecks - Tune system parameters - Optimize application settings - Balance resource allocation ### Common Performance Issues and Solutions 1. **CPU Bottlenecks** - Use `top` and `htop` to identify high CPU processes - Check process priorities with `nice` and `renice` - Monitor CPU scheduling with `schedtool` - Analyze system load with `uptime` 2. **Memory Problems** - Check memory usage with `free` and `vmstat` - Monitor swap usage - Analyze memory leaks - Tune kernel memory parameters 3. **Disk I/O Issues** - Monitor disk activity with `iostat` - Check disk health with `smartctl` - Analyze I/O patterns with `iotop` - Optimize filesystem parameters 4. **Network Performance** - Monitor bandwidth with `iftop` - Analyze connections with `netstat` - Check network errors with `ethtool` - Tune network parameters with `sysctl` ### Quick Reference Commands ```bash # System Overview top # Process activity htop # Interactive process viewer uptime # Load average dmesg # Kernel messages # Memory Analysis free -m # Memory usage vmstat # Virtual memory stats pmap # Process memory map smem # Memory reporting # Disk Operations iostat # I/O statistics iotop # I/O monitoring fdisk -l # Disk partitions df -h # Filesystem usage # Network Analysis netstat # Network statistics ss # Socket statistics iftop # Network bandwidth tcpdump # Packet analysis # Process Management ps aux # Process status pstree # Process tree nice # Process priority renice # Change priority # System Information uname -a # System information lscpu # CPU information lsblk # Block devices lspci # PCI devices ``` ### Additional Resources 1. **Documentation** - Man pages for each tool - Linux kernel documentation - Distribution-specific guides - Tool-specific documentation 2. **Performance Tuning Guides** - Red Hat Performance Tuning Guide - Ubuntu Server Guide - Linux Performance website - Kernel documentation 3. **Monitoring Tools** - Nagios - Zabbix - Prometheus - Grafana 4. **Training Resources** - Linux Foundation courses - Distribution certification programs - Online tutorials - Community forums