线上问题排查常见脚本工具

摘要：从所有的进程中找出最消耗的线程缺省个，打印出其线程栈用法要显示的线程栈数要显示的线程栈数指定的脚本找出库，即文件或目录类目录中的重复类。通过脚本参数指定目录，查找目录下文件，收集文件中文件以分析重复类。

</>复制代码 
从所有的 Java 进程中找出最消耗 CPU 的线程（缺省5个），打印出其线程栈

</>复制代码 
./show-busy-java-threads.sh
./show-busy-java-threads.sh -c <要显示的线程栈数>
./show-busy-java-threads.sh -c <要显示的线程栈数> -p <指定的Java Process>

</>复制代码 
#!/bin/bash
# @Function
# Find out the highest cpu consumed threads of java, and print the stack of these threads.
#
# @Usage
#   $ ./show-busy-java-threads.sh
#
# @author Jerry Lee
readonly PROG=`basename $0`
readonly -a COMMAND_LINE=("$0" "$@")
usage() {
    cat < /dev/null; then
    [ -z "$JAVA_HOME" ] && {
        redEcho "Error: jstack not found on PATH!"
        exit 1
    }
    ! [ -f "$JAVA_HOME/bin/jstack" ] && {
        redEcho "Error: jstack not found on PATH and $JAVA_HOME/bin/jstack file does NOT exists!"
        exit 1
    }
    ! [ -x "$JAVA_HOME/bin/jstack" ] && {
        redEcho "Error: jstack not found on PATH and $JAVA_HOME/bin/jstack is NOT executalbe!"
        exit 1
    }
    export PATH="$JAVA_HOME/bin:$PATH"
fi
readonly uuid=`date +%s`_${RANDOM}_$$
cleanupWhenExit() {
    rm /tmp/${uuid}_* &> /dev/null
}
trap "cleanupWhenExit" EXIT
printStackOfThreads() {
    local line
    local count=1
    while IFS=" " read -a line ; do
        local pid=${line[0]}
        local threadId=${line[1]}
        local threadId0x="0x`printf %x ${threadId}`"
        local user=${line[2]}
        local pcpu=${line[4]}
        local jstackFile=/tmp/${uuid}_${pid}
        [ ! -f "${jstackFile}" ] && {
            {
                if [ "${user}" == "${USER}" ]; then
                    jstack ${pid} > ${jstackFile}
                else
                    if [ $UID == 0 ]; then
                        sudo -u ${user} jstack ${pid} > ${jstackFile}
                    else
                        redEcho "[$((count++))] Fail to jstack Busy(${pcpu}%) thread(${threadId}/${threadId0x}) stack of java process(${pid}) under user(${user})."
                        redEcho "User of java process($user) is not current user($USER), need sudo to run again:"
                        yellowEcho "    sudo ${COMMAND_LINE[@]}"
                        echo
                        continue
                    fi
                fi
            } || {
                redEcho "[$((count++))] Fail to jstack Busy(${pcpu}%) thread(${threadId}/${threadId0x}) stack of java process(${pid}) under user(${user})."
                echo
                rm ${jstackFile}
                continue
            }
        }
        blueEcho "[$((count++))] Busy(${pcpu}%) thread(${threadId}/${threadId0x}) stack of java process(${pid}) under user(${user}):"
        sed "/nid=${threadId0x} /,/^$/p" -n ${jstackFile}
    done
}
ps -Leo pid,lwp,user,comm,pcpu --no-headers | {
    [ -z "${pid}" ] &&
    awk "$4=="java"{print $0}" ||
    awk -v "pid=${pid}" "$1==pid,$4=="java"{print $0}"
} | sort -k5 -r -n | head --lines "${count}" | printStackOfThreads

</>复制代码 
找出Java Lib（Java库，即Jar文件）或Class目录（类目录）中的重复类。

通过脚本参数指定Libs目录，查找目录下Jar文件，
收集Jar文件中Class文件以分析重复类。可以指定多个Libs目录。

注意，只会查找这个目录下Jar文件，不会查找子目录下Jar文件。
因为Libs目录一般不会用子目录再放Jar，这样也避免把去查找不期望Jar。
通过 -c 选项指定 Class 目录，直接收集这个目录下的 Class 文件以分析重复类。可以指定多个目录。

</>复制代码 
# 查找当前目录下所有Jar中的重复类
./show-duplicate-java-classes
查找多个指定目录下所有Jar中的重复类
./show-duplicate-java-classes path/to/lib_dir1 /path/to/lib_dir2
查找多个指定Class目录下的重复类。 class 目录 通过 -c 选项指定
./show-duplicate-java-classes -c path/to/class_dir1 -c /path/to/class_dir2
查找指定Class目录和指定目录下所有Jar中的重复类的Jar
./show-duplicate-java-classes path/to/lib_dir1 /path/to/lib_dir2 -c path/to/class_dir1 -c path/to/class_dir2

</>复制代码 
#!/usr/bin/env python
# -*- coding: utf-8 -*-
__author__ = "tg123"
from glob import glob
from os import walk
from zipfile import ZipFile
from os.path import relpath, isdir
from optparse import OptionParser
def list_jar_file_under_lib_dirs(libs):
    jar_files = set()
    for lib in libs:
        if isdir(lib):
            jar_files |= {f for f in glob(lib + "/*.jar")}
        else:
            jar_files.add(lib)
    return jar_files
def list_class_under_jar_file(jar_file):
    return {f for f in ZipFile(jar_file).namelist() if f.lower().endswith(".class")}
def list_class_under_class_dir(class_dir):
    return {relpath(dir_path + "/" + filename, class_dir)
            for dir_path, _, file_names in walk(class_dir)
            for filename in file_names if filename.lower().endswith(".class")}
def expand_2_class_path(jar_files, class_dirs):
    java_class_2_class_paths = {}
    # list all classes in jar files
    for jar_file in jar_files:
        for class_file in list_class_under_jar_file(jar_file):
            java_class_2_class_paths.setdefault(class_file, set()).add(jar_file)
    # list all classes in class dir
    for class_dir in class_dirs:
        for class_file in list_class_under_class_dir(class_dir):
            java_class_2_class_paths.setdefault(class_file, set()).add(class_dir)
    return java_class_2_class_paths, jar_files | set(class_dirs)
def find_duplicate_classes(java_class_2_class_paths):
    class_path_2_duplicate_classes = {}
    for java_class, class_paths in list(java_class_2_class_paths.items()):
        if len(class_paths) > 1:
            classes = class_path_2_duplicate_classes.setdefault(frozenset(class_paths), set())
            classes.add(java_class)
    return class_path_2_duplicate_classes
def print_class_paths(class_paths):
    print()
    print("=" * 80)
    print("class paths to find:")
    print("=" * 80)
    for idx, class_path in enumerate(class_paths):
        print(("%-3d: %s" % (idx + 1, class_path)))
if __name__ == "__main__":
    optionParser = OptionParser("usage: %prog "
                                "[-c class-dir1 [-c class-dir2] ...] "
                                "[lib-dir1|jar-file1 [lib-dir2|jar-file2] ...]")
    optionParser.add_option("-c", "--class-dir", dest="class_dirs", default=[], action="append", help="add class dir")
    options, libs = optionParser.parse_args()
    if not options.class_dirs and not libs:
        libs = ["."]
    java_class_2_class_paths, class_paths = expand_2_class_path(
        list_jar_file_under_lib_dirs(libs), options.class_dirs)
    class_path_2_duplicate_classes = find_duplicate_classes(java_class_2_class_paths)
    if not class_path_2_duplicate_classes:
        print("COOL! No duplicate classes found!")
        print_class_paths(class_paths)
        exit()
    print("Found duplicate classes in below class path:")
    for idx, jars in enumerate(class_path_2_duplicate_classes):
        print("%-3d(%d@%d): %s" % (idx + 1, len(class_path_2_duplicate_classes[jars]), len(jars), " ".join(jars)))
    print()
    print("=" * 80)
    print("Duplicate classes detail info:")
    print("=" * 80)
    for idx, (jars, classes) in enumerate(class_path_2_duplicate_classes.items()):
        print("%-3d(%d@%d): %s" % (idx + 1, len(class_path_2_duplicate_classes[jars]), len(jars), " ".join(jars)))
        for i, c in enumerate(classes):
            print("
%-3d %s" % (i + 1, c))
    print_class_paths(class_paths)
    exit(1)

参考：https://github.com/oldratlee/...