SystemServer的Watchdog

  在我们的认知中，Watchdog 是 Linux 系统一个很重要的机制，其目的是监测系统运行的情况，一旦出现锁死，死机的情况，能及时重启机器（取决于设置策略），并收集crash dump。在Android的SystemServer中，也存在一个Watchdog用来监控一些重要的线程，一旦这些线程长时间阻塞，就会杀掉SystemServer进程，导致Android重启。
  Watchdog是一个单例：

/frameworks/base/services/core/java/com/android/server/Watchdog.java

    public static Watchdog getInstance() {
        if (sWatchdog == null) {
            sWatchdog = new Watchdog();
        }

        return sWatchdog;
    }

  Watchdog内部维持了一个mHandlerCheckers的ArrayList，这个ArrayList的元素是HandlerChecker。HandlerChecker的构造函数接受三个参数：1.一个线程的Handler；2.名字；3.超时时间。
  可以看到FgThread，SystemServer主线程，UiThread，IoThread和DisplayThread的Handler都被作为入参构建一个HandlerChecker，保存在mHandlerCheckers中，从而纳入Watchdog的监控管理中。

/frameworks/base/services/core/java/com/android/server/Watchdog.java

    private Watchdog() {
        super("watchdog");
        // Initialize handler checkers for each common thread we want to check.  Note
        // that we are not currently checking the background thread, since it can
        // potentially hold longer running operations with no guarantees about the timeliness
        // of operations there.

        // The shared foreground thread is the main checker.  It is where we
        // will also dispatch monitor checks and do other work.
        mMonitorChecker = new HandlerChecker(FgThread.getHandler(),
                "foreground thread", DEFAULT_TIMEOUT);
        mHandlerCheckers.add(mMonitorChecker);
        // Add checker for main thread.  We only do a quick check since there
        // can be UI running on the thread.
        mHandlerCheckers.add(new HandlerChecker(new Handler(Looper.getMainLooper()),
                "main thread", DEFAULT_TIMEOUT));
        // Add checker for shared UI thread.
        mHandlerCheckers.add(new HandlerChecker(UiThread.getHandler(),
                "ui thread", DEFAULT_TIMEOUT));
        // And also check IO thread.
        mHandlerCheckers.add(new HandlerChecker(IoThread.getHandler(),
                "i/o thread", DEFAULT_TIMEOUT));
        // And the display thread.
        mHandlerCheckers.add(new HandlerChecker(DisplayThread.getHandler(),
                "display thread", DEFAULT_TIMEOUT));
    }

/frameworks/base/services/core/java/com/android/server/Watchdog.java

        HandlerChecker(Handler handler, String name, long waitMaxMillis) {
            mHandler = handler;
            mName = name;
            mWaitMax = waitMaxMillis;
            mCompleted = true;
        }

  HandlerChecker接受一个Handler参数，目的是检测Handler对应的Looper有没有长时间阻塞。此外，HandlerChecker还拥有mMonitors成员，用来检测一个线程有没有长时间阻塞在Synchronized关键字上。添加一个Monitor的方式是调用Watchdog#addMonitor，monitor是一个仅包含了moitor方法的接口。这些Monitor都被加在一个名为mMonitorChecker的HandlerChecker中。的我们熟知的ActivityManagerService，WindowManagerService等一些重要的Service类均实现了Watchdog.Monitor的接口。例如：

/frameworks/base/services/core/java/com/android/server/am/ActivityManagerService.java

    /** In this method we try to acquire our lock to make sure that we have not deadlocked */
    public void monitor() {
        synchronized (this) { }
    }

/frameworks/base/services/core/java/com/android/server/wm/WindowManagerService.java

    // Called by the heartbeat to ensure locks are not held indefnitely (for deadlock detection).
    @Override
    public void monitor() {
        synchronized (mWindowMap) { }
    }

  Watchdog继承自Thread类，看下其run方法：

/frameworks/base/services/core/java/com/android/server/Watchdog.java

    @Override
    public void run() {
        boolean waitedHalf = false;
        while (true) {
            final ArrayList<HandlerChecker> blockedCheckers;
            final String subject;
            final boolean allowRestart;
            int debuggerWasConnected = 0;
            synchronized (this) {
                long timeout = CHECK_INTERVAL;
                // Make sure we (re)spin the checkers that have become idle within
                // this wait-and-check interval
                for (int i=0; i<mHandlerCheckers.size(); i++) {
                    HandlerChecker hc = mHandlerCheckers.get(i);
                    hc.scheduleCheckLocked();
                }

                if (debuggerWasConnected > 0) {
                    debuggerWasConnected--;
                }

                // NOTE: We use uptimeMillis() here because we do not want to increment the time we
                // wait while asleep. If the device is asleep then the thing that we are waiting
                // to timeout on is asleep as well and won't have a chance to run, causing a false
                // positive on when to kill things.
                long start = SystemClock.uptimeMillis();
                while (timeout > 0) {
                    if (Debug.isDebuggerConnected()) {
                        debuggerWasConnected = 2;
                    }
                    try {
                        wait(timeout);
                    } catch (InterruptedException e) {
                        Log.wtf(TAG, e);
                    }
                    if (Debug.isDebuggerConnected()) {
                        debuggerWasConnected = 2;
                    }
                    timeout = CHECK_INTERVAL - (SystemClock.uptimeMillis() - start);
                }

                final int waitState = evaluateCheckerCompletionLocked();
                if (waitState == COMPLETED) {
                    // The monitors have returned; reset
                    waitedHalf = false;
                    continue;
                } else if (waitState == WAITING) {
                    // still waiting but within their configured intervals; back off and recheck
                    continue;
                } else if (waitState == WAITED_HALF) {
                    if (!waitedHalf) {
                        // We've waited half the deadlock-detection interval.  Pull a stack
                        // trace and wait another half.
                        ArrayList<Integer> pids = new ArrayList<Integer>();
                        pids.add(Process.myPid());
                        ActivityManagerService.dumpStackTraces(true, pids, null, null,
                                NATIVE_STACKS_OF_INTEREST);
                        waitedHalf = true;
                    }
                    continue;
                }

                // something is overdue!
                blockedCheckers = getBlockedCheckersLocked();
                subject = describeCheckersLocked(blockedCheckers);
                allowRestart = mAllowRestart;
            }

            // If we got here, that means that the system is most likely hung.
            // First collect stack traces from all threads of the system process.
            // Then kill this process so that the system will restart.
            EventLog.writeEvent(EventLogTags.WATCHDOG, subject);

            ArrayList<Integer> pids = new ArrayList<Integer>();
            pids.add(Process.myPid());
            if (mPhonePid > 0) pids.add(mPhonePid);
            // Pass !waitedHalf so that just in case we somehow wind up here without having
            // dumped the halfway stacks, we properly re-initialize the trace file.
            final File stack = ActivityManagerService.dumpStackTraces(
                    !waitedHalf, pids, null, null, NATIVE_STACKS_OF_INTEREST);

            // Give some extra time to make sure the stack traces get written.
            // The system's been hanging for a minute, another second or two won't hurt much.
            SystemClock.sleep(2000);

            // Pull our own kernel thread stacks as well if we're configured for that
            if (RECORD_KERNEL_THREADS) {
                dumpKernelStackTraces();
            }

            // Trigger the kernel to dump all blocked threads, and backtraces on all CPUs to the kernel log
            doSysRq('w');
            doSysRq('l');

            // Try to add the error to the dropbox, but assuming that the ActivityManager
            // itself may be deadlocked.  (which has happened, causing this statement to
            // deadlock and the watchdog as a whole to be ineffective)
            Thread dropboxThread = new Thread("watchdogWriteToDropbox") {
                    public void run() {
                        mActivity.addErrorToDropBox(
                                "watchdog", null, "system_server", null, null,
                                subject, null, stack, null);
                    }
                };
            dropboxThread.start();
            try {
                dropboxThread.join(2000);  // wait up to 2 seconds for it to return.
            } catch (InterruptedException ignored) {}

            IActivityController controller;
            synchronized (this) {
                controller = mController;
            }
            if (controller != null) {
                Slog.i(TAG, "Reporting stuck state to activity controller");
                try {
                    Binder.setDumpDisabled("Service dumps disabled due to hung system process.");
                    // 1 = keep waiting, -1 = kill system
                    int res = controller.systemNotResponding(subject);
                    if (res >= 0) {
                        Slog.i(TAG, "Activity controller requested to coninue to wait");
                        waitedHalf = false;
                        continue;
                    }
                } catch (RemoteException e) {
                }
            }

            // Only kill the process if the debugger is not attached.
            if (Debug.isDebuggerConnected()) {
                debuggerWasConnected = 2;
            }
            if (debuggerWasConnected >= 2) {
                Slog.w(TAG, "Debugger connected: Watchdog is *not* killing the system process");
            } else if (debuggerWasConnected > 0) {
                Slog.w(TAG, "Debugger was connected: Watchdog is *not* killing the system process");
            } else if (!allowRestart) {
                Slog.w(TAG, "Restart not allowed: Watchdog is *not* killing the system process");
            } else {
                Slog.w(TAG, "*** WATCHDOG KILLING SYSTEM PROCESS: " + subject);
                for (int i=0; i<blockedCheckers.size(); i++) {
                    Slog.w(TAG, blockedCheckers.get(i).getName() + " stack trace:");
                    StackTraceElement[] stackTrace
                            = blockedCheckers.get(i).getThread().getStackTrace();
                    for (StackTraceElement element: stackTrace) {
                        Slog.w(TAG, "    at " + element);
                    }
                }
                Slog.w(TAG, "*** GOODBYE!");
                Process.killProcess(Process.myPid());
                System.exit(10);
            }

            waitedHalf = false;
        }
    }

  首先，遍历所有的HandlerChecker，调用其scheduleCheckLocked方法进行阻塞检查。当一个HandlerChecker不是mMonitorChecker且内部的Handler对应的Looper处于空闲状态时，直接返回。mCompleted表示Monitor阻塞检查是否已通过，若为true，表示Monitor阻塞检查已通过。mCompleted初始化时为true，在Monitor阻塞检查前置为false，在Monitor阻塞检查已通过置为true。
  Monitor阻塞检查就是执行HandlerChecker的run函数，就是将mMonitors里面每个Monitor的monitor方法执行一遍。

/frameworks/base/services/core/java/com/android/server/Watchdog.java

        public void scheduleCheckLocked() {
            if (mMonitors.size() == 0 && mHandler.getLooper().isIdling()) {
                // If the target looper is or just recently was idling, then
                // there is no reason to enqueue our checker on it since that
                // is as good as it not being deadlocked.  This avoid having
                // to do a context switch to check the thread.  Note that we
                // only do this if mCheckReboot is false and we have no
                // monitors, since those would need to be executed at this point.
                mCompleted = true;
                return;
            }

            if (!mCompleted) {
                // we already have a check in flight, so no need
                return;
            }

            mCompleted = false;
            mCurrentMonitor = null;
            mStartTime = SystemClock.uptimeMillis();
            mHandler.postAtFrontOfQueue(this);
        }

/frameworks/base/services/core/java/com/android/server/Watchdog.java

        @Override
        public void run() {
            final int size = mMonitors.size();
            for (int i = 0 ; i < size ; i++) {
                synchronized (Watchdog.this) {
                    mCurrentMonitor = mMonitors.get(i);
                }
                mCurrentMonitor.monitor();
            }

            synchronized (Watchdog.this) {
                mCompleted = true;
                mCurrentMonitor = null;
            }
        }
    

  接下来，睡眠30s，醒来后调用evaluateCheckerCompletionLocked检查mHandlerCheckers的各个HandlerChecker的执行状态。如果完成了Monitor阻塞检查，getCompletionStateLocked会返回COMPLETED；否则，根据当前时间和发起Monitor阻塞检查的时间的差值决定getCompletionStateLocked的返回值：小于30s的返回WAITING，大于30s小于60s的返回WAITED_HALF，大于或等于60s的返回OVERDUE。

/frameworks/base/services/core/java/com/android/server/Watchdog.java

    private int evaluateCheckerCompletionLocked() {
        int state = COMPLETED;
        for (int i=0; i<mHandlerCheckers.size(); i++) {
            HandlerChecker hc = mHandlerCheckers.get(i);
            state = Math.max(state, hc.getCompletionStateLocked());
        }
        return state;
    }

  若返回的是COMPLETED，则调用continue进行新的循环，即是重新开始新的检测步骤。若返回的是WAITING，则调用continue继续进行检测步骤，这时evaluateCheckerCompletionLocked返回的是WAITED_HALF，这个时候需要调用ActivityManagerService#dumpStackTraces dump出第一次的trace，然后调用continue继续进行检测步骤。再次执行到这里时，evaluateCheckerCompletionLocked返回的是OVERDUE，之后便会dump出更加详细的信息，进行善后处理工作，因为确实有被监控的线程阻塞了。ActivityManagerService#dumpStackTraces会再次被调用，和第一次被调用不同的是，第一次被调用时会清空当前的traces.txt文件然后写入，第二次被调用时会将内容添加到第一次写入的内容后面。