mongod 启动流程

    根据工作需要，开始研究起了mongo。以前用过一些开源，也看过一些开源代码，也看过别人的源码分析，都是大神，甚是羡慕。上一篇中提到的几篇文章写的很好，突然就感觉到其实读代码写记录是最好的。今天从mongo开始，先从简单的来，中间有错误的话，欢迎大家指出，不胜感激。如果有幸被转载，请注明出处，熬夜写博客真心很痛苦。

    先从mongo编译后生成的程序开始吧，我的是2.5.4的源码。刚开始编完后在bin目录中有很多可执行文件，比如mongo/mongos/mongod/mongostat/mongotop等，既然是可执行文件，一定有main函数，搜索之后就可以发现在哪些目录中。研究源码流程也就从main函数开始。

    mongod是mongo的数据库存储程序，位于src/mongo/db/db.cpp。

    主程序代码很简单：

int main(int argc, char* argv[], char** envp) {
    int exitCode = mongoDbMain(argc, argv, envp);
    ::_exit(exitCode);
}

    三个参数的main函数，第三个参数是环境变量，可以循环将其打印出来，是a=b型的键值对，需要继续解析才能使用。

    mongoDbMain函数：

static int mongoDbMain(int argc, char* argv[], char **envp) {
    static StaticObserver staticObserver;

    getcurns = ourgetns;

    setupSignalHandlers();

    dbExecCommand = argv[0];

    srand(curTimeMicros());

    {
        unsigned x = 0x12345678;
        unsigned char& b = (unsigned char&) x;
        if ( b != 0x78 ) {
            out() << "big endian cpus not yet supported" << endl;
            return 33;
        }
    }

    if( argc == 1 )
        cout << dbExecCommand << " --help for help and startup options" << endl;

    mongo::runGlobalInitializersOrDie(argc, argv, envp);
    startupConfigActions(std::vector<std::string>(argv, argv + argc));
    cmdline_utils::censorArgvArray(argc, argv);

    if (!initializeServerGlobalState())
        ::_exit(EXIT_FAILURE);

    // Per SERVER-7434, startSignalProcessingThread() must run after any forks
    // (initializeServerGlobalState()) and before creation of any other threads.
    startSignalProcessingThread();

    dataFileSync.go();

#if defined(_WIN32)
    if (ntservice::shouldStartService()) {
        ntservice::startService();
        // exits directly and so never reaches here either.
    }
#endif

    StartupTest::runTests();
    initAndListen(serverGlobalParams.port);
    dbexit(EXIT_CLEAN);
    return 0;
}

    staticObserver虽然名字是静态观察，但实现很简单，具体作用还不清楚，大概是头文件中说的那样。

    ourgentns是一个函数，获取当前的ns名，即db.table，会在log中调用。

    setupSignalHandlers是个函数，设置信号的处理，reportEvent和UnhandledException的设置。

    argv[0]是程序名字。    中间一段是测试是否是小端机器，目前不支持大端。

    mongo::runGlobalInitializersOrDie(argc,argv,envp)主要解析命令行，即启动参数，其中会读取配置文件，解析参数并赋值给变量，比如“cpu”，“noauth”，“quota”，“repair path”，“nohints”等。 全局性质的初始化，上面的参数中有需要执行的命令也会在这里执行。在mongo/src/mongo/base/initializer.cpp。

for (size_t i = 0; i < sortedNodes.size(); ++i) {
            InitializerFunction fn = _graph.getInitializerFunction(sortedNodes[i]);
            if (!fn) {
                return Status(ErrorCodes::InternalError,
                              "topSort returned a node that has no associated function: \"" +
                              sortedNodes[i] + '"');
            }
            status = fn(&context);
            if (Status::OK() != status)
                return status;
        }

    其中fn就是要执行的函数。

    censor，文档说是一种算法：Look for an equal sign in arg。

    initializeServerGlobalState，一些认证设置。

    startSignalProcessingThread监听异步信号SIGUSR1，默认会创建boost::thread it( signalProcessingThread );

    dataFileSync.go，定时同步文件。    

    runTests运行一些测试程序。

    initAndListen初始化各个模块，并开始监听事件，MessageServer开始运行。

    dbexit数据库退出。

     其中dataFileSync的go函数是class BackgroundJob的函数。这个类是后台线程分发，并执行子类的run方法。boost::bind比较复杂，简单说就是绑定一些参数到函数上。

BackgroundJob& BackgroundJob::go() {
        boost::thread t( boost::bind( &BackgroundJob::jobBody , this, _status ) );
        return *this;
    }
 

    然后jobBody会执行run。更多BackgroundJob的信息参考头文件说明。

    今天先到这里，明天继续分析完善。

    现在重点看下initAndListen，这个函数开启了各种监听，包括socket，port，还有检测等。

void initAndListen(int listenPort) {
        try {
            _initAndListen(listenPort);
        }
        catch ( DBException &e ) {
            log() << "exception in initAndListen: " << e.toString() << ", terminating" << endl;
            dbexit( EXIT_UNCAUGHT );
        }
        catch ( std::exception &e ) {
            log() << "exception in initAndListen std::exception: " << e.what() << ", terminating" << endl;
            dbexit( EXIT_UNCAUGHT );
        }
        catch ( int& n ) {
            log() << "exception in initAndListen int: " << n << ", terminating" << endl;
            dbexit( EXIT_UNCAUGHT );
        }
        catch(...) {
            log() << "exception in initAndListen, terminating" << endl;
            dbexit( EXIT_UNCAUGHT );
        }
    }

    参数是要监听的端口，在这传入的是serverGlobalParams.port,这里port的值是枚举类型的DefaultDBPort=27017.

void _initAndListen(int listenPort ) {

        Client::initThread("initandlisten");

        bool is32bit = sizeof(int*) == 4;

        {
            ProcessId pid = ProcessId::getCurrent();
            LogstreamBuilder l = log();
            l << "MongoDB starting : pid=" << pid
              << " port=" << serverGlobalParams.port
              << " dbpath=" << storageGlobalParams.dbpath;
            if( replSettings.master ) l << " master=" << replSettings.master;
            if( replSettings.slave )  l << " slave=" << (int) replSettings.slave;
            l << ( is32bit ? " 32" : " 64" ) << "-bit host=" << getHostNameCached() << endl;
        }
        DEV log() << "_DEBUG build (which is slower)" << endl;
        logStartupWarnings();
#if defined(_WIN32)
        printTargetMinOS();
#endif
        logProcessDetails();
        {
            stringstream ss;
            ss << endl;
            ss << "*********************************************************************" << endl;
            ss << " ERROR: dbpath (" << storageGlobalParams.dbpath << ") does not exist." << endl;
            ss << " Create this directory or give existing directory in --dbpath." << endl;
            ss << " See http://dochub.mongodb.org/core/startingandstoppingmongo" << endl;
            ss << "*********************************************************************" << endl;
            uassert(10296,  ss.str().c_str(), boost::filesystem::exists(storageGlobalParams.dbpath));
        }
        {
            stringstream ss;
            ss << "repairpath (" << storageGlobalParams.repairpath << ") does not exist";
            uassert(12590,  ss.str().c_str(),
                    boost::filesystem::exists(storageGlobalParams.repairpath));
        }

        // TODO check non-journal subdirs if using directory-per-db
        checkReadAhead(storageGlobalParams.dbpath);

        acquirePathLock(mongodGlobalParams.repair);
        boost::filesystem::remove_all(storageGlobalParams.dbpath + "/_tmp/");

        FileAllocator::get()->start();

        // TODO:  This should go into a MONGO_INITIALIZER once we have figured out the correct
        // dependencies.
        if (snmpInit) {
            snmpInit();
        }

        MONGO_ASSERT_ON_EXCEPTION_WITH_MSG( clearTmpFiles(), "clear tmp files" );

        dur::startup();

        if (storageGlobalParams.durOptions & StorageGlobalParams::DurRecoverOnly)
            return;

        unsigned long long missingRepl = checkIfReplMissingFromCommandLine();
        if (missingRepl) {
            log() << startupWarningsLog;
            log() << "** WARNING: mongod started without --replSet yet " << missingRepl
                  << " documents are present in local.system.replset" << startupWarningsLog;
            log() << "**          Restart with --replSet unless you are doing maintenance and no"
                  << " other clients are connected." << startupWarningsLog;
            log() << "**          The TTL collection monitor will not start because of this." << startupWarningsLog;
            log() << "**          For more info see http://dochub.mongodb.org/core/ttlcollections" << startupWarningsLog;
            log() << startupWarningsLog;
        }

        if (mongodGlobalParams.scriptingEnabled) {
            ScriptEngine::setup();
            globalScriptEngine->setCheckInterruptCallback( jsInterruptCallback );
            globalScriptEngine->setGetCurrentOpIdCallback( jsGetCurrentOpIdCallback );
        }

        // On replica set members we only clear temp collections on DBs other than "local" during
        // promotion to primary. On pure slaves, they are only cleared when the oplog tells them to.
        // The local DB is special because it is not replicated.  See SERVER-10927 for more details.
        const bool shouldClearNonLocalTmpCollections = !(missingRepl
                                                         || replSettings.usingReplSets()
                                                         || replSettings.slave == SimpleSlave);
        repairDatabasesAndCheckVersion(shouldClearNonLocalTmpCollections);

        if (mongodGlobalParams.upgrade)
            return;

        uassertStatusOK(getGlobalAuthorizationManager()->initialize());

        /* this is for security on certain platforms (nonce generation) */
        srand((unsigned) (curTimeMicros() ^ startupSrandTimer.micros()));

        snapshotThread.go();
        d.clientCursorMonitor.go();
        PeriodicTask::startRunningPeriodicTasks();
        if (missingRepl) {
            // a warning was logged earlier
        }
        else {
            startTTLBackgroundJob();
        }

#ifndef _WIN32
        mongo::signalForkSuccess();
#endif

        if(getGlobalAuthorizationManager()->isAuthEnabled()) {
            // open admin db in case we need to use it later. TODO this is not the right way to
            // resolve this.
            Client::WriteContext c("admin", storageGlobalParams.dbpath);
        }

        getDeleter()->startWorkers();

        // Starts a background thread that rebuilds all incomplete indices. 
        indexRebuilder.go(); 

        listen(listenPort);

        // listen() will return when exit code closes its socket.
        exitCleanly(EXIT_NET_ERROR);
    }

    Client::initThread用于创建一个数据库操作的cleint。

    ProcessId::getCurrent()获取pid并检测是否32位机器。logStartupWarnings() logProcessDetails()记录一些log信息。

    checkReadAhead()检查dbpath。

    acquirePathLock，如果使用每一个db一个目录，则检查非journal的子目录。

    remove_all删除db目录中的临时目录。

    FileAllocator::get()->start()位于mongo/src/mongo/util/file_allocator.cpp.FileAllocator::get()是单例模式，start会执行run方法。

FileAllocator* FileAllocator::get(){
        if ( ! _instance )
            _instance = new FileAllocator();
        return _instance;
    }

void FileAllocator::start() {
        boost::thread t( boost::bind( &FileAllocator::run , this ) );
    }

void FileAllocator::run( FileAllocator * fa ) {
        setThreadName( "FileAllocator" );
        {
            // initialize unique temporary file name counter
            // TODO: SERVER-6055 -- Unify temporary file name selection
            SimpleMutex::scoped_lock lk(_uniqueNumberMutex);
            _uniqueNumber = curTimeMicros64();
        }
        while( 1 ) {
            {
                scoped_lock lk( fa->_pendingMutex );
                if ( fa->_pending.size() == 0 )
                    fa->_pendingUpdated.wait( lk.boost() );
            }
            while( 1 ) {
                string name;
                long size;
                {
                    scoped_lock lk( fa->_pendingMutex );
                    if ( fa->_pending.size() == 0 )
                        break;
                    name = fa->_pending.front();
                    size = fa->_pendingSize[ name ];
                }

                string tmp;
                long fd = 0;
                try {
                    log() << "allocating new datafile " << name << ", filling with zeroes..." << endl;
                    
                    boost::filesystem::path parent = ensureParentDirCreated(name);
                    tmp = fa->makeTempFileName( parent );
                    ensureParentDirCreated(tmp);

#if defined(_WIN32)
                    fd = _open( tmp.c_str(), _O_RDWR | _O_CREAT | O_NOATIME, _S_IREAD | _S_IWRITE );
#else
                    fd = open(tmp.c_str(), O_CREAT | O_RDWR | O_NOATIME, S_IRUSR | S_IWUSR);
#endif
                    if ( fd < 0 ) {
                        log() << "FileAllocator: couldn't create " << name << " (" << tmp << ") " << errnoWithDescription() << endl;
                        uasserted(10439, "");
                    }

#if defined(POSIX_FADV_DONTNEED)
                    if( posix_fadvise(fd, 0, size, POSIX_FADV_DONTNEED) ) {
                        log() << "warning: posix_fadvise fails " << name << " (" << tmp << ") " << errnoWithDescription() << endl;
                    }
#endif

                    Timer t;

                    /* make sure the file is the full desired length */
                    ensureLength( fd , size );

                    close( fd );
                    fd = 0;

                    if( rename(tmp.c_str(), name.c_str()) ) {
                        const string& errStr = errnoWithDescription();
                        const string& errMessage = str::stream()
                                << "error: couldn't rename " << tmp
                                << " to " << name << ' ' << errStr;
                        msgasserted(13653, errMessage);
                    }
                    flushMyDirectory(name);

                    log() << "done allocating datafile " << name << ", "
                          << "size: " << size/1024/1024 << "MB, "
                          << " took " << ((double)t.millis())/1000.0 << " secs"
                          << endl;

                    // no longer in a failed state. allow new writers.
                    fa->_failed = false;
                }
                catch ( const std::exception& e ) {
                    log() << "error: failed to allocate new file: " << name
                          << " size: " << size << ' ' << e.what()
                          << ".  will try again in 10 seconds" << endl;
                    if ( fd > 0 )
                        close( fd );
                    try {
                        if ( ! tmp.empty() )
                            boost::filesystem::remove( tmp );
                        boost::filesystem::remove( name );
                    } catch ( const std::exception& e ) {
                        log() << "error removing files: " << e.what() << endl;
                    }
                    scoped_lock lk( fa->_pendingMutex );
                    fa->_failed = true;
                    // not erasing from pending
                    fa->_pendingUpdated.notify_all();
                    
                    
                    sleepsecs(10);
                    continue;
                }

                {
                    scoped_lock lk( fa->_pendingMutex );
                    fa->_pendingSize.erase( name );
                    fa->_pending.pop_front();
                    fa->_pendingUpdated.notify_all();
                }
            }
        }
    }

    在run中，_pending存放的可能是文件的名字，并且创建临时目录。

dur::startup();

    会开启journal线程，创建持久化目录和文件。

if (mongodGlobalParams.scriptingEnabled) {
            ScriptEngine::setup();
            globalScriptEngine->setCheckInterruptCallback( jsInterruptCallback );
            globalScriptEngine->setGetCurrentOpIdCallback( jsGetCurrentOpIdCallback );
        }

    ScriptEngine::setup()会创建v8脚本引擎，在mongo/src/mongo/scripting/engine_v8.cpp

snapshotThread.go();

    会开启快照线程，定时拍照，默认4秒。在/mongo/src/mongo/db/stats/snapshots.cpp

d.clientCursorMonitor::go();

    d是全局的DGlobals,获得clientCursorMonitor()时会创建ClientCursorMonitor对象，在mongo/src/mongo/db/clientcursor.cpp，主要检测内存状态，默认一分钟一次。

PeriodicTask::startRunningPerioodicTasks();

    启动定期执行任务的线程，创建PeriodicTaskRunner,检测shutdown信号，没有就等待，默认60秒。

if (missingRepl) {
            // a warning was logged earlier
        }
        else {
            startTTLBackgroundJob();
        }

    如果没有设置"--replSet"则之前会有警告，如果设置了这里会直行startTTLBackgroundJob();创建ttl监视，淘汰过期数据。

getDeleter()->startWorkers();

    启动deleter线程，绑定RangeDeleter::doWork()，删除给定namespace的文档，可以immediate也可以lazy删除。

listen(listenPort);

    这是监听socket的开始地方。

void listen(int port) {
        //testTheDb();
        MessageServer::Options options;
        options.port = port;
        options.ipList = serverGlobalParams.bind_ip;

        MessageServer * server = createServer( options , new MyMessageHandler() );
        server->setAsTimeTracker();
        // we must setupSockets prior to logStartup() to avoid getting too high
        // a file descriptor for our calls to select()
        server->setupSockets();

        logStartup();
        startReplication();
        if (serverGlobalParams.isHttpInterfaceEnabled)
            boost::thread web( boost::bind(&webServerThread, new RestAdminAccess() /* takes ownership */));

#if(TESTEXHAUST)
        boost::thread thr(testExhaust);
#endif
        server->run();
    }

    这个监听结构也是消息模式的，类似于android的handler机制。createServer()是关键一句。options包含要监听的ip和port，MyMessageHandler是收到消息后处理的类，继承MessageHandler,返回的是新创建的ProtMessageServer的父类，初始化时就给变量赋值。其中Listener是一个抽象出来的监听类，setupSockets()中会创建，设置，添加socket。

    还会启动log和replication。最后server的run开始真正进入了等待循环。用的select监听并处理数据。

run里面的accepted实际执行的是mongo/src/mongo/util/net/message_server_port.cpp中的acceptedMP().

try {
#ifndef __linux__  // TODO: consider making this ifdef _WIN32
                {
                    HandleIncomingMsgParam* himParam = new HandleIncomingMsgParam(p, _handler);
                    boost::thread thr(boost::bind(&handleIncomingMsg, himParam));
                }
.....

    其中创建了HandleIncomingMsgParam，并绑定处理函数handleIncomingMsg和参数himParam。当有消息过来时就可以处理了。

    先到这里吧。