golang 数组 最后一个_面试必问:Golang高阶Golang协程实现原理

b8bcc675e6ca93756874a1c5f1462c99.gif

1

01

引言

实现并发编程有进程,线程,IO多路复用的方式。(并发和并行我们这里不区分,如果CPU是多核的,可能在多个核同时进行,我们叫并行,如果是单核,需要排队切换,我们叫并发)。

1.1

进程和线程的区别

进程是计算机资源分配的最小单位,进程是对处理器资源(CPU),虚拟内存(1)的抽象,虚拟内存是对主存资源 (Memory) 和文件(2)的抽象,文件是对 I/O 设备的抽象。

虚拟内存是操作系统初始化后内部维护的一个程序加载空间,对于32位操作系统来说,也就是寄存器有32位的比特长度,虚拟内存中每个字节都有一个内存地址,内存地址的指针长度为32位(刚好是寄存器可以存放的位数),算下来2的32次,刚好可以存放4G左右的字节,所以在32位的操作系统上,你的8G内存条只有50%的利用率,所以现在都是64位的操作系统。

其中,CPU,Memory,I/O设备就是我们所说的CPU核,内存,硬盘。

线程是计算机调度的最小单位,也就是CPU大脑调度的最小单位,同个进程下的线程可以共享同个进程分配的计算机资源。同个进程下的线程间切换需要CPU切换上下文,但不需要创建新的虚拟内存空间,不需要内存管理单元切换上下文,比不同进程切换会显得更轻量。

总上所述,实际并发的是线程。首先,每个进程都有一个主线程,因为线程是调度的最小单位,你可以只有一个线程,但是你也可以创建多几个线程,线程调度需要CPU来切换,需要内核层的上下文切换,如果你跑了A线程,然后切到B线程,内核调用开始,CPU需要对A线程的上下文保留,然后切到B线程,然后把控制权交给你的应用层调度。进程切换也需要内核来切换,因为从C进程的主线程切换到D进程的主线程。。。

1.2

进程间通信和线程间通信

那么进程间要通讯呀,而且他们资源不共享,这个时候需要用 IPC(Inter-Process Communication,进程间通信),常用的有信号量,共享内存,套接字,实际百度上说有六种耶。

而同个进程的多个线程共享资源,通讯起来比进程容易多了,因为它们共享了虚拟内存的空间,直接就可以读取内存,现在很多 Python,Java 等编程语言都有这种线程库实现。

至于 IO 多路复用,其实就是维持一个线程队列,然后让一个线程或多个线程,去队列里面拿任务去完成。为什么呢?因为线程的数量是有限的,而且线程间通讯需要点资源,内核也要频繁切换上下文,干脆就弄一个池,有任务就派个小弟出去。

只有一个线程的 IO 多路复用,典型的就是 Redis 和 Nodejs 了,根本不需要切换上下文,一个线程走天下。而多个线程的 IP 多路复用,就是 Golang 协程的实现方式了,协程,自己管理线程,把线程控制到一定的数量,然后构造一个规则状态机来调度任务。

1

02 

Golang 协程 

无论是一个进程下的多个线程,还是不同进程,还是不同进程下的线程,切换时都需要损耗资源,浪费一些资源。所以 Golang 有协程这种东西,就是在语言内部管理自己的一些线程,合理的调度方式,使得线程不那么频繁的切换。

Golang 语言的调度器其实就是通过使用数量合适的线程并在每一个线程上执行更多的工作来降低操作系统和硬件的负载。

2.1

调度器数据结构

Golang调度器有三个主要数据结构。

  1. M,操作系统的线程,被操作系统管理的,原生线程。

  2. G,协程,被Golang语言本身管理的线程,该结构体中包含一些指令或者调度的信息。

  3. P,调度的上下文,运行在M上的调度器。

也就是说,Golang 使用 来对 进行管理,延伸出 的概念。

2.2

G 假线程

Goroutine,也就是 G,只存在于 Go 语言运行时,是对实际操作系统线程的映射,一般是 M:1 映射。也就是说,可能 5  个G,其实真实情况只有1个M。Golang帮你做了调度,帮你进行了抽象。

数据结构定义可以在此查看:/src/runtime/runtime2.go

type g struct {    // Stack parameters.    // stack describes the actual stack memory: [stack.lo, stack.hi).    // stackguard0 is the stack pointer compared in the Go stack growth prologue.    // It is stack.lo+StackGuard normally, but can be StackPreempt to trigger a preemption.    // stackguard1 is the stack pointer compared in the C stack growth prologue.    // It is stack.lo+StackGuard on g0 and gsignal stacks.    // It is ~0 on other goroutine stacks, to trigger a call to morestackc (and crash).    stack       stack   // offset known to runtime/cgo    stackguard0 uintptr // offset known to liblink    stackguard1 uintptr // offset known to liblink    _panic         *_panic // innermost panic - offset known to liblink    _defer         *_defer // innermost defer    m              *m      // current m; offset known to arm liblink    sched          gobuf    syscallsp      uintptr        // if status==Gsyscall, syscallsp = sched.sp to use during gc    syscallpc      uintptr        // if status==Gsyscall, syscallpc = sched.pc to use during gc    stktopsp       uintptr        // expected sp at top of stack, to check in traceback    param          unsafe.Pointer // passed parameter on wakeup    atomicstatus   uint32    stackLock      uint32 // sigprof/scang lock; TODO: fold in to atomicstatus    goid           int64    schedlink      guintptr    waitsince      int64      // approx time when the g become blocked    waitreason     waitReason // if status==Gwaiting    preempt        bool       // preemption signal, duplicates stackguard0 = stackpreempt    paniconfault   bool       // panic (instead of crash) on unexpected fault address    preemptscan    bool       // preempted g does scan for gc    gcscandone     bool       // g has scanned stack; protected by _Gscan bit in status    gcscanvalid    bool       // false at start of gc cycle, true if G has not run since last scan; TODO: remove?    throwsplit     bool       // must not split stack    raceignore     int8       // ignore race detection events    sysblocktraced bool       // StartTrace has emitted EvGoInSyscall about this goroutine    sysexitticks   int64      // cputicks when syscall has returned (for tracing)    traceseq       uint64     // trace event sequencer    tracelastp     puintptr   // last P emitted an event for this goroutine    lockedm        muintptr    sig            uint32    writebuf       []byte    sigcode0       uintptr    sigcode1       uintptr    sigpc          uintptr    gopc           uintptr         // pc of go statement that created this goroutine    ancestors      *[]ancestorInfo // ancestor information goroutine(s) that created this goroutine (only used if debug.tracebackancestors)    startpc        uintptr         // pc of goroutine function    racectx        uintptr    waiting        *sudog         // sudog structures this g is waiting on (that have a valid elem ptr); in lock order    cgoCtxt        []uintptr      // cgo traceback context    labels         unsafe.Pointer // profiler labels    timer          *timer         // cached timer for time.Sleep    selectDone     uint32         // are we participating in a select and did someone win the race?    // Per-G GC state    // gcAssistBytes is this G's GC assist credit in terms of    // bytes allocated. If this is positive, then the G has credit    // to allocate gcAssistBytes bytes without assisting. If this    // is negative, then the G must correct this by performing    // scan work. We track this in bytes to make it fast to update    // and check for debt in the malloc hot path. The assist ratio    // determines how this corresponds to scan work debt.    gcAssistBytes int64}

结构 定义了一个字段 atomicstatus,表示当前这个协程的状态:

// defined constantsconst (    // G status    //    // Beyond indicating the general state of a G, the G status    // acts like a lock on the goroutine's stack (and hence its    // ability to execute user code).    //    // If you add to this list, add to the list    // of "okay during garbage collection" status    // in mgcmark.go too.    //    // TODO(austin): The _Gscan bit could be much lighter-weight.    // For example, we could choose not to run _Gscanrunnable    // goroutines found in the run queue, rather than CAS-looping    // until they become _Grunnable. And transitions like    // _Gscanwaiting -> _Gscanrunnable are actually okay because    // they don't affect stack ownership.    // _Gidle means this goroutine was just allocated and has not    // yet been initialized.    // 刚刚被分配并且还没有被初始化    _Gidle = iota // 0    // _Grunnable means this goroutine is on a run queue. It is    // not currently executing user code. The stack is not owned.    // 没有执行代码、没有栈的所有权、存储在运行队列中    _Grunnable // 1    // _Grunning means this goroutine may execute user code. The    // stack is owned by this goroutine. It is not on a run queue.    // It is assigned an M and a P.    // 可以执行代码、拥有栈的所有权,被赋予了内核线程 M 和处理器 P    _Grunning // 2    // _Gsyscall means this goroutine is executing a system call.    // It is not executing user code. The stack is owned by this    // goroutine. It is not on a run queue. It is assigned an M.    // 正在执行系统调用、拥有栈的所有权、没有执行用户代码,被赋予了内核线程 M 但是不在运行队列上    _Gsyscall // 3    // _Gwaiting means this goroutine is blocked in the runtime.    // It is not executing user code. It is not on a run queue,    // but should be recorded somewhere (e.g., a channel wait    // queue) so it can be ready()d when necessary. The stack is    // not owned *except* that a channel operation may read or    // write parts of the stack under the appropriate channel    // lock. Otherwise, it is not safe to access the stack after a    // goroutine enters _Gwaiting (e.g., it may get moved).    // 由于运行时而被阻塞,没有执行用户代码并且不在运行队列上,但是可能存在于 Channel 的等待队列上    _Gwaiting // 4    // _Gmoribund_unused is currently unused, but hardcoded in gdb    // scripts.    // 暂无作用    _Gmoribund_unused // 5    // _Gdead means this goroutine is currently unused. It may be    // just exited, on a free list, or just being initialized. It    // is not executing user code. It may or may not have a stack    // allocated. The G and its stack (if any) are owned by the M    // that is exiting the G or that obtained the G from the free    // list.    // 没有被使用,没有执行代码,可能有分配的栈    _Gdead // 6    // _Genqueue_unused is currently unused.    // 暂无作用    _Genqueue_unused // 7    // _Gcopystack means this goroutine's stack is being moved. It    // is not executing user code and is not on a run queue. The    // stack is owned by the goroutine that put it in _Gcopystack.    // 栈正在被拷贝、没有执行代码、不在运行队列上    _Gcopystack // 8    // _Gscan combined with one of the above states other than    // _Grunning indicates that GC is scanning the stack. The    // goroutine is not executing user code and the stack is owned    // by the goroutine that set the _Gscan bit.    //    // _Gscanrunning is different: it is used to briefly block    // state transitions while GC signals the G to scan its own    // stack. This is otherwise like _Grunning.    //    // atomicstatus&~Gscan gives the state the goroutine will    // return to when the scan completes.    // 为了更友好,把0加上些前缀    _Gscan         = 0x1000    _Gscanrunnable = _Gscan + _Grunnable // 0x1001    _Gscanrunning  = _Gscan + _Grunning  // 0x1002    _Gscansyscall  = _Gscan + _Gsyscall  // 0x1003    _Gscanwaiting  = _Gscan + _Gwaiting  // 0x1004)

主要有 _Grunnable_Grunning_Gsyscall_Gwaiting 四个状态:

_Grunnable  没有执行代码、没有栈的所有权、存储在运行队列中_Grunning   可以执行代码、拥有栈的所有权,被赋予了内核线程 M 和处理器 P_Gsyscall   正在执行系统调用、拥有栈的所有权、没有执行用户代码,被赋予了内核线程 M 但是不在运行队列上_Gwaiting   由于运行时而被阻塞,没有执行用户代码并且不在运行队列上,但是可能存在于 Channel 的等待队列上

上面进行抽象,Goroutine 可能在等待某些满足条件,处于等待中,当满足条件时会变成可运行状态,等待被调度到真实的线程 M,如果伙伴太多可能需要等很久,等到了会进入运行中,表示正在某个M上执行。

2.3

M 真线程

Golang 默认情况下,调度器可以创建很多线程,但是最多只有 gomaxprocs M真线程能真正正常运行。

allp       []*p  // len(allp) == gomaxprocs; may change at safe points, otherwise immutable  每一个真线程M都会被绑定一个调度器P    allpLock   mutex // Protects P-less reads of allp and all writes    gomaxprocs int32

通常情况下,gomaxprocs 的数量等于核数,如果你的 CPU 有四个核,那么就是最多有4个 M。这样每个核对应一个真线程,不会切换上下文,节省了一些开销,当然你可以改变  runtime.GOMAXPROCS 的值。

type m struct {    g0      *g     // goroutine with scheduling stack 带有调度堆栈信息的G    morebuf gobuf  // gobuf arg to morestack    divmod  uint32 // div/mod denominator for arm - known to liblink    // Fields not known to debuggers.    procid        uint64       // for debuggers, but offset not hard-coded    gsignal       *g           // signal-handling g    goSigStack    gsignalStack // Go-allocated signal handling stack    sigmask       sigset       // storage for saved signal mask    tls           [6]uintptr   // thread-local storage (for x86 extern register)    mstartfn      func()    curg          *g       // current running goroutine    caughtsig     guintptr // goroutine running during fatal signal    p             puintptr // attached p for executing go code (nil if not executing go code)  这就是绑定到M的调度器P    nextp         puintptr    oldp          puintptr // the p that was attached before executing a syscall    id            int64    mallocing     int32    throwing      int32    preemptoff    string // if != "", keep curg running on this m    locks         int32    dying         int32    profilehz     int32    spinning      bool // m is out of work and is actively looking for work    blocked       bool // m is blocked on a note    newSigstack   bool // minit on C thread called sigaltstack    printlock     int8    incgo         bool   // m is executing a cgo call    freeWait      uint32 // if == 0, safe to free g0 and delete m (atomic)    fastrand      [2]uint32    needextram    bool    traceback     uint8    ncgocall      uint64      // number of cgo calls in total    ncgo          int32       // number of cgo calls currently in progress    cgoCallersUse uint32      // if non-zero, cgoCallers in use temporarily    cgoCallers    *cgoCallers // cgo traceback if crashing in cgo call    park          note    alllink       *m // on allm    schedlink     muintptr    mcache        *mcache    lockedg       guintptr    createstack   [32]uintptr // stack that created this thread.    lockedExt     uint32      // tracking for external LockOSThread    lockedInt     uint32      // tracking for internal lockOSThread    nextwaitm     muintptr    // next m waiting for lock    waitunlockf   func(*g, unsafe.Pointer) bool    waitlock      unsafe.Pointer    waittraceev   byte    waittraceskip int    startingtrace bool    syscalltick   uint32    thread        uintptr // thread handle    freelink      *m      // on sched.freem    // these are here because they are too large to be on the stack    // of low-level NOSPLIT functions.    libcall   libcall    libcallpc uintptr // for cpu profiler    libcallsp uintptr    libcallg  guintptr    syscall   libcall // stores syscall parameters on windows    vdsoSP uintptr // SP for traceback while in VDSO call (0 if not in call)    vdsoPC uintptr // PC for traceback while in VDSO call    dlogPerM    mOS}

其中:

curg          *g       // current running goroutine

表示正在真线程 M 上运行的 G

2.4

P:将 G 调度到 M 的调度器

每一个真线程 都会被绑定一个调度器 P

type p struct {    id          int32    status      uint32 // one of pidle/prunning/...  真线程的状态    link        puintptr    schedtick   uint32     // incremented on every scheduler call    syscalltick uint32     // incremented on every system call    sysmontick  sysmontick // last tick observed by sysmon    m           muintptr   // back-link to associated m (nil if idle)    mcache      *mcache    raceprocctx uintptr    deferpool    [5][]*_defer // pool of available defer structs of different sizes (see panic.go)    deferpoolbuf [5][32]*_defer    // Cache of goroutine ids, amortizes accesses to runtime·sched.goidgen.    goidcache    uint64    goidcacheend uint64    // Queue of runnable goroutines. Accessed without lock.    runqhead uint32    runqtail uint32    runq     [256]guintptr    // runnext, if non-nil, is a runnable G that was ready'd by    // the current G and should be run next instead of what's in    // runq if there's time remaining in the running G's time    // slice. It will inherit the time left in the current time    // slice. If a set of goroutines is locked in a    // communicate-and-wait pattern, this schedules that set as a    // unit and eliminates the (potentially large) scheduling    // latency that otherwise arises from adding the ready'd    // goroutines to the end of the run queue.    runnext guintptr    // Available G's (status == Gdead)    gFree struct {        gList        n int32    }    sudogcache []*sudog    sudogbuf   [128]*sudog    tracebuf traceBufPtr    // traceSweep indicates the sweep events should be traced.    // This is used to defer the sweep start event until a span    // has actually been swept.    traceSweep bool    // traceSwept and traceReclaimed track the number of bytes    // swept and reclaimed by sweeping in the current sweep loop.    traceSwept, traceReclaimed uintptr    palloc persistentAlloc // per-P to avoid mutex    _ uint32 // Alignment for atomic fields below    // Per-P GC state    gcAssistTime         int64    // Nanoseconds in assistAlloc    gcFractionalMarkTime int64    // Nanoseconds in fractional mark worker (atomic)    gcBgMarkWorker       guintptr // (atomic)    gcMarkWorkerMode     gcMarkWorkerMode    // gcMarkWorkerStartTime is the nanotime() at which this mark    // worker started.    gcMarkWorkerStartTime int64    // gcw is this P's GC work buffer cache. The work buffer is    // filled by write barriers, drained by mutator assists, and    // disposed on certain GC state transitions.    gcw gcWork    // wbBuf is this P's GC write barrier buffer.    //    // TODO: Consider caching this in the running G.    wbBuf wbBuf    runSafePointFn uint32 // if 1, run sched.safePointFn at next safe point    pad cpu.CacheLinePad}

其中 status 表示调度器的状态:

const (    // P status    // _Pidle means a P is not being used to run user code or the    // scheduler. Typically, it's on the idle P list and available    // to the scheduler, but it may just be transitioning between    // other states.    //    // The P is owned by the idle list or by whatever is    // transitioning its state. Its run queue is empty.    // 处理器没有运行用户代码或者调度器.    _Pidle = iota    // _Prunning means a P is owned by an M and is being used to    // run user code or the scheduler. Only the M that owns this P    // is allowed to change the P's status from _Prunning. The M    // may transition the P to _Pidle (if it has no more work to    // do), _Psyscall (when entering a syscall), or _Pgcstop (to    // halt for the GC). The M may also hand ownership of the P    // off directly to another M (e.g., to schedule a locked G).    // 被线程 M 持有,并且正在执行用户代码或者调度器    _Prunning    // _Psyscall means a P is not running user code. It has    // affinity to an M in a syscall but is not owned by it and    // may be stolen by another M. This is similar to _Pidle but    // uses lightweight transitions and maintains M affinity.    //    // Leaving _Psyscall must be done with a CAS, either to steal    // or retake the P. Note that there's an ABA hazard: even if    // an M successfully CASes its original P back to _Prunning    // after a syscall, it must understand the P may have been    // used by another M in the interim.    // 没有执行用户代码,当前线程陷入系统调用    _Psyscall    // _Pgcstop means a P is halted for STW and owned by the M    // that stopped the world. The M that stopped the world    // continues to use its P, even in _Pgcstop. Transitioning    // from _Prunning to _Pgcstop causes an M to release its P and    // park.    //    // The P retains its run queue and startTheWorld will restart    // the scheduler on Ps with non-empty run queues.    // 被线程 M 持有,当前处理器由于STW(垃圾回收)被停止    _Pgcstop    // _Pdead means a P is no longer used (GOMAXPROCS shrank). We    // reuse Ps if GOMAXPROCS increases. A dead P is mostly    // stripped of its resources, though a few things remain    // (e.g., trace buffers).    // 当前处理器已经不被使用    _Pdead)

处理器执行用户代码时会 _Prunning,当M正在系统调用时,会 _Psyscall,当垃圾回收,Stop the world(STW) 时,会 _Pgcstop

2.5

协程创建,销毁,调度过程

详见:src/runtime/proc.go

// Goroutine scheduler// The scheduler's job is to distribute ready-to-run goroutines over worker threads.//// The main concepts are:// G - goroutine.// M - worker thread, or machine.// P - processor, a resource that is required to execute Go code.//     M must have an associated P to execute Go code, however it can be//     blocked or in a syscall w/o an associated P.//// Design doc at https://golang.org/s/go11sched.

入口:

//go:linkname main_main main.mainfunc main_main()// The main goroutine.func main() {    g := getg()    // Racectx of m0->g0 is used only as the parent of the main goroutine.    // It must not be used for anything else.    g.m.g0.racectx = 0    // Max stack size is 1 GB on 64-bit, 250 MB on 32-bit.    // Using decimal instead of binary GB and MB because    // they look nicer in the stack overflow failure message.    if sys.PtrSize == 8 {        maxstacksize = 1000000000    } else {        maxstacksize = 250000000    }    // Allow newproc to start new Ms.    mainStarted = true    if GOARCH != "wasm" { // no threads on wasm yet, so no sysmon        systemstack(func() {            newm(sysmon, nil)        })    }    // Lock the main goroutine onto this, the main OS thread,    // during initialization. Most programs won't care, but a few    // do require certain calls to be made by the main thread.    // Those can arrange for main.main to run in the main thread    // by calling runtime.LockOSThread during initialization    // to preserve the lock.    lockOSThread()    if g.m != &m0 {        throw("runtime.main not on m0")    }    doInit(&runtime_inittask) // must be before defer    if nanotime() == 0 {        throw("nanotime returning zero")    }    // Defer unlock so that runtime.Goexit during init does the unlock too.    needUnlock := true    defer func() {        if needUnlock {            unlockOSThread()        }    }()    // Record when the world started.    runtimeInitTime = nanotime()    gcenable()    main_init_done = make(chan bool)    if iscgo {        if _cgo_thread_start == nil {            throw("_cgo_thread_start missing")        }        if GOOS != "windows" {            if _cgo_setenv == nil {                throw("_cgo_setenv missing")            }            if _cgo_unsetenv == nil {                throw("_cgo_unsetenv missing")            }        }        if _cgo_notify_runtime_init_done == nil {            throw("_cgo_notify_runtime_init_done missing")        }        // Start the template thread in case we enter Go from        // a C-created thread and need to create a new thread.        startTemplateThread()        cgocall(_cgo_notify_runtime_init_done, nil)    }    doInit(&main_inittask)    close(main_init_done)    needUnlock = false    unlockOSThread()    if isarchive || islibrary {        // A program compiled with -buildmode=c-archive or c-shared        // has a main, but it is not executed.        return    }    fn := main_main // make an indirect call, as the linker doesn't know the address of the main package when laying down the runtime    fn()    if raceenabled {        racefini()    }    // Make racy client program work: if panicking on    // another goroutine at the same time as main returns,    // let the other goroutine finish printing the panic trace.    // Once it does, it will exit. See issues 3934 and 20018.    if atomic.Load(&runningPanicDefers) != 0 {        // Running deferred functions should not take long.        for c := 0; c < 1000; c++ {            if atomic.Load(&runningPanicDefers) == 0 {                break            }            Gosched()        }    }    if atomic.Load(&panicking) != 0 {        gopark(nil, nil, waitReasonPanicWait, traceEvGoStop, 1)    }    exit(0)    for {        var x *int32        *x = 0    }}

我们自己的 main 包下的 main 方法入口执行前,都会先执行运行时的 The main goroutine.,类似于注入。Golang 先准备好各种资源,然后开始执行我们的方法,然后收尾。

从这里开始分析是极好的。

设计文档在:https://golang.org/s/go11sched

作者:花花世界

出处:https://www.cnblogs.com/nima/p/11751393.html

51Reboot 运维开发

Golang 课程

K8S 课程

Python 自动化进阶课程

Python 基础实战课程

运维前端课程

课程试听预约请扫码>>>

18edfc17496937e11f7b87fe3f96e38f.png

文章好看点这里

b298aa36ce26fc6c67d4d472a8fa89d7.gif

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
©️2022 CSDN 皮肤主题:深蓝海洋 设计师:CSDN官方博客 返回首页
评论
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、C币套餐、付费专栏及课程。

余额充值