net/http 库 web 工作原理阅读

net/http 库 web 工作原理阅读

现在阅读net/http 库，解释一些关键功能实现。

相比其他许多语言搭建服务器程序，go语言搭建服务器程序非常方便，利用http包可以很方便地搭建web服务器。下面的代码就可以搭建一个最基本的web服务器：

package main

import (
    "fmt"
    "net/http"
)

func IndexHandler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintln(w, "Hello World!")
}

func main() {
    http.HandleFunc("/", IndexHandler)
    http.ListenAndServe("127.0.0.0:80", nil)
}

根据计算机网络等课程知识,http协议基于客户机(client)-服务器(server)架构，是一种pull式的协议。客户机发出http请求，而服务器收到http请求报文后经过相应处理，返回http应答报文。go语言服务器处理http请求大致分为四步：创建 ServerSocket， 绑定并 listen， accept 连接， 创建 go 线程服务一个连接。

服务器接收和处理http请求的过程中，重要的一步在于路由请求到对应的处理函数。这也就是后端路由(route)的功能。将不同url和method的请求路由到不同处理函数后，处理函数可以根据request中的参数，进行相应处理并返回特定的response报文。这个流程大致就是Clinet -> Requests -> Router -> Handler -> Response -> Clinet

更详细的net/http库处理流程如下图所示：

按照上面的流程，首先跟踪ListenAndServe入口函数，其代码如下：

// ListenAndServe always returns a non-nil error. After Shutdown or Close,
// the returned error is ErrServerClosed.
func (srv *Server) ListenAndServe() error {
	if srv.shuttingDown() {
		return ErrServerClosed
	}
	addr := srv.Addr
	if addr == "" {
		addr = ":http"
	}
	ln, err := net.Listen("tcp", addr)
	if err != nil {
		return err
	}
	return srv.Serve(ln)
}

可见，其检查srv是否关闭，如果关闭则返回ErrServerClosed错误，否则获得srv的地址，然后建立一个该地址的tcp链接(使用net的Listen方法)，之后在没有错误发生的情况下则返回srv.Serve(ln)，让服务器侦听该tcp链接。

可以查看Server结构体的定义如下：

// A Server defines parameters for running an HTTP server.
// The zero value for Server is a valid configuration.
type Server struct {
	Addr    string  // TCP address to listen on, ":http" if empty
	Handler Handler // handler to invoke, http.DefaultServeMux if nil

	// TLSConfig optionally provides a TLS configuration for use
	// by ServeTLS and ListenAndServeTLS. Note that this value is
	// cloned by ServeTLS and ListenAndServeTLS, so it's not
	// possible to modify the configuration with methods like
	// tls.Config.SetSessionTicketKeys. To use
	// SetSessionTicketKeys, use Server.Serve with a TLS Listener
	// instead.
	TLSConfig *tls.Config

	// ReadTimeout is the maximum duration for reading the entire
	// request, including the body.
	//
	// Because ReadTimeout does not let Handlers make per-request
	// decisions on each request body's acceptable deadline or
	// upload rate, most users will prefer to use
	// ReadHeaderTimeout. It is valid to use them both.
	ReadTimeout time.Duration

	// ReadHeaderTimeout is the amount of time allowed to read
	// request headers. The connection's read deadline is reset
	// after reading the headers and the Handler can decide what
	// is considered too slow for the body. If ReadHeaderTimeout
	// is zero, the value of ReadTimeout is used. If both are
	// zero, there is no timeout.
	ReadHeaderTimeout time.Duration

	// WriteTimeout is the maximum duration before timing out
	// writes of the response. It is reset whenever a new
	// request's header is read. Like ReadTimeout, it does not
	// let Handlers make decisions on a per-request basis.
	WriteTimeout time.Duration

	// IdleTimeout is the maximum amount of time to wait for the
	// next request when keep-alives are enabled. If IdleTimeout
	// is zero, the value of ReadTimeout is used. If both are
	// zero, there is no timeout.
	IdleTimeout time.Duration

	// MaxHeaderBytes controls the maximum number of bytes the
	// server will read parsing the request header's keys and
	// values, including the request line. It does not limit the
	// size of the request body.
	// If zero, DefaultMaxHeaderBytes is used.
	MaxHeaderBytes int

	// TLSNextProto optionally specifies a function to take over
	// ownership of the provided TLS connection when an NPN/ALPN
	// protocol upgrade has occurred. The map key is the protocol
	// name negotiated. The Handler argument should be used to
	// handle HTTP requests and will initialize the Request's TLS
	// and RemoteAddr if not already set. The connection is
	// automatically closed when the function returns.
	// If TLSNextProto is not nil, HTTP/2 support is not enabled
	// automatically.
	TLSNextProto map[string]func(*Server, *tls.Conn, Handler)

	// ConnState specifies an optional callback function that is
	// called when a client connection changes state. See the
	// ConnState type and associated constants for details.
	ConnState func(net.Conn, ConnState)

	// ErrorLog specifies an optional logger for errors accepting
	// connections, unexpected behavior from handlers, and
	// underlying FileSystem errors.
	// If nil, logging is done via the log package's standard logger.
	ErrorLog *log.Logger

	// BaseContext optionally specifies a function that returns
	// the base context for incoming requests on this server.
	// The provided Listener is the specific Listener that's
	// about to start accepting requests.
	// If BaseContext is nil, the default is context.Background().
	// If non-nil, it must return a non-nil context.
	BaseContext func(net.Listener) context.Context

	// ConnContext optionally specifies a function that modifies
	// the context used for a new connection c. The provided ctx
	// is derived from the base context and has a ServerContextKey
	// value.
	ConnContext func(ctx context.Context, c net.Conn) context.Context

	disableKeepAlives int32     // accessed atomically.
	inShutdown        int32     // accessed atomically (non-zero means we're in Shutdown)
	nextProtoOnce     sync.Once // guards setupHTTP2_* init
	nextProtoErr      error     // result of http2.ConfigureServer if used

	mu         sync.Mutex
	listeners  map[*net.Listener]struct{}
	activeConn map[*conn]struct{}
	doneChan   chan struct{}
	onShutdown []func()
}

可以上面存储Server的结构体，包含地址Addr，处理函数Handler，TEL配置TLSConfig，互斥锁mu等服务器必要的字段。

可以看到Server的shuttingDownh函数实现如下，其通过s.inShutdown读取32位整数的原子操作判断Server是否关闭：

func (s *Server) shuttingDown() bool {
	// TODO: replace inShutdown with the existing atomicBool type;
	// see https://github.com/golang/go/issues/20239#issuecomment-381434582
	return atomic.LoadInt32(&s.inShutdown) != 0
}

再阅读Server的Serve函数如下，其先使用testHookServerServe函数测试srv和l，之后声明一个包含l这个Listenere的onceCloseListener结构，在Serve结束后关闭该Listener。之后设置通过setupHTTP2_Serve函数进行http协议的设置，再之后通过trackListener来追踪listener，并在函数结束后结束追踪。再之后获得上下文并进行相应设置，如果没有发生错误，则开始无限循环，并尝试Accept发来的http请求，如果发生错误则等待一段时间再尝试。这里的尝试过程采取了和计算机网络中以太网协议类似的一些二进制指数退避思想，连续第一次失败时会睡眠5毫秒后重试，之后每多连续失败一次，睡眠的时间就会翻倍，而睡眠时间增长到超过1秒时就不再增长，以1秒作为睡眠时间，直到可以成功接收到请求。之后，其会通过c := srv.newConn(rw)建立一个新的连接，并且通过go c.serve(connCtx)创建一个新的go程来服务这个连接。这样就实现了TCP控制连接的侦听请求。

func (srv *Server) Serve(l net.Listener) error {
	if fn := testHookServerServe; fn != nil {
		fn(srv, l) // call hook with unwrapped listener
	}

	origListener := l
	l = &onceCloseListener{Listener: l}
	defer l.Close()

	if err := srv.setupHTTP2_Serve(); err != nil {
		return err
	}

	if !srv.trackListener(&l, true) {
		return ErrServerClosed
	}
	defer srv.trackListener(&l, false)

	var tempDelay time.Duration // how long to sleep on accept failure

	baseCtx := context.Background()
	if srv.BaseContext != nil {
		baseCtx = srv.BaseContext(origListener)
		if baseCtx == nil {
			panic("BaseContext returned a nil context")
		}
	}

	ctx := context.WithValue(baseCtx, ServerContextKey, srv)
	for {
		rw, e := l.Accept()
		if e != nil {
			select {
			case <-srv.getDoneChan():
				return ErrServerClosed
			default:
			}
			if ne, ok := e.(net.Error); ok && ne.Temporary() {
				if tempDelay == 0 {
					tempDelay = 5 * time.Millisecond
				} else {
					tempDelay *= 2
				}
				if max := 1 * time.Second; tempDelay > max {
					tempDelay = max
				}
				srv.logf("http: Accept error: %v; retrying in %v", e, tempDelay)
				time.Sleep(tempDelay)
				continue
			}
			return e
		}
		connCtx := ctx
		if cc := srv.ConnContext; cc != nil {
			connCtx = cc(connCtx, rw)
			if connCtx == nil {
				panic("ConnContext returned nil")
			}
		}
		tempDelay = 0
		c := srv.newConn(rw)
		c.setState(c.rwc, StateNew) // before Serve can return
		go c.serve(connCtx)
	}
}

然后可以进一步观察c.serve代码实现如下，具体分析详见下面代码注释：

func (c *conn) serve(ctx context.Context) {
	c.remoteAddr = c.rwc.RemoteAddr().String()//获得字符串形式的远程客户机地址
	ctx = context.WithValue(ctx, LocalAddrContextKey, c.rwc.LocalAddr())//获得go程的上下文
	defer func() {//最后执行该函数
		if err := recover(); err != nil && err != ErrAbortHandler {//通过recover获得错误类型
			const size = 64 << 10
			buf := make([]byte, size)//设置大小为64kB的缓冲区
			buf = buf[:runtime.Stack(buf, false)]//获得运行栈
			c.server.logf("http: panic serving %v: %v\n%s", c.remoteAddr, err, buf)//log记录信息
		}
		if !c.hijacked() {//没有被劫持
			c.close()//关闭连接
			c.setState(c.rwc, StateClosed)
		}
	}()

	if tlsConn, ok := c.rwc.(*tls.Conn); ok {
		if d := c.server.ReadTimeout; d != 0 {//设置Read的ddl
			c.rwc.SetReadDeadline(time.Now().Add(d))
		}
		if d := c.server.WriteTimeout; d != 0 {//设置Write的ddl
			c.rwc.SetWriteDeadline(time.Now().Add(d))
		}
		if err := tlsConn.Handshake(); err != nil {//连接握手
			// If the handshake failed due to the client not speaking
			// TLS, assume they're speaking plaintext HTTP and write a
			// 400 response on the TLS conn's underlying net.Conn.
			if re, ok := err.(tls.RecordHeaderError); ok && re.Conn != nil && tlsRecordHeaderLooksLikeHTTP(re.RecordHeader) {//因连接出错，请求头错误等原因导致请求无效
				io.WriteString(re.Conn, "HTTP/1.0 400 Bad Request\r\n\r\nClient sent an HTTP request to an HTTPS server.\n")//400错误
				re.Conn.Close()//关闭连接
				return
			}
			c.server.logf("http: TLS handshake error from %s: %v", c.rwc.RemoteAddr(), err)//记录log
			return
		}
		c.tlsState = new(tls.ConnectionState)//声明一个新的tls.ConnectionState结构
		*c.tlsState = tlsConn.ConnectionState()//获得连接状态
		if proto := c.tlsState.NegotiatedProtocol; validNPN(proto) {//获得协议
			if fn := c.server.TLSNextProto[proto]; fn != nil {//获得下一个协议
				h := initNPNRequest{ctx, tlsConn, serverHandler{c.server}}
				fn(c.server, tlsConn, h)
			}
			return
		}
	}

	// HTTP/1.x from here on.

	ctx, cancelCtx := context.WithCancel(ctx)//撤销上下文
	c.cancelCtx = cancelCtx
	defer cancelCtx()

	c.r = &connReader{conn: c}
	c.bufr = newBufioReader(c.r)//创建新的redaer
	c.bufw = newBufioWriterSize(checkConnErrorWriter{c}, 4<<10)//创建新的writer 缓冲区为4kB

	for {
		w, err := c.readRequest(ctx)//读请求
		if c.r.remain != c.server.initialReadLimitSize() {//没有超过度限制
			// If we read any bytes off the wire, we're active.
			c.setState(c.rwc, StateActive)
		}
		if err != nil {//发生了错误
			const errorHeaders = "\r\nContent-Type: text/plain; charset=utf-8\r\nConnection: close\r\n\r\n"//声明错误头部

			switch {
			case err == errTooLarge://too large错误
				// Their HTTP client may or may not be
				// able to read this if we're
				// responding to them and hanging up
				// while they're still writing their
				// request. Undefined behavior.
				const publicErr = "431 Request Header Fields Too Large"
				fmt.Fprintf(c.rwc, "HTTP/1.1 "+publicErr+errorHeaders+publicErr)
				c.closeWriteAndWait()
				return

			case isUnsupportedTEError(err)://为unsupported错误
				// Respond as per RFC 7230 Section 3.3.1 which says,
				//      A server that receives a request message with a
				//      transfer coding it does not understand SHOULD
				//      respond with 501 (Unimplemented).
				code := StatusNotImplemented

				// We purposefully aren't echoing back the transfer-encoding's value,
				// so as to mitigate the risk of cross side scripting by an attacker.
				fmt.Fprintf(c.rwc, "HTTP/1.1 %d %s%sUnsupported transfer encoding", code, StatusText(code), errorHeaders)
				return

			case isCommonNetReadError(err)://网络读错误
				return // don't reply

			default://其他情况
				publicErr := "400 Bad Request"
				if v, ok := err.(badRequestError); ok {
					publicErr = publicErr + ": " + string(v)
				}

				fmt.Fprintf(c.rwc, "HTTP/1.1 "+publicErr+errorHeaders+publicErr)
				return
			}
		}

		// Expect 100 Continue support  没有错误
		req := w.req//获得请求
		if req.expectsContinue() {//期望continue
			if req.ProtoAtLeast(1, 1) && req.ContentLength != 0 {
				// Wrap the Body reader with one that replies on the connection
				req.Body = &expectContinueReader{readCloser: req.Body, resp: w}
			}
		} else if req.Header.get("Expect") != "" {//没有expect字段
			w.sendExpectationFailed()
			return
		}

		c.curReq.Store(w)//存储该请求

		if requestBodyRemains(req.Body) {//获得请求的body部分
			registerOnHitEOF(req.Body, w.conn.r.startBackgroundRead)
		} else {
			w.conn.r.startBackgroundRead()
		}

		// HTTP cannot have multiple simultaneous active requests.[*]
		// Until the server replies to this request, it can't read another,
		// so we might as well run the handler in this goroutine.
		// [*] Not strictly true: HTTP pipelining. We could let them all process
		// in parallel even if their responses need to be serialized.
		// But we're not going to implement HTTP pipelining because it
		// was never deployed in the wild and the answer is HTTP/2.
		serverHandler{c.server}.ServeHTTP(w, w.req)//使用Handler处理该请求
		w.cancelCtx()
		if c.hijacked() {
			return
		}
		w.finishRequest()//完成请求
		if !w.shouldReuseConnection() {//要保持连接
			if w.requestBodyLimitHit || w.closedRequestBodyEarly() {
				c.closeWriteAndWait()//保持连接不关闭
			}
			return
		}
		c.setState(c.rwc, StateIdle)
		c.curReq.Store((*response)(nil))

		if !w.conn.server.doKeepAlives() {
			// We're in shutdown mode. We might've replied
			// to the user without "Connection: close" and
			// they might think they can send another
			// request, but such is life with HTTP/1.1.
			return
		}

		if d := c.server.idleTimeout(); d != 0 {
			c.rwc.SetReadDeadline(time.Now().Add(d))
			if _, err := c.bufr.Peek(4); err != nil {
				return
			}
		}
		c.rwc.SetReadDeadline(time.Time{})
	}
}

注意到上面使用了Handler，可以查看其定义如下：

type Handler interface {
	ServeHTTP(ResponseWriter, *Request)
}

Go语言没有继承，类多态等语法，其相应语法特性可以通过接口来实现。接口定义声明了函数签名，任何结构只要实现了与接口函数签名相同的方法，就等同于实现了接口。go的http服务都是基于handler接口处理。任何实现了ServeHTTP方法的结构体都是这样的Handler对象。

http包内还定义了HandlerFunc结构，该结构默认实现了ServeHTTP()方法，在调用http.HandleFunc()的时候就可以将自定义的handler处理函数强制转为HandlerFunc类型。

func (f HandlerFunc) ServeHTTP(w ResponseWriter, r *Request) {
	f(w, r)
}

可以再查看ServeHTTP函数如下：

func (sh serverHandler) ServeHTTP(rw ResponseWriter, req *Request) {
    handler := sh.srv.Handler
    if handler == nil {
        handler = DefaultServeMux
    }
    if req.RequestURI == "*" && req.Method == "OPTIONS" {
        handler = globalOptionsHandler{}
    }
    handler.ServeHTTP(rw, req)
}

该函数首先设定handler为服务器对应的Handler，如果服务器的Handler为nil，则使用DefaultServeMux。之后判断如果请求的url为"*"，并且请求的方法为"OPTIONS"，则设定handler为globalOptionsHandler。之后调用该handler的ServeHTTP函数处理请求。

以上完成了net/http 库 源码阅读与web 工作原理分析。