Prometheus源码系列：指标采集(scrapeManager)

• 指标采集(scrapeManager)简介

　　从上篇文章：Prometheus源码系列：服务发现 (serviceDiscover)，我们已经知道，为了从服务发现(serviceDiscover)实时获取监控服务(targets)，指标采集(scrapeManager)通过协程把管道(chan)获取来的服务(targets)存进一个map类型：map[string][]*targetgroup.Group．其中，map的key是job_name，map的value是结构体targetgroup.Group，该结构体包含该job_name对应的Targets，Labels和Source．

　　指标采集(scrapeManager)获取服务(targets)的变动，可分为多种情况，以服务增加为例，若有新的job添加，指标采集(scrapeManager)会进行重载，为新的job创建一个scrapePool，并为job中的每个target创建一个scrapeLoop．若job没有变动，只增加了job下对应的targets，则只需创建新的targets对应的scrapeLoop．

　　为本文分析的代码都基于版本 v2.7.1，会通过dlv输出多个参数的示例，所用的配置文件：Prometheus.yml配置文件示例．

• 指标采集(scrapeManager)实时获取监控服务

　　指标采集(scrapeManager)获取实时监控服务(targets)的入口函数：scrapeManager.Run(discoveryManagerScrape.SyncCh())：

prometheus/cmd/prometheus/main.go

// Scrape manager.
g.Add(
	func() error {
		// When the scrape manager receives a new targets list
		// it needs to read a valid config for each job.
		// It depends on the config being in sync with the discovery manager so
		// we wait until the config is fully loaded.
		<-reloadReady.C

		err := scrapeManager.Run(discoveryManagerScrape.SyncCh())
		level.Info(logger).Log("msg", "Scrape manager stopped")
		return err
	},
	func(err error) {
		// Scrape manager needs to be stopped before closing the local TSDB
		// so that it doesn't try to write samples to a closed storage.
		level.Info(logger).Log("msg", "Stopping scrape manager...")
		scrapeManager.Stop()
	},
)


// ts即map[string][]*targetgroup.Group
(dlv) p ts["prometheus"]
[]*github.com/prometheus/prometheus/discovery/targetgroup.Group len: 1, cap: 1, [
	*{
		Targets: []github.com/prometheus/common/model.LabelSet len: 1, cap: 1, [
			[...],
		],
		Labels: github.com/prometheus/common/model.LabelSet nil,
		Source: "0",},
]

　　其中包含两个部分：scrapeManager的初始化和起一个协程监控服务(targets)的变化

　　１． scrapeManager的初始化，调用NewManager方法实现：

prometheus/cmd/prometheus/main.go

//fanoutStorage是监控的存储的抽象
scrapeManager = scrape.NewManager(log.With(logger, "component", "scrape manager"), fanoutStorage)

　　(1) NewManager方法了实例化结构体Manager：

prometheus/scrape/manager.go

// NewManager is the Manager constructor
func NewManager(logger log.Logger, app Appendable) *Manager {
	if logger == nil {
		logger = log.NewNopLogger()
	}
	return &Manager{
		append:        app,
		logger:        logger,
		scrapeConfigs: make(map[string]*config.ScrapeConfig),
		scrapePools:   make(map[string]*scrapePool),
		graceShut:     make(chan struct{}),
		triggerReload: make(chan struct{}, 1),
	}
}

　　(2) 结构体Manager维护map类型的scrapePools和targetSets，两者key都是job_name，但scrapePools的value对应结构体scrapepool，而targetSets的value对应的结构体是Group，分别给出了两者的示例输出

prometheus/scrape/manager.go

// Manager maintains a set of scrape pools and manages start/stop cycles
// when receiving new target groups form the discovery manager.
type Manager struct {
	logger    log.Logger  //系统日志
	append    Appendable  //存储监控指标
	graceShut chan struct{}  //退出

	mtxScrape     sync.Mutex // Guards the fields below. 读写锁
	scrapeConfigs map[string]*config.ScrapeConfig  //prometheus.yml的srape_config配置部分，key对应job_name，value对应job_name的配置参数
	scrapePools   map[string]*scrapePool  //key对应job_name，value对应结构体scrapePool，包含该job_name下所有的targets
	targetSets    map[string][]*targetgroup.Group  //key对应job_name，value对应结构体Group，包含job_name对应的Targets，Labels和Source

	triggerReload chan struct{} //若有新的服务(targets)通过服务发现(serviceDisvoer)传过来，会向该管道传值，触发加载配置文件操作，后面会讲到
}


基于job_name：node的targetSets的示例输出：
(dlv) p m.targetSets["node"]
[]*github.com/prometheus/prometheus/discovery/targetgroup.Group len: 1, cap: 1, [
	*{
		Targets: []github.com/prometheus/common/model.LabelSet len: 1, cap: 1, [
	                 [
		                   "__address__": "localhost:9100", 
	                 ],
		],
		Labels: github.com/prometheus/common/model.LabelSet nil,
		Source: "0",},
]

基于job_name：node的scrapePools示例输出：
(dlv) p m.scrapePools
map[string]*github.com/prometheus/prometheus/scrape.scrapePool [
	"node": *{
		appendable: github.com/prometheus/prometheus/scrape.Appendable(*github.com/prometheus/prometheus/storage.fanout) ...,
		logger: github.com/go-kit/kit/log.Logger(*github.com/go-kit/kit/log.context) ...,
		mtx: (*sync.RWMutex)(0xc001be0020),
		config: *(*"github.com/prometheus/prometheus/config.ScrapeConfig")(0xc00048ab40),
		client: *(*"net/http.Client")(0xc000d303c0),
		activeTargets: map[uint64]*github.com/prometheus/prometheus/scrape.Target [],
		droppedTargets: []*github.com/prometheus/prometheus/scrape.Target len: 0, cap: 0, nil,
		loops: map[uint64]github.com/prometheus/prometheus/scrape.loop [],
		cancel: context.WithCancel.func1,
		newLoop: github.com/prometheus/prometheus/scrape.newScrapePool.func2,}, 
]

　　２．在前面已经多次提到，指标采集(scrapeManager)在main.go启动时，会起一个协程运行Run方法，从服务发现(serviceDiscover)实时获取被监控服务(targets)，接下来看下Run方法的具体实现

prometheus/scrape/manager.go

// Run receives and saves target set updates and triggers the scraping loops reloading.
// Reloading happens in the background so that it doesn't block receiving targets updates.
func (m *Manager) Run(tsets <-chan map[string][]*targetgroup.Group) error {
     //定时(5s)更新服务(targets)，结合triggerReload一起使用，即每5s判断一次triggerReload是否更新．
	go m.reloader() 
	for {
		select {
        //通过管道获取被监控的服务(targets)
		case ts := <-tsets:  
			m.updateTsets(ts)

			select {
　　　　　　　//若从服务发现 (serviceDiscover)有服务(targets)变动，则给管道triggerReload传值，并触发reloader()方法更新服务．
			case m.triggerReload <- struct{}{}: 
			default:
			}

		case <-m.graceShut:
			return nil
		}
	}
}

　　以上流程还是比较清晰，若服务发现(serviceDiscovery)有服务(target)变动，Run方法就会向管道triggerReload注入值：m.triggerReload <- struct{}{}中，并起了一个协程，运行reloader方法．用于定时更新服务(targets)．启动这个协程应该是为了防止阻塞从服务发现(serviceDiscover)获取变动的服务(targets)

　　reloader方法启动了一个定时器，在无限循环中每5s判断一下管道triggerReload，若有值，则执行reload方法．

prometheus/scrape/manager.go

func (m *Manager) reloader() {
    //定时器5s
	ticker := time.NewTicker(5 * time.Second)
	defer ticker.Stop()

	for {
		select {
		case <-m.graceShut:
			return
　      // 若服务发现(serviceDiscovery)有服务(targets)变动，就会向管道triggerReload写入值，定时器每5s判断一次triggerReload管道是否有值，若有值，则触发reload方法
		case <-ticker.C:
			select {
			case <-m.triggerReload:
				m.reload()
			case <-m.graceShut:
				return
			}
		}
	}
}

　　reload方法会根据job_name比较targetSets，scrapePools和scrapeConfigs的一致性，并把每个job_name下的类型为[]*targetgroup.Group的groups通过协程传给sp.Sync方法，增加并发．

prometheus/scrape/manager.go

func (m *Manager) reload() {
	m.mtxScrape.Lock()
	var wg sync.WaitGroup
　　//setName对应job_name，
　　//group的结构体包含job_name对应的Targets，Labels和source，这个在上篇文章已经详细介绍
	for setName, groups := range m.targetSets {
		var sp *scrapePool
		existing, ok := m.scrapePools[setName]
　　　　 //若该job_name不在scrapePools中，分为两种情况处理
        //(1)job_name不在scrapeConfigs中，则报错
　　　　 //(2)job_name在scrapeConfigs中，则需要把该job_name加到scrapePools中
		if !ok {
			scrapeConfig, ok := m.scrapeConfigs[setName]
			if !ok {
				level.Error(m.logger).Log("msg", "error reloading target set", "err", "invalid config id:"+setName)
				continue
			}
			sp = newScrapePool(scrapeConfig, m.append, log.With(m.logger, "scrape_pool", setName))
			m.scrapePools[setName] = sp
		} else {
			sp = existing
		}

		wg.Add(1)
		// Run the sync in parallel as these take a while and at high load can't catch up.
		go func(sp *scrapePool, groups []*targetgroup.Group) {
　　　　　　　//把groups转换为targets类型
			sp.Sync(groups)
			wg.Done()
		}(sp, groups)

	}
	m.mtxScrape.Unlock()
	wg.Wait()
}

　　sp.Sync方法引入了Target结构体，把[]*targetgroup.Group类型的groups转换为targets类型，其中每个groups对应一个job_name下多个targets．随后，调用sp.sync方法，同步scrape服务

prometheus/scrape/scrape.go

// Sync converts target groups into actual scrape targets and synchronizes
// the currently running scraper with the resulting set and returns all scraped and dropped targets.
func (sp *scrapePool) Sync(tgs []*targetgroup.Group) {
	start := time.Now()

	var all []*Target
	sp.mtx.Lock()
	sp.droppedTargets = []*Target{}
	for _, tg := range tgs {
　　　　//转换targetgroup.Group类型为Target
		targets, err := targetsFromGroup(tg, sp.config)
		if err != nil {
			level.Error(sp.logger).Log("msg", "creating targets failed", "err", err)
			continue
		}
        // 这里有个疑问，tg对应一个target，为什么返回回来的targets不是对应一个target相关参数，需要用for循环？
		for _, t := range targets {
            //判断Target的有效label是否大于0
			if t.Labels().Len() > 0 {
				all = append(all, t)
			} else if t.DiscoveredLabels().Len() > 0 {
                //若为无效Target，则加入scrapeLoop的droppedTargets中
				sp.droppedTargets = append(sp.droppedTargets, t)
			}
		}
	}
	sp.mtx.Unlock()
	sp.sync(all)

	targetSyncIntervalLength.WithLabelValues(sp.config.JobName).Observe(
		time.Since(start).Seconds(),
	)
	targetScrapePoolSyncsCounter.WithLabelValues(sp.config.JobName).Inc()
}

　Target结构体定义：

// Target refers to a singular HTTP or HTTPS endpoint.
type Target struct {
	// Labels before any processing.
	discoveredLabels labels.Labels
	// Any labels that are added to this target and its metrics.
	labels labels.Labels
	// Additional URL parmeters that are part of the target URL.
	params url.Values

	mtx                sync.RWMutex
	lastError          error
	lastScrape         time.Time
	lastScrapeDuration time.Duration
	health             TargetHealth
	metadata           metricMetadataStore
}

　　sp.sync方法对比新的Target列表和原来的Target列表，若发现不在原来的Target列表中，则新建该targets的scrapeLoop，通过协程启动scrapeLoop的run方法，并发采集存储指标．然后判断原来的Target列表是否存在失效的Target，若存在，则移除

prometheus/scrape/scrape.go

// sync takes a list of potentially duplicated targets, deduplicates them, starts
// scrape loops for new targets, and stops scrape loops for disappeared targets.
// It returns after all stopped scrape loops terminated.
func (sp *scrapePool) sync(targets []*Target) {
	sp.mtx.Lock()
	defer sp.mtx.Unlock()

	var (
		uniqueTargets = map[uint64]struct{}{} 
		interval      = time.Duration(sp.config.ScrapeInterval) //指标采集周期
		timeout       = time.Duration(sp.config.ScrapeTimeout)  //指标采集超时时间
		limit         = int(sp.config.SampleLimit) //指标采集的限额
		honor         = sp.config.HonorLabels  //
		mrc           = sp.config.MetricRelabelConfigs
	)

	for _, t := range targets {
		t := t
		hash := t.hash()
		uniqueTargets[hash] = struct{}{}
　　　　 //若发现不在原来的Target列表中，则新建该target的scrapeLoop．
		if _, ok := sp.activeTargets[hash]; !ok {
			s := &targetScraper{Target: t, client: sp.client, timeout: timeout}
			l := sp.newLoop(t, s, limit, honor, mrc)

			sp.activeTargets[hash] = t
			sp.loops[hash] = l
            //通过协程启动scrapeLoop的run方法，采集存储指标
			go l.run(interval, timeout, nil)
		} else {
			// Need to keep the most updated labels information
			// for displaying it in the Service Discovery web page.
			sp.activeTargets[hash].SetDiscoveredLabels(t.DiscoveredLabels())
		}
	}

	var wg sync.WaitGroup

	// Stop and remove old targets and scraper loops.
    //判断原来的Target列表是否存在失效的Target，若存在则移除
	for hash := range sp.activeTargets {
		if _, ok := uniqueTargets[hash]; !ok {
			wg.Add(1)
			go func(l loop) {

				l.stop()

				wg.Done()
			}(sp.loops[hash])

			delete(sp.loops, hash)
			delete(sp.activeTargets, hash)
		}
	}

	// Wait for all potentially stopped scrapers to terminate.
	// This covers the case of flapping targets. If the server is under high load, a new scraper
	// may be active and tries to insert. The old scraper that didn't terminate yet could still
	// be inserting a previous sample set.
	wg.Wait()
}

　　sp.sync方法起了一个协程运行scrapePool的run方法去采集并存储监控指标(metrics)，run方法实现如下：

prometheus/scrape/scrape.go

func (sl *scrapeLoop) run(interval, timeout time.Duration, errc chan<- error) {
	select {
    //检测超时
	case <-time.After(sl.scraper.offset(interval)):
		// Continue after a scraping offset.
　　//停止，退出
	case <-sl.scrapeCtx.Done():
		close(sl.stopped)
		return
	}

	var last time.Time
　　//设置定时器
	ticker := time.NewTicker(interval)
	defer ticker.Stop()

mainLoop:
	for {
		select {
		case <-sl.ctx.Done():
			close(sl.stopped)
			return
		case <-sl.scrapeCtx.Done():
			break mainLoop
		default:
		}

		var (
			start             = time.Now()
			scrapeCtx, cancel = context.WithTimeout(sl.ctx, timeout)
		)

		// Only record after the first scrape.
		if !last.IsZero() {
			targetIntervalLength.WithLabelValues(interval.String()).Observe(
				time.Since(last).Seconds(),
			)
		}
        //获取上次scrape(拉取)指标(metric)占用空间
		b := sl.buffers.Get(sl.lastScrapeSize).([]byte)
　      //根据上次的占用的空间申请存储空间
		buf := bytes.NewBuffer(b)

        //开始scrape(拉取)指标
		contentType, scrapeErr := sl.scraper.scrape(scrapeCtx, buf)
		cancel()

		if scrapeErr == nil {
			b = buf.Bytes()
			// NOTE: There were issues with misbehaving clients in the past
			// that occasionally returned empty results. We don't want those
			// to falsely reset our buffer size.
			if len(b) > 0 {
　　　　　　　　　//存储本次scrape拉取磁盘占用的空间，留待下次scrape(拉取)使用
				sl.lastScrapeSize = len(b)
			}
		} else {
			level.Debug(sl.l).Log("msg", "Scrape failed", "err", scrapeErr.Error())
			if errc != nil {
				errc <- scrapeErr
			}
		}

		// A failed scrape is the same as an empty scrape,
		// we still call sl.append to trigger stale markers.
        //存储指标
		total, added, appErr := sl.append(b, contentType, start)
		if appErr != nil {
			level.Warn(sl.l).Log("msg", "append failed", "err", appErr)
			// The append failed, probably due to a parse error or sample limit.
			// Call sl.append again with an empty scrape to trigger stale markers.
			if _, _, err := sl.append([]byte{}, "", start); err != nil {
				level.Warn(sl.l).Log("msg", "append failed", "err", err)
			}
		}

		sl.buffers.Put(b)

		if scrapeErr == nil {
			scrapeErr = appErr
		}

		if err := sl.report(start, time.Since(start), total, added, scrapeErr); err != nil {
			level.Warn(sl.l).Log("msg", "appending scrape report failed", "err", err)
		}
		last = start

        //停止scrapeLoop
		select {
		case <-sl.ctx.Done():
			close(sl.stopped)
			return
		case <-sl.scrapeCtx.Done():
			break mainLoop
		case <-ticker.C:
		}
	}

	close(sl.stopped)

	sl.endOfRunStaleness(last, ticker, interval)
}

　　run方法主要实现两个功能：指标采集(scrape)和指标存储．此外，为了实现对象的复用，在采集(scrape)过程中，使用了sync.Pool机制提高性能，即每次采集(scrape)完成后，都会申请和本次采集(scrape)指标存储空间一样的大小的bytes，加入到buffer中，以备下次指标采集(scrape)直接使用．

• 指标采集(scrapeManager)配置初始化和应用

　　指标采集(scrapeManager)调用scrapeManager.ApplyConfig方法，完成配置初始化及应用，具体方法如下：

prometheus/scrape/manager.go

// ApplyConfig resets the manager's target providers and job configurations as defined by the new cfg.
func (m *Manager) ApplyConfig(cfg *config.Config) error {
   //操作前加锁
	m.mtxScrape.Lock()
   //完成后解锁
	defer m.mtxScrape.Unlock()ApplyConfig

　　// 创建一个map，key是job_name，value是结构体config.ScrapeConfig
	c := make(map[string]*config.ScrapeConfig)
	for _, scfg := range cfg.ScrapeConfigs {
		c[scfg.JobName] = scfg
	}
	m.scrapeConfigs = c

    //首次启动不执行
	// Cleanup and reload pool if config has changed.
	for name, sp := range m.scrapePools {
        // 若job_name在scrapePools中，不在scrapeConfigs中，则说明已经更新，停止该job_name对应的scrapePool
		if cfg, ok := m.scrapeConfigs[name]; !ok {
			sp.stop()
			delete(m.scrapePools, name)
		} else if !reflect.DeepEqual(sp.config, cfg) {
            // 若job_name在scrapePools中，也在scrapeConfigs中，但配置有变化，比如target增加或减少，需要重新加载
			sp.reload(cfg)
		}
	}

	return nil
}

　　调用reload方法重新加载配置文件：

prometheus/scrape/scrape.go

// reload the scrape pool with the given scrape configuration. The target state is preserved
// but all scrape loops are restarted with the new scrape configuration.
// This method returns after all scrape loops that were stopped have stopped scraping.
func (sp *scrapePool) reload(cfg *config.ScrapeConfig) {
	start := time.Now()

    // 操作前加锁
	sp.mtx.Lock()
　　// 完成后解锁
	defer sp.mtx.Unlock()

    // 生成client，用于获取指标(metircs)
	client, err := config_util.NewClientFromConfig(cfg.HTTPClientConfig, cfg.JobName)
	if err != nil {
		// Any errors that could occur here should be caught during config validation.
		level.Error(sp.logger).Log("msg", "Error creating HTTP client", "err", err)
	}
	sp.config = cfg
	sp.client = client

	var (
		wg       sync.WaitGroup
		interval = time.Duration(sp.config.ScrapeInterval)
		timeout  = time.Duration(sp.config.ScrapeTimeout)
		limit    = int(sp.config.SampleLimit)
		honor    = sp.config.HonorLabels
		mrc      = sp.config.MetricRelabelConfigs
	)

　　// 停止该scrapePool下对应的所有的oldLoop，更具配置创建所有的newLoop，并通过协程启动．
	for fp, oldLoop := range sp.loops {
		var (
			t       = sp.activeTargets[fp]
			s       = &targetScraper{Target: t, client: sp.client, timeout: timeout}
			newLoop = sp.newLoop(t, s, limit, honor, mrc)
		)
		wg.Add(1)

		go func(oldLoop, newLoop loop) {
			oldLoop.stop()
			wg.Done()

			go newLoop.run(interval, timeout, nil)
		}(oldLoop, newLoop)

		sp.loops[fp] = newLoop
	}

	wg.Wait()
	targetReloadIntervalLength.WithLabelValues(interval.String()).Observe(
		time.Since(start).Seconds(),
	)
}

至此，指标采集(scrapeManager)功能分析结束