[Python-Matplotlib] 动态图表绘制来达到实时监控

一年后重新登录CSDN,才发现原来有人会看这个东西的。 重新调整了下这文章.......

工作期间很少用CSDN 了, 如果有问题想问的可以发邮件给我.....QQ邮箱就是我用户名里的数字...

需求:

当时因为网速的问题, 使我们的系统加载都很慢。 为了知道一天下来到底有多少时间,加载网页时间是很慢的,然后跟公司的IT撕去才写的这个东西。大概就是自动登录系统后, 统计完全加载页面所需要的时间, 然后绘制动态图+表。  

代码部分参考了大神的  https://blog.csdn.net/yc_1993/article/details/54933751。建议先去大神那里看完再来这看, 虽然我当时也是一看就懂, 一做就报错.......

除了大神实现的效果外,我还另外加上了滚动条与暂停功能。代码中还包含了一些小的功能,例如可以对每个数值进行设置y-value。但是我综合考虑了最终的效果,最后都注释掉了,如果有需要的话,可以取消注释来使用。

最终效果如下:

图像会根据数据实时进行更新,也就是数据跟图像都是实时刷新的。表格会显示最近的五个数值。如果直接拉动滚动条的话,可能反应会有点慢。可以先按下 “Pause” 按钮,暂停绘图,然后拉动下方的滚动条来回顾过去的数据。

 

代码分为两个部分,一部分是数据如何产生,第二部分为图表绘制。

1. 数据产生

数据是实时产生的(初始是没有的,需要成功登录网页后获取到相关数据后才有),利用Selenium,每5秒刷新某网页,并调用JS "Windonw.performance.timing“ 来计算完全加载所需时间,并得到当前时间。有关Selenium的使用,可以谷歌相关的用法, 我还另外写了个坑点总结 [Python-Selenium] 入门总结+坑点陈列

2. 图表绘制

利用ion来绘制动态图。那么分为三大部分

  1. 1)  初始值的设定 
  2. 2)框架的定义
  3. 3)数据传入并绘制
  4. 4 ) 关闭ion并显示图像

重点讲解下一下几点

  • 数据如何传入

简单来说就是要让数据的两个因素, dtime还有onloadingtime能够传入,需要结合下面的调用环节来理解。这里就是简单地在函数DetectionPlot那里传入参数,并把新得到的参数,附加成为列表类型。

  • 横纵坐标的设定
self.loadingGraph.xaxis.set_ticks(self.timestamp1)
self.loadingGraph.yaxis.set_ticks(self.loadingValue1)

如果直接设置了set_ticks的话,那么以下这几行代码将不会生效,图标的横纵坐标就会不断地压缩, 而不是动态的调整轴的最小最大值。

self.loadingGraph.set_ylim(
                min(self.loadingValueRange), max(self.loadingValueRange))
self.loadingGraph.set_xlim(
                self.timestampRange[1] - dt.timedelta(minutes=1), self.timestampRange[1])

另外注意一点就是,如果你想横坐标是时间,你需要首先将时间从string类型转换为datetime类型,这样才能够与datetime.timedelta进行相加减。然后利用下面的两行代码,将横坐标的格式定义为datetime类型,那么Matplotlib就能帮你处理。纵坐标也是相同的道理。

self.dateFormat = DateFormatter("%m-%d %H:%M:%S")
self.loadingGraph.xaxis.set_major_formatter(ticker.FuncFormatter(self.dateFormat))
  • 滚动条与暂停功能

这两个部分,就是首先定义相关的位置,然后与功能函数挂钩。Matplotlib的API上有相关的简介。

这里具体说一下,由于横坐标是时间类型,那么怎么设置滚动条涅?

首先Slider定义上包含了一个最小值,跟最大值。这里要对datetime类型进行转换。

功能定义函数上,就要转换为datetime类型,要不然matplotlib识别不了。然后就是利用拉动的变量来定义横坐标。

# Add a scroll bar
        self.axcolor = 'lightgoldenrodyellow'
        self.axsb = fig.add_axes([0.06, 0.25, 0.65, 0.02], facecolor=self.axcolor)

#Slider setting
        self.ssb = Slider(self.axsb, 'Scrollbar',
                              dates.date2num(self.timestampRange[0]),
                              dates.date2num(self.timestampRange[1]))
        self.ssb.on_changed(self.update)

#Define the slider update function
    def update(self,val):
        self.sb=self.ssb.val
        self.xmin_time=dates.num2date(self.sb)-dt.timedelta(minutes=1)
        self.xmax_time=dates.num2date(self.sb)
        self.loadingGraph.axis([self.xmin_time,self.xmax_time,min(self.loadingValueRange), max(self.loadingValueRange)])
        fig.canvas.draw_idle()

 

3. 调用

数据部分由于需要实时返回,因此我使用了yield这个generator。注意不要用return,return只能返回一次值就会停止了。

当需要使用yield回来的两个值时候:

    for times,values in Getdata():
        graph.DetectionPlot(times,values)

你也可以使用静态形式来查看图表的情况,取消注释 以下两段。

timelist = ['08-31 11:17:27', '08-31 11:17:34', '08-31 11:17:41', '08-31 11:18:27',
             '08-31 11:18:47', '08-31 11:19:47','08-31 11:19:49']
onloadlist = ['25032', '6558', '383', '3383', '1111', '2345','5000']

 

#If timelist and onloadlist is not "#", you can use the following to do the static test.
for i in range(len(timelist)):
    graph.DetectionPlot(timelist[i], onloadlist[i])

然后注释掉上面的Getdata()部分。

完整代码如下:

https://github.com/cyx441984694/Matplotlib_in_Real_Time_to_Plot_Webpage_Loading_Time/blob/master/matwithselen.py

from matplotlib.widgets import Button, Slider
import datetime as dt
import matplotlib.pyplot as plt
from matplotlib.dates import DateFormatter, DayLocator
import matplotlib.ticker as ticker
from datetime import datetime
import matplotlib.transforms as transforms
from matplotlib import dates
import getpass
import time
import sys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.chrome.options import Options
from datetime import datetime
from selenium.common.exceptions import NoSuchElementException
from selenium import webdriver
import matplotlib
matplotlib.use('TkAgg')

class streamDetectionPlot(object):
    # Initial the figure parameters
    def __init__(self):
        # Turn the interactive mode on
        plt.ion()
        # initial the plot variable
        self.timestamp = []
        self.loadingValue = []
        self.highlightList = []
        self.highlightListTurnOn = True
        self.loadingValueRange = [0, 1]
        self.timestampRange = [0, 1]
        self.tableValue = [[0, 0]]
        self.timestamp1 = []
        self.loadingValue1 = []

        # initial the figure
        global fig
        fig = plt.figure(figsize=(18, 8), facecolor="white")
        fig.subplots_adjust(left=0.06, right=0.70)
        self.loadingGraph = fig.add_subplot(2, 1, 1)
        ##add_axes(rect,**kwargs) Meaning:##rect: left,bottom,width,height
        self.loadingTable = fig.add_axes([0.72, 0.1, 0.2, 0.8], frameon=False)

        # Add a scroll bar
        self.axcolor = 'lightgoldenrodyellow'
        self.axsb = fig.add_axes([0.06, 0.25, 0.65, 0.02], facecolor=self.axcolor)

        # Add the pause button
        self.pauseax = fig.add_axes([0.8, 0.25, 0.05, 0.02], facecolor=self.axcolor)

    # define the initial plot method
    def initPlot(self):
        # initial two lines
        self.line, = self.loadingGraph.plot_date(self.timestamp, self.loadingValue, fmt="-", color="red",
                                                 label="LoadingValue")
        self.loadingGraph.legend(loc="upper right", frameon=False)
        self.loadingGraph.grid(True)

        # Set the baseline of 5000, horizen use "axhline" while vertical uses "axvline"
        self.baseline = self.loadingGraph.axhline(5000, color='black', lw=1)

        # Add the "5000" to the y-axis label next to the line
        # trans = transforms.blended_transform_factory(
        #     self.loadingGraph.get_yticklabels()[0].get_transform(), self.loadingGraph.transData)
        # self.loadingGraph.text(0, 5000, "{:.0f}".format(5000), color="black", transform=trans, ha="right", va="center")

        # Set the title
        self.loadingGraph.set_title("Live update test")

        # set the x/y label of the first graph
        self.loadingGraph.set_xlabel("datetime")
        self.loadingGraph.set_ylabel("Loading Time")

        # axis format
        self.dateFormat = DateFormatter("%m-%d %H:%M:%S")
        self.loadingGraph.xaxis.set_major_formatter(ticker.FuncFormatter(self.dateFormat))

        ##Configure the table
        self.loadingTableColumnName = ["timestamp", "Loading value"]
        self.loadingTable.set_xticks([])
        self.loadingTable.set_yticks([])

    # define the output method
    def DetectionPlot(self, timestamp, loadingValue):
        if not pause:
            # update the plot value of the graph
            self.timestamp.append(timestamp)
            self.loadingValue.append(loadingValue)

            # From string to datetime
            self.timestamp1 = [datetime.strptime(d, "%m-%d %H:%M:%S") for d in self.timestamp]
            self.loadingValue = [float(i) for i in self.loadingValue]

            # pretty date names
            plt.gcf().autofmt_xdate()

            # update the x/y range
            self.timestampRange = [min(self.timestamp1), max(self.timestamp1) + dt.timedelta(minutes=0.1)]  # datetime style
            self.loadingValueRange = [min(self.loadingValue), max(self.loadingValue) + 1]

            # update the x/y axis limits
            self.loadingGraph.set_ylim(
                min(self.loadingValueRange), max(self.loadingValueRange))
            self.loadingGraph.set_xlim(
                self.timestampRange[1] - dt.timedelta(minutes=1), self.timestampRange[1])

            # Set the y-ticks if the variation is above 1000
            # https://stackoverflow.com/questions/39969217/ytick-overlapping-in-matplotlib
            # self.loadingValue1 = [min(self.loadingValue)]
            # for i in sorted(self.loadingValue[0:]):
            #     if i - self.loadingValue1[-1] > 1000:
            #         self.loadingValue1.append(i)

            # Set the ticks
            # self.loadingGraph.xaxis.set_ticks(self.timestamp1)
            # self.loadingGraph.yaxis.set_ticks(self.loadingValue1)

            # update the two lines
            self.line.set_xdata(self.timestamp1)
            self.line.set_ydata(self.loadingValue)

            # The x-data is being updated, but the plot ranges are not. When I put new data on the plot, it was all out of range. The solution was to add:
            self.loadingGraph.relim()
            self.loadingGraph.autoscale_view()

            # Rotate the x-array
            for tick in self.loadingGraph.get_xticklabels():
                tick.set_rotation(15)

            # Plot the value of y on the plot.
            # for xp, yp in zip(self.timestamp1[1:-1], self.loadingValue[1:-1]):
            #     label = "%s" % yp
            #     self.loadingGraph.text(xp, yp, label, fontsize=8, horizontalalignment='right', verticalalignment='bottom')

            #Clear the text first
            for txt in self.loadingGraph.texts:
                txt.set_visible(False)
            #Set the maximum and minimum value of y on the plot.
            self.max_y, self.min_y = max(self.loadingValue), min(self.loadingValue)
            self.max_x, self.min_x = self.timestamp1[self.loadingValue.index(self.max_y)], self.timestamp1[
                self.loadingValue.index(self.min_y)]
            self.maxlabel, self.minlabel = "Max: %s ms" % self.max_y, "Min: %s ms" % self.min_y
            self.loadingGraph.text(self.max_x, self.max_y, self.maxlabel, fontsize=8, horizontalalignment='right',
                                   verticalalignment='bottom')
            self.loadingGraph.text(self.min_x, self.min_y, self.minlabel, fontsize=8, horizontalalignment='right',
                                   verticalalignment='bottom')


            # update the highlight of the graph. The x will be 1 second varitation
            if int(loadingValue) >= 5000:
                self.highlightList = self.timestamp1
                self.highlightListTurnOn = True
            else:
                self.highlightListTurnOn = False
            if len(self.highlightList) != 0 and self.highlightListTurnOn is False:
                self.loadingGraph.axvspan(
                    self.highlightList[0] - dt.timedelta(seconds=1),
                    self.highlightList[-1] + dt.timedelta(seconds=1),
                    color='r',
                    edgecolor=None,
                    alpha=0.2
                )
                self.highlightList = []
                self.highlightListTurnOn = True

            # Update the table with the latest 5 entries
            self.loadingTableColumnName = ["timestamp", "Loading value"]
            self.loadingTable.text(0.15, 1, "LoadingTable", size=12)
            self.loadingTable = fig.add_axes([0.72, 0.1, 0.2, 0.8], frameon=False)
            self.loadingTable.set_xticks([])
            self.loadingTable.set_yticks([])
            self.tableValue.append([timestamp, loadingValue])

            if len(self.tableValue) >= 6: self.tableValue.pop(0)
            self.loadingTable.table(cellText=self.tableValue,
                                    colWidths=[0.5] * 2,
                                    colLabels=self.loadingTableColumnName,
                                    loc=1,
                                    cellLoc='center'
                                    )

            #Slider setting
            self.ssb = Slider(self.axsb, 'Scrollbar',
                              dates.date2num(self.timestampRange[0]),
                              dates.date2num(self.timestampRange[1]))
            self.ssb.on_changed(self.update)

        #Pause Button setting
        self.button=Button(self.pauseax,'Pause',color='0.85',hovercolor='0.975')
        self.button.on_clicked(self.setpause)


        # plot pause 0.0001 second and then plot the next one.
        plt.pause(1)
        plt.draw()

    #Define the slider update function
    def update(self,val):
        self.sb=self.ssb.val
        self.xmin_time=dates.num2date(self.sb)-dt.timedelta(minutes=1)
        self.xmax_time=dates.num2date(self.sb)
        self.loadingGraph.axis([self.xmin_time,self.xmax_time,min(self.loadingValueRange), max(self.loadingValueRange)])
        fig.canvas.draw_idle()

    #Define the pause button function
    def setpause(self,event):
        global pause
        pause = not pause
        print(pause)

    #Turn off the ion and show the plot.
    def close(self):
        plt.ioff()
        plt.show()



######################Data Part############################

def verificationcode():
    ##Test if the verification page exists or not
    key = input("Please input your verfication code:")
    if driver.find_element_by_id("smc"):
        verification = driver.find_element_by_id("smc")
        verification.send_keys(key)
        # if driver.find_element_by_id("emc"):
        #     emailverification = driver.find_element_by_id("emc")
        #     emailverification.send_keys(key)
    driver.find_element_by_id("save").click()

def Getdata():

    driver.get(source)

    try:
        driver.find_element_by_id("Login").click()
    except NoSuchElementException:
        print("Don't find the login button")

    while not driver.find_elements_by_id("cas2_ilecell"):
        if driver.find_elements_by_id("smc"):
            for i in range(1, 4):
                try:
                    verificationcode()
                    if driver.find_elements_by_id("smc-error"):
                        verificationcode()
                    else:
                        break
                except IOError:
                    break
        elif driver.find_elements_by_id("editPage"):
            print("Too much tries, please try it later")
            driver.quit()
    else:
        # print("Welcome")
        while True:
            try:

                navigationStart = driver.execute_script("return window.performance.timing.navigationStart")
                # domComplete = driver.execute_script("return window.performance.timing.domComplete")
                loadEvent = driver.execute_script("return window.performance.timing. loadEventEnd")
                onloadPerformance = loadEvent - navigationStart
                dtime = datetime.now().strftime("%m-%d %H:%M:%S")
                # print(dtime, ",", onloadPerformance, "ms")
                yield dtime,onloadPerformance

                time.sleep(2)
                driver.refresh()

            except TimeoutException:
                print("It took too long")
                break
        driver.quit()


# timelist = ['08-31 11:17:27', '08-31 11:17:34', '08-31 11:17:41', '08-31 11:18:27',
#             '08-31 11:18:47', '08-31 11:19:47','08-31 11:19:49']
# onloadlist = ['25032', '6558', '383', '3383', '1111', '2345','5000']

if __name__ == '__main__':
    # Get the local user name
    u = getpass.getuser()

    # # initialize Chrome options
    chrome_options = Options()
    # chrome_options.add_argument('--headless')

    chrome_options.add_argument('user-data-dir=C:\\Users\\%s\\AppData\\Local\\Google\\Chrome\\User Data' % (u))

    # chrome_driver_binary="E:\MyDownloads\Download\chromedriver.exe"
    source = "https://na66.salesforce.com/5000y00001SgXm0?srPos=0&srKp=500"
    driver = webdriver.Chrome(chrome_options=chrome_options)

    # Use the graph class
    graph = streamDetectionPlot()
    graph.initPlot()
    pause = False

    # Dynamic Test
    for times,values in Getdata():
        graph.DetectionPlot(times,values)

    # If timelist and onloadlist is not "#", you can use the following to do the static test.
    # for i in range(len(timelist)):
    #     graph.DetectionPlot(timelist[i], onloadlist[i])

    graph.close()

谢谢看到最后,希望对你有帮助。

发布了16 篇原创文章 · 获赞 17 · 访问量 3万+
展开阅读全文

没有更多推荐了,返回首页

©️2019 CSDN 皮肤主题: 大白 设计师: CSDN官方博客

分享到微信朋友圈

×

扫一扫,手机浏览