前言
在我们使用服务器的时候,有时候需要监控文件或文件夹的变化。例如,定期扫描文件夹下是否有某一类型的文件生成。
今天,我们介绍如何使用 Python 来监控文件系统。
在 Python 中,主要有两个监控文件系统的库:
pyinotify:https://github.com/seb-m/pyinotify/wiki
pyinotify 依赖的是 Linux 的 inotify,所以它只能在 linux 中使用
watchdog: http://pythonhosted.org/watchdog/
watchdog 是跨平台(linux、Windows、macOS)的库,对不同平台都进行了封装。但是需要安装依赖库。
通常我们使用的服务器都是 linux 系统,所以为了避免安装依赖包的麻烦,我们使用 pyinotify 库来实现我们的监控文件系统的功能。
介绍
1. 安装
依赖
Linux kernel with inotify ≥ 2.6.13
Python ≥ 2.4 or 3.x
安装
pip install pyinotify
使用
比方说,我们要监视 /tmp 及其所有子目录中的每一个新文件的创建或删除。为了简单起见,我们只在标准输出上打印每个通知的消息
现在有几种读取事件并采取适当操作的策略,其中有四种流行的模式:
使用 的 loop() 方法进行不间断的监视。在这种情况下,当前进程只需要专注于此任务
通过在 Notifier 中使用 timeout 通知程序在方便时显式调用处理方法来进行定期监视。在这种情况下,它不会阻塞您当前的线程,但是如果超时值设置得太小,也可能导致少量的响应性丢失或资源使用增加
生成新线程来独立地监视文件事件,此方法实例化 ThreadedNotifier
通过 AsyncNotifier 类进行异步监控
具体使用哪种策略主要取决于您的需要和您的环境,下面我们来实现这四种策略
共有代码
import pyinotify
# The watch manager stores the watches and provides operations on watches
wm = pyinotify.WatchManager()
# 监视事件
mask = pyinotify.IN_DELETE | pyinotify.IN_CREATE
class EventHandler(pyinotify.ProcessEvent):
def process_IN_CREATE(self, event):
print "Creating:", event.pathname
def process_IN_DELETE(self, event):
print "Removing:", event.pathname
我们定义了一个 EventHandler 类,并继承一个 ProcessEvent 处理事件类。
对于某一 EVENT_TYPE,其对应的 process_EVENT_TYPE 函数将被执行。
EVENT_TYPE 包括:
image.png
1. 不加 timeout 的 Notifier
# 创建事件处理对象
handler = EventHandler()
notifier = pyinotify.Notifier(wm, handler)
wdd = wm.add_watch('/tmp', mask, rec=True)
notifier.loop()
接下去就要添加需要监听的目录
wdd = wm.add_watch('/tmp', mask, rec=True)
最后,执行监听,在使用 Ctrl+C 终止之前,程序都会一直运行
notifier.loop()
2. 使用 timeout 的 Notifier
如果您希望定期检查事件而不是阻塞事件,可以使用较短的超时值来构造 Notifier
notifier = pyinotify.Notifier(wm, handler, timeout=10)
并使用这样的函数检查和处理事件
def quick_check(notifier):
assert notifier._timeout is not None, 'Notifier must be constructed with a short timeout'
notifier.process_events()
while notifier.check_events(): #loop in case more events appear while we are processing
notifier.read_events()
notifier.process_events()
3. 使用 ThreadedNotifier 类
notifier = pyinotify.ThreadedNotifier(wm, EventHandler())
notifier.start()
wdd = wm.add_watch('/tmp', mask, rec=True)
wm.rm_watch(wdd.values())
notifier.stop()
当然,在需要的时候,我们也可以删除监视目录
if wdd['/tmp'] > 0: # test if the wd is valid, ie. if /tmp is being watched, this test is not mandatory though
wm.rm_watch(wdd['/tmp'])
注意:如果 /tmp 中存在子目录,那么还会监听其子目录。如果要删除 /tmp 及其子目录的监听,可以使用 rec=True
wm.rm_watch(wdd['/tmp'], rec=True)
或者
wm.rm_watch(wdd.values())
在我们操作完目录之后,最后需要将监听事件停止
notifier.stop()
4. 使用 AsyncNotifier 类
import asyncore
notifier = pyinotify.AsyncNotifier(wm, EventHandler())
wdd = wm.add_watch('/tmp', mask, rec=True)
asyncore.loop()
这个 AsyncNotifier 类依赖于 Python 标准模块 asyncore,是轮询事件的一个替代方法。
示例
import pyinotify
import sys
class EventHandler(pyinotify.ProcessEvent):
def my_init(self, file_object=sys.stdout):
"""
This is your constructor it is automatically called from
ProcessEvent.__init__(), And extra arguments passed to __init__() would
be delegated automatically to my_init().
"""
self._file_object = file_object
self.sign = False
def process_IN_CREATE(self, event):
"""
This method processes a specific type of event: IN_CREATE. event
is an instance of Event.
"""
if hasattr(event, 'pathname') and event.pathname.endswith('.xls'):
print('%s have been created.\n' % event.pathname, file=self._file_object)
def process_IN_DELETE(self, event):
"""
This method processes a specific type of event: IN_DELETE. event
is an instance of Event.
"""
print('deleting: %s\n' % event.pathname, file=self._file_object)
# pass
def process_IN_CLOSE(self, event):
"""
This method is called on these events: IN_CLOSE_WRITE and
IN_CLOSE_NOWRITE.
"""
# print('closing: %s\n' % event.pathname, file=self._file_object)
pass
def process_IN_CLOSE_WRITE(self, event):
"""
This method processes a specific type of event: IN_CLOSE_WRITE
"""
pass
def process_default(self, event):
"""
Eventually, this method is called for all others types of events.
This method can be useful when an action fits all events.
"""
# print('default processing\n', file=self._file_object)
pass
def watching(path, exclude_path=None, rec=False, read_freq=0, timeout=None):
""" watch files or directories
@args:
path: str or list of str, Path to watch, the path can either be a
file or a directory. Also accepts a sequence (list) of paths.
exclude_path: str or list, predicate (boolean function), which returns True
if the current path must be excluded from being watched. This
argument has precedence over exclude_filter passed to the
class' constructor.
rec: Recursively add watches from path on all its subdirectories,
set to False by default (doesn't follows symlinks in any case)
read_freq: if read_freq == 0, events are read asap, if read_freq is > 0,
this thread sleeps max(0, read_freq - (timeout / 1000)) seconds.
But if timeout is None it may be different because poll is
blocking waiting for something to read.
timeout: see read_freq above. If provided, it must be set in milliseconds
"""
# Instanciate a new WatchManager (will be used to store watches)
wm = pyinotify.WatchManager()
# events types
mask = pyinotify.IN_DELETE | pyinotify.IN_CREATE
# Associate this WatchManager with a Notifier (will be used to report and process events).
notifier = pyinotify.Notifier(wm, EventHandler(), read_freq=read_freq, timeout=timeout)
# Add a new watch on 'path' for some XXX_EVENTS.
if isinstance(path, str) or isinstance(path, list):
print("now starting monitor %s." %path)
wm.add_watch(path, mask, rec=rec, exclude_filter=exclude_path)
else:
raise ValueError("the %s seems not valid path" %(path))
# Loop forever and handle events.
notifier.loop()
if __name__ == '__main__':
path = '~/jupyter/Others/Wechat'
watching(path)
这个脚本监控目录下 .xls 类型文件的创建和任意文件删除
运行
$ python watching.py
now starting monitor ~/jupyter/Others/Wechat.
新建 xls 文件
touch a.xls b.xls c.txt
输出
~/jupyter/Others/Wechat/a.xls have been created.
~/jupyter/Others/Wechat/b.xls have been created.
删除文件
ls | egrep '.*xls|txt' | xargs rm
输出
deleting: ~/jupyter/Others/Wechat/a.xls
deleting: ~/jupyter/Others/Wechat/b.xls
deleting: ~/jupyter/Others/Wechat/c.txt
总结
上面的脚本主要用来监听目录下面文件的创建和删除,如果想要监听其他事件,可以在脚本的基础上进行改下,并添加对应的时间处理。
同时,如果想要定时监控的话,可以设置 read_freq 和 timeout 参数。
计算公式为:
max(0, read_freq - (timeout / 1000))
文件 github 地址:
https://github.com/dxsbiocc/learn/blob/main/Python/utils/watching.py