对 Windows GUI进行自动化控制的工具有很多，比如pywinauto、pyautogui、pywin32、Autoit、airtest、UIAutomation等，UI Automation API是微软提供的自动化框架，可在支持 Windows Presentation Foundation (WPF) 的所有操作系统上使用，支持的应用类型更多。本文介绍封装了UI Automation API的Python uiautomation 模块的使用方法。
Python uiautomation 模块由yinkaisheng 开发，封装了微软 UI Automation API，支持自动化Win32，MFC，WPF，Modern UI(Metro UI), Qt, IE, Firefox, Chrome和基于Electron开发的应用程序。
环境准备
uiautomation安装
最新版uiautomation2.0只支持Python 3版本，但不要使用3.7.6和3.8.1这两个版本，因为comtypes包在这两个版本中不能正常工作。
pip安装uiautomation：
$ pip install uiautomation
检查是否安装成功：
$ pip list | findstr uiautomation
uiautomation 2.0.18
安装完成后，在Python的Scripts(我的路径为C:\Program Files\Python37\Scripts)目录中会有一个文件automation.py，是用来枚举控件树结构的一个脚本。
可运行 automation.py -h查看命令帮助:
$ python automation.py -h
UIAutomation 2.0.18 (Python 3.7.2, 64 bit)
usage
-h show command help
-t delay time, default 3 seconds, begin to enumerate after Value seconds, this must be an integer
you can delay a few seconds and make a window active so automation can enumerate the active window
-d enumerate tree depth, this must be an integer, if it is null, enumerate the whole tree
-r enumerate from root:Desktop window, if it is null, enumerate from foreground window
-f enumerate from focused control, if it is null, enumerate from foreground window
-c enumerate the control under cursor, if depth is < 0, enumerate from its ancestor up to depth
-a show ancestors of the control under cursor
-n show control full name, if it is null, show first 30 characters of control's name in console,
always show full name in log file @AutomationLog.txt
-p show process id of controls
if UnicodeError or LookupError occurred when printing,
try to change the active code page of console window by using chcp or see the log file @AutomationLog.txt
chcp, get current active code page
chcp 936, set active code page to gbk
chcp 65001, set active code page to utf-8
examples:
automation.py -t3
automation.py -t3 -r -d1 -m -n
automation.py -c -t3
进程查看器
对 Windows GUI进行自动化控制需要使用进程查看器工具对GUI界面元素进行定位，定位工具有很多，这里推荐使用微软提供的inspect.exe 或者 Accessibility Insights 这两款工具。
inspect.exe
inspect.exe 是 Windows SDK 自带的一个进程查看器，可以用来查看系统正在运行的进程信息、模块、线程、堆栈跟踪等详细数据。
Windows SDK下载地址为：https://developer.microsoft.com/en-us/windows/downloads/sdk-archive/
建议直接到这里下载inspect.exe：https://github.com/yinkaisheng/Python-UIAutomation-for-Windows/tree/master/inspect
64位系统版本的inspect.exe也可以点击这里下载。
Accessibility Insights
Accessibility Insights 是微软开发的一款辅助功能测试工具。它可以帮助开发者测试 web 应用、Windows 桌面应用和 Android 应用的可访问性，确保这些应用程序符合无障碍标准。
Accessibility Insights获取的控件属性信息没有inspect.exe全面，使用起来更加流畅。下载为：https://accessibilityinsights.io/downloads/
控件对象模型
微软 UIAutomation API定义了支持的控件类型和对应的模型（Pattern），所有支持的控件类型可参考：https://learn.microsoft.com/en-us/windows/win32/winauto/uiauto-controlpatternmapping
| 控件类型 | 必须支持的模型 | 可选模型 | Does not support |
|---|---|---|---|
| Button | None | ExpandCollapse, Invoke, Toggle, Value | None |
| Calendar | Grid, Table | Scroll, Selection | Value |
| CheckBox | Toggle | None | None |
| Edit | None | RangeValue, Text, Value | None |
| List | None | Grid, MultipleView, Scroll, Selection | Table |
| ListItem | SelectionItem | CustomNavigation, ExpandCollapse, GridItem, Invoke, ScrollItem, Toggle, Value | None |
| Menu | None | None | None |
| MenuBar | None | Dock, ExpandCollapse, Transform | None |
| MenuItem | None | ExpandCollapse, Invoke, SelectionItem, Toggle | None |
| RadioButton | SelectionItem | None | Toggle |
| SplitButton | ExpandCollapse, Invoke | None | None |
| Tab | Selection | Scroll | None |
| TabItem | SelectionItem | None | Invoke |
| Table | Grid, GridItem, Table, TableItem | None | None |
| Text | None | GridItem, TableItem, Text | Value |
| TitleBar | None | None | None |
| ToolBar | None | Dock, ExpandCollapse, Transform | None |
python uiautomation库对UIAutomation API定义的各个Control和Pattern进行了封装。
下面来看使用python uiautomation操作Windows自带计算器的例子。
uiautomation库示例
控制计算器
可以使用inspect.exe来定位计算器元素：
示例脚本如下：
import os
import uiautomation as auto
import subprocess
class uiautoCalc(Loggers):
"""uiautomation控制计算器
"""
def __init__(self):
super().__init__()
self.logger = Loggers().myLogger()
auto.uiautomation.DEBUG_SEARCH_TIME =True
auto.uiautomation.SetGlobalSearchTimeout(2) # 设置全局搜索超时时间
self.calcWindow = auto.WindowControl(searchDepth=1, Name='计算器', desc='计算器窗口') # 计算器窗口
if not self.calcWindow.Exists(0,0):
subprocess.Popen('calc')# 设置窗口前置
self.calcWindow = auto.WindowControl(
searchDepth=1, Name='计算器', desc='计算器窗口')
self.calcWindow.SetActive() # 激活窗口
self.calcWindow.SetTopmost(True) # 设置为顶层
def gotoScientific(self):
self.calcWindow.ButtonControl(AutomationId='TogglePaneButton', desc='打开导航').Click(waitTime=0.01)
self.calcWindow.ListItemControl(AutomationId='Scientific', desc='选择科学计算器').Click(waitTime=0.01)
clearButton = self.calcWindow.ButtonControl(AutomationId='clearEntryButton', desc='点击CE清空输入')
if clearButton.Exists(0,0):
clearButton.Click(waitTime=0)
else:
self.calcWindow.ButtonControl(AutomationId='clearButton', desc='点击C清空输入').Click(waitTime=0.01)
def getKeyControl(self):
automationId2key ={'num0Button':'0','num1Button':'1','num2Button':'2','num3Button':'3','num4Button':'4','num5Button':'5','num6Button':'6','num7Button':'7','num8Button':'8','num9Button':'9','decimalSeparatorButton':'.','plusButton':'+','minusButton':'-','multiplyButton':'*','divideButton':'/','equalButton':'=','openParenthesisButton':'(','closeParenthesisButton':')'}
calckeys = self.calcWindow.GroupControl(ClassName='LandmarkTarget')
keyControl ={}
for control, depth in auto.WalkControl(calckeys, maxDepth=3):
if control.AutomationId in automationId2key:
self.logger.info(control.AutomationId)
keyControl[automationId2key[control.AutomationId]]= control
return keyControl
def calculate(self, expression, keyControl):
expression =''.join(expression.split())
if not expression.endswith('='):
expression +='='
for char in expression:
keyControl[char].Click(waitTime=0)
self.calcWindow.SendKeys('{Ctrl}c', waitTime=0.1)
return auto.GetClipboardText()
def calc_demo(self):
"""计算器示例
:return :
"""
self.gotoScientific() # 选择科学计算器
keyControl = self.getKeyControl() # 获取按键控件
result = self.calculate('(1 + 2 - 3) * 4 / 5.6 - 7', keyControl)
print('(1 + 2 - 3) * 4 / 5.6 - 7 =', result)
self.calcWindow.CaptureToImage('calc.png', x=7, y=0, width=-14, height=-7) # 截图
self.calcWindow.GetWindowPattern().Close() # 关闭计算机
if __name__ == "__main__":
ui = uiautoCalc()
ui.calc_demo()
脚本执行动图：
参考文档
-
https://github.com/yinkaisheng/Python-UIAutomation-for-Windows
-
Python UIAutomation文档：https://github.com/yinkaisheng/Python-UIAutomation-for-Windows/blob/master/readme_cn.md
-
GitHub - mhammond/pywin32: Python for Windows (pywin32) Extensions
-
Accessibility tools - Inspect - Win32 apps | Microsoft Learn
1万+
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
