Android抓取文字、文字位置的分析

引文：

因为我弃用原来ATX框架中的uiautomator的东西，所以现在要把 UiSelector().text("XXX")这部分的功能给重新实现下。
所以这篇文章介绍的是抓取到页面中的文字还有文字的位置的方法及其分析。

现有的方法

1，UiSelector.text /自动化测试框架

大多数测试框架使用的方法。好像需要在手机上安装一个测试的app，没动手实践

2，Layout Inspector

android studio->Tools->android->Layout Inspector

显示很友好，没找到开源部分的代码

3，uiautomator dump

adb shell uiautomator dump

dump出来的东西会自动保存在/sdcard/window_dump.xml中，内容大概是这样

<?xml version="1.0" encoding="utf-8"?>
<hierarchy rotation="0">
    ...
    <node index="1" text="" 
    resource-id="com.android.contacts:id/contacts_unavailable_view" 
    class="android.widget.FrameLayout" 
    package="com.android.contacts" 
    content-desc="" checkable="false" checked="false" 
    clickable="false" enabled="true" focusable="false" 
    focused="false" scrollable="false" long-clickable="false" 
    password="false" selected="false" 
    bounds="[0,408][1080,1776]">
        <node index="0" text="" 
        resource-id="com.android.contacts:id/contacts_unavailable_container" class="android.widget.FrameLayout" 
        package="com.android.contacts" 
        content-desc="" checkable="false" checked="false" 
        clickable="false" enabled="true" focusable="false" 
        focused="false" scrollable="false" long-clickable="false" 
        password="false" selected="false" 
        bounds="[0,408][1080,1776]">
            <node index="0" text="" 
            resource-id="com.android.contacts:id/floating_action_button" 
            class="android.widget.ImageButton" 
            package="com.android.contacts" 
            content-desc="添加新联系人" checkable="false" 
            checked="false" clickable="true" 
            enabled="true" focusable="true" focused="false" 
            scrollable="false" long-clickable="false" 
            password="false" selected="false" 
            bounds="[864,1560][1032,1728]"/>
            ...
        </node>
    </node>
    ...
</hierarchy>

数据类型都是类似的，都是node节点，bounds就是各个view的边界了，举个例子

添加新联系人
bounds= 864,1560 1032,1728

所以重心就是 （948，1644）

adb shell input tap 948 1644

就点击到文字了。

ok，下面也是谈实现细节。dump调用链是这样的

DumpCommand:run
-> automationWrapper.getUiAutomation().getRootInActiveWindow()
-> AccessibilityInteractionClient:findAccessibilityNodeInfoByAccessibilityId
-> binder
-> ViewRootImpl.findAccessibilityNodeInfoByAccessibilityId

-> AccessibilityNodeInfoDumper.dumpWindowToFile

序列化的节点node的数据存在这里

AccessibilityNodeInfo.java

可以看到，它是一颗树的节点

public class AccessibilityNodeInfo implements Parcelable {
    ...
    private static final int MAX_POOL_SIZE = 50;
    private static final SynchronizedPool<AccessibilityNodeInfo> sPool =
            new SynchronizedPool<>(MAX_POOL_SIZE);
    ...
}

节点是这样建的

view.java

public AccessibilityNodeInfo createAccessibilityNodeInfoInternal() {
    AccessibilityNodeProvider provider = getAccessibilityNodeProvider();
    if (provider != null) {
        return provider.createAccessibilityNodeInfo(AccessibilityNodeProvider.HOST_VIEW_ID);
    } else {
        AccessibilityNodeInfo info = AccessibilityNodeInfo.obtain(this);
        onInitializeAccessibilityNodeInfo(info);
        return info;
    }
}
public void onInitializeAccessibilityNodeInfoInternal(AccessibilityNodeInfo info) {
    ...
        getDrawingRect(bounds);
        info.setBoundsInParent(bounds);

}

在TextView

Override
public void onInitializeAccessibilityNodeInfoInternal(AccessibilityNodeInfo info) {
    super.onInitializeAccessibilityNodeInfoInternal(info);

    final boolean isPassword = hasPasswordTransformationMethod();
    info.setPassword(isPassword);

    if (!isPassword || shouldSpeakPasswordsForAccessibility()) {
        info.setText(getTextForAccessibility());
    }
    ...
}

所以AccessibilityNodeInfo更像是在View中存了一个副本，这个副本可以用于辅助操作。

// TODO 更细节的分析

4,hierarchyviewer1

这个是在源码中找到的工具，发现它抓取的信息出乎意料的很全，比3中的信息还要多许多，是这样用的

.out/host/linux-x86/bin/hierarchyviewer1

是一个gui工具，打开之后，选中某个元素，可以看到Property中有absolute_x、absolute_y、getHeight、getWidth、mText。更加这些信息我们可以判断找的这个元素是不是在屏幕内，在的话，具体是哪个位置。

/sdk/hierachyviewer/ com.android.hierarchyviewer.scene.ViewHierarchyLoader

package com.android.hierarchyviewer.scene;
public class ViewHierarchyLoader {
    public static ViewHierarchyScene loadScene(IDevice device, Window window) {
            ...
            System.out.println("==> Starting client");

            socket = new Socket();
            socket.connect(new InetSocketAddress("127.0.0.1",
                    DeviceBridge.getDeviceLocalPort(device)));

            out = new BufferedWriter(new OutputStreamWriter(socket.getOutputStream()));
            in = new BufferedReader(new InputStreamReader(socket.getInputStream(), "utf-8"));

            System.out.println("==> DUMP");

            out.write("DUMP " + window.encode());
            out.newLine();
            out.flush();

            Stack<ViewNode> stack = new Stack<ViewNode>();

            boolean setRoot = true;
            ViewNode lastNode = null;
            int lastWhitespaceCount = Integer.MAX_VALUE;

            while ((line = in.readLine()) != null) {
                // debug by yeshen： 
                // System.out.println(line);
                if ("DONE.".equalsIgnoreCase(line)) {
                    break;
                }
                ...
            }
            ...
    }
}

看样子是自己建了一个socket取连本地的socket server，在代码里面打log，发现line的数据是类似这样的：

android.widget.Button@eb7e4a6 
text:mCurTextColor=9,-13224394text:mGravity=2,17 
text:mText=6,创建新联系人 
getEllipsize()=4,null 
text:getScaledTextSize()=4,14.0 
text:getSelectionEnd()=2,-1 
text:getSelectionStart()=2,-1 
text:getTextSize()=4,42.0 
text:getTypefaceStyle()=6,NORMAL 
layout:mBottom=3,144 
theme:com.android.contacts:style/PeopleTheme()=6,forced 
theme:android:style/Theme.DeviceDefault()=6,forced 
fg_=4,null 
mID=24,id/create_contact_button 
drawing:mLayerType=4,NONE 
layout:mLeft=1,0 
measurement:mMeasuredHeight=3,144 
measurement:mMeasuredWidth=3,324 
measurement:mMinHeight=3,144 
measurement:mMinWidth=3,264 
padding:mPaddingBottom=2,30 
padding:mPaddingLeft=2,36 
padding:mPaddingRight=2,36 
padding:mPaddingTop=2,30 
mPrivateFlags_DRAWN=4,0x20 
mPrivateFlags=9,0x1028830 
layout:mRight=3,324 
scrolling:mScrollX=1,0 
scrolling:mScrollY=1,0 
mSystemUiVisibility_SYSTEM_UI_FLAG_VISIBLE=3,0x0 
mSystemUiVisibility=3,0x0 
layout:mTop=1,0 
padding:mUserPaddingBottom=2,30 
padding:mUserPaddingEnd=11,-2147483648 
padding:mUserPaddingLeft=2,36 
padding:mUserPaddingRight=2,36 
padding:mUserPaddingStart=11,-2147483648 
mViewFlags=10,0x18004001 
drawing:getAlpha()=3,1.0 
layout:getBaseline()=2,88 
accessibility:getContentDescription()=4,null 
drawing:getElevation()=3,6.0 
getFilterTouchesWhenObscured()=5,false 
getFitsSystemWindows()=5,false 
layout:getHeight()=3,144 
accessibility:getImportantForAccessibility()=3,yes 
accessibility:getLabelFor()=2,-1 
layout:getLayoutDirection()=22,RESOLVED_DIRECTION_LTR 
layout:layout_gravity=4,NONE 
layout:layout_weight=3,0.0 
layout:layout_bottomMargin=2,45 
layout:layout_endMargin=11,-2147483648 
layout:layout_leftMargin=1,0 
layout:layout_mMarginFlags_LEFT_MARGIN_UNDEFINED_MASK=3,0x4 
layout:layout_mMarginFlags_RIGHT_MARGIN_UNDEFINED_MASK=3,0x8 
layout:layout_mMarginFlags=4,0x0C 
layout:layout_rightMargin=1,0 
layout:layout_startMargin=11,-2147483648 
layout:layout_topMargin=1,0 
layout:layout_height=12,WRAP_CONTENT 
layout:layout_width=12,MATCH_PARENT 
layout:getLocationOnScreen_x()=3,378 
layout:getLocationOnScreen_y()=3,757 
measurement:getMeasuredHeightAndState()=3,144 
measurement:getMeasuredWidthAndState()=3,324 
drawing:getPivotX()=5,162.0 
drawing:getPivotY()=4,72.0 
layout:getRawLayoutDirection()=7,INHERIT 
text:getRawTextAlignment()=7,GRAVITY 
text:getRawTextDirection()=7,INHERIT 
drawing:getRotation()=3,0.0 
drawing:getRotationX()=3,0.0 
drawing:getRotationY()=3,0.0 
drawing:getScaleX()=3,1.0 
drawing:getScaleY()=3,1.0 
getScrollBarStyle()=14,INSIDE_OVERLAY 
drawing:getSolidColor()=1,0 
getTag()=4,null 
text:getTextAlignment()=7,GRAVITY 
text:getTextDirection()=12,FIRST_STRONG 
drawing:getTransitionAlpha()=3,1.0 
getTransitionName()=4,null 
drawing:getTranslationX()=3,0.0 
drawing:getTranslationY()=3,0.0 
drawing:getTranslationZ()=3,0.0 
getVisibility()=7,VISIBLE 
layout:getWidth()=3,324 
drawing:getX()=3,0.0 
drawing:getY()=3,0.0 
drawing:getZ()=3,6.0 
focus:hasFocus()=5,false 
drawing:hasOverlappingRendering()=4,true 
drawing:hasShadow()=4,true 
layout:hasTransientState()=5,false 
isActivated()=5,false 
isClickable()=4,true 
drawing:isDrawingCacheEnabled()=5,false 
isEnabled()=4,true 
focus:isFocusable()=4,true 
isFocusableInTouchMode()=5,false 
focus:isFocused()=5,false 
isHapticFeedbackEnabled()=4,true 
drawing:isHardwareAccelerated()=4,true 
isHovered()=5,false 
isInTouchMode()=4,true 
layout:isLayoutRtl()=5,false 
drawing:isOpaque()=5,false 
isPressed()=5,false 
isSelected()=5,false 
isSoundEffectsEnabled()=4,true 
drawing:willNotCacheDrawing()=5,false 
drawing:willNotDraw()=5,false 

不过这里缺了 absolute_x absolute_y，找了一下代码

com.android.hierarchyviewer.ui.model.PropertiesTableModel

private void loadPrivateProperties(ViewNode node) {
    int x = node.left;
    int y = node.top;
    ViewNode p = node.parent;
    while (p != null) {
        x += p.left - p.scrollX;
        y += p.top - p.scrollY;
        p = p.parent;
    }

    ViewNode.Property property = new ViewNode.Property();
    property.name = "absolute_x";
    property.value = String.valueOf(x);
    privateProperties.add(property);

    property = new ViewNode.Property();
    property.name = "absolute_y";
    property.value = String.valueOf(y);
    privateProperties.add(property);
}

所以我们已经知道了，上面抓到的数据是一个ViewNode，然后递归解析，算出绝对位置。
所以找到文字的位置就要，递归算到自己在根视图的绝对位置，然后再修正下layout:getWidth(),layout:getHeight(),找到view的重点，最后再执行点击。

ok,下面谈实现细节

hierarchyviewer其实是在android上开了一个ViewServer，然后把ViewServer的端口转发到本地，然后在连本地的sockect，用sockect与ViewServer进行通讯。
可以看到，dump出来的信息很全，而且是在用户进程中才有的信息。所以ViewServer会通过mWindowToken中的打印。调用链是这样的：

WMS:viewServerWindowCommand
-> ViewRootImpl:executeCommand
-> ViewDebug:dispatchCommand
-> ViewDebug:dumpViewProperties

打印到目标属性用了注解+反射

private static void dumpViewProperties(Context context, Object view,
        BufferedWriter out, String prefix) throws IOException {

    if (view == null) {
        out.write(prefix + "=4,null ");
        return;
    }

    Class<?> klass = view.getClass();
    do {
        exportFields(context, view, out, klass, prefix);
        exportMethods(context, view, out, klass, prefix);
        klass = klass.getSuperclass();
    } while (klass != Object.class);
}

在每个view中，如果允许dump的话，就加入注解,比如说

@ViewDebug.ExportedProperty(flagMapping = {
    @ViewDebug.FlagToString(mask = SYSTEM_UI_FLAG_LOW_PROFILE,
                            equals = SYSTEM_UI_FLAG_LOW_PROFILE,
                            name = "SYSTEM_UI_FLAG_LOW_PROFILE", 
                            outputIf = true),
    @ViewDebug.FlagToString(mask = SYSTEM_UI_FLAG_HIDE_NAVIGATION,
                            equals = SYSTEM_UI_FLAG_HIDE_NAVIGATION,
                            name = "SYSTEM_UI_FLAG_HIDE_NAVIGATION", 
                            outputIf = true),
    @ViewDebug.FlagToString(mask = PUBLIC_STATUS_BAR_VISIBILITY_MASK,
                            equals = SYSTEM_UI_FLAG_VISIBLE,
                            name = "SYSTEM_UI_FLAG_VISIBLE",
                            outputIf = true)
}, formatToHexString = true)
int mSystemUiVisibility;

hierarchyviewer小结：
hierarchyviewer 就是通过在wms中开启一个viewServer，把server的端口forward到本地端口上。socket连本地端口，发送 DUMP $(WindowHash),通过binder调用到具体的某个view，递归打印出有加注解的信息，回信息给socket，hierarchyviewer（gui）再对信息进行整理，获得绝对位置等信息。

小结

本文提供了四种获取当前显示屏幕的文字信息的方法，简单分析了下原理。

接下来准备拓展下uiautomator，就可以支持文字查找与点击了。实现看这篇文章:

Android抓取文字、文字位置的实现

如果对精确度有高要求，对速度有高要求，可以考虑下移植下hierarchyviewer部分的实现。这部分完全重写需要不少时间，暂时偷个懒吧，不实现了：）

尾声

用文字匹配抢微信红包，查资料的时候发现网上不少这样的文章。下面只是针对这个场景的个人猜想，没有实际调研。

攻击的话，看大多数的文章都提到用Accessibility,其实就是用第三种方法（uiautomator）。可以考虑用第四种方法（hierarchyviewer）。就不会受到Accessibility的限制了。
防守的话，新消息那一栏，用自定义view、非规则view，那么有新红包这个文字信息就抓不到。

---

扫码关注，实时互动