经典 -Writing a UI Automation Provider for a Win32-based Custom Control

 

Do you have a complex custom control that you want to make programmatically accessible, but you aren’t sure how? Custom controls, by their nature, tend to be very diverse: each is typically written for a specific purpose, making it difficult to generalize implementation details. How do you know what to implement? You should consider supporting accessibility for any custom control that performs its own rendering and input management-routing mouse and keyboard input-within the HWND that it owns.

In this article, I walk through the steps you need to follow to implement a server-side provider for a Win32-based custom control using Microsoft® UI Automation. The example I use assumes the custom control has its own Handle to a Window (HWND) and Window Procedure. While the emphasis is on unmanaged C++ implementations, the techniques are equally applicable to managed code.

What is a Provider?

UI Automation presents the UI on the desktop to a client application in the form of a tree of IUIAutomationElement objects, each of which can expose properties describing the UI. The UI itself must supply these properties using APIs and messages. The work of calling the UI-specific APIs and returning the information back to UI Automation is done by a component called a provider.

 

UI Automation comes with providers for the various standard OS controls, such as Win32 button and list controls, already built in. To make a custom control accessible through UI Automation, you need to supply a UI Automation provider for that control.

Exposing Built-in Properties of a Single Element

The first step to making your custom control accessible is to create a class that implements the UI Automation provider interface, IRawElementProviderSimple, as shown in Listing 1.

While UI Automation clients see the entire user interface (UI) as a tree of IUIAutomationElement objects, providers expose information about the UI as a tree of IRawElementProviderSimple objects. The IRawElementProviderSimple interface is the base interface representing and exposing information about a single UI element.

The MyCustomControlProvider class only exposes information for the overall HWND. Here, I complete the implementations of these IRawElementProviderSimple methods. In this example, I expose a custom name and specify a control type:

IFACEMETHODIMP MyCustomControlProvider ::
    get_ProviderOptions(ProviderOptions * pRetVal)
{
    *pRetVal = ProviderOptions_ServerSideProvider
        | ProviderOptions_UseComThreading;
    return S_OK;
}

This method tells UI Automation that the class is a server-side provider rather than a client-side provider. In general, UI Automation uses client-side providers to support built-in controls such as Win32 buttons, while server-side providers expose information about custom controls to UI Automation.

The second provider options flag tells UI Automation to use COM’s threading rules when it makes calls using the interface: incoming calls must come in on the same thread on which you handed out this object, which saves you the trouble of having to do synchronization ourselves. This assumes that CoInitialize was called on the main UI thread during application startup so that the main UI thread is Apartment-threaded.

Patterns are interfaces that UI Automation uses to expose control-specific functionality such as selection or command invocation. I’ll revisit them later, so I’ll set the return value to NULL here:

IFACEMETHODIMP MyCustomControlProvider::
    GetPatternProvider(PATTERNID idPattern, 
    IUnknown ** pRetVal )
{
    *pRetVal = NULL;
    return S_OK;
}

The GetPropertyValue method handles element properties. UI Automation divides properties into two categories: those related to a specific area of functionality-for example, those dealing with selection, which only apply to controls that manage selection-and those that broadly apply to any element, including Name, Enabled, and Control Type. This latter category of property is handled in the GetPropertyValue method.

For now, I’m supplying suitable values for the element’s Name and Control Type properties. These are two of the more important properties, as screen readers typically read the name and type of a control as focus changes, but I can support any number of properties later:

IFACEMETHODIMP MyCustomControlProvider::
    GetPropertyValue(PROPERTYID idProp, 
    VARIANT * pRetVal )
{
    pRetVal->vt = VT_EMPTY;
    if(idProp == UIA_NamePropertyId)
    {
        pRetVal->bstrVal = SysAllocString(L"MyControlName");
        pRetVal->vt = VT_BSTR;
    }
    else if(idProp == UIA_ControlTypePropertyId)
    {
        pRetVal->lVal = UIA_ButtonControlTypeId;
        pRetVal->vt = VT_I4;
    }
    return S_OK;
}

Finally, get_HostRawElementProvider tells UI Automation which HWND this control is associated with:

IFACEMETHODIMP get_HostRawElementProvider(
    IRawElementProviderSimple ** pRetVal )
{
    return UiaHostProviderFromHwnd(_hwnd, 
        pRetVal);
}

Once you’ve implemented the class, you have to ensure that UI Automation can actually access an instance of it. This is done by handling the WM_GETOBJECT message in the window’s Window Procedure, as shown in Listing 2.

In response to WM_GETOBJECT, I create and return an instance of the provider to UI Automation via the UiaReturnRawElement API from UIAutomationCore.dll. Standard COM reference counting rules apply: UiaReturnRawElement calls AddRef() internally to claim its own reference; so I call Release() when I’m done with the WndProc’s reference.

Also, the code here creates a new instance of the provider each time: this is allowable, as is creating a single instance the first time and caching that value for later reuse. Just remember to use the Release() method on that reference in the WM_DESTROY handler.

Note: Throughout this article, I refer to the Inspect Objects tool, which is available in the Active Accessibility 2.0 SDK Tools download on the Microsoft Download Center (http://www.microsoft.com/downloads/details.aspx?familyid=3755582a-a707-460a-bf21-1373316e13f0&;displaylang=en). This tool helps you see what UI Automation sees. You can use it to test and verify your provider implementations.

If I point the Inspect Objects tool at the custom control, it should report the Name and Control Type that I’ve specified above. What’s happening here is that Inspect Objects asks UI Automation for the object at a specified point on the screen. UI Automation determines the HWND, sends a WM_GETOBJECT message to see if the HWND supports UI Automation, and uses the returned provider to obtain more properties.

All this happens only on demand: it’s only when a client requests an object for this specific HWND that the WM_GETOBJECT message is sent. GetPropertyValue is called only when a client requests a property.

Connecting a Provider to Real Control Data

To make this example more realistic, I’m going to modify the provider code to contain a reference to the custom control’s own data so that it can return real data from GetPropertyValue. Depending on how your control works, this might involve a public or internal COM object, window messages, or direct access to the control’s data.

An important issue to address here is that of provider lifetime. UI Automation holds onto a reference to the control on behalf of clients, and clients are not required to release their references at any particular point in time. This means that a provider can outlive its corresponding UI.

One way to deal with this situation is to add an “alive” flag to the provider and set it to a “dead” state when the UI is no longer present. Further incoming calls to the IRawElementProvider can then check this flag and return the defined value UIA_E_ELEMENTNOTAVAILABLE.

Listing 3 shows the new code that takes provider lifetime into account.

Now if you point the Inspect Objects tool at the control, you should see the control’s actual name instead of the string.

Adding Structure by Adding IRawElementProviderFragment

So far, I’ve simply exposed a single element. However, most custom controls have some internal structure, whether items in a ListBox or a tree of elements in a HTML-like control. The next step is to expose that structure to UI Automation by adding the IRawElementProviderFragment interface.

You need to consider what sort of class structure to use to expose your control’s internal structure. If you have a two-level homogenous container-for example, a list that contains only one type of list item-then it may make sense to have one class for the overall parent list and another class for the items. On the other hand, if the control has a more generalized object tree-as is the case with HTML-then it may make more sense to have a hierarchy of provider objects that mirrors the UI’s own class structure. It could have a base class that implements common functionality and derived classes that handle element-specific behavior.

Whatever structure you choose, you need to implement both IRawElementProviderSimple and IRawElementProviderFragment interfaces for each class that represents an element in the structure. As you’ll see later, the root element also needs to implement IRawElementProviderFragmentRoot.

You only need to access the tree structure when a client requests it. Furthermore, you don’t expose a tree structure all at once; instead, you access or traverse it element by element as needed.

Because internal structure is very control-specific, I won’t provide concrete examples in this walk-through. I’ll just outline what each method does. Note that, as with IRawElementProviderSimple, all the methods should check that the corresponding UI element is “alive” first before they go on to do the real work.

The Navigate method is perhaps the most obvious method in this interface. It’s called with a specified direction (Parent, Next or Previous sibling, or First or Last child), and returns an instance of IRawElementProviderFragment that represents the element in the specified direction. This object must also implement IRawElementProviderSimple or else UI Automation would be able to navigate to it but not get any information about it:

IFACEMETHODIMP MyCustomControlProvider::
    Navigate(NavigateDirection direction, 
    IRawElementProviderFragment ** pRetVal );

The returned IRawElementProviderFragment instance can be a new instance of a provider or a cached instance, so long as it obeys COM’s reference counting rules. Note that the object that this is called on continues to point to the same UI element that it was previously pointing to.

The tree structure that is returned by this method should be consistent: if you can navigate to a child, then when you navigate back to the parent, you should end up at a provider instance that represents the same UI element. If you cache providers, this may be the same provider object instance.

The root provider of your structure, the one that occupies the entire HWND, must return NULL when asked for its parent or siblings. You should provide just the subtree structure that corresponds to the UI that your control is providing; UI Automation will do the work of integrating this into the larger tree-of-peer-HWNDs.

Next, for the GetRuntimeId method, UI Automation requires that you return an integer array that uniquely identifies this specific element within the scope of the owning HWND:

IFACEMETHODIMP MyCustomControlProvider::
    GetRuntimeId (SAFEARRAY ** pRetVal);

If you already have a unique integer assigned to each UI element, using that would work perfectly here as a single element array. Be cautious about using a pointer value because you have to take into account that pointers on 64-bit systems require two integers.

Once you have your array, prepend to it the value UiaAppendRuntimeId and use the API’s SafeArrayCreateVector and SafeArrayPutElement functions to convert the C-style array to a SAFEARRAY.

The root element of your control can simply return NULL here: UI Automation automatically constructs Runtime IDs for the host HWND, and from UI Automation’s point of view, the root element of the control is essentially the same element as that HWND.

The get_BoundingRectangle method returns the bounding rectangle coordinates of the control. For this, return the location of the UI element in screen coordinates. If the element is not visible, return a rectangle consisting of all 0s. You may need to use MapWindowPoints to convert from HWND client-relative coordinates to screen coordinates:

IFACEMETHODIMP MyCustomControlProvider::
    get_BoundingRectangle(UiaRect * pRetVal);

As with GetRuntimeId, the root element doesn’t have to do anything here, since its location is the same as the host HWND, and UI Automation can already get that information from the HWND.

You only use the GetEmbeddedFragmentRoots method when your control hosts other HWNDs within it. Since I’m not covering this issue yet, this method returns NULL here.

Again, the root element doesn’t have to do anything because its location is the same as the host HWND:

IFACEMETHODIMP MyCustomControlProvider::
    GetEmbeddedFragmentRoots(SAFEARRAY** pRetVal);

UI Automation calls the SetFocus method when it wants you to set your control’s internal focus state to a particular element. UI Automation automatically focuses the parent HWND first, so you only need to update your control’s internal state:

IFACEMETHODIMP MyCustomControlProvider::
    SetFocus();

The get_FragmentRoot method provides UI Automation with a convenient way to get to the root element of your control without having to call Navigate(Parent) repeatedly:

IFACEMETHODIMP MyCustomControlProvider::
    get_FragmentRoot(
    IRawElementProviderFragmentRoot ** pRetVal);

Next, you need to implement two structure-related methods only on the root node. These are on the IRawElementProviderFragmentRoot and allow UI Automation to request the focused element or the element at a specific screen location:

IFACEMETHODIMP MyCustomControlProvider::
    ElementProviderFromPoint(double x, double y, 
    IRawElementProviderFragment ** pRetVal);

IFACEMETHODIMP MyCustomControlProvider::
    GetFocus(
    IRawElementProviderFragment ** pRetVal);

These methods should return a provider instance representing the focused element or the element at the specified point (in screen coordinates), or NULL if the point or focus is on the root element itself or if the point or focus is not on the element at all. For both these methods, the provider should return the deepest element possible.

Finally, I’ll return to the implementation of IRawElementProviderSimple::get_HostRawElementProvider. You should modify this such that it only returns the HWND for the root. For all child elements within the control, it should return NULL.

You can take a piecemeal strategy for implementing all of these methods. First, start with implementing the Navigate and BoundingRectangle methods. This enables you to get to the root element by hovering the cursor over any part of the control, and then walking through the tree using the Navigation functionality of the Inspect Objects tool. Use the highlight rectangle feature to quickly verify that the locations are being exposed correctly as you navigate. Then you can implement ElementProviderFromPoint. When you hover the mouse over an element within the control, Inspect Objects should go straight to that element instead of stopping at the HWND. Finally, implement the GetRuntimeId and GetFocus methods.

Deciding What Part of the Tree Structure to Expose

When deciding what parts of the tree structure to expose, remember that UI Automation is only interested in elements that are important to an end user. This includes any element that an end user would perceive as being a separate control in its own right. For example, a push-button and its text are typically not considered distinct elements from an end-user’s point of view, even if they are represented as such in a control’s internal structures. In such a case, you should expose just a single element representing the entire button.

list1

#include <UIAutomation.h>

// New class that exposes a custom control to UI Automation. 
// This implements the provider-side interfaces.
class MyCustomControlProvider: public IRawElementProviderSimple
{
   HWND _hwnd;

   // Reference to control data or object model goes here

public:

   MyCustomControlProvider(HWND hwnd): _hwnd(hwnd)
   {
   }

   // Standard implementations of AddRef, Release, 
   // QueryInterface go here

   // IRawElementProviderSimple methods
   IFACEMETHODIMP get_ProviderOptions(ProviderOptions * pRetVal);
   IFACEMETHODIMP GetPatternProvider(PATTERNID idPattern, 
        IUnknown ** pRetVal );
   IFACEMETHODIMP GetPropertyValue(PROPERTYID idProp, 
        VARIANT * pRetVal );
   IFACEMETHODIMP get_HostRawElementProvider(
        IRawElementProviderSimple ** pRetVal );
};

list2

LRESULT MyCustomControlWndProc(HWND hwnd, UINT uMsg, 
    WPARAM wParam, LPARAM lParam)
{
    switch(uMsg)
    {
        case WM_GETOBJECT:
        {
            MyCustomControlProvider * pProvider = 
                new MyCustomControlProvider();
            LRESULT lres = UiaReturnRawElement(hwnd, wParam, 
                lParam, pProvider);
            pProvider->Release();
            return lres;
        }
    }
    return DefWindowProc(hwnd, uMsg, wParam, lParam);
}

list3

class MyCustomControlProvider: public IRawElementProviderSimple
{
    HWND _hwnd;

    // Create a pointer to internal control state, set to 
    // non-NULL in the constructor. 
    // OnUIDisposed sets it to NULL to indicate that UI is no 
    // longer present. Using NULL to indicate this avoids having 
    // a separate flag.
    // Incoming UI Automation interface methods must check this 
    // before use.
    // 
    IMyInternalControlInterface _pControl; 

...

    // Called when corresponding UI is deleted
    void OnUIDisposed();
};

void MyCustomControlProvider::OnUIDisposed()
{
    _pControl = NULL;
}

IFACEMETHODIMP MyCustomControlProvider::GetPropertyValue(
    PROPERTYID idProp, VARIANT * pRetVal )
{
    // Always clear out parameter before anything else
    pRetVal->vt = VT_EMPTY;

    // Now I check to see if the UI is still present
    if(_pControl == NULL)
    {
        return UIA_E_ELEMENTNOTAVAILABLE;
    }

    // UI is still present, so code continues as before
    if(idProp == UIA_NamePropertyId)
    {
        ... // get name from _pControl and return via pRetVal.
    }
...