AppleScript – How can I _get_ UI elements' names, attributes, properties, classes programmatically without “guessing” via AccessibilityInspector?

URL: https://apple.stackexchange.com/questions/340942/applescript-how-can-i-get-ui-elements-names-attributes-properties-classe/

When I started to search for an AppleScript's generic way to "click onto Safari's context menu" of a selected item, there were a lot of questions on "how-to" on several levels of nomenclature.

How can I, without having to use Accessibility Inspector, investigate what "categories" of AppleScript UI elements an application is offering and what additional details AppleScript knows about them?

ANSWER 1:

This is a "study" –an *attempt*– to find pragmatic ways to identify and "use" AS's UI elements:
------------------------------------------------------------------------
Code below works ***in principle*** but must be adapted to individ. case (e.g.: find ***second** image*).    
I am *not an IT professional* so I apologize for a distinct lack of "on error" code – *Sorry!*  
… nor is this meant to be a "Comprehensive Guide" on AppleScript, rather a "Travel Journal".    

*(BTW: AirBook, i7-3740QM, 8GB. Repeat-loops on "entire contents" may take 70+ sec's.    
System: MacOS X High Sierra 10.13.6, Applescript 2.7. Commands may vary with versions.)* 

I started my search here at **Ask Different**, my below "results" mostly apply to thread:   
[How to know the name of UI elements using Accessibility inspector (or any other tool)][1]
     

>***1) How many** UI elements are there in a "Ask Different" window? …*

    "Frame":    tell application "System Events" to tell process "Safari"
                    set frontmost to true
                    set i to 0
    Start:		    set listItems to (entire contents of window 1 as list)
    Next:	   	    repeat with thisItem in listItems
                        set i to i + 1
                    end repeat
                end tell
    "Result" in
    ScriptEditor:	951		           -- display dialog ("UIElems:" & i)
                   ¯¯¯¯¯
>*How many elements are there in which **different classes**? …*    

    Classes:	if (class of item i of listItems is static text) then
                    set classCount to classCount +1    
                end if
    ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯    
    "Results" in ScriptEditor:
    static text (376), group (337), UI element (166), button (51),
    radio button (12), menu button (6), image (2)
    (1 inst. each:) toolbar, scroll area, scroll bar, text area, text field, 
    tab, checkbox, splitter group, splitter
≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈    
>*2) How to **identify** e.g. those 2 images within 951 UI elements & **what** to do:*    

  (From here on only differing code-parts, e.g. "Next" are posted; _/¯ = line break.)

    Next:		set imgNums to {}
                repeat with thisItem in listItems
                    set i to i + 1
    Then:	    	if (class of item i of listItems is image) then
                        set imgNums to imgNums & i
                end repeat
                return imgNums    _/¯    end tell
    ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯    
    "Result" in
    ScriptEditor:	{111, 255}     (≈ 70 sec.! Searching 951 items is tedious…)
                    ¯¯¯¯¯¯¯¯¯¯
>… or just "context" the first …      (≈ 7 sec.!)

    Next:		    if (class of item i of listItems is image) then exit repeat
                end repeat
    Then:	    tell thisItem to perform action "AXShowMenu"    _/¯    end tell
≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈    
>*3) How to get at AppleScript's **"verbal description"** of a UI element?*

    Next:		repeat with thisItem in listItems    _/¯    set i to i + 1
    Then:		    if (class of item i of listItems is image) then
                        return (item i of listItems as list)
		                exit repeat
                    end if    _/¯    end repeat    _/¯    end tell
    ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯    
    "Result" for image 1 in ScriptEditor:
    image 1 of group 15 of group 6 of UI element 1 of scroll area 1 ¬
    of group 1 of group 1 of tab group 1 of splitter group 1 of window ¬
    "applescript - How to know the name of UI elements using Accessibility ¬
    inspector (or any other tool) - Ask Different" of application process ¬
    "Safari" of application "System Events"
    ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯    
    [Last three lines may be shortened to "window 1" – it's still in front …]

– which is this image … <img src="https://i.stack.imgur.com/UsqNF.png" alt="enter image description here"> … of the above mentioned thread.
≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈    
>*4) Which **attributes** are possible for an element, which are factual **properties**?*

    Then:		    if (class of item i of listItems is image) then ¬
				    return (name of attributes of thisItem as list)
    ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯    
    "Result" for "attributes" in ScriptEditor:
	"AXFocused", "AXAccessKey", "AXTitle", "AXElementBusy", "AXPosition", ¬
	"AXLinkedUIElements", "AXSelected", "AXLanguage", "AXStartTextMarker", ¬
	"AXEnabled", "AXEditableAncestor", "AXHighestEditableAncestor", ¬
	"AXVisited", "AXDOMIdentifier", "AXHelp", "AXURL", "AXChildren", "AXRole", ¬
	"AXParent", "AXSelectedTextMarkerRange", "AXTopLevelUIElement", ¬
	"AXDOMClassList", "AXSubrole", "AXDescription", "AXFocusableAncestor", ¬
	"AXValue", "AXBlockQuoteLevel", "AXRoleDescription", "AXSize", "AXWindow", ¬
	"AXEndTextMarker", "AXFrame"

>*Which are the **factual properties** of "image 1"?*

    Next:		    if (class of item i of listItems is image) then ¬
				    return (properties of thisItem as list)
    ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯    
    "Result" for "properties" in ScriptEditor:
	missing value, m.v, {382, 790}, image, "enter image description here", ¬
    "Bild", false, "", {241, 118}, "", {}, true, m.v., "AXImage", "", m.v., ¬
    false, m.v., "enter image description here"
&nbsp; Obviously factual properties are not corresponding directly to attributes (19 to 32).    
>*But **some attributes** actually can return values **if asked directly:***

    Then:		end repeat
                return value of attribute "AXFrame" of thisItem  _/¯  end tell
    ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯    
    "Result" for "AXFrame" in ScriptEditor:	  {382, 790, 623, 908}
>***Which** property is **assigned** to **which** attribute? (For exact "AX…" name, see appendix!)    
... easy solution, found on Apple's developer pages:*

    Next:		if (class of item i of listItems is image) then
				    tell item i of listItems to properties
    ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯    
    "Result" for "Bild" in ScriptEditor:	
    minimum value:missing value, orientation:m. v., position:{382, 790}, ¬
    class:image, accessibility description:"enter image description here", ¬
    role description:"Bild", focused:false, title:"", size:{241, 118}, ¬
    help:"", entire contents:{}, enabled:true, maximum value:m. v., ¬
    role:"AXImage", value:"", subrole:m. v., selected:false, name:m. v., ¬
    description:"enter image description here"
≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈    
>5. Which ***"actions"*** belong to an element (*AXPress, AXShowMenu*…) ?

&nbsp; I DuckDuckGo[ogle]'d (DD'went?) and found a list of ["AXActionConstants" on Apple's site][2]:        

&nbsp; &nbsp;  "AXShowMenu" &nbsp; &nbsp;  "AXPress" &nbsp; &nbsp;  "AXCancel" &nbsp; &nbsp;  "AXConfirm"    

&nbsp; &nbsp;  "AXDecrement" &nbsp; &nbsp;  "AXIncrement" &nbsp; &nbsp;  "AXShowAlternateUI" &nbsp; &nbsp;  "AXShowDefaultUI"    

&nbsp; &nbsp;  "AXPick" &nbsp; &nbsp; &nbsp; Choose a UIElement, like menu item    

&nbsp; &nbsp;  "AXRaise" &nbsp; &nbsp;Cause a window to become as frontmost as possible    


>6. How are ***menu items*** addressed best?    

&nbsp; (I will soon add my view/findings on these remaining issues.)

≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈    
>7. How ***"different"*** are classes –and UI elements– in ***various apps***? 

 - "TextEdit" is an example for minimalistic "functions" of an application window. (Be aware that    
its tools windows (Fonts etc.) have UI elements of their own, many more than the main app!)
 - Its top "bar" consists of 3 coloured buttons, an image representing the document, its title as static text and a tiny down-arrow, a menu button.
 - Apart from these there are (potentially) 2 scroll areas (right &
   bottom) with scroll bars, also value indicators (their size), and
   last not least its "contents", a text area.    


        Classes of window 1 of TextEdit / Finder / iTunes:
        set classNum to classNum & (class of item i of listItems)
        – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – 
        "Result" for "TextEdit" (16):
        button (7), scroll bar (2), scroll area, text area, value indicator, 
        menu button, image, static text
        – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – 
        "Result" for "Finder" window in list view with left area with 
        "devices" etc. , 11 files (208):
        UI element (49), static text (49), image (27), row (26), button (19),
        text field (12), group (6), radio button (4), column (4), ...
        – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – 
        "Result" for "iTunes" (26):
        button (12), radio button (3), static text (3), pop up button (2), 
        radio group (2), slider, value indicator, scroll area, text field
    
 - "Finder" in list view shows icons & tools (= images), files have names, date etc. (static text), tools are also buttons, file names are also text fields (rename).
 - "iTunes" (with no TV items "installed") has got 2 pop up buttons (1: music, movies, TV / 2: next, history, song text) plus several sorts of navigation buttons.
 - All of these applications, though, also have hundreds of menu items: menus, sub-menus, sub-sub-…menus (TextEdit ≈ 350, Finder ≈ 430).   

≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈    
&nbsp; &nbsp; A P P E N D I X
≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈    

>Older code to get at "assignations" – but returns ***exact name!*** ... THAT took **some** testing:*

    Next:		if (class of item i of listItems is image) then
				    repeat with name in (attributes of thisItem as list)
				        try
						    if value of name is "Bild" then exit repeat
					    end try
	     		    end repeat    _/¯    exit repeat    _/¯    end if
		    	try
		    	    i & k & item k of (name of attributes of thisItem as list)
		    	end try
    ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯        
    "Result" for "Bild" in ScriptEditor:	{111, 28, "AXRoleDescription"}
     (Incl.: # of UI element, # of attribute, name of attribute) 
    ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯    
     ({382, 790} => AXPosition, {241, 118} => AXSize, "AXImage" => AXRole …)    
&nbsp; The try-block is necessary since some value-less attributes otherwise cause errors.    
&nbsp; This search on sample page took ≈ 9s. (#111) on *THIS page* ≈ 135s. (#767) !
≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈    

  [1]: https://apple.stackexchange.com/questions/40436/how-to-know-the-name-of-ui-elements-using-accessibility-inspector-or-any-other
  [2]: https://developer.apple.com/documentation/applicationservices/axactionconstants_h/miscellaneous_defines