android-use

This skill enables Claude to control Android devices using ADB (Android Debug Bridge) and the Android Accessibility API (uiautomator). It works by capturing the device's UI hierarchy as structured XML text, parsing interactive elements with their coordinates, and executing actions like tapping, typing, swiping, and navigation.

IMPORTANT: This skill uses TEXT-BASED UI automation, NOT image/screenshot processing.

Why Text-Based (Not Screenshots)?

| Approach | Cost | Speed | Accuracy |

|----------|------|-------|----------|

| Screenshots + Vision | ~$0.15/action | 3-5s | 70-80% |

| Text UI Dump | ~$0.01/action | <1s | 99%+ |

The Android Accessibility API provides structured XML with:

• Element text content (exact, no OCR errors)
• Precise coordinates (deterministic)
• Clickable/focusable state
• Resource IDs for identification

Only use screenshots as a LAST RESORT when:

• UI text is ambiguous or unclear
• You need to verify visual state (colours, images)
• Text dump is empty but screen clearly has content

Before using this skill, ensure:

1. ADB installed: brew install android-platform-tools (macOS) or equivalent
2. Device connected: Via USB with "USB debugging" enabled in Developer Options
3. Device authorised: Accept the RSA key prompt on the device when first connecting

To verify connection:

```bash

adb devices -l

```

To control an Android device:

1. Get screen state - Dump UI hierarchy as text
2. Parse elements - Extract interactive components with coordinates
3. Decide action - Based on text content, find target element
4. Execute action - Tap, type, swipe, back, home
5. Wait & repeat - Allow UI to update, then reassess

Device Management

```bash

# List all connected devices

adb devices -l

# Target specific device (when multiple connected)

adb -s

```

UI Inspection (Text-Based)

```bash

# Dump UI hierarchy to device

adb -s shell uiautomator dump /sdcard/window_dump.xml

# Pull to local machine

adb -s pull /sdcard/window_dump.xml /tmp/screen.xml

# Parse and extract interactive elements

python3 scripts/parse_ui.py /tmp/screen.xml

```

One-liner for convenience:

```bash

adb -s shell uiautomator dump /sdcard/window_dump.xml && \

adb -s pull /sdcard/window_dump.xml /tmp/screen.xml && \

python3 scripts/parse_ui.py /tmp/screen.xml

```

The parser outputs interactive elements like:

```

TAPPABLE:

👆 "Submit" @ (540, 1200)

👆 "Cancel" @ (200, 1200)

👆 [search_button] @ (980, 156)

INPUT FIELDS:

⌨️ "Enter email" @ (540, 600)

TEXT/INFO:

👁️ "Welcome back, John" @ (540, 300)

```

Actions

```bash

# Tap at coordinates

adb shell input tap

# Type text (use %s for spaces)

adb shell input text "hello%sworld"

# Press keys

adb shell input keyevent KEYCODE_HOME # Home button

adb shell input keyevent KEYCODE_BACK # Back button

adb shell input keyevent KEYCODE_ENTER # Enter key

adb shell input keyevent KEYCODE_DEL # Backspace

# Swipe (x1, y1 to x2, y2 over duration_ms)

adb shell input swipe

# Long press (swipe with same start/end)

adb shell input swipe 1000

```

App Management

```bash

# Launch app by package name (simplest)

adb shell monkey -p -c android.intent.category.LAUNCHER 1

# Launch app with specific activity

adb shell am start -n /

# List installed packages

adb shell pm list packages | grep

Step 1: Get Current Screen State (TEXT ONLY)

```bash

adb -s shell uiautomator dump /sdcard/window_dump.xml && \

adb -s pull /sdcard/window_dump.xml /tmp/screen.xml && \

python3 scripts/parse_ui.py /tmp/screen.xml

```

This gives you a text list of all interactive elements with coordinates.

Step 2: Find Target Element

From the parsed output, identify the element you need:

• Match by text content: "Submit", "Login", "Search"
• Match by resource ID: [login_button], [search_field]
• Match by type: Button, EditText, TextView

Step 3: Execute Action

```bash

# To tap "Submit" button at (540, 1200)

adb -s shell input tap 540 1200

# To type in a field - first tap it, then type

adb -s shell input tap 540 600

adb -s shell input text "user@email.com"

```

Step 4: Wait and Repeat

```bash

# Wait 1-2 seconds for UI to update

sleep 2

# Dump again and reassess

adb -s shell uiautomator dump /sdcard/window_dump.xml && \

adb -s pull /sdcard/window_dump.xml /tmp/screen.xml && \

python3 scripts/parse_ui.py /tmp/screen.xml

```

Only take a screenshot if:

1. UI dump returns empty/incomplete but you know the screen has content
2. You need to verify something visual (image loaded, colour state)
3. Text elements are ambiguous and you need visual context
4. User explicitly asks to "see" the screen

```bash

# Screenshot command (USE SPARINGLY - costs ~15x more to process)

adb -s shell screencap -p /sdcard/screen.png && \

adb -s pull /sdcard/screen.png /tmp/screen.png

```

Opening an App and Navigating

```bash

# 1. Launch app

adb -s DEVICE shell monkey -p com.example.app -c android.intent.category.LAUNCHER 1

# 2. Wait for launch

sleep 2

# 3. Get UI state (text)

adb -s DEVICE shell uiautomator dump /sdcard/window_dump.xml

adb -s DEVICE pull /sdcard/window_dump.xml /tmp/screen.xml

python3 scripts/parse_ui.py /tmp/screen.xml

# 4. Find and tap target element based on text output

adb -s DEVICE shell input tap

```

Searching in an App

```bash

# 1. Find search field in UI dump

# Output shows: ⌨️ "Search" @ (540, 200)

# 2. Tap search field

adb shell input tap 540 200

# 3. Type search query

adb shell input text "cheesecake"

# 4. Press enter

adb shell input keyevent KEYCODE_ENTER

```

Scrolling to Find Elements

If an element isn't visible in the UI dump, scroll and re-dump:

```bash

# Scroll down

adb shell input swipe 540 1500 540 500 300

# Wait and re-dump

sleep 1

adb shell uiautomator dump /sdcard/window_dump.xml

# ... parse again

```

Handling Popups/Dialogs

Popups appear in UI dump with their own elements:

```

👆 "Allow" @ (700, 1200)

👆 "Deny" @ (300, 1200)

```

Just tap the appropriate button coordinates.

The parse_ui.py script outputs elements grouped by action type:

TAPPABLE (clickable=true)

Elements that respond to taps: buttons, links, icons

```

👆 "Button Text" @ (x, y)

👆 [resource_id] @ (x, y)

```

INPUT FIELDS (EditText)

Text input fields

```

⌨️ "Placeholder text" @ (x, y)

```

TEXT/INFO (readable)

Non-interactive text elements (limited to 10)

```

👁️ "Display text" @ (x, y)

```

JSON Output

For programmatic use:

```bash

python3 scripts/parse_ui.py /tmp/screen.xml --json

```

Returns:

```json

[

{

"id": "com.app:id/submit_btn",

"text": "Submit",

"type": "Button",

"bounds": "[400,1100][680,1300]",

"center": [540, 1200],

"clickable": true,

"action": "tap"

}

]

```

"error: device not found"

• Check USB connection
• Run adb devices to verify
• Try adb kill-server && adb start-server

UI dump returns empty/incomplete

• Screen may be loading - wait and retry
• Some apps have accessibility restrictions
• Only then consider a screenshot to diagnose

Taps not registering

• Verify coordinates are within screen bounds
• Check if element is actually clickable
• Some overlays may intercept touches

Text input issues

• Replace spaces with %s
• Special characters may need escaping
• For passwords, some apps block programmatic input

Element not found in dump

• It may be off-screen - try scrolling
• It may be in a WebView (limited accessibility)
• It may be loading - wait and retry

Common Package Names

| App | Package |

|-----|---------|

| Uber Eats | com.ubercab.eats |

| DoorDash | com.dd.dasher |

| Chrome | com.android.chrome |

| Settings | com.android.settings |

| Messages | com.google.android.apps.messaging |

Common Keycodes

| Key | Code |

|-----|------|

| Home | KEYCODE_HOME |

| Back | KEYCODE_BACK |

| Enter | KEYCODE_ENTER |

| Delete | KEYCODE_DEL |

| Tab | KEYCODE_TAB |

| Escape | KEYCODE_ESCAPE |

• scripts/parse_ui.py - Parses Android UI XML and outputs interactive elements
• Based on [android-action-kernel](https://github.com/anthropics/android-action-kernel) approach