#Android

Midscene connects to Android devices through adb to automate apps and system interfaces.

This guide covers device connection, model configuration, Playground, and JavaScript SDK integration with @midscene/android.

#See it in action

Prompt: Open the Booking app. Search for a hotel in Tokyo for four adults on Christmas, with a score of 8 or above.

View the full report, or explore more Midscene showcases.

#Get started

#Prepare your Android device

Before scripting, confirm adb can talk to your device and the device trusts your machine.

#Install adb and set ANDROID_HOME

• Install via Android Studio or the command-line tools
• Verify installation:

adb --version

Example output indicates success:

Android Debug Bridge version 1.0.41
Version 34.0.4-10411341
Installed as /usr/local/bin//adb
Running on Darwin 24.3.0 (arm64)

• Set ANDROID_HOME as documented in Android environment variables, then confirm:

echo $ANDROID_HOME

Any non-empty output means it is configured:

/Users/your_username/Library/Android/sdk

#Enable USB debugging and verify the device

In the system settings developer options, enable USB debugging (and USB debugging (Security settings) if present), then connect the device via USB.

Verify the connection:

adb devices -l

Example success output:

List of devices attached
s4ey59	device usb:34603008X product:cezanne model:M2006J device:cezan transport_id:3

#Launch Playground

Playground is the fastest way to validate the connection and try core capabilities such as aiAct, aiQuery, and aiAssert without writing code. It shares the same core as @midscene/android, so anything that works here will behave the same once scripted.

1. Launch the Playground CLI:

npx --yes @midscene/android-playground

2. Click the gear icon in the Playground window, then paste your API key configuration. Refer back to Model configuration if you still need credentials.

#Use the JavaScript SDK

Once Playground works, move to a repeatable script with the JavaScript SDK.

#Configure the model

Set the model configuration through environment variables. See Model strategy for guidance on choosing a model.

export MIDSCENE_MODEL_BASE_URL="https://replace-with-your-model-service-url/v1"
export MIDSCENE_MODEL_API_KEY="replace-with-your-api-key"
export MIDSCENE_MODEL_NAME="replace-with-your-model-name"
export MIDSCENE_MODEL_FAMILY="replace-with-your-model-family"

For all configuration options, see Model configuration.

#Install dependencies

npm install @midscene/android dotenv --save-dev

yarn add @midscene/android dotenv --save-dev

pnpm add @midscene/android dotenv --save-dev

bun add @midscene/android dotenv --save-dev

deno add npm:@midscene/android npm:dotenv --save-dev

#Write a script

Save the following code as ./demo.ts. It opens the browser on the device, searches eBay, and asserts the result list.

import 'dotenv/config'; // load Midscene environment variables from .env if present
import {
  AndroidAgent,
  AndroidDevice,
  getConnectedDevices,
} from '@midscene/android';

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
Promise.resolve(
  (async () => {
    const devices = await getConnectedDevices();
    const device = new AndroidDevice(devices[0].udid);

    const agent = new AndroidAgent(device, {
      aiActionContext:
        'If any location, permission, user agreement, etc. popup, click agree. If login page pops up, close it.',
    });
    await device.connect();

    await agent.aiAct('open browser and navigate to ebay.com');
    await sleep(5000);
    await agent.aiAct('type "Headphones" in search box, hit Enter');
    await agent.aiWaitFor('There is at least one headphone product');

    const items = await agent.aiQuery(
      '{itemTitle: string, price: Number}[], find item in list and corresponding price',
    );
    console.log('headphones in stock', items);

    await agent.aiAssert('There is a category filter on the left');
  })(),
);

#Run the script

npx tsx demo.ts

#View the report

Successful runs print Midscene - report file updated: /path/to/report/some_id.html. Open the generated HTML file in a browser to replay every interaction, query, and assertion.

#Advanced

Use this section to customize device behavior, integrate Midscene into your framework, or troubleshoot adb issues. For detailed constructor parameters, see the Android section of the API reference.

#Extend Midscene on Android

Use defineAction() for custom gestures and pass them through customActions. Midscene will append them to the planner so AI can call your domain-specific action names.

import { getMidsceneLocationSchema, z } from '@midscene/core';
import { defineAction } from '@midscene/core/device';
import { AndroidAgent, AndroidDevice, getConnectedDevices } from '@midscene/android';

const ContinuousClick = defineAction({
  name: 'continuousClick',
  description: 'Click the same target repeatedly',
  paramSchema: z.object({
    locate: getMidsceneLocationSchema(),
    count: z.number().int().positive().describe('How many times to click'),
  }),
  async call(param) {
    const { locate, count } = param;
    console.log('click target center', locate.center);
    console.log('click count', count);
  },
});

const devices = await getConnectedDevices();
const device = new AndroidDevice(devices[0].udid);
await device.connect();

const agent = new AndroidAgent(device, {
  customActions: [ContinuousClick],
});

await agent.aiAct('click the red button five times');

See Integrate with any interface for a deeper explanation of custom actions and action schemas.

#More

• For every Agent method, check the API reference (Common).
• For Android-specific APIs, see API reference (Android).
• Use YAML automation scripts and command-line tools.
• Demo projects
  • Android JavaScript SDK demo: https://github.com/web-infra-dev/midscene-example/blob/main/android/javascript-sdk-demo
  • Android + Vitest demo: https://github.com/web-infra-dev/midscene-example/tree/main/android/vitest-demo

#Complete example (Vitest + AndroidAgent)

import {
  AndroidAgent,
  AndroidDevice,
  getConnectedDevices,
} from '@midscene/android';
import type { TestStatus } from '@midscene/core';
import { ReportMergingTool } from '@midscene/core/report';
import { sleep } from '@midscene/core/utils';
import type ADB from 'appium-adb';
import {
  afterAll,
  afterEach,
  beforeAll,
  beforeEach,
  describe,
  it,
} from 'vitest';

describe('Android Settings Test', () => {
  let page: AndroidDevice;
  let adb: ADB;
  let agent: AndroidAgent;
  let startTime: number;
  let itTestStatus: TestStatus = 'passed';
  const reportMergingTool = new ReportMergingTool();

  beforeAll(async () => {
    const devices = await getConnectedDevices();
    page = new AndroidDevice(devices[0].udid);
    adb = await page.getAdb();
  });

  beforeEach((ctx) => {
    startTime = performance.now();
    agent = new AndroidAgent(page, {
      groupName: ctx.task.name,
    });
  });

  afterEach((ctx) => {
    if (ctx.task.result?.state === 'pass') {
      itTestStatus = 'passed';
    } else if (ctx.task.result?.state === 'skip') {
      itTestStatus = 'skipped';
    } else if (ctx.task.result?.errors?.[0].message.includes('timed out')) {
      itTestStatus = 'timedOut';
    } else {
      itTestStatus = 'failed';
    }
    reportMergingTool.append({
      reportFilePath: agent.reportFile as string,
      reportAttributes: {
        testId: `${ctx.task.name}`,
        testTitle: `${ctx.task.name}`,
        testDescription: 'description',
        testDuration: (Date.now() - ctx.task.result?.startTime!) | 0,
        testStatus: itTestStatus,
      },
    });
  });

  afterAll(() => {
    reportMergingTool.mergeReports('my-android-setting-test-report');
  });

  it('toggle wlan', async () => {
    await adb.shell('input keyevent KEYCODE_HOME');
    await sleep(1000);
    await adb.shell('am start -n com.android.settings/.Settings');
    await sleep(1000);
    await agent.aiAct('find and enter WLAN setting');
    await agent.aiAct(
      'toggle WLAN status *once*, if WLAN is off pls turn it on, otherwise turn it off.',
    );
  });

  it('toggle bluetooth', async () => {
    await adb.shell('input keyevent KEYCODE_HOME');
    await sleep(1000);
    await adb.shell('am start -n com.android.settings/.Settings');
    await sleep(1000);
    await agent.aiAct('find and enter bluetooth setting');
    await agent.aiAct(
      'toggle bluetooth status *once*, if bluetooth is off pls turn it on, otherwise turn it off.',
    );
  });
});

#FAQ

#Why can't I control the device even though I've connected it?

A common error is:

Error:
Exception occurred while executing 'tap':
java.lang.SecurityException: Injecting input events requires the caller (or the source of the instrumentation, if any) to have the INJECT_EVENTS permission.

Make sure USB debugging is enabled and the device is unlocked in developer options.

#Text input is cleared or lost after typing

After entering text, Midscene automatically dismisses the keyboard. The default behavior sends an ESC key event. However, some input fields (especially those inside WebView) listen for the ESC key event, which can cause side effects such as:

• Clearing the text just entered
• Closing the popup/modal containing the input field
• Navigating away from the current page

You can try the following solutions in order of priority:

Option 1: Use the BACK key (Android back button) to dismiss the keyboard

Set keyboardDismissStrategy to 'back-first' to use the Android BACK key instead of ESC to dismiss the keyboard:

const device = new AndroidDevice('device-id', {
  keyboardDismissStrategy: 'back-first',
});

Option 2: Disable auto keyboard dismiss

If your input field also listens for the BACK key, you can disable auto keyboard dismiss entirely and let the AI Agent or subsequent actions manage the keyboard state:

const device = new AndroidDevice('device-id', {
  autoDismissKeyboard: false,
});

With auto dismiss disabled, the keyboard will remain visible and may cover a large portion of the screen. You can work around this by:

• Using aiAct to manually dismiss the keyboard, e.g. await agent.aiAct('tap the collapse button on the keyboard')
• Installing and switching to ADBKeyBoard — a minimal virtual keyboard that takes up very little screen space, so it barely affects screen interactions even when visible

#English text is rewritten by the Android keyboard

If the report shows the correct input parameter, but the app receives different text, missing text, or Chinese/pinyin candidates, the active Android input method may be rewriting the text. This can happen when pure ASCII text goes through the native adb input text path while a Chinese keyboard or autocorrect keyboard is active.

Use the existing imeStrategy option and force all text input through yadb:

const device = new AndroidDevice('device-id', {
  imeStrategy: 'always-yadb',
});

For YAML scripts:

android:
  imeStrategy: always-yadb

Or set the environment variable:

export MIDSCENE_ANDROID_IME_STRATEGY=always-yadb

This is different from text being cleared after typing. If the text is entered correctly and then disappears, check keyboardDismissStrategy or autoDismissKeyboard instead.

#Screenshot of secure pages (e.g. password input) shows black

Some pages (such as password entry screens in banking or payment apps) set FLAG_SECURE to block screenshots. When screencap captures such a page, the secure region appears black.

The yadb tool creates a virtual display via the secure parameter of SurfaceControl.createDisplay, which can help capture secure content in some setups. Whether it can actually capture FLAG_SECURE pages depends on the Android version, ROM, root/hook environment, and device configuration (for example, some setups may require root plus a Magisk hook). Always verify against your actual device.

Use the screenshotStrategy option to force yadb for screenshots:

const device = new AndroidDevice('device-id', {
  screenshotStrategy: 'always-yadb',
});

For YAML scripts:

android:
  screenshotStrategy: always-yadb

Or set the environment variable:

export MIDSCENE_ANDROID_SCREENSHOT_STRATEGY=always-yadb

The default is auto, which tries the screenshot methods in order: scrcpy → adb.takeScreenshot → screencap → yadb. auto does not analyze screenshot content — even if a command succeeds but returns a black frame, it will not automatically switch to yadb. It only moves to the next method when the previous one fails to execute. For secure pages that produce a valid but black image, set always-yadb to bypass the default auto flow (scrcpy, adb.takeScreenshot, and screencap) and use yadb directly. Yadb can only capture the default display (displayId=0), so combining always-yadb with a non-zero displayId throws an error.

#How do I use a custom adb path or remote adb server?

Set the environment variables first:

export MIDSCENE_ADB_PATH=/path/to/adb
export MIDSCENE_ADB_REMOTE_HOST=192.168.1.100
export MIDSCENE_ADB_REMOTE_PORT=5037

You can also provide the same information via the constructor:

const device = new AndroidDevice('s4ey59', {
  androidAdbPath: '/path/to/adb',
  remoteAdbHost: '192.168.1.100',
  remoteAdbPort: 5037,
});