WanderVision: Scenario-Based GUI Testing for Smart TVs via Vision-Language Models

WanderVision performs scenario-based vision-grounded testing for webOS applications for smart TVs, extended from an existing Android GUI testing agent, DroidAgent.

Table of Contents

1. Setup
2. Running WanderVision

Setup

WanderVision requires several Python libraries installed.

Prerequisite

• >=16GB RAM
• >=Python 3.10
• requires a physical webOS TV
• Environment variables: rename .env.example to .env and fill in the following
  • GEMINI_API_KEY: your Gemini API key
  • TV_IP_ADDRESS, TV_HOSTNAME, TV_PORT, TV_PASSWORD: connection info for the target TV
  • WEBOS_USER_ID, WEBOS_USER_PW: credentials for logging into the webOS system
  • APP_CREDENTIALS: a JSON string mapping each app_id to its {id, pw} pair

Clone & Install Dependencies

$ git clone https://github.com/coinse/wandervision.git
$ pip install -r requirements.txt
$ pip install -e . # install wandervision

Run WanderVision

Make sure that your TV is connected and necessary credentials are ready.

$ cd script
$ python run_wandervision.py \
--app_id [APP_NAME] \
--output_dir [OUTPUT_DIR] \
--manual_task [TASK_DESCRIPTION] \
--manual_task_expected_result [EXPECTED_RESULT]

# example:
# python run_wandervision.py \
# --app_id netflix \
# --output_dir ../evaluation/data_new/netflix \
# --manual_task "Check login behavior" \
# --manual_task_expected_result "Verify that the ID/Password input screen is displayed"

• Given app_id, WanderVision initiates with launching the specified application.

• Given manual_task, WanderVision executes actions to achieve the given task.

• Given manual_task_expected_result, the Reflector of WanderVision evaluates if the task was achieved after the execution.

• Dataset used in the evaluation is available at wandervision/DATASET.md. You may replicate the evaluation by using each field of the dataset.

  • app_id: netflix for Netflix, youtube.leanback.v4 for YouTube
  • manual_task: "Goal" for goal-based evaluation, "Goal + Procedure" for procedure-based evaluation
  • manual_task_expected_result: "Expected result" for the task