AndroTMem: From Interaction Trajectories to Anchored Memory in Long-Horizon GUI Agents

  📑 Paper   |   🏠 Project Page   | 🤗 Hugging Face   | 🤖 Model Scope  

If our project helps you, please give us a star ⭐ on GitHub to support us.

📰 News

• 2026-07-05 🚀 We released the inference and evaluation scripts for AndroTMem-Bench, including GPT-4o/Qwen2.5-VL inference and AMS/TCR evaluation.
• 2026-03-19 🌟 We are happy to release the AndroTMem. You can find the AndroTMem from .

---

📢 Overview

AndroTMem is a diagnostic framework for studying interaction memory in long-horizon Android GUI agents.

Unlike prior work that focuses on perception or short workflows, AndroTMem highlights a key bottleneck:

🔥 Failure in long-horizon tasks is primarily caused by memory breakdown, not perception errors.

AndroTMem consists of:

1. Benchmark construction
2. Long-horizon task design with causal dependencies
3. Memory-oriented evaluation (TCR)
4. Anchored State Memory (ASM)

---

✨ Key Contributions

• 🧠 Anchored State Memory (ASM)
  A structured memory mechanism that represents interaction history as causally linked intermediate state anchors.

• 📊 AndroTMem-Bench
  A large-scale benchmark for long-horizon GUI tasks:

  • 1,069 tasks
  • 34,473 interaction steps
  • Avg. 32.1 steps per task (max 65)
  • Cross-app workflows across 50 Android apps :contentReference[oaicite:1]{index=1}

• 🔍 Diagnostic Evaluation Suite

  • Shows performance degradation is dominated by within-task memory failures

---

🧩 Why AndroTMem?

Existing approaches:

• ❌ Full trajectory replay → noisy & redundant
• ❌ Summarization → loses critical dependencies

We propose:

• ✅ Sparse but critical state anchors
• ✅ Causal dependency modeling
• ✅ Targeted retrieval for decision making

---

🧠 Anchored State Memory (ASM)

ASM models interaction history as:

• Intermediate states (anchors)
• Causal relationships between them

Each anchor includes:

• type (e.g., subgoal, dependency)
• content (semantic info)
• evidence (UI grounding)
• links (causal dependencies)

This enables:

• 🎯 Subgoal-aware retrieval
• 🔗 Dependency-aware reasoning
• 📉 Reduced context noise

---

📊 Evaluation: AndroTMem-Bench

Task Characteristics

• Long-horizon workflows (multi-step, multi-app)
• Strong step-to-step causal dependencies
• Requires state reuse across distant steps

Task Types

• Lookup
• Compare & Decide
• Purchase / Order
• Booking
• Communication
• Sharing
• Content Creation
• Configuration

---

📈 Key Findings

• Performance drops significantly as step length increases
• Failures mainly due to:
  • ❌ State loss
  • ❌ State Mis-binding
  • ❌ Context drift
  • ❌ Unverified progress
  • ❌ Interruption Handling Failure

ASM effectively mitigates these issues and improves:

• TCR (Task Completion Rate)
• AMS (Action Matching Score)

---

🔬 Results

Across 12 GUI agents (open & closed source):

• ✅ +5% ~ +30% improvement using ASM
• ✅ Strong robustness in long-horizon settings
• ✅ Better efficiency vs raw trajectory replay

---

📂 Repository Structure

AndroTMem/
├── baseline
├── evaluation/       # Evaluation pipeline
├── scripts/          # Running scripts
├── assets/           # Figures (paper images)
└── README.md

---

🚀 How to Use

This repository includes an inference and evaluation framework for running GUI agents on AndroTMem-Bench. The framework reads multi-step task data in JSONL format, predicts one GUI action per step, writes prediction results to JSONL, and evaluates them with AMS and TCR.

1. Environment Setup

Create a Python environment and install dependencies:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

For the local Qwen backend, install a PyTorch build that matches your CUDA environment. flash-attn is optional and should only be enabled after it is installed successfully.

2. Configure Backends

Two backends are supported:

• gpt4o: closed-source model inference through an OpenAI-compatible API.
• qwen: local open-source inference with Qwen2.5-VL through HuggingFace.

Configure GPT-4o in configs/gpt4o_cfg.py:

"base_url": "https://your-openai-compatible-endpoint/v1",
"api_keys": [
    "your-api-key",
],
"model_name": "gpt-4o",

"input_file": r"PATH/TO/EVALUATION_DATA.jsonl",
"output_dir": r"PATH/TO/OUTPUT_DIR/gpt4o",
"image_base_path": r"PATH/TO/IMAGE_ROOT_DIR",

Configure Qwen in configs/qwen_cfg.py:

"hf_model_id": "Qwen/Qwen2.5-VL-7B-Instruct",

"input_file": r"PATH/TO/EVALUATION_DATA.jsonl",
"output_dir": r"PATH/TO/OUTPUT_DIR/qwen_2_5_vl",
"image_base_path": r"PATH/TO/IMAGE_ROOT_DIR",

Field meanings:

• input_file: input task JSONL file.
• output_dir: directory for prediction JSONL files.
• image_base_path: screenshot root directory. Each image_name in the input data is resolved relative to this path.
• output_filename: output filename template; {version} and {model_name} are supported when configured.
• max_workers: worker count. Use 1 for local Qwen inference on a single GPU.

3. Input Format

The input file is JSONL. Each line is one task:

{"task_id":"task_001","instruction_en":"Open Settings and turn on Wi-Fi.","steps":[{"step_index":0,"image_name":"task_001/step_000.png"},{"step_index":1,"image_name":"task_001/step_001.png"}]}

Required fields:

• task_id: task ID.
• instruction_en or instruction: task instruction. instruction_en is preferred.
• steps: list of task steps.
• steps[].step_index: step index.
• steps[].image_name: screenshot path relative to image_base_path.

4. Run Inference

Run GPT-4o:

python run_eval.py --backend gpt4o --version v1
python run_eval.py --backend gpt4o --version v2
python run_eval.py --backend gpt4o --version v3

Run Qwen2.5-VL:

python run_eval.py --backend qwen --version v1
python run_eval.py --backend qwen --version v2
python run_eval.py --backend qwen --version v3

You can override selected config values from the CLI:

python run_eval.py \
  --backend gpt4o \
  --version v3 \
  --workers 4 \
  --input PATH/TO/EVALUATION_DATA.jsonl \
  --output PATH/TO/OUTPUT_DIR/gpt4o

Prompt versions:

• v1: predicts only the next action; context includes the last three actions.
• v2: predicts action and summary_en; context includes the previous English summary.
• v3: predicts action and milestones; context includes accumulated milestone anchors.

5. Prediction Output

Each output line is one prediction record:

{"task_id":"task_001","step_index":0,"prediction":{"action":{"action":"tap","x":512,"y":320,"value":"","x_end":0,"y_end":0,"direction":"","distance":""}}}

Coordinates in prediction are denormalized from the 0 ~ coord_scale range to original image pixels. The framework also writes per-step execution status and, when available, token/time metrics:

{
  "execution_status": "success",
  "execution_message": "model inference completed",
  "metrics": {
    "e2e_sec": 1.23,
    "ttft_sec": 0.45,
    "tpot_sec": 0.01,
    "prompt_tokens": 1234,
    "completion_tokens": 56
  }
}

If a screenshot is missing, the step is marked as AUTO_PASS and credited as correct:

{
  "prediction": {"action": {"action": "AUTO_PASS"}},
  "execution_status": "error",
  "execution_message": "image missing: example.png",
  "image_missing": true,
  "credited_as_correct": true
}

6. Evaluate Predictions

Run the evaluation script:

python evaluate_predictions.py \
  --pred PATH/TO/PREDICTIONS.jsonl \
  --gt PATH/TO/GROUND_TRUTH.jsonl

The evaluation reports:

• AMS (Action Match): average step-level action matching score.
• TCR (Task Completion Rate): task-level completion score based on the final completion anchor and its recursively critical dependencies from links.
• Token and runtime statistics, including prompt tokens, completion tokens, average E2E time, TTFT, and TPOT.

The report also keeps the following breakdowns:

• By Intent (primary_intent)
• By Task Length
• Task Length x Intent

📜 Citation

If you find this work useful, please cite:

@misc{shi2026androtmeminteractiontrajectoriesanchored,
      title={AndroTMem: From Interaction Trajectories to Anchored Memory in Long-Horizon GUI Agents}, 
      author={Yibo Shi and Jungang Li and Linghao Zhang and Zihao Dongfang and Biao Wu and Sicheng Tao and Yibo Yan and Chenxi Qin and Weiting Liu and Zhixin Lin and Hanqian Li and Yu Huang and Song Dai and Yonghua Hei and Yue Ding and Xiang Li and Shikang Wang and Chengdong Xu and Jingqi Liu and Xueying Ma and Zhiwen Zheng and Xiaofei Zhang and Bincheng Wang and Nichen Yang and Jie Wu and Lihua Tian and Chen Li and Xuming Hu},
      year={2026},
      eprint={2603.18429},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2603.18429}, 
}