LlamaSeg: Image Segmentation via Autoregressive Mask Generation

Paper | Dataset

This repository hosts the official implementation of LlamaSeg, an image segmentation method based on autoregressive mask generation. We also introduce a data annotation pipeline.

---

1. Installation

1. Clone the repository and navigate into it:

   git clone https://github.com/GML-FMGroup/LlamaSeg.git
   cd LlamaSeg
   

2. Create a Conda environment:

   conda create -n llamaseg python=3.10 -y
   conda activate llamaseg

3. Install PyTorch and Torchvision matching your CUDA version. For example, using PyTorch 2.1.2 with CUDA 12.1:

   pip install torch==2.1.2+cu121 torchvision==0.16.2+cu121 -f https://download.pytorch.org/whl/torch_stable.html

4. Install other dependencies:

   pip install -r requirements.txt

---

2. Download and Organize Datasets

2.1 Existing Datasets

Download the following datasets:

• Semantic segmentation datasets: ADE20K, COCO-Stuff
• Referring segmentation datasets: refCOCO, refCOCO+, refCOCOg, refCLEF

Organize them as below:

Dataset
└── Exist_dataset
    ├── ade20k
    │   ├── annotations
    │   └── images
    ├── coco
    │   └── train2017
    │       ├── 000000000009.jpg
    │       └── ...
    ├── cocostuff
    │   └── train2017
    │       ├── 000000000009.png
    │       └── ...
    └── refer_seg
        ├── images
        │   ├── saiapr_tc-12
        │   └── mscoco
        │       └── images
        │           └── train2014
        ├── refclef
        ├── refcoco
        ├── refcoco+
        └── refcocog

Note: Before running any scripts, update all path variables in the scripts to point to these directories.

2.2 Extract Train Data

Run data extraction scripts:

bash utils/extract_exist_data.sh

After extraction, the directory should look like:

Dataset
└── Train_data
    ├── jsonl
    │   ├── ade20k.jsonl
    │   └── ...
    ├── original_image
    │   └── refclef
    │       ├── 25.jpg
    │       └── ...
    └── original_mask
        ├── ade20k
        ├── cocostuff
        ├── refcoco
        ├── refcoco+
        ├── refcocog
        └── refclef

2.3 SA-1B Dataset Preparation

1. Download raw SA-1B images (Download from the official website or OpenDataLab) and place required .jpg files in Dataset/original_image/sa/ by filtering names listed in utils/sa_images.csv.
2. Download the annotated masks and organize them into the following directory structure

Dataset
└── Annotated_sa1b_data
    ├── sa_000000
    │   ├── sa_5265.json
    │   └── ...
    ├── sa_000001
    └── ...

3. Organize SA-1B data:

bash utils/organize_sa1b_data.sh

Final layout:

Dataset
└── Train_data
    ├── jsonl
    │   ├── ade20k.jsonl
    │   ├── ...
    │   ├── sa_000000.jsonl
    │   ├── sa_000001.jsonl
    │   └── ...
    ├── original_image
    │   ├── refclef
    │   └── sa
    │       ├── sa_5265.jpg
    │       └── ...
    └── original_mask
        ├── ade20k
        ├── ...
        └── sa
            ├── sa_5265_sea.png
            └── ...

---

3. Model Download

1. SigLIP2 (256×256):

   huggingface-cli download --resume-download google/siglip2-large-patch16-256 --local-dir path_to_model_save_dir

2. VQGAN:

   wget https://huggingface.co/peizesun/llamagen_t2i/resolve/main/vq_ds16_t2i.pt

3. Janus Pro:

   huggingface-cli download --resume-download deepseek-ai/Janus-Pro-1B --local-dir path_to_model_save_dir

---

4. Training

Update model and data paths in scripts before running.

4.1 LlamaSeg (Parameter Random Init)

• Pretrain on SA and referring datasets:

  cd LlamaSeg
  bash pretrain.sh

• Fine-tune on semantic or referring segmentation data:

  bash sft.sh

4.2 JanusSeg (Janus Pro)

• Pretrain:

  cd JanusSeg
  bash pretrain.sh

• Fine-tune:

  bash sft.sh

---

5. Inference

5.1 LlamaSeg

cd LlamaSeg
bash seg.sh

5.2 JanusSeg

cd JanusSeg
bash seg.sh

---

Citation

If our work helps your research, please cite:

@article{deng2025llamaseg,
  title={LlamaSeg: Image Segmentation via Autoregressive Mask Generation},
  author={Deng, Jiru and Weng, Tengjin and Yang, Tianyu and Luo, Wenhan and Li, Zhiheng and Jiang, Wenhao},
  journal={arXiv preprint arXiv:2505.19422},
  year={2025}
}

Acknowledgments

This repository benefits from the excellent work of LlamaGen, Janus, SigLIP2, Qwen2-VL, SAM, and LISA.