Skip to content

Feat: Add ARM Support and Apple MPS Acceleration - #594

Open
doge-rgb wants to merge 2 commits into
salesforce:mainfrom
doge-rgb:main
Open

Feat: Add ARM Support and Apple MPS Acceleration#594
doge-rgb wants to merge 2 commits into
salesforce:mainfrom
doge-rgb:main

Conversation

@doge-rgb

Copy link
Copy Markdown

Add ARM Support and Apple MPS Acceleration.
Testing:

import torch
from PIL import Image

from lavis.models import load_model_and_preprocess
from lavis.processors import load_processor

raw_image = Image.open("xx.jpg").convert("RGB")

device = torch.device("mps")

captions = ["x", "xxx"]
model, vis_processors, text_processors = load_model_and_preprocess("blip2_image_text_matching", "pretrain", device=device, is_eval=True)
img = vis_processors["eval"](raw_image).unsqueeze(0).to(device)
for caption in captions:
    txt = text_processors["eval"](caption)
    itm_output = model({"image": img, "text_input": txt}, match_head="itm")
    itm_scores = torch.nn.functional.softmax(itm_output, dim=1)
    print(f'{caption}:  {itm_scores[:, 1].item():.3%}')

@salesforce-cla

Copy link
Copy Markdown

Thanks for the contribution! Unfortunately we can't verify the commit author(s): guobentian <g***@s***.com>. One possible solution is to add that email to your GitHub account. Alternatively you can change your commits to another email and force push the change. After getting your commits associated with your GitHub account, sign the Salesforce Inc. Contributor License Agreement and this Pull Request will be revalidated.

@salesforce-cla

Copy link
Copy Markdown

Thanks for the contribution! Before we can merge this, we need @ChatGPT-1 to sign the Salesforce Inc. Contributor License Agreement.

anson416 added a commit to anson416/LAVIS that referenced this pull request Feb 11, 2026
Add dtype alignment in Mlp.forward, Attention, and PatchEmbed to cast input tensors to float16/float32 based on the corresponding layer weights. This prevents dtype mismatches in mixed-precision scenarios when model weights are in a different precision (see salesforce#594). Changes applied to lavis/models/eva_vit.py.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant