Add support for switching embedding models

[ktamaral]

Description

This update adds a new base class to support switching between different embedding models by setting model_type and model_name in the config. The SigLIP embedding model is now supported. Additional embedding models can be supported similarly by writing a new embedding model subclass. Also this update adds support for Mac GPU devices by setting "mps" in the config.

Motivation and Context

• This adds the ability to switch between embedding models for testing, benchmarking, and evaluation

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Code refactoring (no functional changes)
• Performance improvement
• Research contribution (new models, evaluation methods, etc.)
• Other (please describe):

Component(s) Affected

• Backend (Python/FastAPI)
• Frontend - Photographs
• Frontend - Maps
• Frontend - Documents
• CLIP/ML models
• Configuration
• Documentation
• Tests
• Build/deployment

Changes Made

• Adds support for SigLIP
• Adds BaseEmbeddingService class with a factory pattern that can be extended to support additional embedding models
• Updates config schema with backwards compatibility for existing configs
• Adds support for Mac GPU with "mps" config option
• Adds unit tests

Testing

How Has This Been Tested?

• Created SigLip embeddings: Updated config to siglip settings and created new embeddings, tested embeddings in the app, which anecdotally looked a bit more accurate (a search for cats with SigLip embeddings returned cat, cat, cat, cat, cat, instead of the same dataset with Clip embeddings which previously had returned cat, dog, another not cat animal, nature pic with no animal, cat)
• Confirmed CLIP is still working: Created a new set of CLIP embeddings on the same dataset to check for regressions
• Ran unit tests: Updated unit tests for embeddings service coverage

Test Configuration

• Collection type tested: photographs
• Python version: 3.12
• Node version: 20
• OS: macOS
• Embedding models:
  • openai/clip-vit-base-patch32
  • google/siglip-base-patch16-256-multilingual

Screenshots (if applicable)

N/A

Checklist

Code Quality

• My code follows the project's coding standards
• I have run black . and isort . on Python code
• I have run npm run lint on frontend code (if applicable)
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• My changes generate no new warnings or errors

Testing

• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I have tested this locally with actual data

Documentation

• I have updated the documentation accordingly
• I have updated the README if needed
• I have added docstrings to new functions/classes
• I have updated config.json documentation if config changes were made

Dependencies

• I have updated requirements.txt (if Python dependencies changed)
• I have updated package.json (if Node dependencies changed)
• I have documented any new configuration options

Research (if applicable)

• I have included references to relevant papers or research
• I have shared evaluation results or benchmarks
• I have included information about datasets used
• I have documented model training procedures

Breaking Changes

These changes are designed to be backwards compatible, even with older versions of configs without newly introduced properties

Additional Notes

If there is any interest, I can also work on adding classes for SigLIP2 and ColPali

Reviewers Checklist (for maintainers)

• Code quality and style compliance
• Test coverage adequate
• Documentation complete
• No security concerns
• Performance implications acceptable
• Breaking changes documented

[ktamaral]

I'm looking into the linter issue. I'm not sure how or why but I think the changes made by black and isort may be conflicting with each other. For example, running black . and then black --check . passes. Then running isort . and running black --check . again fails. I will look into how to clean up this issue.

(venv) katieamaral@HUIT-MacBook-Pro digital-collections-explorer % black .        
reformatted /Volumes/dev/HUIT/digital-collections-explorer/src/backend/services/__init__.py
reformatted /Volumes/dev/HUIT/digital-collections-explorer/src/backend/api/routes/search.py
reformatted /Volumes/dev/HUIT/digital-collections-explorer/tests/test_embedding_service_factory.py
reformatted /Volumes/dev/HUIT/digital-collections-explorer/src/models/clip/generate_embeddings.py

All done! ✨ 🍰 ✨
4 files reformatted, 21 files left unchanged.
(venv) katieamaral@HUIT-MacBook-Pro digital-collections-explorer % black --check .
All done! ✨ 🍰 ✨
25 files would be left unchanged.
(venv) katieamaral@HUIT-MacBook-Pro digital-collections-explorer % isort .        
Fixing /Volumes/dev/HUIT/digital-collections-explorer/tests/test_embedding_service_factory.py
Fixing /Volumes/dev/HUIT/digital-collections-explorer/src/backend/api/routes/search.py
Fixing /Volumes/dev/HUIT/digital-collections-explorer/src/backend/services/__init__.py
Fixing /Volumes/dev/HUIT/digital-collections-explorer/src/models/clip/generate_embeddings.py
Skipped 6 files
(venv) katieamaral@HUIT-MacBook-Pro digital-collections-explorer % black --check .
would reformat /Volumes/dev/HUIT/digital-collections-explorer/src/backend/services/__init__.py
would reformat /Volumes/dev/HUIT/digital-collections-explorer/src/backend/api/routes/search.py
would reformat /Volumes/dev/HUIT/digital-collections-explorer/tests/test_embedding_service_factory.py
would reformat /Volumes/dev/HUIT/digital-collections-explorer/src/models/clip/generate_embeddings.py

Oh no! 💥 💔 💥
4 files would be reformatted, 21 files would be left unchanged.

[ktamaral]

@hinxcode in regards to the above black / isort compatibility issue, this configuration setting might fix the issue: https://pycqa.github.io/isort/docs/configuration/black_compatibility.html

Let me know if this solution is workable or if I should look into other ways to fix the issue perhaps finding a way to refactor my code. Thanks!

[hinxcode]

LGTM. Will open a follow-up PR for a few smaller cleanup items from review (pyproject.toml, factory singleton, SigLIP scoring formula). Thanks for the work!