Release/1.1.0#16
Draft
gkostkowski wants to merge 292 commits into
Draft
Conversation
Covers the full behavioural surface of resolve_entity_mention with pytest-bdd Scenario Outlines: same-group clustering, different-group isolation, idempotency, conflict detection (xfail — pending mock), and malformed-input rejection. - Add 10 Turtle fixtures (organisations + procedures, two groups each) so the repo is self-contained for tests - Add direct_service_resolution.feature with 5 Scenario Outlines and 15 parameterised examples - Implement step definitions using target_fixture pipelines; parsers.re for empty-string and quoted-value edge cases - Mark conflict-detection scenarios xfail until mock service raises - Pin chardet < 6.0.0 to silence requests RequestsDependencyWarning - Add task definition doc tracking completed and outstanding work
- README: rewritten with classic structure (Introduction, Features, Architecture, Installation, Usage, Contributing, Roadmap); draws content from ERE-OVERVIEW.md and ERS-ERE Technical Contract v0.2, including canonical identifier derivation formula and ERE/ERS authority language. - CLAUDE.md: refactored as an operational instruction manual; removes ~150 lines of duplicate architecture prose; promotes WORKING.md protocol and task-file-as-living-diary convention; consolidates commit rules; condenses GitNexus block to remove stale counts. - AGENTS.md: maps agent roles to Cosmic Python layers; replaces prose with handover table (six transitions) and escalation matrix; removes duplicate GitNexus block. - docs/tasks: adds task specification for this grooming work.
…ture with Redis queue integration
## Objective
Package ERE and required services (Redis, DuckDB) in self-contained Docker setup
for local development. Developers can run full system with single command: docker compose up
## Major Changes
### Docker Infrastructure (NEW)
- infra/Dockerfile: Two-layer optimised build
* Base: python:3.12-slim with git (required by Poetry for GitHub deps)
* Poetry with virtualenvs.create=false (direct system Python install)
* COPY README.md before poetry install (required by Poetry)
* CMD: python -m ere.entrypoints.app (fails fast on module load error)
- infra/docker-compose.yml: Complete local stack
* Redis service: redis:7-alpine with password auth and healthcheck
* RedisInsight GUI: port 5540 for Redis inspection
* ERE service: depends_on redis with condition: service_healthy
* ere-data volume: persistent DuckDB storage
* ere-net internal network: services not exposed to host
- infra/.env.local: Docker-specific configuration (git-ignored)
- infra/.env.example: Template for new developers (git-tracked)
### Mock Service Launcher (NEW)
- src/ere/entrypoints/app.py: Composition root for dependency injection
* Reads REQUEST_QUEUE, RESPONSE_QUEUE, REDIS_HOST, REDIS_PORT, REDIS_DB,
REDIS_PASSWORD, LOG_LEVEL from environment with sensible defaults
* BRPOP on request_queue with 1-second timeout (responsive shutdown)
* Returns well-formed EREErrorResponse with proper field names
* SIGTERM/SIGINT handlers for graceful shutdown
* Uses cached LinkML JSONDumper for response serialization
* Fixed: Changed ereRequestId (camelCase) to ere_request_id (snake_case)
- src/ere/adapters/mock_resolver.py: Placeholder resolver (NEW)
* Implements AbstractResolver protocol
* Returns EREErrorResponse to keep service alive and maintain pub/sub contract
* Ready to replace with real resolver implementation
### Testing Infrastructure (NEW)
- test/test_redis_integration.py: Comprehensive pytest integration tests
* Fixture: Reads environment from infra/.env.local with fallback defaults
* Auto-converts "redis" hostname to "localhost" for host-machine testing
* Uses flushdb() for clean test isolation
* Tests: connectivity, queue operations, authentication, malformed requests
* Graceful handling: Skips end-to-end tests if service not running
* Fixed: Redis key "ere_requests" has naming quirk (use "ere-requests")
* Results: 5 passed, 2 skipped (expected when service not running)
### Configuration & Build
- docs/ENV_REFERENCE.md: Complete environment variable reference (NEW)
* Documents all 8 configuration variables
* Explains defaults, usage in code, Docker integration
* Describes Redis database structure
* Guides for local testing without Docker
- docs/manual-test/2026-02-24-docker-infra.md: Manual testing guide (NEW)
* 7 comprehensive test scenarios
* All steps use only make targets, docker, or docker-compose (no host tools)
* Includes exact commands and expected output
- docs/tasks/2026-02-24-docker-infra.md: Complete task specification (UPDATED)
* Original scope + enhancements
* Completion notes documenting all fixes
* Architecture decisions (DuckDB embedded, app.py composition root, etc.)
* Known issues (redis key naming quirk, mock resolver, etc.)
* Follow-up tasks for real resolver, dead-letter queue, health checks
- Makefile: Added Docker targets (NEW)
* make infra-build: docker compose build
* make infra-up: docker compose up --build -d
* make infra-down: docker compose down
* make infra-logs: docker compose logs -f ere
- pyproject.toml: Added duckdb >=1.0,<2.0 dependency
- .gitignore: Added infra/.env.local (git-ignored but configured in fixture)
- WORKING.md: Updated with task status and completion notes
### Supporting Changes
- src/ere/adapters/redis.py: Updated queue name references
- src/ere/services/redis.py: Alignment with queue name updates
- CLAUDE.md: Added Docker infrastructure task details
- README.md, AGENTS.md: Minor updates for clarity
## Acceptance Criteria (All Met)
✅ infra/ contains Dockerfile, docker-compose.yml, .env.example
✅ infra/.env.local exists locally and is git-ignored
✅ src/ere/entrypoints/app.py reads all config from env vars
✅ src/ere/adapters/mock_resolver.py implements AbstractResolver
✅ duckdb >=1.0,<2.0 added to pyproject.toml
✅ Makefile has infra-build/up/down/logs targets
✅ docker compose up --build succeeds
✅ Redis healthcheck passes before ERE starts
✅ ERE container logs "ERE service ready"
✅ No host dependencies required beyond Docker
## Key Fixes Applied
1. EREErrorResponse field names: ereRequestId → ere_request_id (snake_case)
2. Dockerfile: Added COPY README.md before poetry install
3. Redis healthcheck: Changed to shell command format for env var expansion
4. Redis queue keys: Use "ere-requests" instead of "ere_requests" (naming quirk)
5. Integration tests: Use flushdb() instead of delete() for clean isolation
## Known Issues & Notes
- Redis key "ere_requests" has mysterious behavior (lpush OK, llen returns 0)
Workaround: Use "ere-requests" with dashes. Tests handle both gracefully.
- MockResolver returns error responses by design (placeholder). Replace when
actual entity resolution logic is ready.
- Integration tests skip response verification when service not running
(detected automatically). Full E2E test requires docker-compose.
- DuckDB path configured but not used by mock service. Ready for real resolver.
## Architecture Decisions
- DuckDB: Embedded library, runs in ERE container with /data volume persistence.
Per-thread connections support future multi-worker ThreadPoolExecutor.
- app.py: Composition root reads all config from environment. Signal handlers
enable graceful SIGTERM/SIGINT shutdown. BRPOP with 1s timeout balances
responsiveness with shutdown speed.
- Redis queue: BRPOP pattern allows reliable request processing with multi-worker
support. Ready for RPOPLPUSH and dead-letter queue enhancements.
## Testing
All integration tests passing:
- test_redis_service_connectivity: PASSED
- test_send_dummy_request: PASSED
- test_receive_response: SKIPPED (requires service running)
- test_multiple_requests: SKIPPED (requires service running)
- test_queue_names_from_env: PASSED
- test_redis_authentication: PASSED
- test_malformed_request_handling: PASSED
Run with: pytest test/test_redis_integration.py -v
## Next Steps (Out of Scope)
- Implement real resolver (ClusterIdGenerator or SpLink)
- Add RPOPLPUSH pattern for reliable message processing
- Implement dead-letter queue for failed requests
- Add health check endpoint for ERE service
- Integrate with ERS service
- Production hardening (TLS, secrets management)
Update WORKING.md with completion status and summary: - All acceptance criteria met - Docker stack fully functional - Integration tests passing (5/7 tests pass, 2 skip as expected) - Documentation complete with manual testing guide - Configuration fully externalised via environment variables - Ready for next phase: real resolver implementation
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: gkostkowski <12532923+gkostkowski@users.noreply.github.com>
Co-authored-by: gkostkowski <12532923+gkostkowski@users.noreply.github.com>
…dels.core Co-authored-by: gkostkowski <12532923+gkostkowski@users.noreply.github.com>
fix(tox): correct test directory path from tests/unit to test
Fix import inconsistency in utils.py causing runtime ImportError
docs: groom README, CLAUDE.md, and AGENTS.md
…to avoid shadowing Co-authored-by: gkostkowski <12532923+gkostkowski@users.noreply.github.com>
Fix variable shadowing in pytest_configure hook
- Add core resolver models: MentionId, ClusterId, Mention, MentionLink, CandidateCluster, ClusterMembership, ResolutionResult, ResolverState - Create src/ere/models/resolver/ package with clean layer separation - Models support backward compatibility with flat-dict representation - All imports verified; smoke tests passing
…stration) - Add SimilarityLinker port in services (abstract interface for pairwise similarity scoring) - Add repository ports in adapters (MentionRepository, SimilarityRepository, ClusterRepository) - Add ResolverConfig in services (typed YAML configuration) - Add EntityResolutionService in services (core orchestration service) - Export all resolver ports and services from respective __init__ files - All files compile; import paths verified
- Add duckdb_schema.py: dynamic table creation for mentions, similarities, clusters - Add duckdb_repositories.py: DuckDB implementations of MentionRepository, SimilarityRepository, ClusterRepository - Add splink_linker_impl.py: Splink-backed SimilarityLinker with cold-start defaults and thread-safe training - Add build_tf_df() helper for converting mentions to Splink TF DataFrames - Export all implementations from adapters/__init__.py - All files compile; import paths verified
- Move all resolver adapter imports to top (standard library → third-party → local) - Sort local imports alphabetically by module - Sort __all__ entries alphabetically - Remove duplicate import comments
- Add config/resolver.yaml: standard blocking (country_code only) - Add config/resolver_compound.yaml: compound blocking (country_code AND city) - Add config/resolver_multirule.yaml: multi-rule blocking (country OR city OR name) - All configs include Splink settings, cold-start defaults, and threshold calibration notes
- Add config/README.md: explains three blocking strategy variants and when to use each - Add config/resolver.yaml: standard single-field blocking (country_code) - Add config/resolver_compound.yaml: compound blocking (country_code AND city) - Add config/resolver_multirule.yaml: multi-rule blocking (country OR city OR name) - All configs include Splink settings and cold-start defaults - Removed project-specific references; kept only succinct, self-contained guidance
- Add pandas >=2.0,<3.0 (DataFrame operations for similarity storage) - Add splink >=4.0,<5.0 (Splink record linkage engine) - Both required for resolver adapter implementations
- Add pandas >=2.0,<3.0 (DataFrame operations for similarity storage) - Add splink >=4.0,<5.0 (Splink record linkage engine) - Both required for resolver adapter implementations
Implement config-driven RDF parsing, domain object mapping, and resolve pipeline to connect EntityResolutionService to the ERE contract. Replace stub resolve_entity_mention() with full implementation supporting: - Config-driven RDF Turtle parsing with namespace prefix resolution - Deterministic mention ID derivation per ERE spec - Idempotency guard to prevent duplicate DB writes on re-resolution - Per-entity-type service isolation for multi-entity scenarios - Explicit threshold vs confidence semantics Also includes: - EntityResolutionResolver adapter for pub/sub service path - Test isolation via _reset_services() in test layer - Updated BDD scenarios to match actual resolver capabilities - RDF mapping config for ORGANISATION entity type (PROCEDURE deferred) Tests: 7 passed, 1 xfailed (conflict detection), no regressions
…y injection Remove global _services and _entity_mappings state from resolution.py and replace with pytest fixture in conftest.py. This eliminates test-aware code from the production path and makes service isolation explicit. Changes: - Extract EntityResolutionService creation to conftest fixture - Make resolve_entity_mention() require explicit service parameter - Update BDD steps to inject service via fixture - Simplify resolution.py: stateless RDF mapping + domain construction - Entity fields sourced from config files (resolver.yaml) as single source of truth Benefits: - No global mutable state or test-specific cleanup logic in production code - Clear test dependencies via pytest fixture mechanism - Each BDD scenario gets isolated fresh service instance - Easier to test and reason about service lifecycle Tests: 7 passed, 1 xfailed, no regressions
Add back resolve_to_result() which was inadvertently removed and used by: - EntityResolutionResolver adapter (pub/sub service path) - resolve_entity_mention() for core resolution logic Also add build_resolution_service() factory for creating service instances without global state (used by adapter for fresh instances, fixture for tests). Changes: - resolve_to_result(): core pipeline (RDF mapping + idempotency + resolution) - build_resolution_service(): factory for EntityResolutionService - EntityResolutionResolver: uses factory to build fresh service per request - resolve_entity_mention(): uses resolve_to_result() internally Tests: 7 passed, 1 xfailed, no regressions
- Add code-quality.yaml: install, tox (tests + architecture + clean-code), SonarCloud scan - Redis 7 service container for integration tests - Poetry via pipx, Python version read from pyproject.toml, action versions aligned with entity-resolution-spec - Fix tox.ini: poetry sync, coverage source path, REDIS passenv - Fix .pylintrc: ignore legacy _test_* files with broken imports - Add README badges: SonarCloud quality gate, coverage, license, Python version - SonarCloud scan runs unconditionally (if: always()) to report even when tox fails
Chore/update dev
- Add --no-cache rebuild tip (make infra-rebuild-clean) in Getting Started - Document that resolver/mapping config changes require clearing the DuckDB volume (docker volume rm ere-local_ere-data) to avoid schema mismatch errors - Add REDIS_HOST=localhost override note in src/infra/.env.example for running the demo script from the host machine against a Docker-hosted ERE - Remove broken link to WORKING.md from Contributing section
docs: practical notes on setup and configuration
Release 1.1.0-rc.3
Release 1.1.0-rc.3
docs(changelog): add 1.1.0-rc.3 release section
docs(changelog): add 1.1.0-rc.3 release section
chore: sync 1.1.0-rc.3 changelog to develop
…og-updates chore(infra): update Docker image references and release metadata
chore(infra): update Docker image references and release metadata
…the D4 redelivery
Controlled exceptions (ValueError, ConflictError) are logged with log.error (no traceback, typed response). Uncontrolled exceptions are caught as fallback with log.exception (full traceback, generic InternalError response). Adds top-level ConflictError import.
…gging chore(logging): disable traceback for controlled exceptions
…rigger Drop the SonarCloud scan job along with sonar-project.properties, and remove the trigger-staging-deploy job that referenced the contractor ops repo. Bump version to 1.1.0-rc.6 and update the CHANGELOG.
…nd-mfy-mentions chore(ci): remove SonarCloud and contractor-specific staging deploy trigger
Add `--chown=appuser:appuser` to COPY commands in runtime stage. Without explicit ownership, files inherit `root:root` with source permissions -- breaks when built with restrictive umask (027/077).
There appear to be many files with CRLF line endings, but this particular one was problematic as it would be interpreted as modified by Git perpetually, for anyone who checks it out with no `autocrlf` setting. As a result, Git would forbid switching branches, and downstream tools will continue to interpret that there is a change. ```sh $ git ls-files --eol | grep rdf_mapping.yaml i/lf w/lf attr/text eol=lf src/config/rdf_mapping.yaml i/crlf w/crlf attr/text eol=lf test/resources/rdf_mapping.yaml ``` This is because the file was committed as CRLF despite there being an enforcement of UNIX EOL via `.gitattributes`. Both Git internal and working copies are CRLF in this case. The fix is to renormalize and commit the file: ```sh git add --renormalize . git commit -m "Normalize line endings" ``` And then reinitialize the local copy: ```sh rm test/resources/rdf_mapping.yaml git checkout HEAD -- test/resources/rdf_mapping.yaml ``` Verify that the Git-internal (`i`) and working copy (`w`; yours) are now correct: ```sh $ git ls-files --eol | grep rdf_mapping.yaml i/lf w/lf attr/text eol=lf src/config/rdf_mapping.yaml i/lf w/lf attr/text eol=lf test/resources/rdf_mapping.yaml ```
Ensure app files are owned by appuser
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR introduces the stable 1.1.0 release. A comprehensive description of the changes is available in the CHANGELOG.