Update relational schema: RI model, LocationGroup, step functions, ETL fixes - #280
Open
benhayes21 wants to merge 6 commits into
Open
Update relational schema: RI model, LocationGroup, step functions, ETL fixes#280benhayes21 wants to merge 6 commits into
benhayes21 wants to merge 6 commits into
Conversation
- Fix 134 column case mismatches across _import_account (82), _import_location (39), PolicyDetails (1), and LocationDetail (12) to match OED spec exactly - Add 9 PV (solar panel) fields to _import_location and LocationDetail - Add _staging_riinfo, _staging_riscope, _import_riinfo, _import_riscope tables with full column sets from the ReinsInfo/ReinsScope spec; wire into usp_Database_Load - Add AccountCoverageTerm and AccountPDTerm junction tables; implement account-level term views (vw_account_terms_1Building through _6All, vw_level_11/12/13_term) and replace the TO DO comment in usp_Terms_Load with working account-term load logic - Add StepPolicy table (FK to Policy) and usp_StepPolicy_Load procedure for step-function parametric policy fields, called from usp_Database_Load - load_sql.py: remove .head(10) on location CSV, uncomment usp_Database_Load exec, RI staging table names already match schema Closes #279 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- New StepPolicyTerm table (FK to Term + FK to StepPolicy) linking step-function parametric policies into the standard term hierarchy - vw_step_policy_terms unpivots StepPolicy into per-coverage-type rows (Building/Contents/BuildingContents → CoverageTypeId 1/3/6) carrying DeductibleBuilding/Contents/BuildingContents and PayOutLimit* as Ded/Limit - vw_level_14_term wraps the view with tmpTermId ordering - usp_StepPolicyTerm_Load creates Term records (level 14), StepPolicyTerm links, and fans down to ItemTerm via Policy→Account→Location→Coverage→Item - usp_Database_Load now calls usp_StepPolicyTerm_Load after usp_StepPolicy_Load - Diagram updated: StepPolicyTerm junction added, term levels bumped to 14 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three new tables normalise reinsurance data from OED flat CSVs: - ReinsInfo: treaty master (type, risk/programme terms, inuring priority, attachment basis, dates, CededPercent at treaty level) - ReinsScope: per-scope filter rows (hierarchical fields + FILTER_LEVEL_EXTRA_FIELDS + SS CededPercent scope override) - ReinsScopeLink: pre-resolved entity PKs (Portfolio/Account/Policy/Location) determined by RiskLevel (SEL/ACC/POL/LOC/LGR) RI staging tables (_staging_riinfo, _staging_riscope, _import_riinfo, _import_riscope) added; usp_ReinsInfo_Load and usp_ReinsScope_Load wire into usp_Database_Load. LGR risk level fans out to multiple Location rows. Schema diagram updated with RI tables, connections, legend, and gap analysis resolved card. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Schema fixes: - Move StepPolicyTerm CREATE TABLE to after Term (FK dependency order) - Move orphaned account-terms code (levels 11-13) into usp_Terms_Load body where it belongs; it was floating between view definitions causing CREATE VIEW failures - Fix usp_ReinsScope_Load LGR JOIN: LocGroup is on LocationDetail not Location; add LocationDetail→Location join - Add DELETE cleanup block to usp_Database_Load so re-runs are idempotent load_sql.py: - Replace engine.connect() with engine.begin() so EXEC auto-commits - Add fast_executemany=True to bypass SQL Server 2100-parameter limit (method='multi' sent all rows in one INSERT, hitting the limit) - Read DB password from OED_DB_PASSWORD env var instead of hardcoding .gitignore: add .env / *.env patterns Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
New table LocationGroup(LocationGroupId, LocGroup) normalises the LocGroup string into a proper entity. Location gains a LocationGroupId FK so group membership is a structural relationship rather than a searchable string in LocationDetail. LocGroup removed from LocationDetail. ReinsScopeLink gains a fifth nullable FK (LocationGroupId) for LGR risk level, replacing the previous per-Location fan-out. usp_ReinsScope_Load LGR case now inserts one ReinsScopeLink row per matched LocationGroup; Location.LocationGroupId is the join path to individual locations at query time. New usp_LocationGroup_Load extracts distinct LocGroup values from _import_location and runs before usp_Location_Load. usp_Location_Load updated to LEFT JOIN LocationGroup and populate Location.LocationGroupId. Cleanup block updated to delete LocationGroup after Location. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Column widths: - Condition.CondNumber/CondName/CondTag: VARCHAR(1) → VARCHAR(20/100/20) - LocationDetail+_import_location.LocName: NVARCHAR(20) → NVARCHAR(200) - _import_riinfo.ReinsName: VARCHAR(30) → VARCHAR(200) - #tmpterm_denormalised.TermPeril: VARCHAR(50) → VARCHAR(250) (insurance_conditions has 67-char peril strings) Logic bugs: - usp_Database_Load: TRUNCATE _import_* tables at start so re-runs don't accumulate rows from prior test cases - usp_Condition_Load: join _import_account on CondNumber (was missing) and use CROSS APPLY TOP 1 so multiple layers/rows per condition don't produce duplicate ConditionIds - ConditionCoverageTerm/ConditionPDTerm/ConditionTerm inserts: add WHERE EXISTS ConditionLocation guard so we don't reference TermIds that were filtered out during the Term insert (FK violation fix) - ItemTerm insert in usp_Terms_Load: add MAX(ItemTermId) offset so it doesn't collide with rows already written by usp_StepPolicyTerm_Load Add test suite (SQL/tests/) and export script (SQL/export_sql.py) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Build PreviewYou can find files attached to the below linked Workflow Run URL (Logs).
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #279
Summary
ReinsInfo,ReinsScope, andReinsScopeLinktables with full load procedures covering FAC/QS/SS/PR/CXL/AXL contract types and SEL/ACC/POL/LOC/LGR risk levelsLocGroupinto its own table;Location.LocationGroupIdFK enables clean LGR reinsurance scoping without denormalising the Location rowSQL/tests/): 77 parametrised pytest tests load all 15 OasisLMF validation cases and assert round-trip fidelity for locations, accounts, RI, step policies, and idempotencySQL/export_sql.py): Exports normalised data back to flat OED CSVs from the_import_*tablesload_sql.py: SA password moved toOED_DB_PASSWORDenv var;EXEC usp_Database_Loadnow runs correctly viaengine.begin()Test plan
OED_DB_PASSWORD=<pw> pytest SQL/tests/ -v)usp_Database_Loadtwice produces identical row counts🤖 Generated with Claude Code