diff --git a/libs/sap-mock-data/README.md b/libs/sap-mock-data/README.md index 57e55e74..f0bf93df 100644 --- a/libs/sap-mock-data/README.md +++ b/libs/sap-mock-data/README.md @@ -1,18 +1,18 @@ # SAP Mock Data `sap-mock-data` generates deterministic, interconnected SAP-style master and -transaction data. The pandas generation engine is available through one Python -API and is shared by the CLI, local notebooks, and Databricks notebooks. +transaction data. The `generate_dataset` function generates it. The CLI, +local notebooks, and Databricks notebooks all call that function. -Generation has two layers. Pandas creates and mutates tables. A `TableStore` -decides where those tables live. Delta Lake is the default local store. The -Databricks notebook supplies a Unity Catalog adapter. +Pandas creates and mutates tables. A `TableStore` persists them. Delta Lake +is the default local store. The Databricks notebook supplies a Unity Catalog +adapter. ## Development environment -The library supports Python 3.11 through 3.13 and does not require Nix. Nix is -an optional way to provision the Python and system tools; Python dependencies -come from `pyproject.toml` and are locked in `uv.lock`. +The library supports Python 3.11 through 3.13. Nix is an optional way to +provision the Python and system tools. Python dependencies come from +`pyproject.toml` and are locked in `uv.lock`. ### Without Nix @@ -28,8 +28,7 @@ uv build The checked-in `.python-version` selects Python 3.13. By default, uv downloads that runtime when it is not already installed. -The package also works with standard Python tooling when the lock file is not -needed: +The package also installs with venv and pip, without the lock file: ```console python -m venv .venv @@ -46,13 +45,13 @@ uv sync --frozen --extra spark --group notebook uv run jupyter lab ``` -Local Spark additionally requires a Java 17 or newer JDK available on `PATH` -with `JAVA_HOME` configured. Databricks supplies its own Spark and Java runtime. +Local Spark requires a Java 17 or newer JDK on `PATH` with `JAVA_HOME` set. +Databricks supplies its own Spark and Java runtime. ### With Nix -The default Nix shell provides pinned Python and uv executables. It uses the -same uv-managed Python dependency workflow as the non-Nix setup: +The default Nix shell provides pinned Python and uv executables. The uv +commands are the same as without Nix: ```console nix develop @@ -69,8 +68,8 @@ uv sync --frozen --extra spark --group notebook uv run jupyter lab ``` -PySpark imports are confined to notebooks, so core library and CLI users do not -need Java or Spark in either environment. +Only the notebooks import PySpark. The library and CLI run without Java or +Spark in either environment. ## Library API @@ -93,9 +92,9 @@ print(result.table_count, result.row_counts) Use a new or isolated warehouse path for each run. Generation overwrites its tables and preserves unrelated tables in an existing store. -For embedding or quick checks, use `MemoryTableStore`. The generation API -accepts any pandas-oriented `TableStore`, including the Databricks adapter shown -in `notebooks/databricks/Generate SAP Mock Data.py`. +For embedding or quick checks, use `MemoryTableStore`. `generate_dataset` +accepts any `TableStore` implementation, including the Databricks adapter in +`notebooks/databricks/Generate SAP Mock Data.py`. Scenario selection accepts `demo`, `none`, `all`, a comma-separated string, or a sequence of IDs. Explicit scenarios need configuration strings: @@ -107,6 +106,32 @@ GenerationConfig( ) ``` +### Dataset size + +`scale_factor` is a positive number or one of the size identifiers `S`, +`M`, `L`, `XL`. For a size identifier, `GenerationConfig` samples each count +from the ranges below, seeded by `random_seed`. The same seed produces the +same dataset, apart from the `INJECTED_AT` timestamp in `scenario_metadata`. +A number multiplies the default order, customer, material, vendor, and site +counts. + +| size | products | suppliers | sites | BOM depth | raw materials | customers | orders | +| ---- | -------- | --------- | ------- | --------- | ------------- | --------- | ----------- | +| S | 3 | 3-5 | 1-2 | 1 | 6-10 | 4-8 | 200-400 | +| M | 10-50 | 20-30 | 4-5 | 1 | 27-42 | 25-40 | 4000-6000 | +| L | 100-200 | 50-100 | 20-40 | 1 | 72-112 | 80-120 | 12000-20000 | +| XL | 400-800 | 150-300 | 100-120 | 1 | 180-280 | 250-400 | 40000-80000 | + +- The products column includes two fixed BOM parent materials, and the raw + materials column includes three fixed BOM components. +- `num_customers`, `num_finished_goods`, `num_raw_materials`, `num_vendors`, + `num_sites`, and `num_orders` set their counts directly, whichever form + `scale_factor` takes. +- The first five sites are fixed plants. Further sites are synthesized with + ids from 6000 in steps of 10, and every fifth one is a production plant. +- The demo scenarios target ids that exist at every size. +- Generation time grows with products times sites. `XL` takes over an hour. + ### Currency `GenerationConfig.currency` sets the three-letter currency code used by all @@ -131,5 +156,5 @@ The manifest records schemas, row and null counts, and canonical content hashes. `SparkCatalogStore` and calls the same API. - `notebooks/databricks/Node Impact Analysis.py` runs the node-impact analysis. -Spark serves as a storage adapter. Generation logic lives under -`src/sap_mock_data`; notebooks call that package. +Spark is a storage adapter. Generation logic is in `src/sap_mock_data`. +Notebooks call that package. diff --git a/libs/sap-mock-data/notebooks/databricks/Generate SAP Mock Data.py b/libs/sap-mock-data/notebooks/databricks/Generate SAP Mock Data.py index 8bf95943..9da6e992 100644 --- a/libs/sap-mock-data/notebooks/databricks/Generate SAP Mock Data.py +++ b/libs/sap-mock-data/notebooks/databricks/Generate SAP Mock Data.py @@ -89,10 +89,15 @@ def tables(self) -> list[str]: dbutils.widgets.get("catalog"), dbutils.widgets.get("schema"), ) +scale_widget = dbutils.widgets.get("scale_factor") +try: + scale_factor = float(scale_widget) +except ValueError: + scale_factor = scale_widget result = generate_dataset( GenerationConfig( random_seed=int(dbutils.widgets.get("random_seed")), - scale_factor=float(dbutils.widgets.get("scale_factor")), + scale_factor=scale_factor, scenarios=dbutils.widgets.get("scenarios"), ), store, diff --git a/libs/sap-mock-data/pipeline/Mock-up SAP ERP data - Masterdata.py b/libs/sap-mock-data/pipeline/Mock-up SAP ERP data - Masterdata.py index fa4ea0d8..8328202c 100644 --- a/libs/sap-mock-data/pipeline/Mock-up SAP ERP data - Masterdata.py +++ b/libs/sap-mock-data/pipeline/Mock-up SAP ERP data - Masterdata.py @@ -5,7 +5,6 @@ import numpy as np from faker import Faker import random -import uuid from math import radians, sin, cos, sqrt, asin from datetime import datetime, timedelta import pyspark.sql.functions as F @@ -532,7 +531,7 @@ def generate_sapapo_tr_data(): from_info = PLANT_CONFIG[loc_from] to_info = PLANT_CONFIG[loc_to] - trlid = str(uuid.uuid4()).replace('-', '').upper()[:32] + trlid = f"{random.getrandbits(128):032X}" lane_name = f"{from_info['city']} -> {to_info['city']}" data.append({ diff --git a/libs/sap-mock-data/pipeline/Mock-up SAP ERP data - Transactions.py b/libs/sap-mock-data/pipeline/Mock-up SAP ERP data - Transactions.py index 557e796a..78482a7d 100644 --- a/libs/sap-mock-data/pipeline/Mock-up SAP ERP data - Transactions.py +++ b/libs/sap-mock-data/pipeline/Mock-up SAP ERP data - Transactions.py @@ -281,6 +281,7 @@ def generate_sales_orders(finished_goods, all_customers): print(f"Generating {NUMBER_OF_ORDERS} Sales Orders...") vbak, vbap, vbep = [], [], [] base_date = datetime.now() + india_customer_exists = CUST_INDIA in all_customers for i in range(NUMBER_OF_ORDERS): vbeln = f'{1000000000 + i:010d}' @@ -290,7 +291,7 @@ def generate_sales_orders(finished_goods, all_customers): req_date = (order_date_dt + timedelta(days=7)).strftime('%Y%m%d') # Scenario: India Customer - if i % 100 == 0: kunnr = CUST_INDIA + if i % 100 == 0 and india_customer_exists: kunnr = CUST_INDIA else: kunnr = random.choice(all_customers) order_total = 0.0 diff --git a/libs/sap-mock-data/src/sap_mock_data/cli/main.py b/libs/sap-mock-data/src/sap_mock_data/cli/main.py index 0d439ea4..fa8468f0 100644 --- a/libs/sap-mock-data/src/sap_mock_data/cli/main.py +++ b/libs/sap-mock-data/src/sap_mock_data/cli/main.py @@ -4,15 +4,33 @@ import argparse import json +import math from importlib.metadata import version from pathlib import Path from typing import Sequence from .. import GenerationConfig, generate_dataset +from ..config import SIZE_KNOB_RANGES from ..storage import DeltaTableStore from ..validation import build_manifest, integrity_report +def _scale_factor(value: str) -> float | str: + try: + number = float(value) + except ValueError: + letter = value.strip().upper() + if letter in SIZE_KNOB_RANGES: + return letter + raise argparse.ArgumentTypeError( + f"must be a positive number or one of {', '.join(SIZE_KNOB_RANGES)}; " + f"got {value!r}" + ) + if not (number > 0 and math.isfinite(number)): + raise argparse.ArgumentTypeError(f"must be a positive finite number; got {value}") + return number + + def _scenario_configs(values: Sequence[str]) -> dict[str, str]: configs: dict[str, str] = {} for value in values: @@ -37,11 +55,18 @@ def _parser() -> argparse.ArgumentParser: generate = commands.add_parser("generate", help="generate a dataset") generate.add_argument("output", type=Path, help="Delta warehouse directory") generate.add_argument("--seed", type=int, default=42) - generate.add_argument("--scale-factor", type=float, default=1.0) + generate.add_argument( + "--scale-factor", + type=_scale_factor, + default=1.0, + help="numeric multiplier, or a dataset size: S, M, L, XL", + ) generate.add_argument("--orders", type=int) generate.add_argument("--customers", type=int) generate.add_argument("--finished-goods", type=int) generate.add_argument("--raw-materials", type=int) + generate.add_argument("--vendors", type=int) + generate.add_argument("--sites", type=int) generate.add_argument("--currency", default="EUR") generate.add_argument( "--scenarios", @@ -78,25 +103,28 @@ def _write_json(payload: object, output: Path | None) -> None: def main(argv: Sequence[str] | None = None) -> int: - args = _parser().parse_args(argv) + parser = _parser() + args = parser.parse_args(argv) if args.command == "generate": try: scenario_configs = _scenario_configs(args.scenario_config) - except argparse.ArgumentTypeError as error: - _parser().error(str(error)) - config = GenerationConfig( - random_seed=args.seed, - scale_factor=args.scale_factor, - num_orders=args.orders, - num_customers=args.customers, - num_finished_goods=args.finished_goods, - num_raw_materials=args.raw_materials, - currency=args.currency, - scenarios=args.scenarios, - scenario_configs=scenario_configs, - generate_dirty_data=args.dirty_data, - dirty_data_rate=args.dirty_data_rate, - ) + config = GenerationConfig( + random_seed=args.seed, + scale_factor=args.scale_factor, + num_orders=args.orders, + num_customers=args.customers, + num_finished_goods=args.finished_goods, + num_raw_materials=args.raw_materials, + num_vendors=args.vendors, + num_sites=args.sites, + currency=args.currency, + scenarios=args.scenarios, + scenario_configs=scenario_configs, + generate_dirty_data=args.dirty_data, + dirty_data_rate=args.dirty_data_rate, + ) + except (argparse.ArgumentTypeError, ValueError) as error: + parser.error(str(error)) store = DeltaTableStore(args.output) result = generate_dataset(config, store) print( diff --git a/libs/sap-mock-data/src/sap_mock_data/config.py b/libs/sap-mock-data/src/sap_mock_data/config.py index 6d214f89..d14b1ac4 100644 --- a/libs/sap-mock-data/src/sap_mock_data/config.py +++ b/libs/sap-mock-data/src/sap_mock_data/config.py @@ -2,33 +2,75 @@ from __future__ import annotations +import math +import random from dataclasses import dataclass, field from typing import Mapping, Sequence ALL_SCENARIOS = tuple(f"SCN{i:03d}" for i in range(1, 27)) +# Knob ranges per dataset size. The fixed BOM parent materials put product +# totals at NUM_FINISHED_GOODS + 2 below 20 and + 1 from 20 up. +SIZE_KNOB_RANGES: dict[str, dict[str, tuple[int, int]]] = { + "S": { + "NUM_FINISHED_GOODS": (1, 1), + "NUM_RAW_MATERIALS": (3, 7), + "NUM_VENDORS": (3, 5), + "NUM_CUSTOMERS": (4, 8), + "NUM_ORDERS": (200, 400), + "NUM_SITES": (1, 2), + }, + "M": { + "NUM_FINISHED_GOODS": (8, 49), + "NUM_RAW_MATERIALS": (25, 40), + "NUM_VENDORS": (20, 30), + "NUM_CUSTOMERS": (25, 40), + "NUM_ORDERS": (4000, 6000), + "NUM_SITES": (4, 5), + }, + "L": { + "NUM_FINISHED_GOODS": (99, 199), + "NUM_RAW_MATERIALS": (70, 110), + "NUM_VENDORS": (50, 100), + "NUM_CUSTOMERS": (80, 120), + "NUM_ORDERS": (12000, 20000), + "NUM_SITES": (20, 40), + }, + "XL": { + "NUM_FINISHED_GOODS": (399, 799), + "NUM_RAW_MATERIALS": (178, 278), + "NUM_VENDORS": (150, 300), + "NUM_CUSTOMERS": (250, 400), + "NUM_ORDERS": (40000, 80000), + "NUM_SITES": (100, 120), + }, +} + +# Demo configs reference only ids that exist at every dataset size. DEMO_SCENARIO_CONFIGS: dict[str, str] = { - "SCN001": "MAT-A0008,1000,FG01,500", - "SCN003": "2000,ALL,20250615,30", - "SCN011": "MAT-A0005,1000,25,20250615", - "SCN012": "MAT-NEW01,1000,MAT-A0005", + "SCN001": "MAT-A0001,1000,FG01,500", + "SCN003": "1000,ALL,20250615,30", + "SCN011": "MAT-A0001,1000,25,20250615", + "SCN012": "MAT-NEW01,1000,MAT-A0001", "SCN014": "1000,95,30", "SCN015": "1000,20250615,7,0.3", - "SCN016": "1000,MAT-A0005;MAT-A0008,30", + "SCN016": "1000,MAT-A0001;MAT-A0020,30", "SCN020": "40,30", - "SCN021": "VEND-0005,ALL,0.72,3", - "SCN023": "VEND-0008,MAT-R0010", - "SCN026": "VEND-0008,ALL,0.85,3", + "SCN021": "VEND-0001,ALL,0.72,3", + "SCN023": "VEND-0001,API1", + "SCN026": "VEND-0002,ALL,0.85,3", } @dataclass(frozen=True, slots=True) class GenerationConfig: random_seed: int = 42 - scale_factor: float = 1.0 + scale_factor: float | str = 1.0 num_customers: int | None = None num_finished_goods: int | None = None num_raw_materials: int | None = None + num_vendors: int | None = None + num_sites: int | None = None num_orders: int | None = None moq_finished_min: int = 250 moq_finished_max: int = 1000 @@ -46,9 +88,17 @@ class GenerationConfig: scenario_configs: Mapping[str, str] = field(default_factory=dict) def __post_init__(self) -> None: - if self.scale_factor <= 0: + if isinstance(self.scale_factor, str): + letter = self.scale_factor.strip().upper() + if letter not in SIZE_KNOB_RANGES: + raise ValueError( + f"scale_factor must be a positive number or one of " + f"{', '.join(SIZE_KNOB_RANGES)}; got {self.scale_factor!r}" + ) + object.__setattr__(self, "scale_factor", letter) + elif not (self.scale_factor > 0 and math.isfinite(self.scale_factor)): raise ValueError( - f"scale_factor must be greater than zero; got {self.scale_factor}" + f"scale_factor must be a positive finite number; got {self.scale_factor}" ) if not 0 <= self.delivery_fill_rate <= 1: raise ValueError( @@ -72,6 +122,8 @@ def __post_init__(self) -> None: "num_customers", "num_finished_goods", "num_raw_materials", + "num_vendors", + "num_sites", "num_orders", ): value = getattr(self, name) @@ -105,7 +157,7 @@ def parameters(self) -> dict[str, str]: values = { "RANDOM_SEED": str(self.random_seed), - "SCALE_FACTOR": str(self.scale_factor), + "SCALE_FACTOR": "1" if isinstance(self.scale_factor, str) else str(self.scale_factor), "MOQ_FINISHED_MIN": str(self.moq_finished_min), "MOQ_FINISHED_MAX": str(self.moq_finished_max), "MOQ_RAW_MIN": str(self.moq_raw_min), @@ -119,10 +171,20 @@ def parameters(self) -> dict[str, str]: "GENERATE_DIRTY_DATA": str(self.generate_dirty_data).lower(), "DIRTY_DATA_RATE": str(self.dirty_data_rate), } + if isinstance(self.scale_factor, str): + rng = random.Random(f"{self.random_seed}:{self.scale_factor}") + values.update( + { + knob: str(rng.randint(low, high)) + for knob, (low, high) in SIZE_KNOB_RANGES[self.scale_factor].items() + } + ) optional = { "NUM_CUSTOMERS": self.num_customers, "NUM_FINISHED_GOODS": self.num_finished_goods, "NUM_RAW_MATERIALS": self.num_raw_materials, + "NUM_VENDORS": self.num_vendors, + "NUM_SITES": self.num_sites, "NUM_ORDERS": self.num_orders, } values.update( diff --git a/libs/sap-mock-data/src/sap_mock_data/generation/common.py b/libs/sap-mock-data/src/sap_mock_data/generation/common.py index b9ab3b80..705d996b 100644 --- a/libs/sap-mock-data/src/sap_mock_data/generation/common.py +++ b/libs/sap-mock-data/src/sap_mock_data/generation/common.py @@ -1,4 +1,4 @@ -"""Provide parameter lookup and random-seed helpers.""" +"""Parameters, seeding, and the plant model shared by the generation stages.""" import random @@ -17,6 +17,8 @@ "MOQ_RAW_MIN": "1000", "MOQ_RAW_MAX": "10000", "NUM_ORDERS": "5000", + "NUM_VENDORS": "20", + "NUM_SITES": "5", "HUB_PLANT": "1000", "DELIVERY_FILL_RATE": "0.8", "SAFETY_STOCK_WEEKS": "6", @@ -27,7 +29,7 @@ "DIRTY_DATA_RATE": "0.05", } -PLANT_CONFIG = { +BASE_PLANTS = { "1000": { "name": "Manufacturing Hub", "name2": "Primary Production", @@ -40,6 +42,8 @@ "calendar": "DE", "xpos": 9.1829, "ypos": 48.7758, + "port": False, + "cost_factor": 1.0, }, "2000": { "name": "Regional DC Europe", @@ -53,6 +57,8 @@ "calendar": "DE", "xpos": 8.6821, "ypos": 50.1109, + "port": False, + "cost_factor": 1.15, }, "3000": { "name": "Regional DC Americas", @@ -66,6 +72,8 @@ "calendar": "US", "xpos": -74.1724, "ypos": 40.7357, + "port": True, + "cost_factor": 0.85, }, "4000": { "name": "Regional DC Asia Pacific", @@ -79,6 +87,8 @@ "calendar": "SG", "xpos": 103.8198, "ypos": 1.3521, + "port": True, + "cost_factor": 1.25, }, "5000": { "name": "Secondary Manufacturing", @@ -92,9 +102,123 @@ "calendar": "IE", "xpos": -8.4756, "ypos": 51.8985, + "port": True, + "cost_factor": 1.0, }, } +# Cities for plants beyond the base plants: country, region, city, +# calendar, longitude, latitude, port. +CITY_POOL = ( + ("NL", "ZH", "Rotterdam", "NL", 4.4777, 51.9244, True), + ("DE", "HH", "Hamburg", "DE", 9.9937, 53.5511, True), + ("FR", "ARA", "Lyon", "FR", 4.8357, 45.7640, False), + ("IT", "LOM", "Milan", "IT", 9.1900, 45.4642, False), + ("ES", "CAT", "Barcelona", "ES", 2.1734, 41.3851, True), + ("PL", "MZ", "Warsaw", "PL", 21.0122, 52.2297, False), + ("AT", "W", "Vienna", "AT", 16.3738, 48.2082, False), + ("CH", "BS", "Basel", "CH", 7.5886, 47.5596, False), + ("IE", "D", "Dublin", "IE", -6.2603, 53.3498, True), + ("GB", "MAN", "Manchester", "GB", -2.2426, 53.4808, False), + ("BE", "VAN", "Antwerp", "BE", 4.4025, 51.2194, True), + ("SE", "O", "Gothenburg", "SE", 11.9746, 57.7089, True), + ("US", "IL", "Chicago", "US", -87.6298, 41.8781, False), + ("US", "TX", "Houston", "US", -95.3698, 29.7604, True), + ("CA", "ON", "Toronto", "CA", -79.3832, 43.6532, False), + ("MX", "CMX", "Mexico City", "MX", -99.1332, 19.4326, False), + ("BR", "SP", "Sao Paulo", "BR", -46.6333, -23.5505, False), + ("IN", "MH", "Mumbai", "IN", 72.8777, 19.0760, True), + ("CN", "SH", "Shanghai", "CN", 121.4737, 31.2304, True), + ("JP", "13", "Tokyo", "JP", 139.6917, 35.6895, True), + ("AU", "NSW", "Sydney", "AU", 151.2093, -33.8688, True), + ("ZA", "GP", "Johannesburg", "ZA", 28.0473, -26.2041, False), + ("AE", "DU", "Dubai", "AE", 55.2708, 25.2048, True), + ("KR", "11", "Seoul", "KR", 126.9780, 37.5665, False), +) + +SYNTHESIZED_ID_START = 6000 +SYNTHESIZED_ID_STEP = 10 +MAX_SITES = len(BASE_PLANTS) + (10000 - SYNTHESIZED_ID_START) // SYNTHESIZED_ID_STEP + +# Plants for the active run; empty until configure_plants() runs. +PLANT_CONFIG: dict[str, dict] = {} + + +def build_plants(num_sites: int) -> dict[str, dict]: + """Return plant records: the base plants first, then synthesized plants.""" + if not 1 <= num_sites <= MAX_SITES: + raise ValueError(f"NUM_SITES must be between 1 and {MAX_SITES}; got {num_sites}") + plants = dict(list(BASE_PLANTS.items())[:num_sites]) + for index in range(num_sites - len(BASE_PLANTS)): + country, region, city, calendar, xpos, ypos, port = CITY_POOL[index % len(CITY_POOL)] + production = index % 5 == 4 + suffix = f" {index // len(CITY_POOL) + 1}" if index >= len(CITY_POOL) else "" + plants[f"{SYNTHESIZED_ID_START + SYNTHESIZED_ID_STEP * index}"] = { + "name": f"{'Production Site' if production else 'Regional DC'} {city}{suffix}", + "name2": "Regional Production Site" if production else "Distribution Center", + "country": country, + "region": region, + "city": city, + "street": f"{10 * (index + 1)} Industrial Way", + "postal": f"{10000 + index:05d}", + "plant_type": "PROD" if production else "DC", + "calendar": calendar, + "xpos": xpos, + "ypos": ypos, + "port": port, + "cost_factor": (0.9, 1.0, 1.1, 1.2)[index % 4], + } + return plants + + +def configure_plants() -> None: + """Rebuild PLANT_CONFIG in place for the active run's NUM_SITES.""" + PLANT_CONFIG.clear() + PLANT_CONFIG.update(build_plants(int(param("NUM_SITES")))) + hub = param("HUB_PLANT") + if hub not in PLANT_CONFIG: + raise ValueError(f"HUB_PLANT {hub!r} is not one of the {len(PLANT_CONFIG)} generated plants") + + +ROUTE_ALPHABET = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ" + + +def route_segment(werks: str) -> str: + """Two-character route code segment for a plant. + + Base plants use their leading digits. Synthesized plants use a letter + followed by a base-36 digit, so no two plants share a segment. + """ + if werks in BASE_PLANTS: + return werks[:2] + index = (int(werks) - SYNTHESIZED_ID_START) // SYNTHESIZED_ID_STEP + return chr(ord("A") + index // len(ROUTE_ALPHABET)) + ROUTE_ALPHABET[index % len(ROUTE_ALPHABET)] + + +def route_code(loc_from: str, loc_to: str) -> str: + return f"R{route_segment(loc_from)}{route_segment(loc_to)}" + + +def route_pairs(plants) -> list[tuple[str, str]]: + """Ordered plant pairs that need a delivery route. + + A single plant ships to itself, so it gets a self-loop route.""" + pairs = [(a, b) for a in plants for b in plants if a != b] + return pairs or [(plant, plant) for plant in plants] + + +def production_plants() -> list[str]: + return [werks for werks, plant in PLANT_CONFIG.items() if plant["plant_type"] == "PROD"] + + +def dc_plants() -> list[str]: + return [werks for werks, plant in PLANT_CONFIG.items() if plant["plant_type"] == "DC"] + + +def is_port(werks: str) -> bool: + return PLANT_CONFIG[werks]["port"] + + EU_COUNTRIES = frozenset( { "AT", @@ -126,13 +250,14 @@ "SK", } ) -PORT_PLANTS = frozenset({"3000", "4000", "5000"}) SCALED_KNOBS = { "NUM_ORDERS", "NUM_CUSTOMERS", "NUM_FINISHED_GOODS", "NUM_RAW_MATERIALS", + "NUM_VENDORS", + "NUM_SITES", } @@ -169,7 +294,7 @@ def transport_modes_for_lane( country_from in EU_COUNTRIES and country_to in EU_COUNTRIES ): modes.insert(0, "ROAD") - if loc_from in PORT_PLANTS and loc_to in PORT_PLANTS and distance_km > 200: + if is_port(loc_from) and is_port(loc_to) and distance_km > 200: modes.append("SEA") return tuple(modes) diff --git a/libs/sap-mock-data/src/sap_mock_data/generation/masterdata.py b/libs/sap-mock-data/src/sap_mock_data/generation/masterdata.py index a3a5607d..46352473 100644 --- a/libs/sap-mock-data/src/sap_mock_data/generation/masterdata.py +++ b/libs/sap-mock-data/src/sap_mock_data/generation/masterdata.py @@ -3,14 +3,18 @@ import numpy as np from faker import Faker import random -import uuid from math import radians, sin, cos, sqrt, asin from datetime import datetime, timedelta from .common import ( PLANT_CONFIG, + configure_plants, customs_days, + dc_plants, param, + production_plants, + route_code, + route_pairs, seed_all, transport_modes_for_lane, ) @@ -226,7 +230,6 @@ def generate_mard_data(): def generate_mbew_data(): data = [] - plant_cost_factors = {'1000': 1.0, '2000': 1.15, '3000': 0.85, '4000': 1.25} for matnr in PREDEFINED_MATERIALS: if matnr in FINISHED_GOODS: @@ -237,7 +240,7 @@ def generate_mbew_data(): base_price = round(random.uniform(5, 100), 2) for werks in PREDEFINED_PLANTS: - location_factor = plant_cost_factors.get(werks, 1.0) + location_factor = PLANT_CONFIG[werks]['cost_factor'] std_price = round(base_price * location_factor, 2) mov_avg_price = round(std_price * random.uniform(0.95, 1.05), 2) @@ -369,6 +372,14 @@ def generate_sapapo_loc_data(): }) return pd.DataFrame(data) + +def frame_with_schema(rows, columns, float_columns=()): + """Build a DataFrame that keeps its columns and float dtypes even with no rows.""" + frame = pd.DataFrame(rows, columns=columns) + if frame.empty: + frame = frame.astype({column: 'float64' for column in float_columns}) + return frame + def generate_sapapo_tr_data(): """ Generates /SAPAPO/TR - APO Transportation Lane Header. @@ -385,7 +396,7 @@ def generate_sapapo_tr_data(): from_info = PLANT_CONFIG[loc_from] to_info = PLANT_CONFIG[loc_to] - trlid = str(uuid.uuid4()).replace('-', '').upper()[:32] + trlid = f"{random.getrandbits(128):032X}" lane_name = f"{from_info['city']} -> {to_info['city']}" data.append({ @@ -397,7 +408,7 @@ def generate_sapapo_tr_data(): 'MODEL': 'SAPAPO_MODEL', }) - return pd.DataFrame(data) + return frame_with_schema(data, ['MANDT', 'TRLID', 'LOCFR', 'LOCTO', 'TRNAME', 'MODEL']) def generate_sapapo_trm_data(df_tr): """ @@ -454,7 +465,11 @@ def generate_sapapo_trm_data(df_tr): data.extend(mode_records) - return pd.DataFrame(data) + return frame_with_schema( + data, + ['MANDT', 'TRLID', 'TRMID', 'TRATIME', 'TRACOST', 'TRACOSTCUR', 'PRIFLAG'], + float_columns=('TRATIME', 'TRACOST'), + ) def generate_tvro_data(): """ @@ -467,52 +482,53 @@ def generate_tvro_data(): shipping_types = {'ROAD': '01', 'RAIL': '02', 'SEA': '03', 'AIR': '04'} forwarding_agents = ['DHL', 'KUEHNE', 'DBSCHENK', 'MAERSK', 'FEDEX'] - for loc_from in plants: - for loc_to in plants: - if loc_from == loc_to: - continue + for loc_from, loc_to in route_pairs(plants): - from_info = PLANT_CONFIG[loc_from] - to_info = PLANT_CONFIG[loc_to] + from_info = PLANT_CONFIG[loc_from] + to_info = PLANT_CONFIG[loc_to] - route = f"R{loc_from[:2]}{loc_to[:2]}" + route = route_code(loc_from, loc_to) - distance_km = haversine_km( - from_info['ypos'], from_info['xpos'], - to_info['ypos'], to_info['xpos'] - ) + distance_km = haversine_km( + from_info['ypos'], from_info['xpos'], + to_info['ypos'], to_info['xpos'] + ) - customs_delay = customs_days(from_info['country'], to_info['country']) + customs_delay = customs_days(from_info['country'], to_info['country']) - available_modes = transport_modes_for_lane(loc_from, loc_to, distance_km) - mode = min( - available_modes, - key=lambda candidate: TRANSPORT_MODES[candidate]['cost_per_km'], - ) - vsart = shipping_types[mode] - travel_hours = distance_km / TRANSPORT_MODES[mode]['speed_kmh'] - if mode == 'SEA': - agent = 'MAERSK' - elif mode == 'AIR': - agent = 'FEDEX' - else: - agent = random.choice(['DHL', 'KUEHNE', 'DBSCHENK']) + available_modes = transport_modes_for_lane(loc_from, loc_to, distance_km) + mode = min( + available_modes, + key=lambda candidate: TRANSPORT_MODES[candidate]['cost_per_km'], + ) + vsart = shipping_types[mode] + travel_hours = distance_km / TRANSPORT_MODES[mode]['speed_kmh'] + if mode == 'SEA': + agent = 'MAERSK' + elif mode == 'AIR': + agent = 'FEDEX' + else: + agent = random.choice(['DHL', 'KUEHNE', 'DBSCHENK']) - transit_days = round((travel_hours / 24) + customs_delay, 2) + transit_days = round((travel_hours / 24) + customs_delay, 2) - data.append({ - 'MANDT': '800', - 'ROUTE': route, - 'TRAZTD': transit_days, # Transit duration (calendar days) - 'TDVZTD': transit_days, # Transportation lead time (days) - 'FAHZTD': round(travel_hours, 2), # Travel duration (hours) - 'DISTZ': round(distance_km, 2), # Distance - 'MEDST': 'KM', # Distance unit - 'VSART': vsart, # Shipping type - 'TDLNR': agent, # Forwarding agent - }) + data.append({ + 'MANDT': '800', + 'ROUTE': route, + 'TRAZTD': transit_days, # Transit duration (calendar days) + 'TDVZTD': transit_days, # Transportation lead time (days) + 'FAHZTD': round(travel_hours, 2), # Travel duration (hours) + 'DISTZ': round(distance_km, 2), # Distance + 'MEDST': 'KM', # Distance unit + 'VSART': vsart, # Shipping type + 'TDLNR': agent, # Forwarding agent + }) - return pd.DataFrame(data) + return frame_with_schema( + data, + ['MANDT', 'ROUTE', 'TRAZTD', 'TDVZTD', 'FAHZTD', 'DISTZ', 'MEDST', 'VSART', 'TDLNR'], + float_columns=('TRAZTD', 'TDVZTD', 'FAHZTD', 'DISTZ'), + ) def generate_tvrot_data(df_tvro): """ @@ -521,13 +537,13 @@ def generate_tvrot_data(df_tvro): """ data = [] plants = list(PLANT_CONFIG) + lanes = {route_code(a, b): (a, b) for a, b in route_pairs(plants)} languages = ['E', 'D', 'F'] # English, German, French for _, route in df_tvro.iterrows(): - route_code = route['ROUTE'] + code = route['ROUTE'] - from_plant = route_code[1:3] + '00' - to_plant = route_code[3:5] + '00' + from_plant, to_plant = lanes[code] from_name = PLANT_CONFIG.get(from_plant, {}).get('city', from_plant) to_name = PLANT_CONFIG.get(to_plant, {}).get('city', to_plant) @@ -543,11 +559,30 @@ def generate_tvrot_data(df_tvro): data.append({ 'MANDT': '800', 'SPRAS': lang, - 'ROUTE': route_code, + 'ROUTE': code, 'BEZEI': desc, }) - return pd.DataFrame(data) + return frame_with_schema(data, ['MANDT', 'SPRAS', 'ROUTE', 'BEZEI']) + + +VENDOR_CATEGORY_SHARES = [ + ('API', 0.25), + ('EXCIPIENT', 0.25), + ('PACKAGING', 0.25), + ('CMO', 0.15), + ('LOGISTICS', 0.10), +] + + +def vendor_category(index, num_vendors): + """Assign the 1-based vendor index to a category per VENDOR_CATEGORY_SHARES.""" + cumulative = 0.0 + for category, share in VENDOR_CATEGORY_SHARES: + cumulative += share + if index <= round(num_vendors * cumulative): + return category + return VENDOR_CATEGORY_SHARES[-1][0] def generate_lfa1_data(): @@ -567,20 +602,10 @@ def generate_lfa1_data(): countries = ['DE', 'US', 'IN', 'CN', 'CH', 'IE', 'GB', 'FR'] - num_vendors = 20 + num_vendors = int(param("NUM_VENDORS")) for i in range(1, num_vendors + 1): lifnr = f"VEND-{i:04d}" - - if i <= 5: - v_type = 'API' - elif i <= 10: - v_type = 'EXCIPIENT' - elif i <= 15: - v_type = 'PACKAGING' - elif i <= 18: - v_type = 'CMO' - else: - v_type = 'LOGISTICS' + v_type = vendor_category(i, num_vendors) country = random.choice(countries) name_base = random.choice(vendor_categories[v_type]) @@ -615,12 +640,16 @@ def generate_eina_data(df_mara, df_lfa1): raw_materials = df_mara[df_mara['MTART'] == 'ROH']['MATNR'].tolist() vendors = df_lfa1['LIFNR'].tolist() + # LFA1 stores each vendor's category in NAME2; CMO and LOGISTICS vendors + # supply the OTHER materials. + vendor_types = dict(zip(df_lfa1['LIFNR'], df_lfa1['NAME2'])) vendor_pools = { - 'API': [v for v in vendors if int(v.split('-')[1]) <= 5], - 'EXCIPIENT': [v for v in vendors if 6 <= int(v.split('-')[1]) <= 10], - 'PACKAGING': [v for v in vendors if 11 <= int(v.split('-')[1]) <= 15], - 'OTHER': [v for v in vendors if int(v.split('-')[1]) >= 16], + 'API': [v for v in vendors if vendor_types[v] == 'API'], + 'EXCIPIENT': [v for v in vendors if vendor_types[v] == 'EXCIPIENT'], + 'PACKAGING': [v for v in vendors if vendor_types[v] == 'PACKAGING'], + 'OTHER': [v for v in vendors if vendor_types[v] in ('CMO', 'LOGISTICS')], } + vendor_pools = {category: pool or vendors for category, pool in vendor_pools.items()} materials_by_category = {category: [] for category in vendor_pools} for matnr in raw_materials: @@ -777,36 +806,41 @@ def generate_crhd_data(): data = [] work_centers = [ - ('DISP01', 'Dispensing & Weighing 1', '001', 16, 5, 95, ['1000', '5000']), - ('DISP02', 'Dispensing & Weighing 2', '001', 16, 5, 95, ['1000']), - ('GRAN01', 'Wet Granulation Line 1', '002', 16, 5, 85, ['1000', '5000']), - ('GRAN02', 'Dry Granulation Line 1', '002', 16, 5, 88, ['1000']), - ('BLND01', 'Blending Station 1', '002', 16, 5, 92, ['1000', '5000']), - ('BLND02', 'Blending Station 2', '002', 16, 5, 92, ['1000']), - ('COMP01', 'Tablet Press Line 1', '003', 24, 7, 80, ['1000', '5000']), - ('COMP02', 'Tablet Press Line 2', '003', 24, 7, 82, ['1000']), - ('COMP03', 'Tablet Press Line 3', '003', 16, 5, 78, ['1000']), - ('COAT01', 'Film Coating Line 1', '003', 16, 5, 85, ['1000', '5000']), - ('COAT02', 'Film Coating Line 2', '003', 16, 5, 85, ['1000']), - ('ENCAP01', 'Encapsulation Line 1', '003', 16, 5, 88, ['1000']), - ('FILL01', 'Liquid Filling Line 1', '004', 16, 5, 82, ['1000']), - ('FILL02', 'Liquid Filling Line 2', '004', 16, 5, 82, ['1000', '5000']), - ('STER01', 'Sterile Filling Line 1', '005', 16, 5, 75, ['1000']), - ('STER02', 'Sterile Filling Line 2', '005', 16, 5, 75, ['1000']), - ('PACK01', 'Primary Packaging Line 1', '006', 24, 7, 90, ['1000', '5000']), - ('PACK02', 'Primary Packaging Line 2', '006', 24, 7, 90, ['1000']), - ('PACK03', 'Secondary Packaging Line 1', '006', 16, 5, 92, ['1000', '5000']), - ('PACK04', 'Secondary Packaging Line 2', '006', 16, 5, 92, ['1000']), - ('QCLAB01', 'Quality Control Lab 1', '007', 16, 5, 85, ['1000', '5000']), - ('QCLAB02', 'Quality Control Lab 2', '007', 16, 5, 85, ['1000']), - ('INSP01', 'Incoming Inspection', '008', 8, 5, 95, ['2000', '3000', '4000']), - ('REPK01', 'Repackaging Station', '006', 8, 5, 90, ['2000', '3000', '4000']), + ('DISP01', 'Dispensing & Weighing 1', '001', 16, 5, 95, 'PROD'), + ('DISP02', 'Dispensing & Weighing 2', '001', 16, 5, 95, 'HUB'), + ('GRAN01', 'Wet Granulation Line 1', '002', 16, 5, 85, 'PROD'), + ('GRAN02', 'Dry Granulation Line 1', '002', 16, 5, 88, 'HUB'), + ('BLND01', 'Blending Station 1', '002', 16, 5, 92, 'PROD'), + ('BLND02', 'Blending Station 2', '002', 16, 5, 92, 'HUB'), + ('COMP01', 'Tablet Press Line 1', '003', 24, 7, 80, 'PROD'), + ('COMP02', 'Tablet Press Line 2', '003', 24, 7, 82, 'HUB'), + ('COMP03', 'Tablet Press Line 3', '003', 16, 5, 78, 'HUB'), + ('COAT01', 'Film Coating Line 1', '003', 16, 5, 85, 'PROD'), + ('COAT02', 'Film Coating Line 2', '003', 16, 5, 85, 'HUB'), + ('ENCAP01', 'Encapsulation Line 1', '003', 16, 5, 88, 'HUB'), + ('FILL01', 'Liquid Filling Line 1', '004', 16, 5, 82, 'HUB'), + ('FILL02', 'Liquid Filling Line 2', '004', 16, 5, 82, 'PROD'), + ('STER01', 'Sterile Filling Line 1', '005', 16, 5, 75, 'HUB'), + ('STER02', 'Sterile Filling Line 2', '005', 16, 5, 75, 'HUB'), + ('PACK01', 'Primary Packaging Line 1', '006', 24, 7, 90, 'PROD'), + ('PACK02', 'Primary Packaging Line 2', '006', 24, 7, 90, 'HUB'), + ('PACK03', 'Secondary Packaging Line 1', '006', 16, 5, 92, 'PROD'), + ('PACK04', 'Secondary Packaging Line 2', '006', 16, 5, 92, 'HUB'), + ('QCLAB01', 'Quality Control Lab 1', '007', 16, 5, 85, 'PROD'), + ('QCLAB02', 'Quality Control Lab 2', '007', 16, 5, 85, 'HUB'), + ('INSP01', 'Incoming Inspection', '008', 8, 5, 95, 'DC'), + ('REPK01', 'Repackaging Station', '006', 8, 5, 90, 'DC'), ] objid_counter = 1000 - for arbpl, ktext, capacity_category, hours_day, days_week, efficiency, plants in work_centers: - for werks in plants: + plants_for_role = { + 'HUB': [param("HUB_PLANT")], + 'PROD': production_plants(), + 'DC': dc_plants(), + } + for arbpl, ktext, capacity_category, hours_day, days_week, efficiency, role in work_centers: + for werks in plants_for_role[role]: objid_counter += 1 available_hours = hours_day * days_week @@ -879,7 +913,7 @@ def generate_plko_data(df_mara, df_crhd): producible = df_mara[df_mara['MTART'].isin(['FERT', 'HALB'])]['MATNR'].tolist() - prod_plants = ['1000', '5000'] + prod_plants = production_plants() plnnr_counter = 1000000 @@ -1067,6 +1101,7 @@ def generate(wh): print(f"Seed: {RANDOM_SEED}") print(f"Universe: {NUM_CUSTOMERS} customers, {NUM_FINISHED_GOODS} finished goods, {NUM_RAW_MATERIALS} raw materials") + configure_plants() PREDEFINED_PLANTS = list(PLANT_CONFIG) PREDEFINED_STORAGE_LOCATIONS = ['0001', 'FG01', 'RM01', 'WH01', 'QA01', 'ALT1'] PREDEFINED_CUSTOMERS = [f'CUST{i:05d}' for i in range(1, NUM_CUSTOMERS + 1)] diff --git a/libs/sap-mock-data/src/sap_mock_data/generation/transactions.py b/libs/sap-mock-data/src/sap_mock_data/generation/transactions.py index d3edf420..b62d7458 100644 --- a/libs/sap-mock-data/src/sap_mock_data/generation/transactions.py +++ b/libs/sap-mock-data/src/sap_mock_data/generation/transactions.py @@ -10,6 +10,8 @@ from .common import ( PLANT_CONFIG, + configure_plants, + route_code, customs_days, param, seed_all, @@ -120,6 +122,7 @@ def generate_sales_orders(finished_goods, all_customers): print(f"Generating {NUMBER_OF_ORDERS} Sales Orders...") vbak, vbap, vbep = [], [], [] base_date = datetime.now() + india_customer_exists = CUST_INDIA in all_customers for i in range(NUMBER_OF_ORDERS): vbeln = f'{1000000000 + i:010d}' @@ -128,7 +131,7 @@ def generate_sales_orders(finished_goods, all_customers): order_date = order_date_dt.strftime('%Y%m%d') req_date = (order_date_dt + timedelta(days=7)).strftime('%Y%m%d') - if i % 100 == 0: kunnr = CUST_INDIA + if i % 100 == 0 and india_customer_exists: kunnr = CUST_INDIA else: kunnr = random.choice(all_customers) order_total = 0.0 @@ -261,8 +264,7 @@ def haversine_km(lat1, lon1, lat2, lon2): return R * 2 * asin(sqrt(a)) def get_route_code(from_plant, to_plant): - """Generate route code in format R{FROM}{TO}.""" - return f"R{from_plant[:2]}{to_plant[:2]}" + return route_code(from_plant, to_plant) def get_best_transport_mode(from_plant, to_plant, distance_km): """Determine the best transport mode based on cost.""" @@ -298,7 +300,7 @@ def generate_shipments(df_likp, df_lips, df_vbap): suffixes=('', '_VBAP') ) if 'WERKS_VBAP' in df_delivery.columns: - df_delivery['WERKS'] = df_delivery['WERKS_VBAP'].fillna(df_delivery.get('WERKS', '1000')) + df_delivery['WERKS'] = df_delivery['WERKS_VBAP'].fillna(df_delivery.get('WERKS', HUB_PLANT)) delivery_groups = df_delivery.groupby('VBELN') @@ -310,17 +312,17 @@ def generate_shipments(df_likp, df_lips, df_vbap): shipment_counter += 1 first_item = del_items.iloc[0] - source_plant = first_item.get('WERKS', '1000') + source_plant = first_item.get('WERKS', HUB_PLANT) if pd.isna(source_plant) or source_plant == '': - source_plant = '1000' # Hub plant + source_plant = HUB_PLANT delivery_date_str = first_item.get('LFDAT', datetime.now().strftime('%Y%m%d')) dest_plants = [p for p in PLANT_CONFIG if p != source_plant] - dest_plant = random.choice(dest_plants) if dest_plants else '2000' + dest_plant = random.choice(dest_plants) if dest_plants else source_plant - from_info = PLANT_CONFIG.get(source_plant, PLANT_CONFIG['1000']) - to_info = PLANT_CONFIG.get(dest_plant, PLANT_CONFIG['2000']) + from_info = PLANT_CONFIG.get(source_plant, PLANT_CONFIG[HUB_PLANT]) + to_info = PLANT_CONFIG.get(dest_plant, PLANT_CONFIG[HUB_PLANT]) distance_km = haversine_km( from_info['ypos'], from_info['xpos'], @@ -916,7 +918,7 @@ def generate_purchase_orders(df_eina, df_eine, df_matdoc, num_months=12): 'EBELP': '00010', # Item number 'MATNR': matnr, 'TXZ01': f"Raw Material {matnr}", - 'WERKS': '1000', # Plant + 'WERKS': HUB_PLANT, # Plant 'LGORT': 'RM01', # Storage location (raw materials) 'MENGE': order_qty, # Order quantity 'MEINS': 'PC', # Unit @@ -1095,6 +1097,8 @@ def generate(wh): global RANDOM_SEED, NUMBER_OF_ORDERS, HUB_PLANT, DELIVERY_FILL_RATE, SAFETY_STOCK_WEEKS global SUPPLIER_RELIABILITY_RATE, UNRELIABLE_MATERIALS_STR global PLANTS, PRICE_LOOKUP, PRICE_FALLBACK, BATCH_INVENTORY, AVAILABLE_STOCK, network_schema + global BATCH_COUNTER + BATCH_COUNTER = 1000000 RANDOM_SEED = int(param("RANDOM_SEED")) NUMBER_OF_ORDERS = int(param("NUM_ORDERS")) @@ -1117,6 +1121,7 @@ def generate(wh): FINISHED_PRODUCTS = [row['MATNR'] for row in df_mara[df_mara["MTART"] == "FERT"][['MATNR']].drop_duplicates().to_dict("records")] ALL_CUSTOMERS = [row['KUNNR'] for row in wh.read("kna1")[['KUNNR']].drop_duplicates().to_dict("records")] + configure_plants() PLANTS = list(PLANT_CONFIG) SALES_MARKUP = 0.35 # 35% markup on cost for selling price diff --git a/libs/sap-mock-data/src/sap_mock_data/scenarios/definitions.py b/libs/sap-mock-data/src/sap_mock_data/scenarios/definitions.py index 697cc372..3b29047e 100644 --- a/libs/sap-mock-data/src/sap_mock_data/scenarios/definitions.py +++ b/libs/sap-mock-data/src/sap_mock_data/scenarios/definitions.py @@ -9,7 +9,7 @@ "SCENARIO_TYPE": "INVENTORY", "DESCRIPTION": "A batch deviation impacts inventory at the Manufacturing site (1000) - removes a particular batch of one product permanently", "IMPACTED_NODE": "1000", - "IMPACTED_PRODUCTS": "MAT-A0008", + "IMPACTED_PRODUCTS": "MAT-A0001", "IMPACTED_BATCH": "BATCH-2025-001", "NODE_OFFLINE": False, "NODE_CAPACITY_PCT": 100, @@ -36,7 +36,7 @@ "NEW_FACILITY": "", "NETWORK_VOLATILITY": "LOW", "AI_DECISION_OPTIONS": "Accept loss|Expedite production|Source from alternate supplier|Reallocate from other products", - "DATA_EVIDENCE": "inventory_impacted.csv: Filter Product_Code=MAT-A0008, Location=1000 to see affected stock | supply_network.csv: Location_ID=1000 shows Node_Status=IMPACTED | matdoc table: MBLNR starting with SCN001 contains the 344 adjustment movement" + "DATA_EVIDENCE": "inventory_impacted.csv: Filter Product_Code=MAT-A0001, Location=1000 to see affected stock | supply_network.csv: Location_ID=1000 shows Node_Status=IMPACTED | matdoc table: MBLNR starting with SCN001 contains the 344 adjustment movement" }, { "SCENARIO_ID": "SCN002", @@ -78,7 +78,7 @@ "SCENARIO_NAME": "Fire Damage", "SCENARIO_TYPE": "INVENTORY", "DESCRIPTION": "A fire removes ALL inventory (all products, all batches) at one location and takes the node offline temporarily", - "IMPACTED_NODE": "2000", + "IMPACTED_NODE": "1000", "IMPACTED_PRODUCTS": "ALL", "IMPACTED_BATCH": "ALL", "NODE_OFFLINE": True, @@ -106,7 +106,7 @@ "NEW_FACILITY": "", "NETWORK_VOLATILITY": "LOW", "AI_DECISION_OPTIONS": "Reroute through alternate node|Direct ship from hub|Delay customer orders|Split shipments", - "DATA_EVIDENCE": "supply_network.csv: Location_ID=2000 shows Node_Status=OFFLINE, Node_Capacity_Pct=0 | transportation_lanes.csv: Lanes to/from 2000 show Lane_Status=BLOCKED | inventory_impacted.csv: ALL products at Location=2000 destroyed | matdoc table: SCN003 records with BWART=551" + "DATA_EVIDENCE": "supply_network.csv: Location_ID=1000 shows Node_Status=OFFLINE, Node_Capacity_Pct=0 | transportation_lanes.csv: Lanes to/from 1000 show Lane_Status=BLOCKED | inventory_impacted.csv: ALL products at Location=1000 destroyed | matdoc table: SCN003 records with BWART=551" }, { "SCENARIO_ID": "SCN004", @@ -359,7 +359,7 @@ "SCENARIO_TYPE": "PRODUCTION", "DESCRIPTION": "Permanent increase in demand - planned issues increase and receipt/production plans need to be increased accordingly", "IMPACTED_NODE": "1000", - "IMPACTED_PRODUCTS": "MAT-A0005", + "IMPACTED_PRODUCTS": "MAT-A0001", "IMPACTED_BATCH": "N/A", "NODE_OFFLINE": False, "NODE_CAPACITY_PCT": 100, @@ -386,7 +386,7 @@ "NEW_FACILITY": "", "NETWORK_VOLATILITY": "MEDIUM", "AI_DECISION_OPTIONS": "Increase production schedule|Add overtime shifts|Qualify additional capacity|Adjust safety stock targets", - "DATA_EVIDENCE": "vbak table: Sales orders with BSTNK like 'SCN011-%' (ERNAM=SCENARIO) | vbap table: Order items for MATNR=MAT-A0005, WERKS=1000 showing +25% demand | vbep table: Schedule lines with delivery dates | scenario_metadata table: SCN011 scenario_type=PRODUCTION, demand_type=PERMANENT" + "DATA_EVIDENCE": "vbak table: Sales orders with BSTNK like 'SCN011-%' (ERNAM=SCENARIO) | vbap table: Order items for MATNR=MAT-A0001, WERKS=1000 showing +25% demand | vbep table: Schedule lines with delivery dates | scenario_metadata table: SCN011 scenario_type=PRODUCTION, demand_type=PERMANENT" }, { "SCENARIO_ID": "SCN012", @@ -416,7 +416,7 @@ "DEMAND_CHANGE_TYPE": "NEW_PRODUCT", "NEW_PRODUCT_ID": "MAT-NEW01", "CAPACITY_CONSTRAINT": "Shared line with existing products", - "COMPETING_PRODUCTS": "MAT-A0005,MAT-A0008", + "COMPETING_PRODUCTS": "MAT-A0001,MAT-A0020", "REGULATORY_EVENT": "", "NEW_FACILITY": "", "NETWORK_VOLATILITY": "MEDIUM", @@ -486,7 +486,7 @@ "DEMAND_CHANGE_TYPE": "NONE", "NEW_PRODUCT_ID": "", "CAPACITY_CONSTRAINT": "Sterile facility at 95% utilization", - "COMPETING_PRODUCTS": "MAT-A0018,MAT-A0022", + "COMPETING_PRODUCTS": "MAT-A0001,B1_TAB1", "REGULATORY_EVENT": "", "NEW_FACILITY": "", "NETWORK_VOLATILITY": "HIGH", @@ -521,7 +521,7 @@ "DEMAND_CHANGE_TYPE": "NONE", "NEW_PRODUCT_ID": "", "CAPACITY_CONSTRAINT": "Filling line down - backup available at reduced speed", - "COMPETING_PRODUCTS": "MAT-A0005,MAT-A0008,MAT-A0010,MAT-A0012", + "COMPETING_PRODUCTS": "MAT-A0001,MAT-A0020,B1_TAB1", "REGULATORY_EVENT": "", "NEW_FACILITY": "", "NETWORK_VOLATILITY": "HIGH", @@ -534,7 +534,7 @@ "SCENARIO_TYPE": "PRODUCTION", "DESCRIPTION": "Several commercial products with different service levels compete for time on the same production line", "IMPACTED_NODE": "1000", - "IMPACTED_PRODUCTS": "MAT-A0005,MAT-A0008,MAT-A0010", + "IMPACTED_PRODUCTS": "MAT-A0001,MAT-A0020", "IMPACTED_BATCH": "N/A", "NODE_OFFLINE": False, "NODE_CAPACITY_PCT": 100, @@ -556,12 +556,12 @@ "DEMAND_CHANGE_TYPE": "NONE", "NEW_PRODUCT_ID": "", "CAPACITY_CONSTRAINT": "Single line shared by 3 products", - "COMPETING_PRODUCTS": "MAT-A0005,MAT-A0008,MAT-A0010", + "COMPETING_PRODUCTS": "MAT-A0001,MAT-A0020", "REGULATORY_EVENT": "", "NEW_FACILITY": "", "NETWORK_VOLATILITY": "MEDIUM", "AI_DECISION_OPTIONS": "Optimize by service level|Optimize by margin|Fixed rotation schedule|Dynamic scheduling based on inventory", - "DATA_EVIDENCE": "CONFIG ONLY - No transactional injection | COMPETING_PRODUCTS=MAT-A0005,MAT-A0008,MAT-A0010 share line | production_orders.csv: Competing orders | material_master.csv: Product priorities" + "DATA_EVIDENCE": "CONFIG ONLY - No transactional injection | COMPETING_PRODUCTS=MAT-A0001,MAT-A0020 share line | production_orders.csv: Competing orders | material_master.csv: Product priorities" }, { "SCENARIO_ID": "SCN017", @@ -709,7 +709,7 @@ "SCENARIO_TYPE": "SUPPLIER", "DESCRIPTION": "Supplier of one core excipient material has drifted from agreed SLA - delivery quantities and on-time performance declining", "IMPACTED_NODE": "1000", - "IMPACTED_PRODUCTS": "MAT-R0005", + "IMPACTED_PRODUCTS": "ALL", "IMPACTED_BATCH": "N/A", "NODE_OFFLINE": False, "NODE_CAPACITY_PCT": 100, @@ -721,7 +721,7 @@ "IMPACT_DURATION_DAYS": 0, "TLANES_AFFECTED": False, "ALT_LINE_SAME_LOCATION": False, - "IMPACTED_SUPPLIER": "VEND-0005", + "IMPACTED_SUPPLIER": "VEND-0001", "SUPPLIER_ISSUE": "SLA_DRIFT", "METRIC_TREND": "DECLINE", "CURRENT_RELIABILITY": "0.72", @@ -736,7 +736,7 @@ "NEW_FACILITY": "", "NETWORK_VOLATILITY": "LOW", "AI_DECISION_OPTIONS": "Engage supplier for improvement plan|Qualify alternate supplier|Increase safety stock|Adjust production schedule", - "DATA_EVIDENCE": "supplier_master.csv: Vendor_ID=VEND-0005 supplies MAT-R0005 | ekbe table: Filter LIFNR=VEND-0005 to see declining OTIF (OTIF_ONTIME, OTIF_INFULL columns) | CURRENT_RELIABILITY=0.72 (down from baseline 0.95)" + "DATA_EVIDENCE": "supplier_master.csv: Vendor_ID=VEND-0001 supplies API1 | ekbe table: Filter LIFNR=VEND-0001 to see declining OTIF (OTIF_ONTIME, OTIF_INFULL columns) | CURRENT_RELIABILITY=0.72 (down from baseline 0.95)" }, { "SCENARIO_ID": "SCN022", @@ -779,7 +779,7 @@ "SCENARIO_TYPE": "SUPPLIER", "DESCRIPTION": "API supplier receives FDA 483 for a different client - no immediate change but all supply from this supplier requires review", "IMPACTED_NODE": "1000", - "IMPACTED_PRODUCTS": "MAT-R0010", + "IMPACTED_PRODUCTS": "API1", "IMPACTED_BATCH": "ALL", "NODE_OFFLINE": False, "NODE_CAPACITY_PCT": 100, @@ -791,7 +791,7 @@ "IMPACT_DURATION_DAYS": 90, "TLANES_AFFECTED": False, "ALT_LINE_SAME_LOCATION": False, - "IMPACTED_SUPPLIER": "VEND-0008", + "IMPACTED_SUPPLIER": "VEND-0001", "SUPPLIER_ISSUE": "FDA_483", "METRIC_TREND": "STABLE", "CURRENT_RELIABILITY": "0.95", @@ -806,7 +806,7 @@ "NEW_FACILITY": "", "NETWORK_VOLATILITY": "LOW", "AI_DECISION_OPTIONS": "Immediate quality review of existing supply|Request supplier documentation and CAPA|Prepare backup supplier qualification|No action pending review", - "DATA_EVIDENCE": "supplier_master.csv: Vendor_ID=VEND-0008 supplies MAT-R0010 | SUPPLIER_ISSUE=FDA_483, REVIEW_REQUIRED=True | ekbe table: No OTIF impact yet (CURRENT_RELIABILITY=0.95) but all batches flagged for review" + "DATA_EVIDENCE": "supplier_master.csv: Vendor_ID=VEND-0001 supplies API1 | SUPPLIER_ISSUE=FDA_483, REVIEW_REQUIRED=True | ekbe table: No OTIF impact yet (CURRENT_RELIABILITY=0.95) but all batches flagged for review" }, { "SCENARIO_ID": "SCN024", @@ -884,7 +884,7 @@ "SCENARIO_TYPE": "SUPPLIER", "DESCRIPTION": "API supplier with previous CAPA failures shows improvement - situation recovering and reliability increasing", "IMPACTED_NODE": "1000", - "IMPACTED_PRODUCTS": "MAT-R0010", + "IMPACTED_PRODUCTS": "ALL", "IMPACTED_BATCH": "N/A", "NODE_OFFLINE": False, "NODE_CAPACITY_PCT": 100, @@ -896,7 +896,7 @@ "IMPACT_DURATION_DAYS": 30, "TLANES_AFFECTED": False, "ALT_LINE_SAME_LOCATION": False, - "IMPACTED_SUPPLIER": "VEND-0008", + "IMPACTED_SUPPLIER": "VEND-0002", "SUPPLIER_ISSUE": "CAPA_IMPROVEMENT", "METRIC_TREND": "IMPROVE", "CURRENT_RELIABILITY": "0.85", @@ -911,7 +911,7 @@ "NEW_FACILITY": "", "NETWORK_VOLATILITY": "LOW", "AI_DECISION_OPTIONS": "Resume normal ordering volumes|Continue enhanced monitoring|Reduce safety stock buffer|Update supplier qualification status", - "DATA_EVIDENCE": "supplier_master.csv: Vendor_ID=VEND-0008 supplies MAT-R0010 | ekbe table: Filter LIFNR=VEND-0008 to see improving OTIF trend | CURRENT_RELIABILITY=0.85 (recovering), SUPPLIER_ISSUE=CAPA_IMPROVEMENT, METRIC_TREND=IMPROVE" + "DATA_EVIDENCE": "supplier_master.csv: Vendor_ID=VEND-0002 supplies its materials | ekbe table: Filter LIFNR=VEND-0002 to see improving OTIF trend | CURRENT_RELIABILITY=0.85 (recovering), SUPPLIER_ISSUE=CAPA_IMPROVEMENT, METRIC_TREND=IMPROVE" } ] diff --git a/libs/sap-mock-data/src/sap_mock_data/scenarios/injection.py b/libs/sap-mock-data/src/sap_mock_data/scenarios/injection.py index bc22d37b..1d71ea87 100644 --- a/libs/sap-mock-data/src/sap_mock_data/scenarios/injection.py +++ b/libs/sap-mock-data/src/sap_mock_data/scenarios/injection.py @@ -4,7 +4,7 @@ import random from datetime import datetime, timedelta -from ..generation.common import param, widget, seed_all +from ..generation.common import PLANT_CONFIG, configure_plants, param, seed_all, widget INVENTORY_SCENARIO_DEFINITIONS = { @@ -2192,7 +2192,7 @@ def inject_high_volatility(scenario_id, config, df_vbak, df_vbap, df_vbep, df_kn customers = df_kna1['KUNNR'].tolist() finished_goods = df_mara[df_mara['MTART'] == 'FERT']['MATNR'].tolist() - plants = ['1000', '2000', '3000', '4000'] + plants = list(PLANT_CONFIG) start_date = datetime.now() new_vbak = [] @@ -2424,6 +2424,7 @@ def generate(wh): SCN019_CONFIG = widget("SCN019_CONFIG", "") SCN020_CONFIG = widget("SCN020_CONFIG", "") + configure_plants() seed_all(RANDOM_SEED) print(f"Seed: {RANDOM_SEED}") diff --git a/libs/sap-mock-data/src/sap_mock_data/validation/integrity.py b/libs/sap-mock-data/src/sap_mock_data/validation/integrity.py index 92858fb4..efa9de63 100644 --- a/libs/sap-mock-data/src/sap_mock_data/validation/integrity.py +++ b/libs/sap-mock-data/src/sap_mock_data/validation/integrity.py @@ -13,6 +13,10 @@ ("vbap", "VBELN", "vbak", "VBELN"), ("vbep", "VBELN", "vbak", "VBELN"), ("ekpo", "EBELN", "ekko", "EBELN"), + ("vbak", "KUNNR", "kna1", "KUNNR"), + ("vbap", "MATNR", "mara", "MATNR"), + ("eina", "LIFNR", "lfa1", "LIFNR"), + ("vttk", "ROUTE", "tvro", "ROUTE"), ) diff --git a/libs/sap-mock-data/tests/test_cli.py b/libs/sap-mock-data/tests/test_cli.py new file mode 100644 index 00000000..7217b7c0 --- /dev/null +++ b/libs/sap-mock-data/tests/test_cli.py @@ -0,0 +1,42 @@ +import unittest +from contextlib import redirect_stderr +from io import StringIO + +from sap_mock_data.cli.main import _parser, main + + +class ScaleFactorArgumentTests(unittest.TestCase): + def parse(self, value): + return _parser().parse_args(["generate", "out", "--scale-factor", value]).scale_factor + + def test_numbers_and_identifiers_parse(self) -> None: + self.assertEqual(self.parse("0.5"), 0.5) + self.assertEqual(self.parse("3"), 3.0) + self.assertEqual(self.parse("s"), "S") + self.assertEqual(self.parse("XL"), "XL") + + def assert_usage_error(self, argv, fragment) -> None: + stderr = StringIO() + with redirect_stderr(stderr), self.assertRaises(SystemExit) as exit_info: + main(argv) + self.assertEqual(exit_info.exception.code, 2) + self.assertIn(fragment, stderr.getvalue()) + + def test_unknown_identifier_is_a_usage_error(self) -> None: + self.assert_usage_error( + ["generate", "out", "--scale-factor", "banana"], "one of S, M, L, XL" + ) + + def test_non_finite_and_non_positive_numbers_are_usage_errors(self) -> None: + for value in ("nan", "inf", "0", "-2"): + self.assert_usage_error( + ["generate", "out", "--scale-factor", value], "positive finite number" + ) + + def test_config_rejections_are_usage_errors(self) -> None: + self.assert_usage_error(["generate", "out", "--vendors", "0"], "num_vendors") + self.assert_usage_error(["generate", "out", "--currency", "EURO"], "currency") + + +if __name__ == "__main__": + unittest.main() diff --git a/libs/sap-mock-data/tests/test_generation_quality.py b/libs/sap-mock-data/tests/test_generation_quality.py index f73e67b4..b0202203 100644 --- a/libs/sap-mock-data/tests/test_generation_quality.py +++ b/libs/sap-mock-data/tests/test_generation_quality.py @@ -6,7 +6,6 @@ from sap_mock_data.generation.common import ( EU_COUNTRIES, PLANT_CONFIG, - PORT_PLANTS, customs_days, ) from sap_mock_data.storage import MemoryTableStore @@ -18,8 +17,9 @@ def setUpClass(cls) -> None: cls.store = MemoryTableStore() with redirect_stdout(StringIO()): generate_dataset( - GenerationConfig(scale_factor=0.1, scenarios="demo"), cls.store + GenerationConfig(scale_factor=0.1, num_sites=5, scenarios="demo"), cls.store ) + cls.plants = dict(PLANT_CONFIG) def test_generated_columns_have_no_corruption_marker(self) -> None: malformed = [ @@ -88,15 +88,15 @@ def test_transport_lanes_use_valid_modes_and_customs_delays(self) -> None: self.assertEqual(set(preferred["TRMID"]), {"ROAD", "SEA", "AIR"}) for lane in preferred.itertuples(): - country_from = PLANT_CONFIG[lane.LOCFR]["country"] - country_to = PLANT_CONFIG[lane.LOCTO]["country"] + country_from = self.plants[lane.LOCFR]["country"] + country_to = self.plants[lane.LOCTO]["country"] if lane.TRMID == "ROAD": self.assertTrue( country_from == country_to or {country_from, country_to}.issubset(EU_COUNTRIES) ) if lane.TRMID == "SEA": - self.assertTrue({lane.LOCFR, lane.LOCTO}.issubset(PORT_PLANTS)) + self.assertTrue(self.plants[lane.LOCFR]["port"] and self.plants[lane.LOCTO]["port"]) self.assertEqual(customs_days("DE", "DE"), 0) self.assertEqual(customs_days("DE", "IE"), 0) diff --git a/libs/sap-mock-data/tests/test_scale_sizes.py b/libs/sap-mock-data/tests/test_scale_sizes.py new file mode 100644 index 00000000..57dfa5fa --- /dev/null +++ b/libs/sap-mock-data/tests/test_scale_sizes.py @@ -0,0 +1,252 @@ +import os +import unittest +from contextlib import redirect_stdout +from io import StringIO + +from sap_mock_data import GenerationConfig, generate_dataset +from sap_mock_data.config import DEMO_SCENARIO_CONFIGS, SIZE_KNOB_RANGES +from sap_mock_data.context import GenerationContext +from sap_mock_data.generation.common import BASE_PLANTS, build_plants, param +from sap_mock_data.storage import MemoryTableStore + +# Target ranges per size. BOM depth is fixed at 1 for every size. +SIZE_RANGES = { + "S": {"products": (1, 3), "suppliers": (3, 5), "sites": (1, 2)}, + "M": {"products": (10, 50), "suppliers": (20, 30), "sites": (4, 5)}, + "L": {"products": (100, 200), "suppliers": (50, 100), "sites": (20, 40)}, + "XL": {"products": (400, 800), "suppliers": (150, 300), "sites": (100, 120)}, +} + + +def measure(store): + mara = store.read("mara") + stpo = store.read("stpo") + parents = set(store.read("mast")["MATNR"]) + children = set(stpo["IDNRK"]) + return { + "products": mara[mara["MTART"] == "FERT"]["MATNR"].nunique(), + "suppliers": store.read("lfa1")["LIFNR"].nunique(), + "sites": store.read("t001w")["WERKS"].nunique(), + "bom_depth": 2 if parents & children else 1, + } + + +def generate_at(size): + store = MemoryTableStore() + with redirect_stdout(StringIO()): + generate_dataset(GenerationConfig(scale_factor=size, scenarios=()), store) + return measure(store) + + +class ScaleFactorValidationTests(unittest.TestCase): + def test_identifiers_normalize_case_and_whitespace(self) -> None: + for raw, letter in (("S", "S"), ("m", "M"), (" l ", "L"), ("xl", "XL")): + self.assertEqual(GenerationConfig(scale_factor=raw).scale_factor, letter) + + def test_unknown_letters_are_rejected(self) -> None: + with self.assertRaisesRegex(ValueError, "positive number or one of"): + GenerationConfig(scale_factor="banana") + + def test_non_positive_and_non_finite_numbers_are_rejected(self) -> None: + for value in (0, -1, float("nan"), float("inf")): + with self.assertRaises(ValueError): + GenerationConfig(scale_factor=value) + + def test_identifiers_sample_knobs_within_their_ranges(self) -> None: + for size, ranges in SIZE_KNOB_RANGES.items(): + for seed in (1, 42, 999): + parameters = GenerationConfig(scale_factor=size, random_seed=seed).parameters() + for knob, (low, high) in ranges.items(): + self.assertTrue( + low <= int(parameters[knob]) <= high, f"{size}.{knob} seed={seed}" + ) + + def test_sampling_is_deterministic_per_seed(self) -> None: + first = GenerationConfig(scale_factor="M", random_seed=7).parameters() + second = GenerationConfig(scale_factor="M", random_seed=7).parameters() + self.assertEqual(first, second) + + def test_sampling_varies_across_seeds(self) -> None: + knobs = list(SIZE_KNOB_RANGES["M"]) + samples = { + tuple(GenerationConfig(scale_factor="M", random_seed=seed).parameters()[knob] + for knob in knobs) + for seed in range(10) + } + self.assertGreater(len(samples), 1) + + def test_numbers_multiply_without_size_adjustment(self) -> None: + with GenerationContext(GenerationConfig(scale_factor=0.1)).activate(): + self.assertEqual(param("NUM_VENDORS"), "2") + self.assertEqual(param("NUM_CUSTOMERS"), "3") + with GenerationContext(GenerationConfig(scale_factor=0.001)).activate(): + self.assertEqual(param("NUM_VENDORS"), "1") + + def test_explicit_knobs_override_the_size_values(self) -> None: + config = GenerationConfig(scale_factor="S", num_customers=77, num_vendors=9, num_sites=3) + self.assertEqual(config.parameters()["NUM_CUSTOMERS"], "77") + self.assertEqual(config.parameters()["NUM_VENDORS"], "9") + self.assertEqual(config.parameters()["NUM_SITES"], "3") + + def test_tiny_numeric_factor_generates_a_minimal_dataset(self) -> None: + store = MemoryTableStore() + with redirect_stdout(StringIO()): + generate_dataset(GenerationConfig(scale_factor=0.001, scenarios=()), store) + self.assertEqual(store.read("lfa1")["LIFNR"].nunique(), 1) + + +class PlantModelTests(unittest.TestCase): + def test_base_plants_come_first_and_unchanged(self) -> None: + self.assertEqual(build_plants(5), BASE_PLANTS) + self.assertEqual(list(build_plants(2)), ["1000", "2000"]) + + def test_synthesized_plants_are_well_formed(self) -> None: + plants = build_plants(60) + self.assertEqual(len(plants), 60) + fields = set(BASE_PLANTS["1000"]) + for werks, plant in plants.items(): + self.assertEqual(len(werks), 4, werks) + self.assertEqual(set(plant), fields, werks) + production = [w for w, plant in plants.items() if plant["plant_type"] == "PROD"] + self.assertIn("1000", production) + self.assertGreater(len(production), 2) + + TABLE_KEYS = { + "t001w": ["WERKS"], "sapapo_loc": ["LOCNO"], "sapapo_tr": ["TRLID"], + "sapapo_trm": ["TRLID", "TRMID"], "tvro": ["ROUTE"], "tvrot": ["ROUTE", "SPRAS"], + "crhd": ["OBJID"], "kako": ["KAPID"], "plko": ["PLNNR"], "plpo": ["PLNNR", "VORNR"], + "marc": ["MATNR", "WERKS"], "mbew": ["MATNR", "BWKEY"], "mard": ["MATNR", "WERKS", "LGORT", "CHARG"], + "vttk": ["TKNUM"], "vttp": ["TKNUM", "TPNUM"], "vtts": ["TKNUM", "TSNUM"], + } + + def test_synthesized_plants_keep_every_key_unique(self) -> None: + store = MemoryTableStore() + with redirect_stdout(StringIO()): + generate_dataset(GenerationConfig(scale_factor="S", num_sites=60, scenarios=()), store) + for table, key in self.TABLE_KEYS.items(): + duplicates = int(store.read(table).duplicated(subset=key).sum()) + self.assertEqual(duplicates, 0, f"{table} has {duplicates} duplicate {'+'.join(key)}") + tvro, tvrot, vttk = store.read("tvro"), store.read("tvrot"), store.read("vttk") + self.assertEqual(len(tvro), 60 * 59) + self.assertFalse(tvrot["BEZEI"].str.contains(r"\b6\d{3}\b").any()) + self.assertTrue(set(vttk["ROUTE"]) <= set(tvro["ROUTE"])) + + def test_hub_outside_the_generated_plants_is_rejected(self) -> None: + with redirect_stdout(StringIO()), self.assertRaisesRegex(ValueError, "HUB_PLANT '3000'"): + generate_dataset( + GenerationConfig(scale_factor="S", num_sites=2, hub_plant="3000"), MemoryTableStore() + ) + + def test_single_site_dataset_generates(self) -> None: + store = MemoryTableStore() + with redirect_stdout(StringIO()): + generate_dataset(GenerationConfig(scale_factor="S", num_sites=1, scenarios="demo"), store) + self.assertEqual(store.read("t001w")["WERKS"].tolist(), ["1000"]) + self.assertEqual(set(store.read("marc")["WERKS"]), {"1000"}) + self.assertEqual(set(store.read("vbap")["WERKS"]), {"1000"}) + tvro, vttk = store.read("tvro"), store.read("vttk") + self.assertEqual(tvro["ROUTE"].tolist(), ["R1010"]) + self.assertTrue(set(vttk["ROUTE"]) <= set(tvro["ROUTE"])) + + +class SmallDatasetTests(unittest.TestCase): + """The S size generates the full pipeline with observable scenarios.""" + + @classmethod + def setUpClass(cls) -> None: + cls.store = MemoryTableStore() + with redirect_stdout(StringIO()): + generate_dataset( + GenerationConfig(scale_factor="S", scenarios="demo"), cls.store + ) + + def test_material_universe_is_s_sized(self) -> None: + mara = self.store.read("mara") + finished = mara[mara["MTART"] == "FERT"]["MATNR"].nunique() + self.assertLessEqual(finished, 4) + + def test_vendor_count_is_s_sized(self) -> None: + self.assertTrue(3 <= self.store.read("lfa1")["LIFNR"].nunique() <= 5) + + def test_every_sales_order_names_an_existing_customer(self) -> None: + customers = set(self.store.read("kna1")["KUNNR"]) + self.assertTrue(set(self.store.read("vbak")["KUNNR"]) <= customers) + + def test_every_eina_record_names_an_existing_vendor(self) -> None: + vendors = set(self.store.read("lfa1")["LIFNR"]) + self.assertTrue(set(self.store.read("eina")["LIFNR"]) <= vendors) + + def test_demo_scenarios_leave_observable_evidence(self) -> None: + metadata = self.store.read("scenario_metadata") + self.assertGreaterEqual(metadata["SCENARIO_ID"].nunique(), 8) + mara = self.store.read("mara") + self.assertIn("MAT-NEW01", set(mara["MATNR"])) + matdoc = self.store.read("matdoc") + self.assertTrue(matdoc["MBLNR"].astype(str).str.startswith("SCN").any()) + + def test_material_scenarios_target_existing_materials(self) -> None: + materials = set(self.store.read("mara")["MATNR"]) + for scenario_material in ("MAT-A0001", "MAT-A0020", "API1"): + self.assertIn(scenario_material, materials) + + def test_scenario_catalog_names_only_ids_that_exist(self) -> None: + catalog = self.store.read("scenario_config") + catalog = catalog[catalog["SCENARIO_ID"].isin(DEMO_SCENARIO_CONFIGS)] + plants = set(self.store.read("t001w")["WERKS"]) + materials = set(self.store.read("mara")["MATNR"]) + vendors = set(self.store.read("lfa1")["LIFNR"]) + for row in catalog.itertuples(index=False): + if row.IMPACTED_NODE not in ("", "ALL"): + self.assertIn(row.IMPACTED_NODE, plants, row.SCENARIO_ID) + if row.IMPACTED_SUPPLIER: + self.assertIn(row.IMPACTED_SUPPLIER, vendors, row.SCENARIO_ID) + for field in (row.IMPACTED_PRODUCTS, row.COMPETING_PRODUCTS): + for material in filter(None, field.split(",")): + if material != "ALL": + self.assertIn(material, materials, f"{row.SCENARIO_ID}: {material}") + + def test_supplier_scenario_targets_an_existing_supply_relationship(self) -> None: + vendor, material = DEMO_SCENARIO_CONFIGS["SCN023"].split(",") + eina = self.store.read("eina") + self.assertTrue(((eina["LIFNR"] == vendor) & (eina["MATNR"] == material)).any()) + + +class SizeTableTests(unittest.TestCase): + """Generated datasets land in the size table's ranges.""" + + def assert_size(self, size) -> None: + measured = generate_at(size) + for dimension, (low, high) in SIZE_RANGES[size].items(): + self.assertTrue( + low <= measured[dimension] <= high, + f"{size} {dimension}: {measured[dimension]} outside [{low}, {high}]", + ) + self.assertEqual(measured["bom_depth"], 1) + + def test_s_dataset_matches_the_size_table(self) -> None: + self.assert_size("S") + + @unittest.skipUnless( + os.environ.get("SAP_MOCK_SIZE_VALIDATION") == "1", + "M and L generation adds about 45 seconds; set SAP_MOCK_SIZE_VALIDATION=1", + ) + def test_m_dataset_matches_the_size_table(self) -> None: + self.assert_size("M") + + @unittest.skipUnless( + os.environ.get("SAP_MOCK_SIZE_VALIDATION") == "1", + "M and L generation adds about 45 seconds; set SAP_MOCK_SIZE_VALIDATION=1", + ) + def test_l_dataset_matches_the_size_table(self) -> None: + self.assert_size("L") + + @unittest.skipUnless( + os.environ.get("SAP_MOCK_XL_VALIDATION") == "1", + "XL generation takes over an hour; set SAP_MOCK_XL_VALIDATION=1", + ) + def test_xl_dataset_matches_the_size_table(self) -> None: + self.assert_size("XL") + + +if __name__ == "__main__": + unittest.main()