Feat/deep detectors

.github/workflows/python-app.yml

@@ -30,7 +30,7 @@ jobs:
         run: uv run --dev prek run -a
 
       - name: Test with pytest
-        run: uv run --dev pytest -s
+        run: uv run pytest -s --run-ignored
 
       # integration tests for DetectMateService
       - name: Checkout DetectMateService

docs/detectors.md

@@ -92,6 +92,8 @@ List of detectors:
 * [Rule Based](detectors/rule_based.md): Detect anomalies based in a set of rules.
 * [Bigram Frequency](detectors/bigram_frequency.md): Detect bigram-frequency-based anomalies in the logs.
 * [Charset](detectors/charset.md): Detect new characters in the variables in the logs.
+* [Deeplog](detectors/deeplog.md): Detect anomalies of a sequence of evend IDs with a LSTM.
+* [LogBert](detectors/logbert.md): Detect anomalies of a sequence of evend IDs with a Transformer.
 * [SCVS Detector](detectors/scvs_detector.md): Detect anomalies by looking at different sequence count vectors.
 * [ECVC Detector](detectors/ecvc_detector.md): Detect anomalies by calculating the distance between different sequence count vectors.
 
@@ -104,6 +106,8 @@ detectors:
   NewValueDetector:
     method_type: new_value_detector
     auto_config: False
+    data_use_configure: None  # Data used for configuration
+    data_use_training: 199  # Data used for training
     params: {}  # global parameters
     events:  # event-specific configuration
       1:  # event_id

docs/detectors/deeplog.md

@@ -0,0 +1,63 @@
+# Deeplog Detector
+
+The Deeplog Detector is inspired from [Deeplog paper](https://dl.acm.org/doi/10.1145/3133956.3134015).
+
+|            | Schema                     | Description        |
+|------------|----------------------------|--------------------|
+| **Input**  | [ParserSchema](../schemas.md) | Structured log  |
+| **Output** | [DetectorSchema](../schemas.md) | Combined alert / finding |
+
+## Description
+Deep learning method that looks at the event ID sequence
+
+## Configuration
+
+```yaml
+detectors:
+    DeeplogDetector:
+        method_type: deeplog_detector
+        auto_config: False
+        data_use_training: 10
+        window_size: 3
+        hyperparameters:
+            Model: 
+                hidden_dim: 64
+                n_layers: 2
+            Train: 
+                seed: 0
+                batch_size: 2048
+                learning_rate: 0.01
+                epochs: 10
+                patience: 3
+            Finetune: 
+                - ["Model", "hidden_dim", [128, 256, 512]]
+                - ["Model", "n_layers", [1, 2, 3]]
+                - ["Train", "learning_rate", [0.01, 0.02, 0.03]]
+```
+
+## Example usage
+
+```python
+from detectmatelibrary.detectors.deeplog_detector import DeeplogDetector
+
+import detectmatelibrary.schemas as schemas
+
+detector = DeeplogDetector(name="DeeplogDetector", config=cfg)
+
+test_data = schemas.ParserSchema({
+    "parserType": "test",
+    "EventID": 12,
+    "template": "test template",
+    "variables": ["adsasd", "asdasd"],
+    "logID": "2",
+    "parsedLogID": "2",
+    "parserID": "test_parser",
+    "log": "test log message",
+    "logFormatVariables": {"level": "CRITICAL"}
+})
+output = schemas.DetectorSchema()
+
+result = detector.detect(test_data, output)
+
+```
+Go back [Index](../index.md)

docs/detectors/logbert.md

@@ -0,0 +1,68 @@
+# LogBert Detector
+
+The LogBert Detector is inspired from [LogBert paper](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9534113).
+
+|            | Schema                     | Description        |
+|------------|----------------------------|--------------------|
+| **Input**  | [ParserSchema](../schemas.md) | Structured log  |
+| **Output** | [DetectorSchema](../schemas.md) | Combined alert / finding |
+
+## Description
+Deep learning method that looks at the event ID sequence
+
+## Configuration
+
+```yaml
+detectors:
+    LogBertDetector:
+        method_type: logbert_detector
+        auto_config: False
+        data_use_training: 10
+        window_size: 4
+        hyperparameters:
+            Model: 
+                hidden: 32
+                num_heads: 2
+                n_layers: 1
+                dropout: 0.0
+                max_len: 1000
+            Train: 
+                seed: 0
+                batch_size: 256
+                learning_rate: 0.01
+                epochs: 10
+                mask_per: 0.4
+                alpha: 0.0
+                patience: 3
+            Finetune: 
+                - ["Model", "hidden", [64, 128, 256]]
+                - ["Model", "n_layers", [1, 2, 3]]
+                - ["Train", "learning_rate", [0.002, 0.001, 0.005]]
+```
+
+## Example usage
+
+```python
+from detectmatelibrary.detectors.logbert_detector import LogBertDetector
+
+import detectmatelibrary.schemas as schemas
+
+detector = LogBertDetector(name="LogBertDetector", config=cfg)
+
+test_data = schemas.ParserSchema({
+    "parserType": "test",
+    "EventID": 12,
+    "template": "test template",
+    "variables": ["adsasd", "asdasd"],
+    "logID": "2",
+    "parsedLogID": "2",
+    "parserID": "test_parser",
+    "log": "test log message",
+    "logFormatVariables": {"level": "CRITICAL"}
+})
+output = schemas.DetectorSchema()
+
+result = detector.detect(test_data, output)
+
+```
+Go back [Index](../index.md)

mkdocs.yml

@@ -31,6 +31,8 @@ nav:
     - BiGram Frequency: detectors/bigram_frequency.md
     - CharSet: detectors/charset.md
     - Value Range: detectors/value_range.md
+    - Deeplog Detector: detectors/deeplog.md
+    - LogBert Detector: detectors/logbert.md
     - SCVS Detector: detectors/scvs_detector.md
     - ECVC Detector: detectors/ecvc_detector.md
   - Alert Aggregation Methods:

pyproject.toml

@@ -17,6 +17,9 @@ dependencies = [
     "msgpack>=1.0.0",
     "fsspec>=2024.1.0",
     "pyarrow>=24.0.0",
+    "jax>=0.11.0",
+    "flax>=0.12.8",
+    "optax>=0.2.8",
 ]
 
 [dependency-groups]

pytest.ini

@@ -0,0 +1,5 @@
+[pytest]
+# You can use addopts to change the default terminal behavior globally
+addopts = --strict-markers
+markers =
+    ignored: tests that should only run on demand

src/detectmatelibrary/common/_core_op/_fit_logic.py

@@ -67,7 +67,7 @@ class FitLogicState(Enum):
     def describe(self) -> str:
         descriptions = [
             "Configuring",
-            "Training.",
+            "Training",
             "Default"
         ]
         return descriptions[self.value]

src/detectmatelibrary/common/deeplearning_detector.py

@@ -0,0 +1,90 @@
+
+from detectmatelibrary.common.detector import CoreDetector, CoreDetectorConfig
+
+from detectmatelibrary.utils.deep_learning.imodel import DeepModel
+from detectmatelibrary.utils.data_buffer import BufferMode
+
+from detectmatelibrary import schemas
+
+from typing import Any
+
+
+class DeepLearningDetectorConfig(CoreDetectorConfig):
+    window_size: int = 10
+    validation_per: float = 0.2
+    finetune_epochs: int = 2
+
+    hyperparameters: dict[str, Any] = {
+        "Model": {
+
+        },
+        "Train": {
+
+        },
+        "Finetune": [],
+    }
+
+
+def build_seq(input_: list[schemas.ParserSchema]) -> tuple[int]:
+    return tuple([in_["EventID"] for in_ in input_])
+
+
+class DeepLearningDetector(CoreDetector):
+    def __init__(
+        self,
+        model_cls: DeepModel,
+        name: str = "CoreDetector",
+        config: DeepLearningDetectorConfig = DeepLearningDetectorConfig()
+    ) -> None:
+
+        if isinstance(config, dict):
+            config = DeepLearningDetectorConfig.from_dict(config, name)
+
+        super().__init__(
+            name=name,
+            buffer_mode=BufferMode.WINDOW,
+            buffer_size=config.window_size,
+            config=config
+        )
+        self.config: DeepLearningDetectorConfig
+        self.model: DeepModel = model_cls(config=self.config.hyperparameters)  # type: ignore
+
+        self.train_seqs: list[tuple[int]] = []
+        self.config_seqs: list[tuple[int]] = []
+        self.stats: dict[str, float | int] = {}
+        self.top_k: int = 0
+
+    def train(self, input_: list[schemas.ParserSchema]) -> None:  # type: ignore
+        self.train_seqs.append(build_seq(input_))
+
+    def configure(self, input_: list[schemas.ParserSchema]) -> None:  # type: ignore
+        self.config_seqs.append(build_seq(input_))
+
+    def set_configuration(self) -> None:
+        self.model.finetune(
+            self.config_seqs, var_per=self.config.validation_per, epochs=self.config.finetune_epochs
+        )
+        self.config_seqs = []
+
+    def post_train(self) -> None:
+        self.stats = self.model.train(self.train_seqs, var_per=self.config.validation_per)
+        self.train_seqs = []
+
+        if "top_k" in self.stats:
+            self.top_k = int(self.stats["top_k"])
+            print(self.model)
+            print("Top k assigned", self.top_k)
+
+    def detect(
+        self,
+        input_: list[schemas.ParserSchema],  # type: ignore
+        output_: schemas.DetectorSchema,
+    ) -> bool:
+
+        alert = self.model.check_anomaly(build_seq(input_), top_k=self.top_k)
+        if alert:
+            output_["score"] = 1.0
+            output_["description"] = f"{self.name} found an anomaly in the sequence"
+            return True
+
+        return False

src/detectmatelibrary/detectors/deeplog_detector.py

@@ -0,0 +1,46 @@
+from detectmatelibrary.common.deeplearning_detector import (
+    DeepLearningDetectorConfig, DeepLearningDetector
+)
+
+from detectmatelibrary.utils.deep_learning.deeplog import DeepLog
+
+
+from typing import Any
+
+
+class DeeplogDetectorConfig(DeepLearningDetectorConfig):
+    method_type: str = "deeplog_detector"
+
+    hyperparameters: dict[str, Any] = {  # type: ignore
+        "Model": {
+            "hidden_dim": 64,
+            "n_layers": 2,
+        },
+        "Train": {
+            "seed": 0,
+            "batch_size": 2048,
+            "learning_rate": 0.01,
+            "epochs": 10,
+            "patience": 3,
+        },
+        "Finetune": [
+            ["Model", "hidden_dim", [128, 256, 512]],
+            ["Model", "n_layers", [1, 2, 3]],
+            ["Train", "learning_rate", [0.01, 0.02, 0.03]],
+        ],
+    }
+
+
+class DeeplogDetector(DeepLearningDetector):
+    def __init__(
+        self,
+        name: str = "DeeplogDetector",
+        config: DeeplogDetectorConfig | dict[str, Any] = DeeplogDetectorConfig(),
+    ) -> None:
+
+        if isinstance(config, dict):
+            config = DeeplogDetectorConfig.from_dict(config, name)
+
+        super().__init__(
+            name=name, model_cls=DeepLog, config=config 
+        )

src/detectmatelibrary/detectors/logbert_detector.py

@@ -0,0 +1,52 @@
+from detectmatelibrary.common.deeplearning_detector import (
+    DeepLearningDetectorConfig, DeepLearningDetector
+)
+
+from detectmatelibrary.utils.deep_learning.logbert import LogBert
+
+
+from typing import Any
+
+
+class LogBertDetectorConfig(DeepLearningDetectorConfig):
+    method_type: str = "logbert_detector"
+
+    hyperparameters: dict[str, Any] = {  # type: ignore
+        "Model": {
+            "n_embed": 10,
+            "hidden": 32,
+            "num_heads": 2,
+            "n_layers": 1,
+            "dropout": 0.0,
+            "max_len": 1000,
+        },
+        "Train": {
+            "seed": 0,
+            "batch_size": 256,
+            "learning_rate": 0.01,
+            "epochs": 10,
+            "mask_per": 0.4,
+            "alpha": 0.0,
+            "patience": 3,
+        },
+        "Finetune": [
+            ["Model", "hidden", [64, 128, 256]],
+            ["Model", "n_layers", [1, 2, 3]],
+            ["Train", "learning_rate", [0.002, 0.001, 0.005]],
+        ]
+    }
+
+
+class LogBertDetector(DeepLearningDetector):
+    def __init__(
+        self,
+        name: str = "LogBertDetector",
+        config: LogBertDetectorConfig | dict[str, Any] = LogBertDetectorConfig(),
+    ) -> None:
+
+        if isinstance(config, dict):
+            config = LogBertDetectorConfig.from_dict(config, name)
+
+        super().__init__(
+            name=name, model_cls=LogBert, config=config 
+        )