Fast Parser Engine (fast_parser)

A high-performance, multi-threaded log analytics engine built in Rust and exposed as a native Python extension via PyO3 and Maturin.

The engine ingests newline-delimited data streams (server logs, clickstream events, etc.), parses them in parallel using Rayon's work-stealing thread pool, and surfaces both successfully parsed events and structured parse errors to the caller — no silent data loss.

Key Architectural Features

• GIL-Free Multithreading via Rayon: py.allow_threads drops Python's GIL for the entire parallel region. Rayon's work-stealing scheduler distributes lines across all CPU cores with automatic load balancing — no manual thread management, no thread storms from unbounded spawning.

• Zero-Copy Slicing: &str references and splitn / split_once boundary operations parse layout headers without duplicating memory or forcing heap re-allocations until an owned value is required.

• Pre-Allocated Hashing: Byte-boundary scanning estimates the incoming HashMap capacity before insertion, bypassing costly incremental rehashing.

• Structured Error Observability: Malformed lines are collected into ParseError objects (with line_number, raw, and reason fields) and returned alongside successful events in a StreamResult. Callers have full visibility into parse failures without sacrificing throughput.

• Thread-Safe GIL Re-Acquisition: Python::with_gil is called exactly once, after all Rayon workers have finished, to wrap results into Py<Event> smart pointers for consumption in Python.

Record Format

Each line must follow:

<timestamp>|<event_type>|<key>:<value>[,<key>:<value>...]

Example:

1719338400|USER_LOGIN|user_id:alice,ip:10.0.0.1,status:success

Tech Stack

Layer	Technology
Core parser	Rust 2021 edition
Parallelism	Rayon 1.x
FFI bindings	PyO3 0.22 + Maturin 1.x
Python target	3.10+

Installation

# Build and link the native extension into your active virtual environment
maturin develop --release

Usage

from fast_parser import parallel_parse_log_stream

streams = [open(f).read() for f in log_files]
results = parallel_parse_log_stream(streams)

for i, result in enumerate(results):
    print(f"Stream {i}: {result.event_count()} events, {result.error_count()} errors")
    for err in result.errors:
        print(f"  Line {err.line_number}: {err.reason!r} — {err.raw!r}")

Testing

# Run the full integration test suite
pytest tests/test_extension.py -v

# Run benchmarks comparing Rust vs pure-Python baseline
pytest tests/test_benchmark.py --benchmark-compare