Streaming Data Structures Library

A Python library that implements efficient streaming algorithms for real-time data processing. Designed for high-volume continuous data streams, this library provides practical tools to analyze data without storing it entirely in memory.

---

🚀 Features

• Sliding Window Maximum – Compute the maximum element in every moving window efficiently.
• Streaming Median – Maintain and retrieve the median in real-time using two heaps.
• Top-K Frequent Elements – Track the most frequent elements dynamically.
• Reservoir Sampling – Randomly sample items from a continuous data stream.
• Modular, production-ready implementation in Python.
• Clean and readable object-oriented design.

---

🧱 Installation

Clone the repository:

git clone https://github.com/<your-username>/streaming-dsa-lib.git
cd streaming-dsa-lib

(Optional) create and activate a virtual environment:

python -m venv venv
source venv/bin/activate  # On Mac/Linux
venv\Scripts\activate     # On Windows

Install dependencies:

pip install -r requirements.txt

---

📦 Library Structure

streaming-dsa-lib/
│
├── streaming/                       # Core streaming algorithms
│   ├── __init__.py
│   ├── sliding_window.py
│   ├── streaming_median.py
│   ├── topk_frequent.py
│   ├── reservoir_sampling.py
│
├── tests/                           # Unit tests
│   ├── test_sliding_window.py
│   ├── test_streaming_median.py
│   ├── test_topk_frequent.py
│   ├── test_reservoir_sampling.py
│
├── examples/                        # Demo examples
│   ├── demo_sliding_window.py
│   ├── demo_streaming_median.py
│   ├── demo_topk_frequent.py
│   ├── demo_reservoir_sampling.py
│
├── requirements.txt
├── setup.py
├── LICENSE
├── README.md
└── StreamingDataStructuresLibrary.docx

---

🧠 Usage Examples

1️⃣ Sliding Window Maximum

from streaming.sliding_window import SlidingWindowMax

sw = SlidingWindowMax(k=3)
for val in [10, 6, 9, 8, 5]:
    sw.add(val)
    print(sw.get_max())

Output:

10
10
10
9
9

2️⃣ Streaming Median

from streaming.streaming_median import StreamingMedian

sm = StreamingMedian()
for val in [5, 15, 1, 3]:
    sm.add(val)
    print(sm.get_median())

Output:

5
10.0
5
4.0

3️⃣ Top-K Frequent Elements

from streaming.topk_frequent import TopKFrequent

stream = ['A', 'B', 'A', 'C', 'A', 'B']
topk = TopKFrequent(k=2)
for val in stream:
    topk.add(val)
print(topk.get_topk())

Output:

[('A', 3), ('B', 2)]

4️⃣ Reservoir Sampling

from streaming.reservoir_sampling import ReservoirSampler

sampler = ReservoirSampler(k=5)
for i in range(1, 21):
    sampler.add(i)
print(sampler.get_samples())

Output:

[4, 12, 7, 18, 9]

---

🧪 Testing

Run all tests using:

pytest

---

📘 License

This project is licensed under the MIT License – see the LICENSE file for details.

---

👤 Author

Teja Sai Eswar Reddy
Student, IIT Kharagpur – ECE Department (3rd Year)
Project: Streaming Data Structures Library for Efficient Real-Time Analytics