🤖 DarkForums Telegram Bot

Telegram bot canggih untuk scraping data dari DarkForums (https://darkforums.st/) menggunakan Playwright dengan implementasi yang aman dan scalable.

✨ Fitur Utama

🔍 Web Scraping

Playwright Integration: Scraping dinamis dengan browser automation
Anti-Detection: Stealth mode, rotating user agents, random delays
Error Handling: Robust retry logic dan error recovery
Screenshot Support: Capture bukti visual dari scraping

🤖 Telegram Bot

Rich Commands: 10+ commands dengan inline keyboards
Real-time Updates: Live data dari forum
User Management: Authentication dan authorization
Rate Limiting: Intelligent throttling untuk mencegah spam

🛡️ Security & Privacy

Data Encryption: Sensitive data protection
User Authentication: Whitelist-based access control
Proxy Support: Anonymous scraping capabilities
Secure Logging: No sensitive data in logs

🗄️ Database & Caching

SQLite/PostgreSQL: Flexible database options
Intelligent Caching: Redis-compatible caching system
Data Persistence: Reliable data storage
Cleanup Automation: Automatic old data removal

📊 Monitoring & Analytics

Prometheus Metrics: Comprehensive monitoring
Grafana Dashboards: Beautiful visualizations
Health Checks: System status monitoring
Performance Tracking: Response times dan usage statistics

🚀 Quick Start

Prerequisites

Python 3.11+
Docker & Docker Compose (optional)
Telegram Bot Token
DarkForums Account

1. Installation

Option A: Automatic Setup (Recommended)

# Clone repository
git clone <repository-url>
cd telegram-darkforums-bot

# Run automatic setup (Windows)
run.bat install
run.bat setup

# Or manual setup (Linux/Mac)
python run.py --setup

Option B: Manual Setup

# Create virtual environment
python -m venv venv
source venv/bin/activate  # Linux/Mac
# or
venv\Scripts\activate     # Windows

# Install dependencies
pip install -r requirements.txt
playwright install chromium

# Setup environment
cp .env.example .env
# Edit .env with your credentials

2. Configuration

Edit .env file dengan credentials Anda:

# Telegram Bot
TELEGRAM_BOT_TOKEN=your_bot_token_here
TELEGRAM_ADMIN_ID=your_admin_user_id

# DarkForums
DARKFORUMS_USERNAME=your_username
DARKFORUMS_PASSWORD=your_password

# Optional: Advanced settings
HEADLESS_MODE=true
RATE_LIMIT_REQUESTS=10
LOG_LEVEL=INFO

3. Running the Bot

Development Mode

# Windows
run.bat dev

# Linux/Mac
python run.py --dev

Production Mode

# Windows
run.bat prod

# Linux/Mac
python run.py --prod

Docker (Recommended for Production)

# Build and run
docker-compose up -d

# View logs
docker-compose logs -f telegram-bot

# Stop
docker-compose down

📱 Bot Commands

Public Commands

Command	Description	Example
`/start`	Initialize bot dan show welcome message	`/start`
`/help`	Show available commands dan usage	`/help`
`/search <keyword>`	Search posts by keyword	`/search python`
`/latest [count]`	Get latest posts (default: 10)	`/latest 5`
`/user <username>`	Get user profile information	`/user john_doe`
`/stats`	Show forum statistics	`/stats`

User Commands

Command	Description	Example
`/history [count]`	Show your search history	`/history 5`
`/clear`	Clear your search history	`/clear`

Admin Commands (Admin Only)

Command	Description	Example
`/status`	Show bot system status	`/status`
`/broadcast <message>`	Send message to all users	`/broadcast Hello!`
`/ban <user_id>`	Ban user from using bot	`/ban 123456789`
`/unban <user_id>`	Unban user	`/unban 123456789`
`/logs [lines]`	Show recent log entries	`/logs 50`

🏗️ Architecture

📁 telegram-darkforums-bot/
├── 📁 src/
│   ├── 📁 bot/           # Telegram bot logic
│   ├── 📁 scraper/       # Web scraping components
│   ├── 📁 database/      # Database models & operations
│   ├── 📁 utils/         # Utilities & helpers
│   └── 📄 main.py        # Application entry point
├── 📁 tests/             # Test suites
├── 📁 monitoring/        # Prometheus & Grafana configs
├── 📁 docker/            # Docker configurations
├── 📄 requirements.txt   # Python dependencies
├── 📄 docker-compose.yml # Docker orchestration
└── 📄 run.py            # Development runner

Core Components

🤖 Bot Application (`src/bot/`)

Application: Main bot orchestration
Handlers: Command dan message processing
Middleware: Authentication, rate limiting, security
Commands: Command definitions dan validation

🔍 Scraper (`src/scraper/`)

PlaywrightManager: Browser automation
DarkForumsScraper: Forum-specific scraping logic
DataParser: HTML parsing dan data extraction

🗄️ Database (`src/database/`)

Models: SQLAlchemy ORM models
Operations: Database CRUD operations

🛠️ Utils (`src/utils/`)

Config: Environment configuration
Logger: Structured logging
Helpers: Common utilities

🐳 Docker Deployment

Services

telegram-bot: Main application
db: PostgreSQL database (optional)
redis: Caching layer (optional)
prometheus: Metrics collection (optional)
grafana: Monitoring dashboards (optional)

Production Deployment

# Production with monitoring
docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d

# Scale bot instances
docker-compose up -d --scale telegram-bot=3

# Update deployment
docker-compose pull
docker-compose up -d --force-recreate

📊 Monitoring

Metrics Available

Bot uptime dan health status
Command usage statistics
Scraping success/failure rates
Response times dan performance
Memory dan CPU usage
Database operations
Rate limiting events
Error rates dan types

Accessing Dashboards

Grafana: http://localhost:3000 (admin/admin123)
Prometheus: http://localhost:9090

🧪 Testing

# Run all tests
run.bat test
# or
python run.py --test

# Run specific test file
pytest tests/test_bot.py -v

# Run with coverage
pytest --cov=src --cov-report=html

Test Coverage

Unit tests for all components
Integration tests for bot commands
Scraper functionality tests
Database operations tests
Mock external dependencies

🔧 Development

Setup Development Environment

# Install development dependencies
pip install -r requirements-dev.txt

# Install pre-commit hooks
pre-commit install

# Run code formatting
black src/ tests/
flake8 src/ tests/
mypy src/

Code Style

Black: Code formatting
Flake8: Linting
MyPy: Type checking
isort: Import sorting

Contributing

Fork repository
Create feature branch
Make changes dengan tests
Run quality checks
Submit pull request

🛡️ Security Considerations

Best Practices Implemented

✅ No hardcoded credentials
✅ Environment variable configuration
✅ Input validation dan sanitization
✅ Rate limiting dan throttling
✅ User authentication
✅ Secure logging (no sensitive data)
✅ Docker security best practices
✅ Proxy support for anonymity

Security Features

Anti-Detection: Stealth browsing techniques
Rate Limiting: Prevent abuse dan detection
Data Encryption: Protect sensitive information
Access Control: Whitelist-based user management
Audit Logging: Track all activities

📝 Configuration Reference

Environment Variables

Required

TELEGRAM_BOT_TOKEN=        # Your Telegram bot token
TELEGRAM_ADMIN_ID=         # Your Telegram user ID
DARKFORUMS_USERNAME=       # DarkForums username
DARKFORUMS_PASSWORD=       # DarkForums password

Optional

# Playwright Settings
HEADLESS_MODE=true         # Run browser in headless mode
BROWSER_TYPE=chromium      # Browser type (chromium/firefox/webkit)
PAGE_TIMEOUT=30000         # Page load timeout (ms)

# Rate Limiting
RATE_LIMIT_REQUESTS=10     # Max requests per window
RATE_LIMIT_WINDOW=60       # Rate limit window (seconds)

# Database
DATABASE_URL=sqlite:///data/bot.db  # Database connection string

# Logging
LOG_LEVEL=INFO             # Logging level
LOG_FILE=logs/bot.log      # Log file path

# Caching
CACHE_TTL=3600            # Cache TTL (seconds)
CACHE_MAX_SIZE=1000       # Max cache entries

# Proxy (Optional)
PROXY_SERVER=             # Proxy server URL
PROXY_USERNAME=           # Proxy username
PROXY_PASSWORD=           # Proxy password

🚨 Troubleshooting

Common Issues

Bot Not Starting

# Check logs
run.bat logs
# or
docker-compose logs telegram-bot

# Verify configuration
run.bat status

Scraping Failures

Check DarkForums credentials
Verify internet connection
Check if site is accessible
Review rate limiting settings

Database Issues

# Reset database
rm data/bot.db
run.bat setup

# Check database status
sqlite3 data/bot.db ".tables"

Docker Issues

# Rebuild containers
docker-compose down
docker-compose build --no-cache
docker-compose up -d

# Check container logs
docker-compose logs -f

Performance Optimization

Memory Usage

Adjust CACHE_MAX_SIZE
Enable database cleanup
Monitor container resources

Scraping Speed

Adjust PAGE_TIMEOUT
Use headless mode
Optimize selectors

📄 License

MIT License - see LICENSE file for details.

🤝 Support

Issues: GitHub Issues
Discussions: GitHub Discussions
Documentation: Wiki

🙏 Acknowledgments

Playwright - Web automation framework
python-telegram-bot - Telegram Bot API wrapper
SQLAlchemy - Database ORM
Docker - Containerization platform

⚠️ Disclaimer: This bot is for educational purposes only. Please respect the terms of service of the target website dan use responsibly.

🔒 Privacy: This bot does not store personal data beyond what's necessary for functionality. All data is encrypted dan handled securely.

Name		Name	Last commit message	Last commit date
Latest commit History 4 Commits
monitoring		monitoring
node_modules		node_modules
src		src
tests		tests
.env.example		.env.example
.gitignore		.gitignore
Dockerfile		Dockerfile
Dockerfile.debian		Dockerfile.debian
PROJECT_PROMPT.md		PROJECT_PROMPT.md
README.md		README.md
VPS-DEPLOYMENT.md		VPS-DEPLOYMENT.md
bypass_ddos_nodejs.js		bypass_ddos_nodejs.js
debug_search.js		debug_search.js
deploy-vps.sh		deploy-vps.sh
docker-compose.minimal.yml		docker-compose.minimal.yml
docker-compose.production.yml		docker-compose.production.yml
docker-compose.simple.yml		docker-compose.simple.yml
docker-compose.yml		docker-compose.yml
init.sql		init.sql
package-lock.json		package-lock.json
package.json		package.json
requirements.txt		requirements.txt
run.bat		run.bat
run.py		run.py
test_ddos_nodejs.js		test_ddos_nodejs.js
test_playwright.js		test_playwright.js

Folders and files

Latest commit

History

Repository files navigation