A composable Reddit crawler for the command line. Fetch posts and comments from any source, shape the data through a lazy filter pipeline, and emit it in any format — all without writing a single line of Python.
# stream the top 500 posts from r/MachineLearning as JSONL
praw-cli posts r/MachineLearning --sort top --time year --limit 500 --format jsonl
# search across r/programming, keep only high-signal posts, project to four fields
praw-cli search "New web framework" --sub r/programming \
--filter "score>=100" --filter "num_comments>=10" \
--fields id,title,score,url --format csv --output results.csv
# full comment tree of a submission, depth-first, minimum score 5
praw-cli comments https://reddit.com/r/Python/comments/xyz/ \
--depth 10 --min-score 5 --format jsonl
# fetch a user's recent posts as a terminal table
praw-cli user spez --mode posts --limit 20 --format table
# re-process a saved dataset offline — no API, no credentials needed
praw-cli input posts.jsonl --filter "score>=500" --format csv --output filtered.csv- Constant Memory Footprint (Lazy Pipelines): Typical scrapers buffer huge arrays in memory, causing out-of-memory crashes on large crawls. praw-cli processes data lazily item-by-item (
Iterator[Record]). Streaming 100,000 posts uses the same constant memory as streaming 10. - Resumable Extractions (Checkpointing): Long-running scrapes often fail halfway due to network drops or API limits. praw-cli checkpoints progress, letting you
--resumeinterrupted sessions without refetching from scratch. - Offline Re-processing (
input): Apply new filters, change output formats, or extract field subsets from any previously saved.jsonl,.json, or.csvfile — without touching the API or needing credentials. Pipe from stdin too.
- Data Science-Focused DSL: Stop writing custom Python scripts just to filter text. Chain
--filterconditions directly in the CLI:- NLP Cleaning: Keep long-form content using
selftext len>= 500. - Temporal Sorting: Restrict dates natively using ISO-8601 strings, like
created_utc >= 2024-01-01. - Targeted Mining: Target specific topics with keyword groupings (
has,has_all) or regular expressions (title ~= \bbot\b).
- NLP Cleaning: Keep long-form content using
- Field Projections: Keep output datasets lightweight and clean by extracting only the schema columns you need (e.g.,
--fields id,title,score,author).
- Diverse Output Formats: Stream outputs directly to
jsonl(preferred for streaming/Pandas), standardjson,csv(for R/Excel), or rich console tables and Markdown reports. - Multi-Sink Pipeline: Write the raw data to a
.jsonldatabase while simultaneously writing a preview to a.csvsummary in a single pass.
- Crawl Manifests: Every execution automatically generates a
.manifest.jsondetailing exact parameters, versioning, records filtered, and a config fingerprint, preventing configuration drift in research environments. - Deterministic Sampling: Extract reproducible subsets of huge subreddits using systematic or randomized sampling (e.g., Bernoulli trial at
rate = 0.1with a fixed seed).
Requires Python 3.12 or later and a Reddit API application (free, read-only access is sufficient for most use cases).
Install the package directly using pip or pipx:
pip install praw-cliIf you want to run it locally or contribute to development:
git clone https://github.com/othonhugo/praw-cli
cd praw-cli
uv sync
source .venv/bin/activateFull documentation is available in the docs/ directory.
- Getting Started: Installation and Authentication
- Usage Guide:
- Development: