Pathfinder

Pathfinder is a Linux-first CLI inventory tool for large reconnaissance URL datasets. Phase 1 does not scan, enrich, or make HTTP requests. It normalizes URLs into a durable application inventory:

Host -> Endpoint -> Parameters -> Metadata

Commands

pathfinder init
pathfinder import urls.txt
cat urls.txt | pathfinder import
pathfinder stats
pathfinder tree
pathfinder reset
pathfinder reset --force

By default Pathfinder stores data in pathfinder.db in the current directory. Use --database before the command to choose another location:

pathfinder --database recon.db init

Architecture

The CLI is intentionally thin. Processing is split into independent stages so later phases can add exporters, enrichment, search, and metadata without reshaping the application.

pathfinder/
  cli/          Typer commands and Rich output
  core/         Import orchestration
  parser/       URL parsing with urllib.parse
  normalizer/   Host, path, query, and percent-encoding normalization
  models/       Typed dataclasses shared between layers
  storage/      SQLite schema and repository methods
  tree/         In-memory hierarchy construction and Rich rendering
  stats/        Statistics queries
  config/       YAML configuration loading extension point
  utils/        Logging setup

Import Behavior

Pathfinder accepts URLs from a text file or stdin. Malformed URLs are counted and skipped without stopping the import. Duplicate endpoint imports merge naturally.

Normalization rules include:

• hostnames are lowercased
• empty paths become /
• duplicate path slashes collapse
• trailing slashes are removed except for /
• path case is preserved
• fragments are ignored
• query parameter names are stored without values

/login, /login/, /login?id=1, and /login?next=home become one endpoint with parameters id and next.

Database

SQLite tables are normalized around projects, imports, hosts, endpoints, and parameters. Endpoint uniqueness is (host_id, path). Parent endpoints are stored to support fast tree rendering without recursive database lookups. Parent nodes that were not directly imported are marked as unobserved, so statistics count only endpoints seen in input.

Development

Install from GitHub
git clone https://github.com/<your-username>/pathfinder.git
cd pathfinder

python -m venv .venv

# Linux/macOS
source .venv/bin/activate

# Windows
.venv\Scripts\activate

pip install --upgrade pip
pip install .

Verify the installation:

pathfinder --help

Community

Bug reports, feature requests, and pull requests are welcome.

If you have ideas for improvements, find a bug, or want to contribute code, please open an issue or submit a pull request on GitHub. Community contributions of all sizes are appreciated.

If Pathfinder is useful in your workflow, consider starring the repository to help others discover the project.

Roadmap Additional export formats Advanced search capabilities Metadata enrichment Extensible plugin architecture Improved reporting and visualization Workflow integrations