Magnet lists an account's public repositories, or searches GitHub for them, filters the results, and downloads the matches in parallel as source archives extracted into one directory per repository.
It is a snapshot tool, not a mirror: it retrieves the current contents of each default branch, without git history. This makes it fast and inexpensive for backups, bulk code search, compliance sweeps, and offline analysis, but not suitable if commits, branches, or tags are required.
$ magnet rust-lang --language rust --min-stars 500 --only-original
Scanning repositories for: rust-lang
41 of 289 repositories match
[ 1/41] rust - 412.7 MiB
[ 2/41] cargo - 38.1 MiB
[ 3/41] rustlings - 6.4 MiB
...
Downloaded: 41
Total size: 1.1 GiB
Elapsed: 3m 12s
Running it again fetches only what has changed.
Requires Rust 1.88 or newer (edition 2024), a C compiler, and cmake, which the bundled TLS stack needs at build time. The resulting binary has no runtime TLS dependency.
git clone https://github.com/naseridev/magnet
cd magnet
cargo build --releaseThe compiled binary is located at target/release/magnet.
Anonymous access to the GitHub API is limited to 60 requests per hour, which is sufficient to list roughly 6000 repositories. A personal access token raises this limit to 5000 requests per hour. No scopes are required for public repositories.
export GITHUB_TOKEN=ghp_...
magnet rust-langThe token can also be passed directly with --token, though the environment variable keeps it out of shell history and process listings.
Archive downloads are served from github.com rather than through the REST API and do not consume API quota. Only listing and searching count against the limit.
# Everything an account has published
magnet torvalds
# Popular original Rust projects, eight downloads at a time
magnet rust-lang --language rust --min-stars 500 --only-original --parallel 8
# Search GitHub instead of naming an account
magnet --search "http client" --language rust --min-stars 500
# Preview results before downloading
magnet acme --regex "^(core|api)-" --dry-run
# Keep a backup current, unattended
magnet acme --daemon --interval 6h --output /backups --log-file /var/log/magnet.logCommand forms:
magnet [OPTIONS] <ACCOUNT>
magnet [OPTIONS] --search <TEXT>
magnet [OPTIONS] <ACCOUNT> --search <TEXT>
| Option | Short | Description | Default |
|---|---|---|---|
| --search TEXT | Search GitHub instead of listing one account | off | |
| --language LANG | -l | Match primary language, case insensitive | any |
| --min-stars N | -s | Minimum star count | 0 |
| --min-forks N | Minimum fork count | 0 | |
| --max-size MIB | -m | Skip larger repositories; also caps the transfer | unlimited |
| --topic TOPIC | Require this topic; repeatable | any | |
| --license SPDX | Require this SPDX license identifier | any | |
| --only-original | -o | Skip forks | off |
| --exclude-archived | Skip archived repositories | off | |
| --regex PATTERN | -r | Repository name must match this pattern | any |
| --pushed-after DATE | Pushed on or after YYYY-MM-DD | any | |
| --pushed-before DATE | Pushed on or before YYYY-MM-DD | any | |
| --created-after DATE | Created on or after YYYY-MM-DD | any | |
| --created-before DATE | Created on or before YYYY-MM-DD | any | |
| --qualifier K:V | Raw search qualifier, passed through; repeatable | none | |
| --max-repos N | Stop after this many matches | unlimited | |
| --sort FIELD | Ordering: best-match, stars, forks, updated | best-match | |
| --order DIR | Sort direction: asc or desc | desc |
| Option | Short | Description | Default |
|---|---|---|---|
| --output DIR | -O | Output root | . |
| --parallel N | -p | Concurrent downloads, 1 to 256 | 3 |
| --sync | -S | Refresh repositories with unknown or outdated state | off |
| --force | -f | Re-download everything, ignoring the manifest | off |
| --dry-run | -n | Report what would be downloaded; write nothing | off |
| --no-resume | Always transfer archives from the start | off | |
| --timeout SECONDS | Abandon a transfer stalled for this long | 60 | |
| --retries N | Retries for transient errors, 0 to 10 | 3 |
| Option | Short | Description | Default |
|---|---|---|---|
| --token TOKEN | -t | Personal access token; also GITHUB_TOKEN | none |
| --daemon | -D | Run continuously on a schedule | off |
| --interval DURATION | Time between cycles in daemon mode | 6h | |
| --max-cycles N | Stop the daemon after this many cycles | unlimited | |
| --log-file PATH | Append timestamped log lines to this file | none | |
| --quiet | -q | Errors only | off |
| --verbose | -v | Per-repository diagnostics, including filter reasons | off |
| --api-base URL | REST API root, for GitHub Enterprise | https://api.github.com | |
| --archive-base URL | Archive host, for GitHub Enterprise | https://github.com |
Filters combine with AND and apply in both listing and search mode.
- --max-size is compared against the size GitHub reports, which covers the packed repository including history, so a large-history, small-tree repository may be filtered out even though its archive would be small. The same value also caps the actual transfer and extracted tree.
- --regex patterns are unanchored and case sensitive. Use ^...$ to anchor and (?i) for case insensitive matching.
- --language matches only the primary language GitHub infers. Repositories with no detected language are excluded whenever --language is set.
- --license matches the SPDX identifier. Repositories with no identifiable license never satisfy a license filter.
- --topic may be repeated; every named topic must be present.
--verbose prints the reason each repository was rejected, which is the fastest way to debug an unexpectedly small match count.
--search queries the GitHub Search API instead of listing a single account. Filter flags serve double duty: they become server-side search qualifiers and are also applied client-side to the results.
magnet --search "http client" --language rust --min-stars 500 --only-original
magnet acme --search parser
magnet --search "static site generator" --topic rust --dry-run--dry-run prints each match with its stars, forks, language, license, and description, which is the intended way to refine a query before downloading.
| Flag | Qualifier |
|---|---|
| ACCOUNT | user: |
| --language | language: |
| --min-stars | stars:>= |
| --min-forks | forks:>= |
| --max-size | size:<= (converted to KiB) |
| --topic | topic: (one per flag) |
| --license | license: |
| --only-original | fork:false |
| --exclude-archived | archived:false |
| --pushed-after / --pushed-before | pushed: |
| --created-after / --created-before | created: |
| --qualifier | passed through verbatim |
Anything not covered by a dedicated flag can be expressed with --qualifier:
magnet --search cli --qualifier "good-first-issues:>5" --qualifier is:publicThe Search API returns at most 1000 results regardless of the true match count; magnet reports the true total and warns when results are truncated. The search endpoint has its own rate limit: 30 requests per minute authenticated, 10 unauthenticated. Each page of 100 results costs one request. Regular expressions are not supported by the Search API; --regex is applied to results after they are returned, so it narrows a search rather than driving it.
Every successful download is recorded in a manifest at the root of the output directory (.magnet.json), along with the repository's pushed_at timestamp. On the next run, magnet compares that timestamp against the API and downloads only what is new or has changed.
The manifest is written atomically and flushed as the run progresses, so an interrupted run retains everything already completed. Failed repositories are not recorded and are retried on the next run.
A directory with no manifest entry, such as one created before this feature existed or added by hand, has an unknown state and is left alone and reported as skipped. Pass --sync to adopt such directories by downloading them once, after which normal change detection applies. --force ignores the manifest entirely and re-downloads everything. Deleting a repository directory causes it to be downloaded again on the next run even if the manifest reports it as current.
Archives are streamed to a scratch file next to their destination. If a transfer is interrupted, the partial file is retained, and the next run continues it with an HTTP Range request rather than starting over.
Before extraction, the archive is validated: its length is checked against the advertised Content-Length, and its zip central directory and entry headers are parsed. Extraction begins only after validation succeeds.
Three failure modes are handled explicitly:
- A partial that resumes into an unreadable archive is deleted and re-fetched from the start.
- A server that ignores Range and returns 200 has its response written from offset zero instead of appended.
- A stale partial larger than the current archive produces 416, which restarts the transfer rather than failing it.
Pass --no-resume to always transfer from the beginning.
Magnet can run unattended, either as a long-lived service or from cron.
magnet acme --daemon --interval 6h --output /backups --log-file /var/log/magnet.logEach cycle lists the account, compares every repository against the manifest, and downloads only what changed. Between cycles the process sleeps. Durations accept s, m, h, and d suffixes. --max-cycles bounds the run, which is useful for a smoke test. Each cycle is logged with an RFC 3339 timestamp:
2026-07-31T02:00:00Z [INFO] backup service started for acme, interval 6h 0m
2026-07-31T02:00:00Z [INFO] cycle 1 started
2026-07-31T02:04:11Z [INFO] cycle 1 finished in 4m 11s: 3 downloaded, 117 unchanged, 0 skipped, 0 failed
2026-07-31T02:04:11Z [INFO] next cycle at 2026-07-31T08:04:11Z
A cycle that fails outright, for example because the API is unreachable, is logged at ERROR and the service waits for the next interval rather than exiting. Only a configuration problem stops it before the first cycle.
SIGINT and SIGTERM stop the service cleanly. New downloads are not started, downloads in flight are allowed to finish, the manifest is flushed, and the process exits with status 0. Because the manifest is flushed continuously and interrupted transfers leave resumable partials behind, a service killed outright still resumes correctly on restart. This makes it safe to run under systemd:
[Service]
ExecStart=/usr/local/bin/magnet acme --daemon --interval 6h --output /backups
Environment=GITHUB_TOKEN=ghp_...
Restart=always
KillSignal=SIGTERMA single run is suitable for cron directly. Use --quiet so only errors reach the mail spool, and --log-file to keep a record:
0 2 * * * /usr/local/bin/magnet acme --quiet --output /backups --log-file /var/log/magnet.logExit code 2 indicates that some repositories failed and will be retried on the next run.
Listing an account places every repository under one directory:
<output>/<account>/
.magnet.json
repository-one/
README.md
src/
repository-two/
Search results span accounts and are nested by owner to keep repositories with the same name apart:
<output>/
.magnet.json
alice/
parser/
bob/
parser/
Each repository directory contains the extracted default-branch archive with GitHub's wrapper directory removed. Unix permission bits, including the executable bit, are preserved.
Repositories are installed by renaming a fully extracted staging directory into place, so a directory in the output is always a complete snapshot. An interrupted run leaves no partial directories behind, which allows the next run to trust what is already present.
| Code | Meaning |
|---|---|
| 0 | Everything requested succeeded |
| 1 | The run could not be completed: bad arguments, unknown account, exhausted rate limit |
| 2 | The run completed, but at least one repository failed |
Progress is written to stdout; warnings and failures are written to stderr.
Rate limiting. Magnet checks the remaining quota before starting and stops immediately if the window is exhausted, rather than issuing requests that will all fail. A 403 or 429 with the primary quota exhausted is reported immediately, since it cannot clear before the window resets. Secondary abuse detection limits and 5xx responses are retried with exponential backoff, honoring Retry-After when provided.
Private repositories are not supported. Magnet lists /users/{account}/repos, which returns only public repositories regardless of token.
Empty repositories have no downloadable archive and are reported as individual failures; the rest of the run continues.
Symlinks inside archives are skipped rather than recreated, since a symlink entry can point outside the destination and allow a later entry to be written through it. Entries whose paths would escape the destination are dropped as well. Both are counted and reported under --verbose.
GitHub Enterprise is supported by pointing --api-base and --archive-base at the target instance.
cargo test # unit tests plus every integration suite
cargo test --lib # unit tests only
cargo test --test sync # one suite: sync, search, resume, daemon, integration, style
cargo test extraction # a single test by name substring
cargo clippy --all-targets -- -D warnings
cargo fmtThe crate is split into a library and a thin binary so the full pipeline can be driven from tests. Integration suites run against an in-process stub of the GitHub API (tests/support/mod.rs) and never touch the network.
| Module | Responsibility |
|---|---|
| cli | Argument parsing; turns flags into a filter, client config, and download options |
| github | REST client, pagination, search, retry policy, streaming and resumable downloads |
| search | Typed search qualifiers rendered into a GitHub search expression |
| filter | Client-side selection, with a reason recorded for every rejection |
| download | Bounded-concurrency orchestration, scratch files, atomic install |
| archive | Zip validation and extraction |
| manifest | Record of what was downloaded, used for change detection |
| daemon | Interval scheduler for unattended backups |
| log | Timestamped log lines to a file and optionally stderr |
| shutdown | Cooperative cancellation shared across tasks |
| report | Verbosity-aware output |
| error, time, util | Error types, RFC 3339 timestamps, formatting helpers |
Two conventions apply throughout: the codebase carries no comments, and long dashes are not used in source, documentation, or program output; this is enforced by tests/style.rs. Spelling follows American English.
Running the binary from the repository root writes downloaded repositories into the working tree, which is not gitignored. Use --output to write elsewhere.
GPL-3.0-or-later. See LICENSE.
Magnet downloads publicly available material through GitHub's documented interfaces and bypasses no access control. Users remain responsible for complying with GitHub's Terms of Service and with the licenses of whatever they download.