Skip to content

fix(ignore-io): use GitHub gitignore templates - #1273

Open
vjymisal0 wants to merge 4 commits into
tj:mainfrom
vjymisal0:fix/gitignore-io-cloudflare
Open

vjymisal0 wants to merge 4 commits into
tj:mainfrom
vjymisal0:fix/gitignore-io-cloudflare

Conversation

@vjymisal0

Copy link
Copy Markdown
Contributor

Summary

git-ignore-io currently fetches templates through the Toptal-hosted gitignore.io endpoint, which can reject requests with Cloudflare. Use GitHub's public gitignore repository as the source instead. Template names are resolved case-insensitively so existing cached lists continue to work.

  • Fetch the template list from GitHub's Contents API.
  • Fetch individual templates from raw.githubusercontent.com.
  • Preserve multi-template requests and update generated documentation.

Test plan

  • bash -n bin/git-ignore-io
  • ./check_integrity.sh ignore-io
  • Manually fetched Python and Java templates, including from a legacy lowercase cache.

Fixes #1272

@spacewander spacewander left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's the AI review result:


The PR replaces gitignore.io with GitHub's github/gitignore repo in git-ignore-io. The reviewer verified against live GitHub endpoints and found the core idea works, but two issues block merge:

High — Global/ templates are broken (silent 404s):

  • ~70 of the most-used templates (Vim, Emacs, macOS, Windows, JetBrains, Xcode, VirtualEnv, etc.) live in the Global/ subdirectory, not the repo root
  • The PR only lists/fetches root-level *.gitignore files, so git ignore-io vim — the exact example in the updated man page — silently returns nothing
  • curl -fsSL swallows the 404, so users get zero output, no error
  • This is a regression: gitignore.io previously served all of these

Medium — rate limits & silent failures:

  • Unauthenticated GitHub Contents API = 60 req/hour, and gi() now hits it on every invocation even with a valid cache
  • update_gi_list writes an empty cache file on failure, breaking -l/-L/-s until next successful -u
  • git ignore-io -r typo truncates .gitignore to empty with no warning (pre-existing risk, now more likely due to silent 404s)

Minor:

  • man/git-ignore-io.md example still shows the old toptal URL (would revert on regeneration); the updated .1/.html example URL is itself a 404
  • "Initial gitignore.io list" banner at bin/git-ignore-io:86 not updated

Proposed Fix Plan

  1. Include Global/ templates: switch to the git trees API (git/trees/main?recursive=1) in one call, filter for *.gitignore, and store each file's full path (or build URLs from path) so both root and Global/ templates resolve
  2. Respect the cache: only call the API when the cache is missing/expired or -u is passed, avoiding the 60 req/hour ceiling
  3. Error handling: on curl failure, print an error to stderr and skip cache truncation / refuse -r write instead of silently emptying files
  4. Docs: update the .md example, fix the 404 example URL (use e.g. Global/Vim.gitignore), and update the init banner string

One tradeoff to confirm: trees API responses are large (~1MB for the whole repo) — acceptable for a one-off -u refresh, which is why I'd cache and only fetch on demand.

Signed-off-by: vjymisal0 <misalvijay153@gmail.com>
@vjymisal0

Copy link
Copy Markdown
Contributor Author

Thanks for the detailed review @spacewander. I switched list refreshes to the recursive git tree API so Global/* templates such as Global/Vim are cached and fetched correctly, and template generation now uses the cache instead of hitting GitHub on every invocation. Refreshes are atomic and report failures without overwriting a valid cache; --replace/--append also avoid modifying files when a requested template cannot be fetched. Updated the generated man pages and init banner. Verified with live -u, vim, and missing-template scenarios, plus bash syntax and diff checks. Pushed as bb9fb05.

@spacewander spacewander left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Main issue (must fix before merge)

Legacy caches resolve but then 404 — the PR's stated goal of keeping existing cached lists working isn't met:

  • raw.githubusercontent.com is case-sensitive (Python.gitignore → 200, python.gitignore → 404)
  • Legacy gitignore.io caches store all-lowercase names (python, vim, ...), so lookup succeeds but fetch 404s
  • Stale/invalid caches are never auto-repaired — check_list_exist only refreshes missing files; a user whose last update_gi_list failed under old code has a cache full of Cloudflare HTML that's never refreshed

Proposed fix plan (for bin/git-ignore-io)

  1. Legacy/invalid cache detection + refresh: in check_list_exist, treat the cache as invalid if it doesn't look like a GitHub template list (e.g. single comma-joined line, or no valid path entries) and force re-download. This fixes both lowercase legacy caches and HTML-poisoned caches in one shot.
  2. Fallback retry on 404 (defense in depth): after resolving a name, if the exact-case fetch 404s, retry with the correct-cased name from a refreshed list.

Comment thread bin/git-ignore-io Outdated
Comment thread bin/git-ignore-io

@spacewander spacewander left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Before I could merge it, could you update https://github.com/tj/git-extras/blob/main/Commands.md#git-ignore-io too?

Signed-off-by: vjymisal0 <misalvijay153@gmail.com>
@vjymisal0

Copy link
Copy Markdown
Contributor Author

Thanks @spacewander! I updated Commands.md to reference the GitHub gitignore templates and corrected the git ignore-io vim example to use the Global/Vim template URL and current template content. Pushed as cb859c0 with DCO sign-off; git diff --check passes.

@vjymisal0

Copy link
Copy Markdown
Contributor Author

Thanks @spacewander for the detailed review. The branch now uses the recursive GitHub tree endpoint, includes Global templates, refreshes invalid/legacy caches safely, preserves the cache between invocations, handles case-insensitive template lookup and 404 failures, and updates the generated documentation examples. I verified the branch is clean and pushed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Request sent by git-ignore-io is rejected by CloudFlare

2 participants