Skip to content

Detailed Workflow

Jump to bottom
Andreas edited this page Nov 18, 2024 · 1 revision
workflow (2)

GitHub Actions manage the scheduling of cron jobs across the data infrastructure. The process begins with data collection (scraper), followed by data processing (summarizer) and automated topics extraction and vector embeddings (llm-fine-tuning). All workflows are documented in their respective project README files.

Daily Processes (UTC)

  • 00:00 – Scrape mailing lists and Delving Bitcoin (scraper)
  • 00:30 – Update author synonyms (scraper)
  • 01:00 – Generate summaries and XML (summarizer)
  • 02:00 – Push summaries to Elasticsearch (summarizer)
  • 02:30 – Push combined summaries to Elasticsearch (summarizer)
  • 03:00 – Generate homepage.json (summarizer)
  • 03:30 – Generate newsletter.json (summarizer), extract topics (llm-fine-tuning)
  • 04:00 – Update submodule (tldr), push topics and vector embeddings to Elasticsearch (llm-fine-tuning)

Weekly Processes

  • Monday 12:00 PM – Send newsletter (tldr)
  • Sunday 00:00 AM – Scrape Bitcoin StackExchange (currently not working)
  • Wednesday 01:00 PM – Scrape Bitcoin Talk Forum, Bitcoin Transcripts, and Bitcoin Optech (scraper)

The Bitcoin Transcripts workflow is covered for completeness in the bottom left of the diagram. A detailed breakdown of its workflow is available here.

Clone this wiki locally