This repository has code to crawl a standalone website and export the static content into text format. I created this script to index the pages into a vector database I used Python 3.12

pip install scrapy

0. This is a extended version of Scrapy Start project template
1. Create a crawled_data directory in the root project
2. run 'scrapy crawl tn'