Skip to content

Latest commit

 

History

26 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

made with rust

aws-crawler

CLI program in Rust that crawls a website using the Spider crate and uploads the crawled data to an S3 bucket using the AWS SDK S3 crate.

How-to

To use aws-crawler:

cargo run website_to_crawl bucket_name region

Please double-check that you have correctly set up your AWS credentials, and that you have the necessary permissions to create and write to an S3 bucket in the specified region.

About

Crawl websites and upload crawled data to S3

Topics

Resources

Stars

7 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages