Welcome to the official web crawler repository for Burplist built using Scrapy.
Growing up in a frugal family, I would spend hours browsing online, looking for the best bang for my bucks. Needless to say, the process was super exhausting and slowly turns into frustration.
So then I thought, why not just create a search engine for craft beers?
This software is only used for research purposes, users must abide by the relevant laws and regulations of their location, please do not use it for illegal purposes. The user shall bear all the consequences caused by illegal use.
- 10+ unique spiders for top craft beer sites in Singapore
- Sentry integration
- ScrapeOps integration
- Scraper API for proxy requests
- Automated random user agent rotation
- Colored logging
- Data deduplication pipeline
- Database migration with Alembic
- Delayed requests middleware
See this documentation on how to use Burplist.
For guidance on setting up a development environment and how to make a contribution, read the contributing guidelines.