Video Games Web Scraper
Video Games Web Scraper is a project that crawls websites and APIs and extracts video game related data from their pages.
This project uses an open-source and collaborative framework named Scrapy.
Sources
- VideoGameGeek (
vgg)
Installation
I strongly recommend that you install this project in a dedicated virtual environment to avoid conflicting with your system packages.
See Virtual Environments and Packages on how to create and use your virtual environment.
Use the package manager pip to install the requirements of this project.
pip install -r requirements.txt
Usage
You can start crawling a source using a spider.
scrapy crawl <spider>
VideoGameGeek
Spiders
vgg-gamesvgg-hotitems
Developer Resources
Initialize your Development Environment
pip install -r requirements.txt
Create and Run Tests
See the Spiders Contracts for more instructions on how to create tests for spiders and then run:
scrapy check
Scrapy Documentation
See the Scrapy Documentation for more instructions on how to create and modify spiders.
Contributing
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
Please make sure to update tests as appropriate.
Support
If you enjoy this repository, please star this repository. By starring a repository, it shows appreciation to the repository maintainer for their work. Many of GitHub's repository rankings depend on the number of stars a repository has.
License
albert-marrero
