Introduction

A dead simple crawler to get books information from Douban.

Pre-requesites

  • Python 3
  • Install dependencies from requirements.txt
  • (Optional) Install Anaconda to handle environment

Usage

  1. Run get_tags to fetch all the trending tags.

# This will generate a file tags.csv
python app.py get_tags
  1. Run crawl_books to start crawling the books by the tags from the previous step.
python app.py crawl_books -i tags.csv

Certainly, you can create the tags.csv without using the get_tags script. You might want to make sure the tags you specified can lead to any actual result of books.

License

MIT © mogita

GitHub

View Github