Media-WebScraper
This program scrapes information and images for movies and TV shows.
Summary
For more information on the program, read the WebScrape_help text file (this can also be accessed while running the program).
For a given list of media, the program will scrape and save general information, images and any episode information for each media.
General Information (default):
Saved as a .txt file
This will scrape general information:
- Title
- Release date
- Runtime
- Genre
- Director
- Cast
- Plot description
Additional information saved:
- Source database used for scrape
- ID for media in source database
- Poster image link
Images (default):
Saved as a .jpg file
This will scrape the poster.
Episode Information (if specified):
Saved as a .csv file
This will scrape information for each episode for a TV show:
- Season number
- Episode number
- Episode title
- Episode air date
- Episode description
Features:
- Multithreaded scraping for media in list to greatly improve the time taken when scraping for large media lists.
- Can generate a media list from folders and files in a specified directory or from user input.
- Can specify save location for scraped data.
- Can specify search tags for media list for a more accurate scrape.
- Can choose to scrape all episode information for a TV show.
- Can detect if data is already scraped which allows for scraping new media from an already scraped list of media very efficient.
- Can recover missing scraped files if one or more are missing without rescraping all data.
- Can retry the scrape before exiting the program if there were any incomplete scrapes (successfully scraped files will not be altered or rescraped).
- Currently only supports scraping data from IMDb.
Usage:
For more information on the program, read the WebScrape_help text file (this can also be accessed while running the program).
Currently a terminal-based program.
Running the program using python:
- Requirements: Python 3.2+ (additional libraries: requests, beautifulsoup4)
Running the program from bundled executable file (created using pyinstaller):
- Requirements: Windows 10
- Creates a ‘temp’ folder containing extracted libraries and support files in the same location as the program while running.
- The temporary files will delete automatically but if the program is closed abruptly, the files will remain.
- The ‘temp’ folder can be manually deleted after closing the program.
- (As of pyinstaller v4.7, a one-file bundled executable will leave any temp ‘_MEIxxxxxx’ folders if the program is force closed)
Updates:
For information on version history, read the HISTORY markdown file.
