Script that crawls meta data from ICLR OpenReview webpage. Tutorials on installing and using Selenium and ChromeDriver on Ubuntu.
gazpacho is a web scraping library. It replaces requests and BeautifulSoup for most projects. gazpacho is small, simple, fast, and consistent. You should use it!
scrapes medias, likes, followers, tags and all metadata. Inspired by instagram-php-scraper
The web is full of data. Transistor is a web scraping framework for collecting, storing, and using targeted data from structured web pages.
Crawl and Visualize ICLR 2019 OpenReview Data.
A self-hosted tool which manages your YouTube subscriptions, and downloads files automatically.
High Performance Python Data driven programming framework for Web Crawler,ETL,Data pipeline work.
Give it a blockchain based crypto wallet address and it will crawl 3 levels deep in transaction data to plot a graph out of the information.
Script requires Python3.5 Audio on wicket fallright now only works in Linux
Web crawling framework for everyone. Written with asyncio, uvloop and aiohttp.
A web crawler written in python.
An intelligent proxy pool for humanities, only supports Python 3.6.
Photon is a lightning fast web crawler which extracts URLs, files, intel & endpoints from a target.
Read a packet capture, extract HTTP requests and turn them into cURL commands for replay.
Spidy (/spˈɪdi/) is the simple, easy to use command line web crawler.
Cryptocurrency historical market price data scraper written in Python.
OSINT Tool: Generate username lists from companies on LinkedIn.
Python Script for 'searching' and 'downloading' hundreds of Google images to the local hard disk!
With Grab you can build web scrapers of various complexity, from simple 5-line scripts to complex asynchronous website crawlers processing millions of web pages.
MechanicalSoup automatically stores and sends cookies, follows redirects, and can follow links and submit forms.
PyQuery-based scraping micro-framework. Supports Python 2.x and 3.x.
Cola is a high-level distributed crawling framework, used to crawl pages and extract structured data from websites.