/ Data Visualization

Full-featured web UI for monitoring and controlling Scrapyd servers cluster

Full-featured web UI for monitoring and controlling Scrapyd servers cluster

ScrapydWeb

Full-featured web UI for monitoring and controlling Scrapyd servers cluster, with Scrapy log analysis and visualization supported.

Features

  • Multinode Scrapyd Servers

    • Group, filter and select any numbers of nodes
    • Execute command on multinodes with one click
  • Scrapy Log Analysis

    • Stats collection
    • Progress visualization
    • Logs categorization
  • All Scrapyd API Supported

    • Deploy project, Run Spider, Stop job
    • List projects/versions/spiders/running_jobs
    • Delete version/project
  • Others

    • Basic auth for web UI
    • Accessing Scrapyd servers protected by basic auth

Maintainer

Installation

To install ScrapydWeb, simply use pip:

pip install scrapydweb

Start Up

  1. Run scrapydweb -h to get help,
    and a config file named scrapydweb_settings.py would be copied to the working directory,
    then you can custom config with it.
  2. Run scrapydweb
  3. Visit http://127.0.0.1:5000 (It's recommended to use Google Chrome to get better experience.)

Screenshot

  • Overview
    overview

  • Dashboard
    dashboard

  • Log Analysis

    • Stats collection
      log_stats

    • Progress visualization
      log_chart

    • Logs categorization
      log_extracted

  • Deploy Project
    deploy

  • Run Spider
    run

  • Manage Projects
    manage

GitHub