Full-featured web UI for monitoring and controlling Scrapyd servers cluster

ScrapydWeb

Full-featured web UI for monitoring and controlling Scrapyd servers cluster, with Scrapy log analysis and visualization supported.

Features

• Multinode Scrapyd Servers

  • Group, filter and select any numbers of nodes
  • Execute command on multinodes with one click

• Scrapy Log Analysis

  • Stats collection
  • Progress visualization
  • Logs categorization

• All Scrapyd API Supported

  • Deploy project, Run Spider, Stop job
  • List projects/versions/spiders/running_jobs
  • Delete version/project

• Others

  • Basic auth for web UI
  • Accessing Scrapyd servers protected by basic auth

Maintainer

• my8100
• simplety (Front-End)

Installation

To install ScrapydWeb, simply use pip:

pip install scrapydweb

Start Up

1. Run scrapydweb -h to get help,
   and a config file named scrapydweb_settings.py would be copied to the working directory,
   then you can custom config with it.
2. Run scrapydweb
3. Visit http://127.0.0.1:5000 (It's recommended to use Google Chrome to get better experience.)

Screenshot

• Overview
  

• Dashboard
  

• Log Analysis

  • Stats collection
    

  • Progress visualization
    

  • Logs categorization
    

• Deploy Project
  

• Run Spider
  

• Manage Projects
  

GitHub