This is the Linux app named Scrapy-Redis whose latest release can be downloaded as Version0.7.1Released.zip. It can be run online in the free hosting provider OnWorks for workstations.
Download and run online this app named Scrapy-Redis with OnWorks for free.
Follow these instructions in order to run this app:
- 1. Downloaded this application in your PC.
- 2. Enter in our file manager https://www.onworks.net/myfiles.php?username=XXXXX with the username that you want.
- 3. Upload this application in such filemanager.
- 4. Start the OnWorks Linux online or Windows online emulator or MACOS online emulator from this website.
- 5. From the OnWorks Linux OS you have just started, goto our file manager https://www.onworks.net/myfiles.php?username=XXXXX with the username that you want.
- 6. Download the application, install it and run it.
SCREENSHOTS
Ad
Scrapy-Redis
DESCRIPTION
You can start multiple spider instances that share a single redis queue. Best suitable for broad multi-domain crawls. Scraped items gets pushed into a redis queued meaning that you can start as many as needed post-processing processes sharing the items queue. Scheduler + Duplication Filter, Item Pipeline, Base Spiders. Default requests serializer is pickle, but it can be changed to any module with loads and dumps functions. Note that pickle is not compatible between python versions. Version 0.3 changed the requests serialization from marshal to cPickle, therefore persisted requests using version 0.2 will not able to work on 0.3. The class scrapy_redis.spiders.RedisSpider enables a spider to read the urls from redis. The urls in the redis queue will be processed one after another, if the first request yields more requests, the spider will process those requests before fetching another url from redis.
Features
- Distributed crawling/scraping
- Distributed post-processing
- Scrapy plug-and-play components
- Python 2.7, 3.4 or 3.5 required
- Redis >= 2.8 required
- Scheduler + Duplication Filter, Item Pipeline, Base Spiders
Programming Language
Python
This is an application that can also be fetched from https://sourceforge.net/projects/scrapy-redis.mirror/. It has been hosted in OnWorks in order to be run online in an easiest way from one of our free Operative Systems.
