This is the Linux app named crwlr whose latest release can be downloaded as v1.2.2.zip. It can be run online in the free hosting provider OnWorks for workstations.
Download and run online this app named crwlr with OnWorks for free.
Follow these instructions in order to run this app:
- 1. Downloaded this application in your PC.
- 2. Enter in our file manager https://www.onworks.net/myfiles.php?username=XXXXX with the username that you want.
- 3. Upload this application in such filemanager.
- 4. Start the OnWorks Linux online or Windows online emulator or MACOS online emulator from this website.
- 5. From the OnWorks Linux OS you have just started, goto our file manager https://www.onworks.net/myfiles.php?username=XXXXX with the username that you want.
- 6. Download the application, install it and run it.
SCREENSHOTS
Ad
crwlr
DESCRIPTION
This library provides kind of a framework and a lot of ready-to-use, so-called steps, that you can use as building blocks, to build your own crawlers and scrapers with. Before diving into the library, let's have a look at the terms crawling and scraping. For most real-world use cases, those two things go hand in hand, which is why this library helps with and combines both. A (web) crawler is a program that (down)loads documents and follows the links in it to load them as well. A crawler could just load actually all links it is finding (and is allowed to load according to the robots.txt file), then it would just load the whole internet (if the URL(s) it starts with are no dead end). Or it can be restricted to load only links matching certain criteria (on same domain/host, URL path starts with "/foo",...) or only to a certain depth. A depth of 3 means 3 levels deep. Links found on the initial URLs provided to the crawler are level 1 and so on.
Features
- Crawler Politeness (respecting robots.txt, throttling,...)
- Get absolute links from HTML documents
- Get sitemaps from robots.txt and get all URLs from those sitemaps
- Crawl (load) all pages of a website
- Use any HTTP methods (GET, POST,...) and send any headers or body
- Iterate over paginated list pages
Programming Language
PHP
Categories
This is an application that can also be fetched from https://sourceforge.net/projects/crwlr.mirror/. It has been hosted in OnWorks in order to be run online in an easiest way from one of our free Operative Systems.
