This is the command htdig that can be run in the OnWorks free hosting provider using one of our multiple free online workstations such as Ubuntu Online, Fedora Online, Windows online emulator or MAC OS online emulator
PROGRAM:
NAME
htdig - retrieve HTML documents for ht://Dig search engine
SYNOPSIS
htdig [options]
DESCRIPTION
Htdig retrieves HTML documents using the HTTP protocol and gathers information from these
documents which can later be used to search these documents. This program can be referred
to as the search robot.
OPTIONS
- Get the list of URLs to start indexing from standard input. This will override the
default parameter start_url specified in the config file and the file supplied to
the -m option.
-a Use alternate work files. Tells htdig to append .work to database files, causing a
second copy of the database to be built. This allows the original files to be used
by htsearch during the indexing run.
-c configfile
Use the specified configfile instead of the default.
-h maxhops
Restrict the dig to documents that are at most maxhops links away from the starting
document. This only works if option -i is also given.
-i Initial. Do not use any old databases. Old databases will be erased before running
the program.
-m filename
Minimal run. Only index the URLs given in the file filename, ignoring all others.
URLs in the file should be formatted one URL per line.
-s Print statistics about the dig after completion.
-t Create an ASCII version of the document database. This database is easy to parse
with other programs so that information can be extracted from it for purposes other
than searching. One could gather some interesting statistics from this database.
Fieldname Value
u URL
t Title
a State
(0 normal, 1 not found, 2 not indexed, 3 obsolete)
m Time of last modification reported by the server
s Document Size in bytes
H Excerpt of the document
h Meta Description
l Time of last retrieval
L Count of links in the document or of outgoing links
b Number of links to the document, also called
incoming links or backlinks
c Hop count of this document
g Signature of this document
(used to detect duplicates)
e E-Mail address to use for a notification from htnotify
n Date on which such notification is sent
S Subject of the notifcation message
d The text of Incoming links pointing to this document
(e.g. <a href="/docURL">description</a>)
A Anchors in the document (i.e. <A NAME=...)
-u username:password
Tells htdig to send the supplied username and password with each HTTP request. The
credentials will be encoded using the ´Basic´ authentication method. There HAS to
be a colon (:) between the username and password.
-v Verbose mode. This increases the verbosity of the program. Using more than 2 is
probably only useful for debugging purposes. The default verbose mode (using only
one -v) gives a nice progress report while digging. Please consult the section
below on the exact format of the progress report.
FORMAT OF THE PROGRESS REPORT GIVEN IN VERBOSE MODE
A line is shown for each URL, with 3 numbers before the URL and some symbols after
the URL. The first number is the number of documents parsed so far, the second is
the DocID for this document, and the third is the hop count of the document (number
of hops from one of the start_url documents). Signification of the symbols printed
after the url:
"*" is printed for a link already visited
"+" is printed for a new link just queued
"-" is output for a link rejected for any of a number of reasons. To find out what
those reasons are, you need to run htdig with at least 3 -v options, i.e. -vvv.
If there are no "*", "+" or "-" symbols after the URL, it doesn't mean the document was
not parsed or was empty, but only that no links to other documents were found within it.
With more verbose output, these symbols will get interspersed in several lines of
debugging output.
FILES
/etc/htdig/htdig.conf
The default configuration file.
Use htdig online using onworks.net services