./www/p5-WWW-Robot [configurable web traversal engine]
[+] Add this package to your ports tracker

[ CVSweb ] [ Homepage ] [ RSS feed ]

Version: 0.026, Package name: p5-WWW-Robot-0.026
Maintained by: The OpenBSD ports mailing-list
Master sites:
Description
This module implements a configurable web traversal engine, for a robot
or other web agent. Given an initial web page (URL), the Robot will get
the contents of that page, and extract all links on the page, adding
them to a list of URLs to visit.

Features of the Robot module include:

* Follows the Robot Exclusion Protocol.
* Supports the META element proposed extensions to the Protocol.
* Implements many of the Guidelines for Robot Writers.
* Configurable.
* Builds on standard Perl 5 modules for WWW, HTTP, HTML, etc.


Filesize: 18.106 KB
Version History (View Complete History)
  • (2011-03-18) Updated to version: p5-WWW-Robot-0.026
  • (2007-04-13) Package added to openports.se, version p5-WWW-Robot-0.025 (created)
[show/hide] View available PLISTS (Can be a lot of data)

CVS Commit History:

   2016-03-20 13:57:24 by Christian Weisgerber | Files touched by this commit (1617)
Log message:
Remove SHARED_ONLY from all CPAN Perl ports and the cpan and perl MODULES.
Add WANTLIB+=perl and PKG_ARCH=* as appropriate.
   2015-01-17 20:16:09 by Christian Weisgerber | Files touched by this commit (3349)
Log message:
Drop remaining MD5/RMD160/SHA1 checksums.
   2011-03-18 03:27:22 by Jasper Lievisse Adriaanse | Files touched by this commit (3)
Log message:
- update to 0.26
- remve USE_GROFF
- set MODCPAN_EXAMPLES
from gleydson soares
   2011-03-18 03:27:22 by Jasper Lievisse Adriaanse | Files touched by this commit (3)
Log message:
- update to 0.26
- remve USE_GROFF
- set MODCPAN_EXAMPLES
from gleydson soares
   2011-03-18 03:27:22 by Jasper Lievisse Adriaanse | Files touched by this commit (3)
Log message:
- update to 0.26
- remve USE_GROFF
- set MODCPAN_EXAMPLES
from gleydson soares
   2010-12-03 04:46:54 by Antoine Jacoutot | Files touched by this commit (371)
Log message:
Bump after recent cpan.port.mk change.
ok jasper@
(final commit)
   2010-11-22 01:37:07 by Marc Espie | Files touched by this commit (859)
Log message:
new depends
   2010-10-19 02:02:58 by Marc Espie | Files touched by this commit (243)
Log message:
USE_GROFF=Yes
   2007-09-15 14:38:27 by Michael Erdely | Files touched by this commit (258)
Log message:
Remove surrounding quotes from
COMMENT-*/ERRORS/NO_REGRESS/PERMIT_*/REGRESS_IS_INTERACTIVE
Change '.include "bsd.port.mk"' to '.include <bsd.port.mk>' while here
(ok naddy@)
   2007-04-05 11:26:27 by Marc Espie | Files touched by this commit (815)
Log message:
more base64 checksums