Tcl Library Source Code

View Ticket
Login
Ticket UUID: 1510449
Title: Feature request: Robots exclusion rules parser/checker
Type: RFE Version: None
Submitter: nobody Created on: 2006-06-22 08:25:22
Subsystem: None Assigned To: nobody
Priority: 5 Medium Severity:
Status: Open Last Modified: 2006-07-03 18:23:11
Resolution: Closed By:
    Closed on:
Description:
It would be nice to see an extension that allows
parsing of a webserver's robots.txt file to see if our
client is allowed to access a specific resource. Such
an extension should return a boolean value.

This feature would prove very useful when making web
spiders (in general, web clients requiring no user
interaction).
User Comments: mic42 added on 2006-07-03 18:23:11:
Logged In: YES 
user_id=302287

I have partial code that does this. I'll see if i can beat
it into usable shape.