Tk Library Source Code

View Ticket
Login
Ticket UUID: 1510449
Title: Feature request: Robots exclusion rules parser/checker
Type: RFE Version: None
Submitter: nobody Created on: 2006-06-22 08:25:22
Subsystem: None Assigned To: aku
Priority: 5 Medium Severity: Minor
Status: Closed Last Modified: 2026-01-29 18:54:43
Resolution: Not Applicable Here Closed By: m.n.summerfield
    Closed on: 2026-01-29 18:54:43
Description:
It would be nice to see an extension that allows
parsing of a webserver's robots.txt file to see if our
client is allowed to access a specific resource. Such
an extension should return a boolean value.

This feature would prove very useful when making web
spiders (in general, web clients requiring no user
interaction).
User Comments: m.n.summerfield added on 2026-01-29 18:54:43:
Parsing robots.txt seems inappropriate for Tklib. (However, it might be considered appropriate for Tcllib.)

mic42 added on 2006-07-03 18:23:11:
Logged In: YES 
user_id=302287

I have partial code that does this. I'll see if i can beat
it into usable shape.