| Ticket UUID: | 1510449 | |||
| Title: | Feature request: Robots exclusion rules parser/checker | |||
| Type: | RFE | Version: | None | |
| Submitter: | nobody | Created on: | 2006-06-22 08:25:22 | |
| Subsystem: | None | Assigned To: | aku | |
| Priority: | 5 Medium | Severity: | Minor | |
| Status: | Closed | Last Modified: | 2026-01-29 18:54:43 | |
| Resolution: | Not Applicable Here | Closed By: | m.n.summerfield | |
| Closed on: | 2026-01-29 18:54:43 | |||
| Description: |
It would be nice to see an extension that allows parsing of a webserver's robots.txt file to see if our client is allowed to access a specific resource. Such an extension should return a boolean value. This feature would prove very useful when making web spiders (in general, web clients requiring no user interaction). | |||
| User Comments: |
m.n.summerfield added on 2026-01-29 18:54:43:
Parsing robots.txt seems inappropriate for Tklib. (However, it might be considered appropriate for Tcllib.) mic42 added on 2006-07-03 18:23:11: Logged In: YES user_id=302287 I have partial code that does this. I'll see if i can beat it into usable shape. | |||
Home
Blog
Docs
Downloads
Tickets
Timeline
Branches
Tags
Wiki