| Ticket UUID: | 1510449 | |||
| Title: | Feature request: Robots exclusion rules parser/checker | |||
| Type: | RFE | Version: | None | |
| Submitter: | nobody | Created on: | 2006-06-22 08:25:22 | |
| Subsystem: | None | Assigned To: | nobody | |
| Priority: | 5 Medium | Severity: | ||
| Status: | Open | Last Modified: | 2006-07-03 18:23:11 | |
| Resolution: | Closed By: | |||
| Closed on: | ||||
| Description: |
It would be nice to see an extension that allows parsing of a webserver's robots.txt file to see if our client is allowed to access a specific resource. Such an extension should return a boolean value. This feature would prove very useful when making web spiders (in general, web clients requiring no user interaction). | |||
| User Comments: |
mic42 added on 2006-07-03 18:23:11:
Logged In: YES user_id=302287 I have partial code that does this. I'll see if i can beat it into usable shape. | |||
Home
Blog
Docs
Downloads
Tickets
Timeline
Branches
Tags
Wiki