Dead example URL in urlib.robotparser documentation

[lexzlei]

Documentation

In the Python documentation for urllib.robotparser, the example currently references a page that is no longer available (musi-cal.com). The example code now points to an inactive website:

>>> import urllib.robotparser
>>> rp = urllib.robotparser.RobotFileParser()
>>> rp.set_url("http://www.musi-cal.com/robots.txt")
>>> rp.read()
>>> rrate = rp.request_rate("*")
>>> rrate.requests
3
>>> rrate.seconds
20
>>> rp.crawl_delay("*")
6
>>> rp.can_fetch("*", "http://www.musi-cal.com/cgi-bin/search?city=San+Francisco")
False
>>> rp.can_fetch("*", "http://www.musi-cal.com/")
True

Additionally, the current robots.txt file at http://www.musi-cal.com/robots.txt contains:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Because of this, both can_fetch() calls now return True, which doesn't align with the expected output from the example.

Proposed fix:
Update the example in urlib.robotparser.rst to replace the outdated musi-cal.com URL with a valid URL (e.g. https://www.python.org).

I would be happy to work on this issue and put together a PR for the update.

Linked PRs

• gh-141444: Doc: update urllib.robotparser example to use python.org robots.txt #141461

[hugovk]

Thanks for the report, a PR is welcome.

Yes, https://www.python.org/robots.txt sounds like a good replacement.

[lexzlei]

Thanks for linking PR #141461 . Since a fix is already in progress, I won’t submit a duplicate PR.