Serhiy Storchaka cb7ef18d70 gh-88375, gh-111788: Fix parsing errors and normalization in robotparser (GH-138502) ...

* Don't fail trying to parse weird patterns.
* Don't fail trying to decode non-UTF-8 "robots.txt" files.
* No longer ignore trailing "?" in patterns and URLs.
* Distinguish raw special characters "?", "=" and "&" from the
  percent-encoded ones.
* Remove tests that do nothing.

2025-09-05 18:58:42 +03:00

16 KiB

Raw Blame History

View Raw