Skip to content

bpo-46337: Urllib.parse scheme-specific behavior without reliance on URL scheme #30520

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
oldaccountdeadname wants to merge 23 commits into from
Closed

bpo-46337: Urllib.parse scheme-specific behavior without reliance on URL scheme #30520

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
ee890ed
bpo-46337: Enumerate URL types for urllib's scheme-based behavior
oldaccountdeadname Jan 10, 2022
53c6ccc
bpo-46337: Allow caller modification of url classes
oldaccountdeadname Jan 10, 2022
41d3b58
📜🤖 Added by blurb_it.
blurb-it[bot] Jan 10, 2022
eee880c
bpo-46337: Fix grammar of doc comment.
oldaccountdeadname Feb 26, 2022
f9b59dd
bpo-46337: document SchemeClass behavior
oldaccountdeadname Feb 26, 2022
1691a1e
add newline to end of news file
oldaccountdeadname Feb 28, 2022
c7ae936
bpo-46337: fix doctest formatting
oldaccountdeadname Mar 12, 2022
4fc9059
doc: fix some doctests
oldaccountdeadname Mar 12, 2022
c07600c
bpo-43677: fixup PEP 8 style
oldaccountdeadname Mar 13, 2022
07a8576
Merge branch 'main' into urllib-custom-schemes
JelleZijlstra Mar 29, 2022
2c4aa3a
urllib: use an enum.FLAG for SchemeClass's.
oldaccountdeadname Mar 30, 2022
5f81d16
urllib: rename SchemeClass to SchemeFlag
oldaccountdeadname Mar 30, 2022
226bbe9
urllib: remove SchemeFlag.NONE
oldaccountdeadname Mar 31, 2022
ff88881
doc/urllib: remove old references to SchemeClass
oldaccountdeadname Mar 31, 2022
bf64df0
news: update news file to use SchemeFlag name
oldaccountdeadname Mar 31, 2022
6a09c38
doc/urllib: fix formatting and wording
oldaccountdeadname Mar 31, 2022
9d7cfb5
urllib: expose enums SchemeFlag variants directly
oldaccountdeadname Mar 31, 2022
81d3414
urllib: add UNIVERSAL SchemeFlag
oldaccountdeadname Mar 31, 2022
677ed1a
use None rather than SchemeFlag in public API
oldaccountdeadname Apr 21, 2022
0ec4a4e
do not import from enum
oldaccountdeadname Apr 21, 2022
b25e0e8
doc: correct urljoin signature
oldaccountdeadname Apr 21, 2022
2123ad7
make flags parameter keyword-only
oldaccountdeadname Apr 21, 2022
9f50dfb
s/classes/flags
oldaccountdeadname Apr 21, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
120 changes: 88 additions & 32 deletions Doc/library/urllib.parse.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,8 @@ Resource Locators. It supports the following URL schemes: ``file``, ``ftp``,
``gopher``, ``hdl``, ``http``, ``https``, ``imap``, ``mailto``, ``mms``,
``news``, ``nntp``, ``prospero``, ``rsync``, ``rtsp``, ``rtspu``, ``sftp``,
``shttp``, ``sip``, ``sips``, ``snews``, ``svn``, ``svn+ssh``, ``telnet``,
``wais``, ``ws``, ``wss``.
``wais``, ``ws``, ``wss``. The behavior of other schemes may be controlled with
a ``UrlFlag`` passed to dependent functions.

The :mod:`urllib.parse` module defines functions that fall into two broad
categories: URL parsing and URL quoting. These are covered in detail in
Expand All @@ -37,30 +38,35 @@ URL Parsing
The URL parsing functions focus on splitting a URL string into its components,
or on combining URL components into a URL string.

.. function:: urlparse(urlstring, scheme='', allow_fragments=True)
.. function:: urlparse(urlstring, scheme='', allow_fragments=True, *, flags=None)

Parse a URL into six components, returning a 6-item :term:`named tuple`. This
corresponds to the general structure of a URL:
``scheme://netloc/path;parameters?query#fragment``.
Each tuple item is a string, possibly empty. The components are not broken up
into smaller parts (for example, the network location is a single string), and %
Parse a URL into six components with respect to given scheme classes,
returning a 6-item :term:`named tuple`. This corresponds to the general
structure of a URL: ``scheme://netloc/path;parameters?query#fragment``. Each
tuple item is a string, possibly empty. The components are not broken up into
smaller parts (for example, the network location is a single string), and %
escapes are not expanded. The delimiters as shown above are not part of the
result, except for a leading slash in the *path* component, which is retained if
present. For example:
result, except for a leading slash in the *path* component, which is retained
if present.

The scheme of the URL determines whether or not parameters are parsed as
distinct from the path. To override the scheme and parse parameters anyway,
pass the corresponding SchemeFlag.

For example:

.. doctest::
:options: +NORMALIZE_WHITESPACE

>>> from urllib.parse import urlparse
>>> from urllib.parse import urlparse, PARAMS
>>> urlparse("scheme://netloc/path;parameters?query#fragment")
ParseResult(scheme='scheme', netloc='netloc', path='/path;parameters', params='',
query='query', fragment='fragment')
ParseResult(scheme='scheme', netloc='netloc', path='/path;parameters', params='', query='query', fragment='fragment')
>>> urlparse("scheme://netloc/path;parameters?query#fragment", flags=PARAMS)
ParseResult(scheme='scheme', netloc='netloc', path='/path', params='parameters', query='query', fragment='fragment')
>>> o = urlparse("http://docs.python.org:80/3/library/urllib.parse.html?"
... "highlight=params#url-parsing")
>>> o
ParseResult(scheme='http', netloc='docs.python.org:80',
path='/3/library/urllib.parse.html', params='',
query='highlight=params', fragment='url-parsing')
ParseResult(scheme='http', netloc='docs.python.org:80', path='/3/library/urllib.parse.html', params='', query='highlight=params', fragment='url-parsing')

@gpshead gpshead Apr 14, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do not reformat doctest examples. these were formatted to be narrow to avoid horizontal scrollbars in documentation on most common displays and to keep the .rst itself <80 columns when possible.

reformatting is unrelated to the change at hand and distracts from the actual change.

@oldaccountdeadname oldaccountdeadname Apr 21, 2022

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Restoring the original formatting causes the doctest to fail, I should've broken that out into a separate and clear commit... I'm doctesting with .python -m doctest. Is that wrong, or is there some other way I can keep the old linebreaks?

@ethanfurman ethanfurman Apr 21, 2022 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gpshead re behavior_overrides vs flags: aren't flags usually behavior overrides? ssl, socket, _pydecimal, _osx_support, and re all use flags, while doctest uses compileflags, _pyio use dec_flags, and subprocess uses creationflags.

My first choice here would be a simple flags, and it should be easily understood that the flags given will modify the parsing behavior of urlparse. Would it be more precise to call it uri_flags? At any rate, behavior_overrides is no less generic and much more verbose than flags.

>>> o.scheme
'http'
>>> o.netloc
Expand All @@ -82,14 +88,11 @@ or on combining URL components into a URL string.

>>> from urllib.parse import urlparse
>>> urlparse('//www.cwi.nl:80/%7Eguido/Python.html')
ParseResult(scheme='', netloc='www.cwi.nl:80', path='/%7Eguido/Python.html',
params='', query='', fragment='')
ParseResult(scheme='', netloc='www.cwi.nl:80', path='/%7Eguido/Python.html', params='', query='', fragment='')
>>> urlparse('www.cwi.nl/%7Eguido/Python.html')
ParseResult(scheme='', netloc='', path='www.cwi.nl/%7Eguido/Python.html',
params='', query='', fragment='')
ParseResult(scheme='', netloc='', path='www.cwi.nl/%7Eguido/Python.html', params='', query='', fragment='')
>>> urlparse('help/Python.html')
ParseResult(scheme='', netloc='', path='help/Python.html', params='',
query='', fragment='')
ParseResult(scheme='', netloc='', path='help/Python.html', params='', query='', fragment='')

The *scheme* argument gives the default addressing scheme, to be
used only if the URL does not specify one. It should be the same type
Expand Down Expand Up @@ -152,11 +155,9 @@ or on combining URL components into a URL string.
>>> from urllib.parse import urlparse
>>> u = urlparse('//www.cwi.nl:80/%7Eguido/Python.html')
>>> u
ParseResult(scheme='', netloc='www.cwi.nl:80', path='/%7Eguido/Python.html',
params='', query='', fragment='')
ParseResult(scheme='', netloc='www.cwi.nl:80', path='/%7Eguido/Python.html', params='', query='', fragment='')
>>> u._replace(scheme='http')
ParseResult(scheme='http', netloc='www.cwi.nl:80', path='/%7Eguido/Python.html',
params='', query='', fragment='')
ParseResult(scheme='http', netloc='www.cwi.nl:80', path='/%7Eguido/Python.html', params='', query='', fragment='')


.. versionchanged:: 3.2
Expand Down Expand Up @@ -348,19 +349,22 @@ or on combining URL components into a URL string.
with an empty query; the RFC states that these are equivalent).


.. function:: urljoin(base, url, allow_fragments=True)
.. function:: urljoin(base, url, allow_fragments=True, *, flags=None)

Construct a full ("absolute") URL by combining a "base URL" (*base*) with
another URL (*url*). Informally, this uses components of the base URL, in
particular the addressing scheme, the network location and (part of) the
path, to provide missing components in the relative URL. For example:
Construct a full ("absolute") URL by combining a "base URL"
(*base*) with another URL (*url*), and with behavior given by a
``SchemeFlag`` flag. Informally, this uses components of the base
URL, in particular the addressing scheme, the network location and
(part of) the path, to provide missing components in the relative
URL. For example:

>>> from urllib.parse import urljoin
>>> urljoin('http://www.cwi.nl/%7Eguido/Python.html', 'FAQ.html')
'http://www.cwi.nl/%7Eguido/FAQ.html'

The *allow_fragments* argument has the same meaning and default as for
:func:`urlparse`.
The *allow_fragments* argument has the same meaning and default as
for :func:`urlparse`. As in :func:`urlparse`, a ``SchemeFlag`` may
be given to override behavior inferred by the scheme.

.. note::

Expand Down Expand Up @@ -543,6 +547,58 @@ operating on :class:`bytes` or :class:`bytearray` objects:

.. versionadded:: 3.2

Special URL Behaviors and Scheme Flags
--------------------------------------

:mod:`urllib.parse` recognizes three special properties of URLs, namely relative
addressing (used in, for instance, the ``ftp``, ``http``, or ``gopher``
protocols), netloc-sensitive resolution (used in the ``ftp``, ``http``, or
``git`` protocols), and URLs that may contain parameters (for instance, ``ftp``
or ``telnet``).

Relative addressing allows resolution of relative URLs, and netloc-sensitive
addressing allows resolution with respect to the netloc (domain name) of a URL.
As HTTP URLs have both behaviors by default, this is demonstrated in the
following example:

>>> from urllib.parse import urljoin
>>> urljoin('http://example.org/post/x/', '../y/')
'http://example.org/post/y/'

Additionally, if it is not indicated that a URL is sensitive to parameters
(those specified after a semicolon in the path), then they'll be treated as part
of the path rather than as a distinct component.

Without specifying optional parameters or modifying global variables, Python
will guess what parameters to apply based on the scheme. Schemes associated with
each are specified by three lists in :mod:`urllib.parse`:

* ``urllib.parse.uses_relative``
* ``urllib.parse.uses_netloc``
* ``urllib.parse.uses_params``

In addition, any function that takes a ``flags`` parameter (for
instance, :func:`urlparse` and :func:`urljoin`) may override the
behavior of the ``uses`` lists, for instance, parsing a custom or
widely unused scheme with the same behavior as that of HTTP:

>>> from urllib.parse import urljoin, NETLOC, RELATIVE
>>> urljoin(
... 'my-protocol://example.org/post/x/', '../y/',
... flags=(NETLOC | RELATIVE))
'my-protocol://example.org/post/y/'

Also provided is the ``UNIVERSAL`` flag, which will use all
recognizable elements of a URL (``RELATIVE``, ``NETLOC``, and
``PARAMS``). It is exactly equivalent to the logical or of all other
flags.

For reference, the following scheme classes are present:

* ``urllib.parse.RELATIVE``
* ``urllib.parse.NETLOC``
* ``urllib.parse.PARAMS``
* ``urllib.parse.UNIVERSAL``

URL Quoting
-----------
Expand Down
11 changes: 8 additions & 3 deletions Lib/test/test_urlparse.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -213,12 +213,12 @@ def _encode(t):
split = (scheme,) + split
self.checkRoundtrips(url, parsed, split)

def checkJoin(self, base, relurl, expected):
def checkJoin(self, base, relurl, expected, flags=urllib.parse.SchemeFlag(0)):
str_components = (base, relurl, expected)
self.assertEqual(urllib.parse.urljoin(base, relurl), expected)
self.assertEqual(urllib.parse.urljoin(base, relurl, flags=flags), expected)
bytes_components = baseb, relurlb, expectedb = [
x.encode('ascii') for x in str_components]
self.assertEqual(urllib.parse.urljoin(baseb, relurlb), expectedb)
self.assertEqual(urllib.parse.urljoin(baseb, relurlb, flags=flags), expectedb)

def test_unparse_parse(self):
str_cases = ['Python', './Python','x-newscheme://foo.com/stuff','x://y','x:/y','x:/','/',]
Expand Down Expand Up @@ -417,6 +417,11 @@ def test_urljoins(self):
self.checkJoin('svn+ssh://pathtorepo/dir1', 'dir2', 'svn+ssh://pathtorepo/dir2')
self.checkJoin('ws://a/b','g','ws://a/g')
self.checkJoin('wss://a/b','g','wss://a/g')
self.checkJoin(

@gpshead gpshead Apr 14, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make these new SchemeFlag specific test methods instead of appending to an existing long one.

'nonsensebase://net.loc/url/', '..',
'nonsensebase://net.loc/',
flags=(urllib.parse.SchemeFlag.RELATIVE | urllib.parse.SchemeFlag.NETLOC),

@ethanfurman ethanfurman Mar 31, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove the SchemeFlag portion as its enums will be available at the global level.

For those that tend to forget, "global" means "module" in Python.

@gpshead gpshead Apr 14, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add more test cases that explicitly cover PARAMS and UNIVERSAL behaviors.

)

# XXX: The following tests are no longer compatible with RFC3986
# self.checkJoin(SIMPLE_BASE, '../../../g','http://a/../g')
Expand Down
72 changes: 63 additions & 9 deletions Lib/urllib/parse.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
"""

from collections import namedtuple
import enum
import functools
import re
import sys
Expand All @@ -39,12 +40,35 @@
"parse_qsl", "quote", "quote_plus", "quote_from_bytes",
"unquote", "unquote_plus", "unquote_to_bytes",
"DefragResult", "ParseResult", "SplitResult",
"SchemeFlag", "RELATIVE", "NETLOC", "PARAMS", "UNIVERSAL",

@gpshead gpshead Apr 14, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets not pollute __all__ with the CONSTANT_NAMES. People shouldn't really use from urllib.parse import * but if they do they shouldn't get these, just SchemeFlag.

@oldaccountdeadname oldaccountdeadname Apr 21, 2022

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ethanfurman thoughts? I remember you suggested that these should be exported as such for code like

urlparse(uri_string, flags=UNIVERSAL)

or similar. I'm fine either way, but do agree that the namespace would be cleaner were the flags not exported individually.

@ethanfurman ethanfurman Apr 21, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Putting them in globals() is not for the from ... import * case, since, as @gpshead said, folks should not be doing that; putting them in globals() is to enable urlparse.RELATIVE usage, much like we have re.IGNORECASE and not re.RegexFlag.IGNORECASE.

"DefragResultBytes", "ParseResultBytes", "SplitResultBytes"]

# A classification of schemes.
# The empty string classifies URLs with no scheme specified,
# being the default value returned by “urlsplit” and “urlparse”.

class SchemeFlag(enum.Flag):
"""SchemeFlag is an enum with the members RELATIVE, NETLOC, and
PARAMS. These describe methods for URL resolution, usually by
scheme. These resolution classes determine, namely, whether a
scheme supports, respectively, relative addressing, preserving the
netloc (domain name), and preserving the parameters.
"""
RELATIVE = enum.auto()
NETLOC = enum.auto()
PARAMS = enum.auto()
UNIVERSAL = RELATIVE | NETLOC | PARAMS

def __repr__(self):
return f'{self.__module__}.{self._name_}'

__str__ = __repr__

RELATIVE, NETLOC, PARAMS = SchemeFlag
# UNIVERSAL must be assigned separately as it's a combination of other variants.
UNIVERSAL = SchemeFlag.UNIVERSAL


@ethanfurman ethanfurman Mar 31, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change to:

class SchemeFlag(Flag):
    """SchemeFlag is an enum with the members RELATIVE, NETLOC, and
    PARAMS. These describe methods for URL resolution, usually by
    scheme. These resolution classes determine, namely, whether a
    scheme supports, respectively, relative addressing, preserving the
    netloc (domain name), and preserving the parameters.
    """
    RELATIVE = auto()
    NETLOC = auto()
    PARAMS = auto()
    UNIVERSAL = RELATIVE | NETLOC | PARAMS
    #
    def __repr__(self):
        return f"{self.module}.{self._name_}"
    __str__ = __repr__
RELATIVE, NETLOC, PARAMS, UNIVERSAL = SchemeFlag

uses_relative = ['', 'ftp', 'http', 'gopher', 'nntp', 'imap',
'wais', 'file', 'https', 'shttp', 'mms',
'prospero', 'rtsp', 'rtspu', 'sftp',
Expand All @@ -60,6 +84,30 @@
'https', 'shttp', 'rtsp', 'rtspu', 'sip', 'sips',
'mms', 'sftp', 'tel']


def _scheme_classes(scheme, overrides=None):
"""Find out what scheme classes a given scheme fits in.

This consults the variables uses_relative, uses_netloc, and
uses_params. It returns a set of all the classes that apply, with
at least the unique classes specified by the optional overrides
parameter.
"""
if overrides is None:
overrides = SchemeFlag(0)

if scheme in uses_relative:
overrides |= RELATIVE

if scheme in uses_netloc:
overrides |= NETLOC

if scheme in uses_params:
overrides |= PARAMS

@ethanfurman ethanfurman Mar 31, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove the three SchemeFlag references above (again, global level access to the enum members).

return overrides


# These are not actually used anymore, but should stay for backwards
# compatibility. (They are undocumented, but have a public-looking name.)

Expand Down Expand Up @@ -363,7 +411,7 @@ def _fix_result_transcoding():
_fix_result_transcoding()
del _fix_result_transcoding

def urlparse(url, scheme='', allow_fragments=True):
def urlparse(url, scheme='', allow_fragments=True, *, flags=None):
"""Parse a URL into 6 components:
<scheme>://<netloc>/<path>;<params>?<query>#<fragment>

Expand All @@ -386,7 +434,8 @@ def urlparse(url, scheme='', allow_fragments=True):
url, scheme, _coerce_result = _coerce_args(url, scheme)
splitresult = urlsplit(url, scheme, allow_fragments)
scheme, netloc, url, query, fragment = splitresult
if scheme in uses_params and ';' in url:
scheme_classes = _scheme_classes(scheme, overrides=flags)
if PARAMS in scheme_classes and ';' in url:
url, params = _splitparams(url)
else:
params = ''
Expand Down Expand Up @@ -500,7 +549,9 @@ def urlunsplit(components):
empty query; the RFC states that these are equivalent)."""
scheme, netloc, url, query, fragment, _coerce_result = (
_coerce_args(*components))
if netloc or (scheme and scheme in uses_netloc and url[:2] != '//'):

scheme_classes = _scheme_classes(scheme)
if netloc or (scheme and (NETLOC in scheme_classes) and url[:2] != '//'):
if url and url[:1] != '/': url = '/' + url
url = '//' + (netloc or '') + url
if scheme:
Expand All @@ -511,23 +562,26 @@ def urlunsplit(components):
url = url + '#' + fragment
return _coerce_result(url)

def urljoin(base, url, allow_fragments=True):
def urljoin(base, url, allow_fragments=True, *, flags=None):
"""Join a base URL and a possibly relative URL to form an absolute
interpretation of the latter."""
interpretation of the latter. Some logic may be enabled by setting
the flags variable."""
Comment thread
ethanfurman marked this conversation as resolved.
if not base:
return url
if not url:
return base

base, url, _coerce_result = _coerce_args(base, url)
bscheme, bnetloc, bpath, bparams, bquery, bfragment = \
urlparse(base, '', allow_fragments)
urlparse(base, '', allow_fragments, flags=flags)
scheme, netloc, path, params, query, fragment = \
urlparse(url, bscheme, allow_fragments)
urlparse(url, bscheme, allow_fragments, flags=flags)

if scheme != bscheme or scheme not in uses_relative:
scheme_classes = _scheme_classes(scheme, overrides=flags)

if scheme != bscheme or RELATIVE not in scheme_classes:
return _coerce_result(url)
if scheme in uses_netloc:
if NETLOC in scheme_classes:
if netloc:
return _coerce_result(urlunparse((scheme, netloc, path,
params, query, fragment)))
Expand Down
1 change: 1 addition & 0 deletions Misc/NEWS.d/next/Library/2022-01-10-22-03-02.bpo-46337.qkFZAw.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Expose scheme-specific URL options in ``urllib.parse.urljoin`` and ``urllib.parse.urlparse`` via the enum ``urllib.parse.SchemeFlag``.