Add params to given URL in Python

passing parameters in url python
python build url with parameters
python get url parameters
selenium add url parameter python
python-urllib
how to add parameters to url python
url path python
python change url parameters

Suppose I was given a URL. It might already have GET parameters (e.g. http://example.com/search?q=question) or it might not (e.g. http://example.com/).

And now I need to add some parameters to it like {'lang':'en','tag':'python'}. In the first case I'm going to have http://example.com/search?q=question&lang=en&tag=python and in the second — http://example.com/search?lang=en&tag=python.

Is there any standard way to do this?

There are a couple of quirks with the urllib and urlparse modules. Here's a working example:

try:
    import urlparse
    from urllib import urlencode
except: # For Python 3
    import urllib.parse as urlparse
    from urllib.parse import urlencode

url = "http://stackoverflow.com/search?q=question"
params = {'lang':'en','tag':'python'}

url_parts = list(urlparse.urlparse(url))
query = dict(urlparse.parse_qsl(url_parts[4]))
query.update(params)

url_parts[4] = urlencode(query)

print(urlparse.urlunparse(url_parts))

ParseResult, the result of urlparse(), is read-only and we need to convert it to a list before we can attempt to modify its data.

Add params to given URL in Python, In Python, how can you add the variables to a URL? param_value): """Given a URL, set or replace a query parameter and return the modified  Python: Add query parameters to a URL I was recently trying to automate adding a query parameter to a bunch of URLS and came across a neat approach a long way down this StackOverflow answer , that uses the PreparedRequest class from the requests library .

Why

I've been not satisfied with all the solutions on this page (come on, where is our favorite copy-paste thing?) so I wrote my own based on answers here. It tries to be complete and more Pythonic. I've added a handler for dict and bool values in arguments to be more consumer-side (JS) friendly, but they are yet optional, you can drop them.

How it works

Test 1: Adding new arguments, handling Arrays and Bool values:

url = 'http://stackoverflow.com/test'
new_params = {'answers': False, 'data': ['some','values']}

add_url_params(url, new_params) == \
    'http://stackoverflow.com/test?data=some&data=values&answers=false'

Test 2: Rewriting existing args, handling DICT values:

url = 'http://stackoverflow.com/test/?question=false'
new_params = {'question': {'__X__':'__Y__'}}

add_url_params(url, new_params) == \
    'http://stackoverflow.com/test/?question=%7B%22__X__%22%3A+%22__Y__%22%7D'
Talk is cheap. Show me the code.

Code itself. I've tried to describe it in details:

from json import dumps

try:
    from urllib import urlencode, unquote
    from urlparse import urlparse, parse_qsl, ParseResult
except ImportError:
    # Python 3 fallback
    from urllib.parse import (
        urlencode, unquote, urlparse, parse_qsl, ParseResult
    )


def add_url_params(url, params):
    """ Add GET params to provided URL being aware of existing.

    :param url: string of target URL
    :param params: dict containing requested params to be added
    :return: string with updated URL

    >> url = 'http://stackoverflow.com/test?answers=true'
    >> new_params = {'answers': False, 'data': ['some','values']}
    >> add_url_params(url, new_params)
    'http://stackoverflow.com/test?data=some&data=values&answers=false'
    """
    # Unquoting URL first so we don't loose existing args
    url = unquote(url)
    # Extracting url info
    parsed_url = urlparse(url)
    # Extracting URL arguments from parsed URL
    get_args = parsed_url.query
    # Converting URL arguments to dict
    parsed_get_args = dict(parse_qsl(get_args))
    # Merging URL arguments dict with new params
    parsed_get_args.update(params)

    # Bool and Dict values should be converted to json-friendly values
    # you may throw this part away if you don't like it :)
    parsed_get_args.update(
        {k: dumps(v) for k, v in parsed_get_args.items()
         if isinstance(v, (bool, dict))}
    )

    # Converting URL argument to proper query string
    encoded_get_args = urlencode(parsed_get_args, doseq=True)
    # Creating new parsed result object based on provided with new
    # URL arguments. Same thing happens inside of urlparse.
    new_url = ParseResult(
        parsed_url.scheme, parsed_url.netloc, parsed_url.path,
        parsed_url.params, encoded_get_args, parsed_url.fragment
    ).geturl()

    return new_url

Please be aware that there may be some issues, if you'll find one please let me know and we will make this thing better

How to add parameters to the URL string in Python, from requests.models import PreparedRequest req = PreparedRequest(). And now let's use use this class to add a query parameter to a URL. If you want to add a parameter into the existing URL. You can use urlsplit() and urlunsplit() to break apart and rebuild a URL, then use urlencode() on the parsed query string: from urllib.parse import urlencode, parse_qs, urlsplit, urlunsplit def set_query_parameter(url, param_name, param_value): """Given a URL, set or replace a query

You want to use URL encoding if the strings can have arbitrary data (for example, characters such as ampersands, slashes, etc. will need to be encoded).

Check out urllib.urlencode:

>>> import urllib
>>> urllib.urlencode({'lang':'en','tag':'python'})
'lang=en&tag=python'

In python3:

from urllib import parse
parse.urlencode({'lang':'en','tag':'python'})

Python: Add query parameters to a URL · Mark Needham, How do you add a variable to a URL in Python? possible duplicate of Add params to given URL in Python – Ignacio Vazquez-Abrams Apr 23 '11 at 22:47 add a comment | 3 Answers 3

You can also use the furl module https://github.com/gruns/furl

>>> from furl import furl
>>> print furl('http://example.com/search?q=question').add({'lang':'en','tag':'python'}).url
http://example.com/search?q=question&lang=en&tag=python

Use URL Variables to Pass Data, The markers parameter lets us add markers to the map. The standard for URL query strings when multiple values have the same key is to repeat the key, In other words, From the Google API's explanation:. Stack Overflow Public questions and answers; Python requests call with URL using parameters. This is the proper way to add parameters to a URL using requests:

Outsource it to the battle tested requests library.

This is how I will do it:

from requests.models import PreparedRequest
url = 'http://example.com/search?q=question'
params = {'lang':'en','tag':'python'}
req = PreparedRequest()
req.prepare_url(url, params)
print(req.url)

How to encode URLs in Python, Changed in version 3.2: Add encoding and errors parameters. Changed in version This is similar to urlparse() , but does not split the params from the URL​. How to make Python do the tedious work of creating URL query strings. Most Web APIs require you to pass in configuration values via a URL query string. Creating these strings is a matter of reading the API’s documentation, and then either doing the mind-numbing work of manually creating the query strings. Or using Python’s urllib parsing

Creating URL query strings in Python, The urlparse module is renamed to urllib.parse in Python 3. This is similar to urlparse() , but does not split the params from the URL. These subclasses add the attributes described in those functions, as well as provide an additional  urlparse.urlsplit (urlstring [, scheme [, allow_fragments]]) ¶ This is similar to urlparse(), but does not split the params from the URL. This should generally be used instead of urlparse() if the more recent URL syntax allowing parameters to be applied to each segment of the path portion of the URL (see RFC 2396) is wanted. A separate function is needed to separate the path segments and parameters.

urllib.parse — Parse URLs into components, This is similar to urlparse(), but does not split the params from the URL. This should This is useful to pass a dictionary of form fields to a POST request. Python provided a getopt module that helps you parse command-line options and arguments. This module provides two functions and an exception to enable command line argument parsing. This module provides two functions and an exception to enable command line argument parsing.

20.16. urlparse — Parse URLs into components, The urllib.parse module defines the following functions: This is similar to urlparse(), but does not split the params from the URL. This should tuples to a “​url-encoded” string, suitable to pass to urlopen() above as the optional data argument. To reverse this encoding process, parse_qs () and parse_qsl () are provided in this module to parse query strings into Python data structures. Refer to urllib examples to find out how urlencode method can be used for generating query string for a URL or data for POST.

Comments
  • You probably want to use urlparse.parse_qs instead of parse_qsl. The latter returns a list whereas you want a dict. See docs.python.org/library/urlparse.html#urlparse.parse_qs.
  • @florian : At least in python 2.7 you then need to call urlencode as urllib.urlencode(query, doseq=True). Otherwise, parameters that existed in the original url are not preserved correctly (because they are returned as tuples from @parse_qs@
  • I've rewritten this to work in Python 3 as well. Code here.
  • The results of urlparse() and urlsplit() are actually namedtuple instances. Thus you can assign them directly to a variable and use url_parts = url_parts._replace(query = …) to update it.
  • Caution - this implementation removes repeated query parameters that some RESTful services use. With a little modification this can be fixed. query = urlparse.parse_qsl(url_parts[4]) query += params.items() But then if you want to replace exiting query params using dict, takes a little more.
  • Perhaps add a try except with from urllib.parse to include Python 3 support? Thanks for the snippet, very useful!
  • Maybe add imports too ?
  • In python 3, this has been moved to urllib.parse.urlencode
  • Thanks for the hint to furl. That is so much nicer than the urllib stuff!
  • Can you please give some brief example?
  • f.read() will show you the HTML page. To see the calling url, f.geturl()
  • -1 for using a HTTP request for parsing a URL (which is actually basic string manipulation). Plus the actual problem is not considered, because you need to know how the URL looks like to be able to append the query string correctly.
  • Either the author edited question either this answer is not related to it.
  • I know you mentioned "simple cases", but to clarify: it won't work properly if there is an ? in the anchor (#?stuff).