Python requests: URL base in Session

python requests session
python-requests session close
python requests proxy
python requests session set-cookie
python http request
python requests codes
python requests 3
python requests 2.21 0

When using a Session, it seems you need to provide the full URL each time, e.g.

session = requests.Session()
session.get('http://myserver/getstuff')
session.get('http://myserver/getstuff2')

This gets a little tedious. Is there a way to do something like:

session = requests.Session(url_base='http://myserver')
session.get('/getstuff')
session.get('/getstuff2')

requests_toolbelt.sessions.BaseUrlSession https://github.com/requests/toolbelt/blob/f5c86c51e0a01fbc8b3b4e1c286fd5c7cb3aacfa/requests_toolbelt/sessions.py#L6

NOTE: This uses urljoin from standard lib. Beware of urljoin's behavior.

In [14]: from urlparse import urljoin

In [15]: urljoin('https://localhost/api', '/resource')
Out[15]: 'https://localhost/resource'

In [16]: urljoin('https://localhost/api', 'resource')
Out[16]: 'https://localhost/resource'

In [17]: urljoin('https://localhost/api/', '/resource')
Out[17]: 'https://localhost/resource'

In [18]: urljoin('https://localhost/api/', 'resource')
Out[18]: 'https://localhost/api/resource'

OR

import requests 
from functools import partial

def PrefixUrlSession(prefix=None):                                                                                                                                                                                                                                                                                                                 
     if prefix is None:                                                                                                                                                                                                                                                                                                                             
         prefix = ""                                                                                                                                                                                                                                                                                                                                
     else:                                                                                                                                                                                                                                                                                                                                          
         prefix = prefix.rstrip('/') + '/'                                                                                                                                                                                                                                                                                                          

     def new_request(prefix, f, method, url, *args, **kwargs):                                                                                                                                                                                                                                                                                      
         return f(method, prefix + url, *args, **kwargs)                                                                                                                                                                                                                                                                                            

     s = requests.Session()                                                                                                                                                                                                                                                                                                                         
     s.request = partial(new_request, prefix, s.request)                                                                                                                                                                                                                                                                                            
     return s             

Specialized Sessions, A Session with a URL that all requests will use as a base. Let's start by looking at an example: >>> from requests_toolbelt import sessions >>> s = sessions. This allows you to specify the base url for the HTTP client and to only specify the resource path at the time of the request. from requests_toolbelt import sessions http = sessions . BaseUrlSession ( base_url = "https://api.org" ) http . get ( "/list" ) http . get ( "/list/item" )

This feature has been asked on the forums a few times 1, 2, 3. The preferred approach as documented here, is subclassing, as follows:

from requests import Session
from urlparse import urljoin

class LiveServerSession(Session):
    def __init__(self, prefix_url=None, *args, **kwargs):
        super(LiveServerSession, self).__init__(*args, **kwargs)
        self.prefix_url = prefix_url

    def request(self, method, url, *args, **kwargs):
        url = urljoin(self.prefix_url, url)
        return super(LiveServerSession, self).request(method, url, *args, **kwargs)

You would use this simply as follows:

baseUrl = 'http://api.twitter.com'
with LiveServerSession(baseUrl) as s:
    resp = s.get('/1/statuses/home_timeline.json')

requests.sessions, hist = [] # keep track of history url = self.get_redirect_target(resp) Session() >>> s.get('https://httpbin.org/get') <Response [200]> Or as a context manager::  The requests module can help us build the URLS and manipulate the URL value dynamically. Any sub-directory of the URL can be fetched programmatically and then some part of it can be substituted with new values to build new URLs. Build_URL. The below example uses urljoin to fetch the different subfolders in the URL path. The urljoin method is used to add new values to the base URL.

You could just subclass request.Session and overload its __init__ and request methods like this:

# my_requests.py
import requests


class SessionWithUrlBase(requests.Session):
    # In Python 3 you could place `url_base` after `*args`, but not in Python 2.
    def __init__(self, url_base=None, *args, **kwargs):
        super(SessionWithUrlBase, self).__init__(*args, **kwargs)
        self.url_base = url_base

    def request(self, method, url, **kwargs):
        # Next line of code is here for example purposes only.
        # You really shouldn't just use string concatenation here,
        # take a look at urllib.parse.urljoin instead.
        modified_url = self.url_base + url

        return super(SessionWithUrlBase, self).request(method, modified_url, **kwargs)

And then you could use your subclass instead of requests.Session in your code:

from my_requests import SessionWithUrlBase


session = SessionWithUrlBase(url_base='https://stackoverflow.com/')
session.get('documentation')  # https://stackoverflow.com/documentation

Also you could monkey-patch requests.Session to avoid modifying existing codebase (this implementation should be 100% compatible), but be sure to do actual patching before any code calls requests.Session():

# monkey_patch.py
import requests


class SessionWithUrlBase(requests.Session):
    ...

requests.Session = SessionWithUrlBase

And then:

# main.py
import requests
import monkey_patch


session = requests.Session()
repr(session)  # <monkey_patch.SessionWithUrlBase object at ...>

Developer Interface, url – URL for the new Request object. params – (optional) Dictionary, list of tuples or A CookieJar containing all currently outstanding cookies set on this session. Base class that all auth implementations derive from Allows client-code to call dict(RequestsCookieJar) and get a vanilla python dict of key value pairs. This page provides Python code examples for requests.Session. base_url, apikey): # The URL in the config should end in /MAAS/, but the api is behind /MAAS/api/2.0

I don't see a built-in way to do this, but you can use wrapper functions to add the functionality you want:

from functools import wraps
import inspect
import requests
from requests.compat import urljoin

def _base_url(func, base):
    '''Decorator for adding a base URL to func's url parameter'''

    @wraps(func)
    def wrapper(*args, **kwargs):
        argname = 'url'
        argspec = inspect.getargspec(func)

        if argname in kwargs:
            kwargs[argname] = urljoin(base, kwargs[argname])
        else:
            # Find and replace url parameter in positional args. The argspec
            # includes self while args doesn't, so indexes have to be shifted
            # over one
            for i, name in enumerate(argspec[0]):
                if name == argname:
                    args = list(args)
                    args[i-1] = urljoin(base, args[i-1])
                    break

        return func(*args, **kwargs)
    return wrapper

def inject_base_url(func):
    '''Decorator for adding a base URL to all methods that take a url param'''

    @wraps(func)
    def wrapper(*args, **kwargs):
        argname = 'base_url'

        if argname in kwargs:
            obj = args[0]

            # Add base_url decorator to all methods that have a url parameter
            for name, method in inspect.getmembers(obj, inspect.ismethod):
                argspec = inspect.getargspec(method.__func__)

                if 'url' in argspec[0]:
                    setattr(obj, name, _base_url(method, kwargs[argname]))

            del kwargs[argname]

        return func(*args, **kwargs)
    return wrapper

# Wrap requests.Session.__init__ so it takes a base_url parameter
setattr(
    requests.Session,
    '__init__',
    inject_base_url(getattr(requests.Session, '__init__'))
)

Now you can specify a base URL when you construct a new requests.Session object:

s = requests.Session(base_url='http://stackoverflow.com')
s.get('questions')      # http://stackoverflow.com/questions
s.post('documentation') # http://stackoverflow.com/documentation

# With no base_url, you get the default behavior
s = requests.Session()
s.get('http://google.com')

Optional base url attribute on Session object · Issue #133 · psf , add an optional base url attribute to the Session object ? For example; auth = ('​user', 'pass') with requests.session(url='http://api.twitter.com',  The GET method indicates that you’re trying to get or retrieve data from a specified resource. To make a GET request, invoke requests.get (). To test this out, you can make a GET request to GitHub’s Root REST API by calling get () with the following URL: >>>. >>> requests.get('https://api.github.com') <Response [200]>.

requests.Session Python Example, This page provides Python code examples for requests.Session. def __init__(​self, url, mutual_auth, cert=None, verify='true', **kwargs): self._logger = logging. Requests will allow you to send HTTP/1.1 requests using Python. With it, you can add content like headers, form data, multipart files, and parameters via simple Python libraries. It also allows you to access the response data of Python in the same way.

Python Requests Tutorial | Using Requests Library in Python, Requests is a Python module you can use to send all kinds of HTTP requests. POST Requests; Sending Cookies and Headers; Session Objects And to see the actual response URL, you can use the req.url property. return_response – (optional) If False, an un-sent Request object will returned. session – (optional) A Session object to be used for the request. config – (optional) A configuration dictionary. verify – (optional) if True, the SSL cert will be verified. A CA_BUNDLE path can also be provided.

Python requests deep dive - Anthony Shaw, We needed to provide a set of base classes that would handle HTTP and HTTPS Calling requests.request in turn creates a Session. However, the above code will lose some of the advantages of having a Requests Session object. In particular, Session-level state such as cookies will not get applied to your request. To get a PreparedRequest with that state applied, replace the call to Request.prepare() with a call to Session.prepare_request(), like this:

Comments
  • I'm sure there is some rationale for removing that api piece of the URI, but holy cow, if it weren't early in the morning with a fresh cup of coffee, I'd be stuck on that for a couple hours wondering why my requests call wasn't working.
  • i like this answer, but it works only when base url has no sublevels like because urljoin overwrites them with what is provided as url to get and post methods. i needed it in my case, so i replaced urljoin call with simple string concatenation