Python web scraping with requests - after login

scrape website with login python beautifulsoup
python login to website requests
python requests login session
python script to open webpage and login
python login to website example\
python login to website javascript
python requests click button
web scraping intranet python

I have a python requests/beatiful soup code below which enables me to login to a url successfully. However, after logon, to get the data I need would normally have to manually have to:

1) click on 'statement' in the first row:

2) Select dates, click 'run statement':

3) view data:

This is the code that I have used to logon to get to step 1 above:

import requests
from bs4 import BeautifulSoup

logurl = ""
posturl = ''

with requests.Session() as s:
    s.headers = {"User-Agent":"Mozilla/5.0"}
    res = s.get(logurl)
    soup = BeautifulSoup(res.text,"html.parser")

    arg_names =[]
    for name in"[name='p_arg_names']"):

    values = {
        'p_flow_id': soup.select_one("[name='p_flow_id']")['value'],
        'p_flow_step_id': soup.select_one("[name='p_flow_step_id']")['value'],
        'p_instance': soup.select_one("[name='p_instance']")['value'],
        'p_page_submission_id': soup.select_one("[name='p_page_submission_id']")['value'],
        'p_request': 'LOGIN',
        'p_t01': 'solar',
        'p_arg_names': arg_names,
        'p_t02': 'password',
        'p_md5_checksum': soup.select_one("[name='p_md5_checksum']")['value'],
        'p_page_checksum': soup.select_one("[name='p_page_checksum']")['value']
    s.headers.update({'Referer': logurl})
    r =, data=values)
    print (r.content)

My question is, (beginner speaking), how could I skip steps 1 and 2 and simply do another headers update and post using the final URL using selected dates as form entries (headers and form info below)? (The referral header is step 2 above):


Edit 1: network request from csv file download:

Use selenium webdriver, it has a lot of good functions to handle web services.

Scraping Data behind Site Logins with Python, Using the Requests library to scrape data behind a website's login page By using Chrome's inspect tool and clicking on the login form, I'm  A great frustration in my web scraping journey has been finding a page tucked away behind a login. I didn’t actually think it was possible to scrape a page locked away like this so I didn’t bother Googling it. Using the requests module to pull data from a page behind a login is relatively simple. It does however require a little bit of HTML

Selenium is gonna be your best bet for automated browser interactions. It can be used not only to scrape data from websites but also to interact with different forms and such. I highly recommended it as I have used it quite a bit in the past. If you already have pip and python installed go ahead and type

pip install selenium

That will install selenium but you also need to install either geckodriver (for Firefox) or chromedriver (for chrome) Then you should be up and running!

How to scrape a website that requires login with Python, The code from this tutorial can be found on my Github. We will perform the following steps: Extract the details that we need for the login; Perform login to the site; Scrape the required data import requests from lxml import html. How to scrape a website that requires login with Python I’ve recently had to perform some web scraping from a site that required login. It wasn’t very straight forward as I expected so I’ve decided to write a tutorial for it.

As others have recommended, Selenium is a good tool for this sort of task. However, I'd try to suggest a way to use requests for this purpose as that's what you asked for in the question.

The success of this approach would really depend on how the webpage is built and how data files are made available (if "Save as CSV" in the view data is what you're targeting).

If the login mechanism is cookie-based, you can use Sessions and Cookies in requests. When you submit a login form, a cookie is returned in the response headers. You add the cookie to request headers in any subsequent page requests to make your login stick.

Also, you should inspect the network request for "Save as CSV" action in the Developer Tools network pane. If you can see a structure to the request, you may be able to make a direct request within your authenticated session, and use a statement identifier and dates as the payload to get your results.

Python web scraping with requests - after login, As others have recommended, Selenium is a good tool for this sort of task. However, I'd try to suggest a way to use requests for this purpose as  Logging in With Requests Stephen Brennan • 02 March 2016. One of my favorite types of quick side projects are ones that involve web scraping with Python. Obviously, the Internet houses a ton of useful data, and you may want to fetch lots of that data to use within your own programs.

Logging in With Requests • Stephen Brennan, So how would you go about simple web scraping in Python? On each subsequent request, your browser sends it back to the web site. After a login, these lines of code above simply extract the information from the page to show that the login was successful. Conclusion. The process of logging into websites using Python is quite easy, however the setup of websites are not the same therefore some sites would prove more difficult to log into than others.

Logging Into Websites With Python – Linux Hint, Therefore if you intend web scraping a website, you could come across the This would be done with the Requests and BeautifulSoup Python libraries. You do this by right clicking on one of the login boxes and clicking inspect element. As you do more web scraping, you will find that the <a> is used for hyperlinks. Now that we’ve identified the location of the links, let’s get started on coding! Python Code. We start by importing the following libraries. import requests import urllib.request import time from bs4 import BeautifulSoup

Web Scraping Behind Authentication With Python - Better , As mentioned, I will use Python for this, with the requests library. I will only focus on this in this guide. Create a file, I'll call it for now. Install the  Web scraping is a big field, and you have just finished a brief tour of that field, using Python as you guide. You can get pretty far using just requests and BeautifulSoup , but as you followed along, you may have come up with few questions:

  • Thanks. I added the csv file download network inspection. Think its doable to do a direct request?
  • I think it's doable - use the sessions and cookies, and then use BeautifulSoup to get the link for "Save as CSV" and do a GET request to with the payload.
  • Thanks. If there is anyway you could add to my script, even with a basic framework of what you mean above (even just added comments for what goes where), it would help me immensely.