Make requests using Python over Tor

tor python change ip
python tor web scraping
torrequest python
how to make anonymous requests using torrequests and python
tor-request
socks5h://localhost:9050
requests.exceptions.invalidschema: missing dependencies for socks support.
python tor selenium

I want to make multiple GET requests using Tor to a webpage. I want to use a different ipaddress for each request.

import socks
import socket
socks.set_default_proxy(socks.SOCKS5, "127.0.0.1", 9150)
socket.socket = socks.socksocket
import requests
print (requests.get('http://icanhazip.com')).content

Using this, I made one request. How can I change the ipaddress to make another?

Here is the code you want to use (download the stem package using pip install stem)

from stem import Signal
from stem.control import Controller

with Controller.from_port(port = 9051) as controller:
    controller.authenticate(password='your password set for tor controller port in torrc')
    print("Success!")
    controller.signal(Signal.NEWNYM)
    print("New Tor connection processed")

Good luck and hopefully that works.

Using Tor with the Python Requests Library - Jason Rigden, Doing HTTP requests anonymously with Python and Tor. Check out my podcast, This is how we normally make requests with the Request library: But using the proxies through the Session object reveals nothing. A picture  There is an issue on windows where the setting for the controlport in the torrc file is ignored if tor was installed using the following command: tor --service install To resolve the issue, after editing your torrc file, type the following commands: tor --service remove tor --service install -options ControlPort 9051

There are 2 aspects to your question -

  1. Making requests using Tor
  2. Renewing the connection as per requirement (in your case, after every request)

Part 1

The first one is easy to do with the latest (upwards of v2.10.0) requests library with an additional requirement of requests[socks] for using the socks proxy.

Installation -

pip install requests requests[socks]

Basic usage -

import requests

def get_tor_session():
    session = requests.session()
    # Tor uses the 9050 port as the default socks port
    session.proxies = {'http':  'socks5://127.0.0.1:9050',
                       'https': 'socks5://127.0.0.1:9050'}
    return session

# Make a request through the Tor connection
# IP visible through Tor
session = get_tor_session()
print(session.get("http://httpbin.org/ip").text)
# Above should print an IP different than your public IP

# Following prints your normal public IP
print(requests.get("http://httpbin.org/ip").text)

Part 2

To renew the Tor IP, i.e. to have a fresh visible exit IP, you need to be able to connect to the Tor service through it's ControlPort and then send a NEWNYM signal.

Normal Tor installation does not enable the ControlPort by default. You'll have to edit your torrc file and uncomment the corresponding lines.

ControlPort 9051
## If you enable the controlport, be sure to enable one of these
## authentication methods, to prevent attackers from accessing it.
HashedControlPassword 16:05834BCEDD478D1060F1D7E2CE98E9C13075E8D3061D702F63BCD674DE

Please note that the HashedControlPassword above is for the password "password". If you want to set a different password, replace the HashedControlPassword in the torrc by noting the output from tor --hash-password "<new_password>" where <new_password> is the password that you want to set.

................................................................................

Warning for Windows users: see post here.

There is an issue on windows where the setting for the controlport in the torrc file is ignored if tor was installed using the following command:

tor --service install

To resolve the issue, after editing your torrc file, type the following commands:

tor --service remove
tor --service install -options ControlPort 9051

................................................................................

Okay, so now that we have Tor configured properly, you will have to restart Tor if it is already running.

sudo service tor restart

Tor should now be up & running on the 9051 ControlPort through which we can send commands to it. I prefer to use the official stem library to control Tor.

Installation -

pip install stem

You may now renew the Tor IP by calling the following function.

Renew IP -

from stem import Signal
from stem.control import Controller

# signal TOR for a new connection 
def renew_connection():
    with Controller.from_port(port = 9051) as controller:
        controller.authenticate(password="password")
        controller.signal(Signal.NEWNYM)

To verify that Tor has a new exit IP, just rerun the code from Part 1. For some reason unknown to me, you need to create a new session object in order to use the new IP.

session = get_tor_session()
print(session.get("http://httpbin.org/ip").text)

How To Make Anonymous Requests using TorRequests and Python , Learn how to make anonymous requests to scrape websites through Tor, the on using TorRequests and Python. Tor is quite useful when you have to use requests without revealing your IP address, especially when you are web scraping. This tutorial will use a wrapper in python that helps you with the same. TOR is short for “The Onion Project”, a worldwide network of servers used by U.S. Navy.

You can use torrequest library (shameless plug). It's available on PyPI.

from torrequest import TorRequest

with TorRequest() as tr:
  response = tr.get('http://ipecho.net/plain')
  print(response.text)  # not your IP address

  tr.reset_identity()

  response = tr.get('http://ipecho.net/plain')
  print(response.text)  # another IP address, not yours

Use Tor with Python – Sylvain Durand, To check if it works, simply run the following command from a terminal: curl --​socks5 sudo apt-get install build-essential libssl-dev libffi-dev python-dev The Tor controller must also be configured to request identity renewal: Make requests using Python over Tor. I want to make multiple GET requests using Tor to a webpage. I want to use a different ipaddress for each request.

Requests supports proxies using the SOCKS protocol from version 2.10.0.

import requests
proxies = {
    'http': 'socks5://localhost:9050',
    'https': 'socks5://localhost:9050'
}
url = 'http://httpbin.org/ip'
print(requests.get(url, proxies=proxies).text)

A step-by-step guide how to use Python with Tor and Privoxy · GitHub, Make requests using Python over Tor · Crawling anonymously with Tor in Python Tor and Privoxy (web proxy configured to route through tor) Docker container  I want to make multiple GET requests using Tor to a webpage. I want to use a different ipaddress for each request. import socks import socket socks.set_default_proxy(socks.SOCKS5, &quot;127.0.0.1&quot;, 9150) socket.socket = socks.socksocket import requests print (requests.ge

This answer complete the one of Ashish Nitin Patil for windows (feel free to update this answer)

Part 2
ControlPort 9051
## If you enable the controlport, be sure to enable one of these
## authentication methods, to prevent attackers from accessing it.
HashedControlPassword 16:05834BCEDD478D1060F1D7E2CE98E9C13075E8D3061D702F63BCD674DE

The HashedControlPassword above is the password. If you want to set a different password in the console navigate to \Tor Browser\Browser\TorBrowser\Tor and type the following commands: tor.exe --hash-password password_XYZ | more). It will give you something like HashedControlPassword 16:54C092A8... This is your password. Now you can add it to the torrc file (Tor Browser\Browser\TorBrowser\Data\Tor\torrc).

You will need then to restart Tor:

tor --service remove
tor --service install -options ControlPort 9051

To check if that works type netstat -an you will now see that port 9051 is open.

Notice that tor --service install -... will create Tor Win32 Service. For some reason, it seems you have to stop the service to use the browser (run services.msc)

EDIT: you will find many pieces of information here (About port number & proxy, Tor, Privoxy, Auto switch user agent...).

Python Requests + Tor (Socks5) · GitHub, http://docs.python-requests.org/en/master/user/advanced/#proxies old but still better solution:http://dae.me/blog/1959/using-pythons-urllib2-or-requests-with-a-​socks5-proxy/ itJunky commented over 2 years ago • edited over 2 years ago. That is your HTTPS request via Tor, no DNS leaks. Your STEM request must not use proxy - Tor does not allow LAN IP's through SOCKS, and it's correct. You must not use any proxies sending signal NEWNYM. To capture HTML output use cURL options like RETURN_TRANSFER - see docs to fit your needs

Avoiding Webscraping Throttling Using Python and Tor as a Proxy , broadly when using Python to make requests to hide your actual IP. a script for scraping through Tor and switching IP's between requests. A step-by-step guide how to use Python with Tor and Privoxy. Tested on Ubuntu 18.04 Docker container. The Dockerfile is a single line FROM ubuntu:18.04. Alternatively, you can simply run docker run -it ubuntu:18.04 bash. NOTE: stopping services didn't work for me for some reason. That's why there is kill $(pidof <service name>) after each failed service <service name> stop to kill it.

torpy · PyPI, Pure python tor protocol implementation. Torpy can be used to communicate with clearnet hosts or hidden services through the Tor Network. Features. No Stem Create Socks5 proxy to relay requests via the Tor Network: The following code is 100% working on Python 3.4 (you need to keep TOR Browser open wil using this code) This script connects to TOR through socks5 get the IP from checkip.dyn.com, change identity and resend the request to get a the new IP (loops 10 times)

Using Tor to make HTTPS requests in Python, amnesia@amnesia: ~$ cat checkTor.py #!/usr/bin/env python import pycurl curl Your STEM request must not use proxy - Tor does not allow LAN IP's through  I'm the OP. According to the answer given by Tisho, this worked for me: import urllib, urllib2 ##Download SocksiPy - A Python SOCKS client module.

Comments
  • Isn't that what tor is supposed to do, i.e. obfuscate or otherwise make it difficult to determine the sender's IP address?
  • Use stem stem.torproject.org
  • @mhawke it does, but once the connection handshake is made it is persistent in that set of ip address hops that it connects through, thus why the op would want to possibly change the ip address again.
  • I have tried this solution but get this error: error(61, 'Connection refused'). It seems port 9051 is not open on my machine, I'm on a mac and have been trying to do everything I can find to open this port but still get this error. Any ideas?
  • @kflaw you need to uncomment your control port in torrc then restart tor. By default 9051 control port turned off because it is not needed for regular tor use.
  • thanks for the reply, I forgot to mention that I did uncomment the control port in torrc as well as the authenticate with hash password. What am I missing?
  • @kflaw Can you post a link to your code in codepen or something like that so I can repeat your error? Thanks
  • here is a link to a gist - I've tried several code snippets I found in similar posts but all get stuck on trying to establish the control port. gist.github.com/kflaw/1a41654993e9692914a7fa18f8819a8f
  • I would not use requesocks as requests now handles socks proxy out of the box via the proxies= parameter.
  • Thanks @jamescampbell for the info, updating my answer to reflect the same.
  • @NOPdaCALL How do you know it isn't working? What is the error that you get? Need specifics.
  • part1 (to test it) you will need pip install request[socks] (if you get an error, pip uninstall request and try to install again)
  • @JinSnow that's more of a Scrapy question than Tor, see stackoverflow.com/questions/45009940/… for more help.
  • tr.reset_identity() does not change the IP address. Am I missing something? Do I need to change the tor config file or something?