How to stop NodeJS "Request" module changes request when using proxy

stop node server programmatically
node.js stop execution
how to stop npm server in cmd
how to stop nodejs application
how to stop node.js server-side javascript
stop express server
how to stop node js application in linux
how to stop node server in visual studio code

Sorry if this comes off as confusing.

I have written a script using the NodeJS request module that runs and performs a function on a website then returns with the data. This script works perfectly fine when I do not use a proxy by setting it to false. This is not a task that is NOT allowed to be done with Selenium/puppeteer

proxy: false

However, when I set a (working) proxy. It fails to perform the same task and is detected by the website firewall/antibot software.

proxy: http://xx.xxx.xx.xx:3128

Some things to note:

  • I have tried many (20+) different proxy providers (Residential and Datacenter) and they all have this issue
  • The issue does not occur if that proxy is set globally on my system
  • The issue does not occur if that proxy is set in a chrome extension
  • The SSL cipher suites do not match Chrome but they still don't match when not using a proxy so I assume that isn't the issue
  • It is very important to keep consistency in the header order

The question basically is. Does the request module change anything when using a proxy such as the header order?

Here is an image of what happens when it passes/fails.

The only difference is changing the proxy that causes this to fail. One request being made with, one request being made without.

url    : url,
simple : false,
forever: true,
resolveWithFullResponse: true,
gzip: true,
headers: {
    'Host'             : 'www.sitename.com',
    'Connection'       : 'keep-alive',
    'Upgrade-Insecure-Requests': '1',
    'User-Agent'       : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36',
    'Accept'           : 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
    'Accept-encoding'  : 'gzip, deflate, br',
    'Accept-Language'  : 'en-GB,en-US;q=0.9,en;q=0.8',
},
method : 'GET',
jar: globalJar,
simple: false,
followRedirect: false,
followAllRedirects: false, 

According to the proxies documentation of the request module:

By default, when proxying http traffic, request will simply make a standard proxied http request. This is done by making the url section of the initial line of the request a fully qualified url to the endpoint.

Instead you can use a http tunnel by setting:

tunnel : true

in the request module proxy settings.

It could be that in your case, you are making a standard proxied http request, whereas when using a proxy globally on your system or a chrome extension a http tunnel is created.

From the documentation:

Note that, when using a tunneling proxy, the proxy-authorization header and any headers from custom proxyHeaderExclusiveList are never sent to the endpoint server, but only to the proxy server.

Stop node.js program from command line, So to summarize you can exit by: Typing . exit in nodejs REPL. Pressing <ctrl>-C twice. pressing <ctrl>-D . process. exit(0) meaning a natural exit from REPL. If you want to return any other status you can return a non zero number. process. kill(process. Likewise, we can use process.abort to exit Node.js. This method works in the REPL as well as scripts and applications. The difference between process.abort, process.kill, and process.exit is that process.abort always exits Node.js immediately and generates a core file. In addition, no event callbacks will run.

After deactivating my old account I wanted to come back and give an actual answer to this question now I fully understand the answer. What I was asking one year ago was not possible, The antibot was fingerprinting me through the TLS ClientHello (And even slightly on the TCP/frame level).

To start, I wrote my a wrapper called request-curl which wrapped libcurl/curl binaries into a single library with the same format as request-promise, this gave me much more control over the request (preventing encoding, http2/proxy support and further session/TLS control) this still only let me reach a medicore rank of the 687th most popular ClientHello (https://client.tlsfingerprint.io:8443/). It wasn't good enough.

I had to move language. NodeJS is too much of a high-level language to allow for a really deep control (had to modify packets being sent from Layer 3). So as the answer to my question.

This is not yet possible to do in NodeJS - Let alone with the now unmaintained request.js library.

For anyone reading this, if you want to forge perfect requests to bypass antibot security you must move to a different language: I recommend utls in Golang or BouncyCastle in c#. Godspeed to you as it took me a year to really know how to do this. Even then, there's more internal issues these languages have and features they do not yet supposed (Go doesn't support 'basic' header-ordering, you need to monkey-patch/modify internals etc, utls doesn't easily support proxies). The list goes on and on.

If you're not already too deep into it, it's one hell of a rabbithole and I recommend you do not enter it.

how to stop node.js server, Learn how to terminate a Node.js app in the best possible way. Published Sep 11 , 2018. There are various ways to terminate a Node.js application. There are various ways to terminate a Node.js application. When running a program in the console you can close it with ctrl-C, but what I want to discuss here is programmatically exiting. Let’s start with the most drastic one, and see why you’re better off not using it.

How to exit from a Node.js program, js program started at the command line can be as simple as waiting for the script to finish executing. Implicitly, the Node.js process will exit when it reaches the end� For reference, here are the different signals you can send to stop a process and what they mean: (from kill man page) 1 HUP (hang up) 2 INT (interrupt) 3 QUIT (quit) 6 ABRT (abort) 9 KILL (non-catchable, non-ignorable kill) 14 ALRM (alarm clock) 15 TERM (software termination signal)

You're using the http-scheme for you request, but if the webserver redirects http to https and if the proxy-server is not configured to accept redirects (to https) then the problem might only be about the scheme respectively the URL you enter.

So the proxy had to be configured to accept redirects or the URL has to be checked manually in the case of faults and then adjusted in the case of a redirect.

Here you can read about redirects on one proxy-server (Apache Traffic Server), the scenario there includes more redirects than I described above: https://docs.trafficserver.apache.org/en/4.2.x/admin/reverse-proxy-http-redirects.en.html#handling-origin-server-redirect-responses

If you still encounter problems the server-logs of the proxy-server would be helpful.

EDIT: According to he page @Jannes Botis linked there exist still more proxy-settings that might be able to support or disrupt the desired functionality, so the whole issue is perhaps about configuring the proxy-server correct. Here are a few settings that are directly related to redirects:

followRedirect - follow HTTP 3xx responses as redirects (default: true). This property can also be implemented as function which gets response object as a single argument and should return true if redirects should continue or false otherwise.
followAllRedirects - follow non-GET HTTP 3xx responses as redirects (default: false)
followOriginalHttpMethod - by default we redirect to HTTP method GET. you can enable this property to redirect to the original HTTP method (default: false)
maxRedirects - the maximum number of redirects to follow (default: 10)
removeRefererHeader - removes the referer header when a redirect happens (default: false). Note: if true, referer header set in the initial request is preserved during redirect chain.

It's quite possible that other settings of the proxy-server have impact on fail or success of your scenario too.

How to Exit in Node.js, While working on a full-stack application it might happen that you end up with a bunch of node processes running at the same time. It could be� Hi, You can stop the server by killing the process. In Windows, run CMD and type taskkill /F /IM node.exe This will kill (stop) all Node. js processes. And then you can restart it.

Stop Node Server: How to kill a node process, I maintain several droplets on DigitalOcean simultaneously and each run Node.js processes. If you haven't given DigitalOcean a try, you can� I've checked online and all the articles say to make sure that "Node.js Core" is enabled in Preferences->Languages & Frameworks->JavaScript->Libraries. I've checked and it is. So is there some other step I need to do to make sure that WS recognises require() as being known (because it's part of NodeJS).

How to Stop a Hanging NodeJS Process, You build nodejs app with express v4x then start your app by npm start My question is how to stop the app Is there npm stopEDIT to inclu Exit status 2. Answers: Yes, npm provides for a stop script too: npm help npm-scripts. prestop, stop, poststop: Run by the npm stop command. Set one of the above in your package.json, and then use npm stop. npm help npm-stop. You can make this really simple if you set in app.js, process.title = myApp;

How to stop app that node.js express 'npm start', From a normal Windows command prompt, ctrl+c will stop a node server running. However, if I start the command prompt via the VS Code� Recent in Node-js. How do I determine the current operating system with Node.js? Jul 20 ; How to install a private NPM module without my own registry?

Comments
  • You need to show how are you using proxy-
  • proxy: http://xx.xxx.xx.xx:3128 @MarcosCasagrande The way it's documented into the request library
  • I don't think the order of HTTP headers is important. If you want to check your headers you can use httpbin.org/anything.
  • Hi @t.m.adam - I understand is very normal circumstances header order isn't important. In this circumstance, the header order is important and will prevent execution. I will do a diagram to further help
  • stackoverflow.com/questions/23585371/… couldn't this question help you?
  • Unfortunately, the same error happens. Setting a tunnel, creating an agent through a tunnel (all methods). All break :(
  • @ConorReid Can you access other urls besides this one? I see you are using also 'Upgrade-Insecure-Requests', is the website url "http" or "https"? If it supports https, try setting tunnel: false. You can see the tunnel setting here: github.com/request/request#requestoptions-callback
  • The proxy is not adding headers. I have checked on my own web server. I have used 20+ free/paid proxies aswell and again. A lot of people seem to not understand that this works set globally and as a chrome extension. If headers are the issue it wouldn't work then either.