Does the Node.js "request" library support an async-iterable response stream?

I'm somewhat new to Node.js libraries and I'm trying figure how to use async iteration over an HTTP response stream. My overall goal is to read a large response stream and process it as chunks arrive, currently via a generator function. I cannot store the entire response in memory for processing.

I'm using the request library to execute the HTTP request as follows.

const request = require("request");

// contrived chunk-by-chunk stream processing 
async function* getChunks(stream) {
  for await (const chunk of stream) {
    yield chunk[0];
  }
}

async function doWork() {
  var response = request.get("https://pastebin.com/raw/x4Nn0Tby");
  for await (c of getChunks(response)) {
    console.log(c);
  }
}

When I run doWork(), I get an error stating that the stream variable of getChunks() is not async-iterable.

TypeError: stream is not async iterable

This is surprising, as I thought that all readable-streams are generally async-iterable, and that the request library returns a stream when no callback is provided. When I replace request.get(...) with fs.createReadStream(...) to some local file, all works as expected.

Perhaps the request library doesn't support this. If so, what do I need to do to process HTTP response streams via async-iteration?

Using Node.js 11.13 and request 2.88.0.

I did some more experimenting with the request and request-promise-native libraries and don't think this is possible under the current implementation. The resulting stream does not appear to be async-iterable at all. Furthermore, a proper implementation will need to await for the response to return before processing the stream (as suggested by @JBone's answer). But if you call await request.get(...), you retrieve the entire contents of the response, which is undesirable for large responses.

const r = require("request");
const rpn = require("request-promise-native");

// contrived chunk-by-chunk stream processing 
async function* getChunks(stream) {
  for await (const chunk of stream) {
    yield chunk[0];
  }
}

async function doWork() {
  const url = "https://pastebin.com/raw/x4Nn0Tby";
  const response = r.get(url);         // returns a non-async-iterable object.
  const response2 = await rp.get(url); // returns the contents of url

  for await (c of getChunks(response)) {  // yields response not async-iterable error.
    console.log(c);
  }
}

My solution to this problem was to replace usage of request and request-promise-native with the axios library. The libraries are functionally similar, but axios allows you to specify that a request should resolve to a stream; as expected, the stream is async-iterable.

const axios = require("axios");

async function doWork() {
  var response = await axios.request({
    method: "GET",
    url: "https://pastebin.com/raw/x4Nn0Tby",
    responseType: "stream",
  });

  for await (c of getChunks(response.data)) {  // async-iteration over response works as expected.
    console.log(c);
  }
}

Why Use Node.js? A Comprehensive Tutorial with Examples, Node.js is easily employed as a server-side proxy where it can handle a large amount of simultaneous connections in a non-blocking manner. It's especially useful� What Can Node.js Do? Node.js can generate dynamic page content; Node.js can create, open, read, write, delete, and close files on the server; Node.js can collect form data; Node.js can add, delete, modify data in your database

Simple answer: No, it doesn't. You might want to use a promise-based wrapper around request, such as request-promise, which then also works with async/await.

Details: Please note that request has been deprecated by its creator, and hence will be discontinued. This means, that sooner or later, you will most probably need to switch to another solution, such as axios, superagent or needle, to name a few.

Of course it's up to you to evaluate these modules and figure out which best suits your needs, but my personal recommendation would be to start with axios, as I had very good experiences with it in the past, however, YMMV.

Node.js - Introduction, Node. js uses an event-driven, non-blocking I/O model that makes it lightweight and efficient, perfect for data-intensive real-time applications that run across distributed devices. js applications are written in JavaScript, and can be run within the Node. js runtime on OS X, Microsoft Windows, and Linux. Though.js is the standard filename extension for JavaScript code, the name "Node.js" doesn't refer to a particular file in this context and is merely the name of the product. Node.js has an event-driven architecture capable of asynchronous I/O.

Seems like you will have to use other alternatives just like they mentioned in the request module documentation that you can find here https://www.npmjs.com/package/request

request supports both streaming and callback interfaces natively. If you'd like 
request to return a Promise instead, you can use an alternative interface wrapper for 
request. These wrappers can be useful if you prefer to work with Promises, or if 
you'd like to use async/await in ES2017.

Several alternative interfaces are provided by the request team, including:

request-promise (uses Bluebird Promises)
request-promise-native (uses native Promises)
request-promise-any (uses any-promise Promises)`

my response based on the below question:

I think you can create async await custom method that does it.

async function doMyWork() {
try {
 const response = await myOwnRequest(url); 
 } catch (e) {
   console.log ('the error', e);
 }  
}

function myOwnRequest(url) {
  return new Promise(function (resolve, reject) {
   const resp = request.get(url);
   if(resp) {
    resolve();
   } else {
     reject();
   }
});
}

Node.js, WARNING: This does not throw an AssertionError! assert.deepEqual(/a/gi All errors thrown by the assert module will be instances of the AssertionError class. Node.js® is a JavaScript runtime built on Chrome’s V8 JavaScript engine. Node.js uses an event-driven, non-blocking I/O model that makes it lightweight and efficient. Node.js’ package ecosystem, npm, is the largest ecosystem of open source libraries in the world.

The axios' stream option didn't work for me using the sample code in the above answer on axios 0.19.0. Could be an issue between the chair and keyboard, but in any case...here is an alternative approach using request.

I ended up adapting request streaming to an async generator (with a buffer in between of course). This allows a "streaming" type interface where reads and writes of the data can be interleaved...it doesn't guarantee low memory consumption. request pipes ("pushes") to our Writable as fast as it can and there isn't a way for us to pause this or flip it into a "pull" type interface (as far as I know). So if we are reading data out of the buffer slower than data is being written in: the buffer will get very big and memory usage will be high.

So if it is critical to keep memory usage down and you parse large files from http sources...then probably doing some monitoring/reporting on buffer size while "streaming" in order to see if you consumption code is faster or slower than the stream so you know if the buffer will get huge or stay small. Of course if you test with a very slow http server...then all bets are off.

This could possibly be addressed by setting a fixed buffer size and making _write block until some more reading happens (making room in the buffer)...ie so request has to wait to write more data to the pipe. However request may buffer internally...so this won't help with memory consumption if the data is piling up on requests' end anyway. Would have to check.

Sample code:

const request = require('request'),
    Writable = require('stream').Writable,
    EventEmitter = require('events');

module.exports = function (url, MAX_BYTES=1024) {
    var response = new ResponseBuffer(MAX_BYTES);

    request
        .get(url)
        .on('error', function(err) { throw err; })
        .pipe(response)
        .on('error', function(err) { throw err; });

    return response.reader();
};

class ResponseBuffer extends Writable {
    constructor (MAX_BYTES=1024) {
        super();
        this.buffer = '';
        this.open = true;
        this.done = null;  // callback to call when done reading.
        this.MAX_BYTES = MAX_BYTES;
        this.events = new EventEmitter();
    }
    _write(chunk, enc, next) {
        this.buffer += chunk;
        this.events.emit('data');
        next();
    }
    _final(done) {
        this.open = false; // signal to reader to return after buffer empty.
        return done();
    }
    async * reader () {
        while (true) {
            if (this.buffer.length == 0) {
                // buffer empty and Writable !open. return.
                if (!this.open) { return; }
                else { // buffer empty.  wait for data.
                    await new Promise(resolve => this.events.once('data', resolve));
                }
            }
            let read_bytes = this.buffer.length < this.MAX_BYTES ? this.buffer.length : this.MAX_BYTES;
            yield this.buffer.slice(0, read_bytes);
            this.buffer = this.buffer.slice(read_bytes);
        }
    }
}

Then use it like so:

const httpModule = require('./path/to/above/module');
var httpGen = httpModule('https://www.google.com'),
    chunk;
for await (chunk of httpGen) {
    // do something with chunk.
}

An alternative approach (if you are concerned about memory usage specifically) is to just download to disk (streaming to file writer) and then read incrementally off disk (you can async iter a fs.createReadStream(...))

Node.js v14.8.0 Documentation, How do I start with Node.js after I installed it? Once we have installed Node.js, let's build our first web server. Create a file named app.js containing the following � Node.js is an open-source and cross-platform JavaScript runtime environment. It is a popular tool for almost any kind of project! Node.js runs the V8 JavaScript engine, the core of Google Chrome, outside of the browser. This allows Node.js to be very performant. A Node.js app is run in a single process, without creating a new thread for every request.

Getting Started Guide, In the following "hello world" example, many connections can be handled connection, the callback is fired, but if there is no work to be done, Node.js will sleep. With Node.js, an open source runtime system invented in 2009 by Ryan Dahl, that reach has extended to the server side. Node.js has become wildly popular, with coders everywhere using it to create

About, You will either need to write the code yourself, or you can avoid reinventing the wheel and use a web framework! Introducing Express. Express is� Very simply, NodeJS is a very thin wrapper around the same Javascript Engine your browser runs. You can run practically any Javascript you would writing in your browser in Node. NodeJS has several built in libraries of objects/functions that allow you to: organize your code into modules.

Node.js, You can use any language that transpiles to JavaScript, for example TypeScript and CoffeeScript. Node.js incorporates the Google Chrome V8� Node.js® is a JavaScript runtime built on Chrome's V8 JavaScript engine.

Comments
  • What version of axios are you using? With 0.19.0 I get response is not async iterable. I couldn't find any docs indicating that axios would support this. Any ideas?
  • @mattpr: I was using 0.18.0 when I wrote the sample code above. As I recall, you need to set the response type to stream for this to work properly.
  • Thank you for the suggestion. I'm not sure that I need a Promise returned from request.get() as I really just need to operate on the resulting stream (which I assumed supported await natively like fs.createReadStream()). If a Promise is returned (via request-promise-native), then calling await request.get(...) resolves the entire response stream into a string.
  • @Steve Guildi Please see if my updated answer can help you.
  • @SteveGuidi was the above (updated) answer any help? let me know if you want me to update with anything
  • There is a small error in myOwnRequest -- resolve() needs to take the object that is being returned, namely resp. Otherwise, the Promise behaves as expected. Unfortunately, the result of request.get(...) is just not async-iterable at all, regardless if it is returned via Promise or directly. I ended up using the axios library instead, which returns streams that are async-iterable (see my answer for an example).