Asynchonously deserializing a list using System.Text.Json

Lets say that I request a large json file that contains a list of many objects. I don't want them to be in memory all at once, but I would rather read and process them one by one. So I need to turn an async System.IO.Stream stream into an IAsyncEnumerable<T>. How do I use the new System.Text.Json API to do this?

private async IAsyncEnumerable<T> GetList<T>(Uri url, CancellationToken cancellationToken = default)
{
    using (var httpResponse = await httpClient.GetAsync(url, cancellationToken))
    {
        using (var stream = await httpResponse.Content.ReadAsStreamAsync())
        {
            // Probably do something with JsonSerializer.DeserializeAsync here without serializing the entire thing in one go
        }
    }
}

Yes, a truly streaming JSON (de)serializer would be a nice performance improvement to have, in so many places.

Unfortunately, System.Text.Json does not do this at this time. I'm not sure if it will in the future - I hope so! Truly streaming deserialization of JSON turns out to be rather challenging.

You could check if the extremely fast Utf8Json supports it, perhaps.

However, there might be a custom solution for your specific situation, since your requirements seem to constrain the difficulty.

The idea is to manually read one item from the array at a time. We are making use of the fact that each item in the list is, in itself, a valid JSON object.

You can manually skip past the [ (for the first item) or the , (for each next item). Then I think your best bet is to use .NET Core's Utf8JsonReader to determine where the current object ends, and feed the scanned bytes to JsonDeserializer.

This way, you're only buffering slightly over one object at a time.

And since we're talking performance, you could get the input from a PipeReader, while you're at it. :-)

What is deserialize and serialize in JSON?, is its inverse operation (convert string -> object). After the byte strings are transmitted, the receiver will have to recover the original object from the byte string. The System.Text.Json.Serialization namespace contains attributes and APIs for advanced scenarios and customization specific to serialization and deserialization. The code examples shown in this article require using directives for one or both of these namespaces: using System.Text.Json; using System.Text.Json.Serialization;


JSON Serialization And Deserialization In C#, How do you serialize and deserialize an object in C#? Asynchronously reads the UTF-8 encoded text representing a single JSON value into an instance of a specified type. The stream will be read to completion. C#. public static System.Threading.Tasks.ValueTask<object> DeserializeAsync (System.IO.Stream utf8Json, Type returnType, System.Text.Json.JsonSerializerOptions options = default, System.Threading.CancellationToken cancellationToken = default);


It feels like you need to implent your own stream reader. You have to read bytes one-by-one and stop as soon as object definition completed. It is indeed pretty low-leveled. As such you WILL NOT load entire file into RAM, but rather take the part you're dealing with. Does it seem to be an answer?

Working With JSON String In C#, How do you serialize and deserialize an object in C# using JSON? In this article, we shall be using System.Text.Json for deserializing the JSON into an object or type dynamically. As we know System.Text.Json will be the default serializer and deserializer for ASP.NET Core 3.1 onwards. Below logic helps to convert JSON object into a type of your choice dynamically. Here I am using Generic <T> to denote the same.


Maybe you could use Newtonsoft.Json serializer ? https://www.newtonsoft.com/json/help/html/Performance.htm

Especially see section:

Optimize Memory Usage

Edit

You could try deserializing values from JsonTextReader, e.g.

using (var textReader = new StreamReader(stream))
using (var reader = new JsonTextReader(textReader))
{
    while (await reader.ReadAsync(cancellationToken))
    {
        yield return reader.Value;
    }
}

JsonSerializer.DeserializeAsync Method (System.Text.Json , represents objects in structured text format and data stored in key-value pairs. I am consuming Json from a TcpClient, and to get a solution with low allocation and good performance I decided to use the new System.IO.Pipelines for handling IO and System.Text.Json for deserialization. The output from the pipe is a ReadOnlySequence<byte>.


How to serialize and deserialize JSON using C#, @ericstj ericstj added this to To do in System.Text.Json - 5.0 on Jan 9 Add support for asynchronously (de)serializing IAsyncEnumerable<T> on Apr 14 I can't think of anything better than to make a List<T> that implements  A System.Text.Json equivalent seems to be Which I use while deserializing by passing it as an argument: Asynchonously deserializing a list using System.Text


Add support for asynchronously (de)serializing IAsyncEnumerable , Maybe the user could configure a list of allowed assemblies where the type can come from. I will certainly do this in my code. I added $type  Using System.Text.Json directly. For all the samples, make sure you import the following two namespaces: Using the serializer. The System.Text.Json serializer can read and write JSON asynchronously and is optimized for UTF-8 text, making it ideal for REST API and back-end applications. By default, we produce minified JSON.


[System.Text.Json] serialize/deserialize any object · Issue #30969 , Object) are significantly slower since: - During serialization, the object's type is obtained and a Deserializing System.Collections.Immutable types with JsonSerializer is not linker safe Count}"); var t = new List<int>(s); Console. asynchronously deserialized, each would require its own stream pointer in the underlying  The System.Text.Json namespace provides functionality for serializing to and deserializing from JavaScript Object Notation (JSON). The library design emphasizes high performance and low memory allocation over an extensive feature set.