[C#]如何使用Newton.Json从流中反序列化json数据

有个程序需要从大文件反序列化json，但是发现读取字符串很慢，于是想从stream反序列化这样至少比读字符串快，于是找到这个文章。

Can Json.NET serialize / deserialize to / from a stream?

Ask Question

Asked 11 years, 1 month ago

Modified 4 months ago

Viewed 237k times

186

I have heard that Json.NET is faster than DataContractJsonSerializer, and wanted to give it a try...

But I couldn't find any methods on JsonConvert that take a stream rather than a string.

For deserializing a file containing JSON on WinPhone, for example, I use the following code to read the file contents into a string, and then deserialize into JSON. It appears to be about 4 times slower in my (very ad-hoc) testing than using DataContractJsonSerializer to deserialize straight from the stream...

// DCJS
DataContractJsonSerializer dc = new DataContractJsonSerializer(typeof(Constants));
Constants constants = (Constants)dc.ReadObject(stream);

// JSON.NET
string json = new StreamReader(stream).ReadToEnd();
Constants constants = JsonConvert.DeserializeObject<Constants>(json);

Am I doing it wrong?

• .net
• serialization
• json.net

Share

Improve this question

Follow

edited Dec 28, 2018 at 16:35

Tim Cooper

155k3737 gold badges322322 silver badges275275 bronze badges

asked Nov 16, 2011 at 19:41

Omri Gazitt

3,42822 gold badges2121 silver badges2727 bronze badges

Add a comment

6 Answers

Sorted by:

                                              Highest score (default)                                                                   Trending (recent votes count more)                                                                   Date modified (newest first)                                                                   Date created (oldest first)                              

339

The current version of Json.net does not allow you to use the accepted answer code. A current alternative is:

public static object DeserializeFromStream(Stream stream)
{
    var serializer = new JsonSerializer();

    using (var sr = new StreamReader(stream))
    using (var jsonTextReader = new JsonTextReader(sr))
    {
        return serializer.Deserialize(jsonTextReader);
    }
}

Documentation: Deserialize JSON from a file stream

Share

Improve this answer

Follow

edited Jan 30, 2015 at 20:35

James Newton-King

47.3k2424 gold badges109109 silver badges129129 bronze badges

answered Jul 22, 2013 at 12:52

user2535425

• 5

  JsonTextReader will close its StreamReader by default, so this example could be simplified a bit by constructing the StreamReader in the call to the JsonTextReader constructor. 

  – Oliver Bock

   Jun 29, 2016 at 6:31

• 3

  Actually, I have an OutOfMemory exception and I already use this code, pretty much exactly. Which, I believe, goes to say, this is not a guarantee - if the deserialized object is large enough, and you're stuck in a 32-bit process, you may still get memory errors with this code 

  – PandaWood

   Oct 21, 2016 at 3:03

• 1

  i am getting an error "The type or namespace name 'JsonTextReader' could not be found" ...any suggestions? 

  – hnvasa

   Sep 22, 2017 at 0:24 

• 6

  I needed to add stream.Position = 0; to correctly deserialize my json. 

  – hybrid2102

   Mar 21, 2019 at 14:54
• The documentation in the link does not mention anything about JsonTextReader 

  – John Henckel

   Nov 17 at 20:16

Add a comment

114

public static void Serialize(object value, Stream s)
{
    using (StreamWriter writer = new StreamWriter(s))
    using (JsonTextWriter jsonWriter = new JsonTextWriter(writer))
    {
        JsonSerializer ser = new JsonSerializer();
        ser.Serialize(jsonWriter, value);
        jsonWriter.Flush();
    }
}

public static T Deserialize<T>(Stream s)
{
    using (StreamReader reader = new StreamReader(s))
    using (JsonTextReader jsonReader = new JsonTextReader(reader))
    {
        JsonSerializer ser = new JsonSerializer();
        return ser.Deserialize<T>(jsonReader);
    }
}

Share

Improve this answer

Follow

edited May 4, 2016 at 17:37

Jim Buck

2,3552525 silver badges4141 bronze badges

answered Mar 27, 2014 at 14:02

ygaradon

2,16822 gold badges2020 silver badges2727 bronze badges

• 4

  Thanks! This helped me avoid an OutOfMemoryException I was getting when I was serializing a very large object collection to a string, and then writing that string into my stream (instead of just serializing directly to the stream). 

  – Jon Schneider

   Apr 14, 2016 at 19:50

• 9

  Why flush? Doesn't the Dispose call caused by the using block already do that? 

  – Şafak Gür

   Nov 17, 2017 at 8:45

• 9

  Side note, because it might help others: if you use JsonSerializer ser = JsonSerializer.Create(settings); you can define which settings to use during de/serialization. 

  – mike

   May 15, 2018 at 14:12

• 6

  One potential issue with this Serialize implementation is that it closes the Stream passed as an argument, which depending on the application can be a problem. With .NET 4.5+ you can avoid this problem by using a StreamWriter constructor overload with a parameter leaveOpen that lets you leave the stream open. 

  – Joe

   Aug 29, 2019 at 13:23
• @ŞafakGür It may not. Seen that happening to many times, so as a reflex I just flush, like the sample. 

  – Erdogan Kurtur

   Sep 7, 2021 at 11:52

Add a comment

63

UPDATE: This no longer works in the current version, see below for correct answer (no need to vote down, this is correct on older versions).

Use the JsonTextReader class with a StreamReader or use the JsonSerializer overload that takes a StreamReader directly:

var serializer = new JsonSerializer();
serializer.Deserialize(streamReader);

Share

Improve this answer

Follow

edited May 23, 2017 at 12:34

CommunityBot

111 silver badge

answered Nov 16, 2011 at 20:29

Paul Tyng

7,87811 gold badge3333 silver badges5757 bronze badges

• 27

  Pretty sure this no longer works. You have to use a JsonReader or TextReader 

  – BradLaney

   Jul 7, 2012 at 0:08

• 12

  You may want to include the version number this is still working on so people know when to scroll down. 

  – PoeHaH

   Oct 24, 2017 at 5:00
• @BradLaney yup JsonTextReader(givenStreamReader) is the way to go now 

  – Antoine Meltzheim

   Apr 18, 2018 at 12:28
• Thank you for taking the time to edit your answer re it's working status and answer recommendation 

  – Nick Bull

   Mar 7, 2019 at 10:43

• 4

  "no need to vote down, this is correct on older versions" - It's no longer relevant to most people, so it should be voted down as a better answer exists. 

  – Ed S.

   Oct 7, 2020 at 17:15 

Add a comment

32

I've written an extension class to help me deserializing from JSON sources (string, stream, file).

public static class JsonHelpers
{
    public static T CreateFromJsonStream<T>(this Stream stream)
    {
        JsonSerializer serializer = new JsonSerializer();
        T data;
        using (StreamReader streamReader = new StreamReader(stream))
        {
            data = (T)serializer.Deserialize(streamReader, typeof(T));
        }
        return data;
    }

    public static T CreateFromJsonString<T>(this String json)
    {
        T data;
        using (MemoryStream stream = new MemoryStream(System.Text.Encoding.Default.GetBytes(json)))
        {
            data = CreateFromJsonStream<T>(stream);
        }
        return data;
    }

    public static T CreateFromJsonFile<T>(this String fileName)
    {
        T data;
        using (FileStream fileStream = new FileStream(fileName, FileMode.Open))
        {
            data = CreateFromJsonStream<T>(fileStream);
        }
        return data;
    }
}

Deserializing is now as easy as writing:

MyType obj1 = aStream.CreateFromJsonStream<MyType>();
MyType obj2 = "{\"key\":\"value\"}".CreateFromJsonString<MyType>();
MyType obj3 = "data.json".CreateFromJsonFile<MyType>();

Hope it will help someone else.

Share

Improve this answer

Follow

answered Jul 29, 2013 at 13:42

Tok'

56588 silver badges1111 bronze badges

• 2

  Against: it will pollute all strings with the extension methods. Workarounds: Only declare Using SomeJsonHelpersNamespace where needed or remove the this keyword and use JsonHelpers.CreateFromJsonString(someJsonString) Pro: it's so easier to use :) 

  – Tok'

   Jul 17, 2017 at 10:24 

• 2

  Although it could be seen as "polluting", almost half the extensions in String object could be seen the same way. This extends an object in a manner seen as useful to anyone that would consistently change from string(json) to JSON. 

  – vipersassassin

   Apr 3, 2018 at 21:44
• Also using Encoding.Default is bad as it will behave differently on different machines (see the big warning at the Microsoft docu). JSON is expected to be UTF-8 and this is what JsonSerializer expects. Thus it should be Encoding.UTF8. The code as is will produce garbled strings or fail to deserialize if non-ASCII characters are used. 

  – ckuri

   Oct 25, 2019 at 7:11 

Add a comment

17

I arrived at this question looking for a way to stream an open ended list of objects onto a System.IO.Stream and read them off the other end, without buffering the entire list before sending. (Specifically I'm streaming persisted objects from MongoDB over Web API.)

@Paul Tyng and @Rivers did an excellent job answering the original question, and I used their answers to build a proof of concept for my problem. I decided to post my test console app here in case anyone else is facing the same issue.

using System;
using System.Diagnostics;
using System.IO;
using System.IO.Pipes;
using System.Threading;
using System.Threading.Tasks;
using Newtonsoft.Json;

namespace TestJsonStream {
    class Program {
        static void Main(string[] args) {
            using(var writeStream = new AnonymousPipeServerStream(PipeDirection.Out, HandleInheritability.None)) {
                string pipeHandle = writeStream.GetClientHandleAsString();
                var writeTask = Task.Run(() => {
                    using(var sw = new StreamWriter(writeStream))
                    using(var writer = new JsonTextWriter(sw)) {
                        var ser = new JsonSerializer();
                        writer.WriteStartArray();
                        for(int i = 0; i < 25; i++) {
                            ser.Serialize(writer, new DataItem { Item = i });
                            writer.Flush();
                            Thread.Sleep(500);
                        }
                        writer.WriteEnd();
                        writer.Flush();
                    }
                });
                var readTask = Task.Run(() => {
                    var sw = new Stopwatch();
                    sw.Start();
                    using(var readStream = new AnonymousPipeClientStream(pipeHandle))
                    using(var sr = new StreamReader(readStream))
                    using(var reader = new JsonTextReader(sr)) {
                        var ser = new JsonSerializer();
                        if(!reader.Read() || reader.TokenType != JsonToken.StartArray) {
                            throw new Exception("Expected start of array");
                        }
                        while(reader.Read()) {
                            if(reader.TokenType == JsonToken.EndArray) break;
                            var item = ser.Deserialize<DataItem>(reader);
                            Console.WriteLine("[{0}] Received item: {1}", sw.Elapsed, item);
                        }
                    }
                });
                Task.WaitAll(writeTask, readTask);
                writeStream.DisposeLocalCopyOfClientHandle();
            }
        }

        class DataItem {
            public int Item { get; set; }
            public override string ToString() {
                return string.Format("{{ Item = {0} }}", Item);
            }
        }
    }
}

Note that you may receive an exception when the AnonymousPipeServerStream is disposed, I ignored this as it isn't relevant to the problem at hand.

Share

Improve this answer

Follow

answered Feb 24, 2014 at 22:39

Blake Mitchell

2,56711 gold badge2525 silver badges2222 bronze badges

• 1

  I need to modify this so that I can get any complete JSON object. My server and client communicate by sending snippets of JSON so the client could send {"sign in":{"username":"nick"}}{"buy item":{"_id":"32321123"}} and it needs to see this as two fragments of JSON signaling an event each time it reads a fragment. In nodejs this can be done in 3 lines of code. 

  – Nick Sotiros

   Jun 26, 2016 at 6:35

Add a comment

-1

another option that is handy when you are running out of memory is to periodically flush

/// <summary>serialize the value in the stream.</summary>
/// <typeparam name="T">the type to serialize</typeparam>
/// <param name="stream">The stream.</param>
/// <param name="value">The value.</param>
/// <param name="settings">The json settings to use.</param>
/// <param name="bufferSize"></param>
/// <param name="leaveOpen"></param>
public static void JsonSerialize<T>(this Stream stream,[DisallowNull] T value, [DisallowNull] JsonSerializerSettings settings, int bufferSize=1024, bool leaveOpen=false)
{
    using (var writer = new StreamWriter(stream,encoding: System.Text.Encoding.UTF32,bufferSize,leaveOpen))
    using (var jsonWriter = new JsonTextWriter(writer))
    {
        var ser = JsonSerializer.Create( settings );
        ser.Serialize(jsonWriter, value);
        jsonWriter.Flush();
    }
}

/// <summary>serialize the value in the stream asynchronously.</summary>
/// <typeparam name="T"></typeparam>
/// <param name="stream">The stream.</param>
/// <param name="value">The value.</param>
/// <param name="settings">The settings.</param>
/// <param name="bufferSize">The buffer size, in bytes, set -1 to not flush till done</param>
/// <param name="leaveOpen"> true to leave the stream open </param>
/// <param name="cancellationToken">Propagates notification that operations should be canceled.</param>
public static Task JsonSerializeAsync<T>(this Stream stream,[DisallowNull] T value, [DisallowNull] JsonSerializerSettings settings, int bufferSize=1024, bool leaveOpen=false, CancellationToken cancellationToken=default)
{
    using (var writer = new StreamWriter(stream,encoding: System.Text.Encoding.UTF32,bufferSize: bufferSize,leaveOpen: leaveOpen))
    using (var jsonWriter = new JsonTextWriter(writer))
    {
        var ser = JsonSerializer.Create( settings );
        ser.Serialize(jsonWriter, value);
        return  jsonWriter.Flush();
    }
    //jsonWriter.FlushAsnc with my version gives an error on the stream
    return Task.CompletedTask;
}

You can test/ use it like so:

[TestMethod()]
public void WriteFileIntoJsonTest()
{
    var file = new FileInfo(Path.GetTempFileName());
    try
    {
        var list = new HashSet<Guid>();
        for (int i = 0; i < 100; i++)
        {
            list.Add(Guid.NewGuid());
        }
        file.JsonSerialize(list);


        var sr = file.IsValidJson<List<Guid>>(out var result);
        Assert.IsTrue(sr);
        Assert.AreEqual<int>(list.Count, result.Count);
        foreach (var item in result)
        {
            Assert.IsFalse(list.Add(item), $"The GUID {item} should already exist in the hash set");
        }
    }
    finally
    {
        file.Refresh();
        file.Delete();
    }
}

you'd need to create the extension methods, here is the whole set:

public static class JsonStreamReaderExt
    {
       static JsonSerializerSettings _settings ;
        static JsonStreamReaderExt()
        {
            _settings = JsonConvert.DefaultSettings?.Invoke() ?? new JsonSerializerSettings();
            _settings.ConstructorHandling = ConstructorHandling.AllowNonPublicDefaultConstructor;
            _settings.DateTimeZoneHandling = DateTimeZoneHandling.Utc;
            _settings.DateFormatHandling = DateFormatHandling.IsoDateFormat ;
        }

    /// <summary>
    /// serialize the value in the stream.
    /// </summary>
    /// <typeparam name="T"></typeparam>
    /// <param name="stream">The stream.</param>
    /// <param name="value">The value.</param>
    public static void JsonSerialize<T>(this Stream stream,[DisallowNull] T value)
    {
        stream.JsonSerialize(value,_settings);
    }

    /// <summary>
    /// serialize the value in the file .
    /// </summary>
    /// <typeparam name="T"></typeparam>
    /// <param name="file">The file.</param>
    /// <param name="value">The value.</param>
    public static void JsonSerialize<T>(this FileInfo file,[DisallowNull] T value)
    {
        if (string.IsNullOrEmpty(file.DirectoryName)==true && Directory.Exists(file.DirectoryName) == false)
        { 
            Directory.CreateDirectory(file.FullName);
        }