How JavascriptNode.js engine store JSON in memory? - Stack Overflow

admin管理员组

文章数量:1332389

I have a 70 MB JSON file, which I read from a Node.js script and assign to a variable.

console.log(process.memoryUsage());
let data=require('../newJSON.json');
console.log(process.memoryUsage());

Output:

{ rss: 28184576,
  heapTotal: 6283264,
  heapUsed: 4199672,
  external: 8252 }
{ rss: 724721664,
  heapTotal: 695595008,
  heapUsed: 663708016,
  external: 8252 }

It seems that 70 MB JSON takes 632 MB of memory. I am interested in understanding how does JSON is stored into memory by Node Js/ Javascript?

I have a 70 MB JSON file, which I read from a Node.js script and assign to a variable.

console.log(process.memoryUsage());
let data=require('../newJSON.json');
console.log(process.memoryUsage());

Output:

{ rss: 28184576,
  heapTotal: 6283264,
  heapUsed: 4199672,
  external: 8252 }
{ rss: 724721664,
  heapTotal: 695595008,
  heapUsed: 663708016,
  external: 8252 }

It seems that 70 MB JSON takes 632 MB of memory. I am interested in understanding how does JSON is stored into memory by Node Js/ Javascript?

• javascript
• json
• node.js
• memory-management
• v8

Share Follow asked Jul 10, 2017 at 8:00 anandanand 7271111 silver badges2020 bronze badges 8

• 1 I'm curious too! And if you save your JSON as a .js file with the only content being module.exports = <json> then it takes even ~2x more memory! – Lukasz Wiktor Commented Jul 10, 2017 at 8:49
• Yeah! for me module.exports caused FatalProcessOutOfMemory issue. – anand Commented Jul 10, 2017 at 9:08
• What kind of JSON is it, what's the structure and average element size? – Bergi Commented Jul 10, 2017 at 9:42
• In my case, it's a nested JSON with 5 sub level. – anand Commented Jul 10, 2017 at 9:52
• e.g. : {"8 char": {''6 char": {"1 char": { "8 char": {"1 char":<0 or 1> ,"1 char":<0 or 1>} } } } }. (8 char : average 8 characters in key) – anand Commented Jul 10, 2017 at 9:53

 |  Show 3 more ments

1 Answer 1

Sorted by: Reset to default 8

First off, JSON is just a string representation of objects. There is nothing special about "JSON objects" -- the JSON parser parses the JSON string and creates regular JavaScript objects from it. This:

var a = JSON.parse('{"foo": "bar"}');

and this:

var a = new Object(); a.foo = "bar";

are pletely equivalent.

---

Object storage in memory is plicated, because modern JavaScript engines have pretty nifty optimizations for various different circumstances depending on what your code is doing.

JSON string length and size of the corresponding object in memory are not strictly correlated; in most cases the JSON representation is expected to be smaller, sometimes by a lot. E.g. for the innermost nesting of your example: "a":0, takes 6 bytes, whereas for one more property in the created object, you need:

• one pointer for the property's name, "a"
• one pointer for the property's attributes (writable, enumerable, configurable)
• one pointer for the property's value, 0
• assuming the object is in dictionary mode: on average, approximately two pointers of slack

On a 64-bit platform, that adds up to ~40 bytes.

If you look at an entire object of similar shape: {"a":0,"b":1} is 13 characters, whereas the memory requirement is:

• "map" pointer
• "elements" pointer (unused)
• out-of-object "properties" pointer (unused)
• value of first property (0)
• value of second property (1)
• the object's "map": 11 pointers (could be shared with other objects of the same shape, but if you have only one such object, there's nothing to share it with)
• the object's map's property descriptors: 10 pointers

In total, 26 pointers or 208 bytes.

Lastly, there's a chance that some of the memory usage you see is from temporary objects that the GC will clean up over time.

本文标签： How JavascriptNodejs engine store JSON in memoryStack Overflow