An Overview of Buffers in Node.js

Buffers are Node.js' built-in type for storing arbitrary binary data. Because most Node.js developers don't use buffers much beyond occasionally reading data from a file, buffers are a common source of confusion. In this article, I'll demonstrate how buffers work in Node.js, and describe a neat use case for buffers with MongoDB and Mongoose.

Working With Buffers

The Buffer class is a global variable in Node.js. Here's how you create a Buffer with 1 byte using the Node.js shell.

$ node -v
v8.9.4
$ node
> Buffer.alloc(1)
<Buffer 00>
>

Node.js introduced the Buffer.alloc() function in v5.10.0, so the above code won't work in older versions of Node.js. The above buffer isn't very useful, it just contains a single '0' byte. Let's create a more interesting buffer using the Buffer.from() function. Note that 0x is how you write a hexadecimal number in JavaScript, so console.log(0xf) prints "15".

const buf = Buffer.from([
  0x48,
  0x65,
  0x6c,
  0x6c,
  0x6f,
  0x2c,
  0x20,
  0x57,
  0x6f,
  0x72,
  0x6c,
  0x64
]);

Generally, the first thing to do with a buffer is to see what it contains using the toString() function. The toString() function takes an encoding parameter that tells Node.js how you want to interpret your buffer. For example, below is how you print the above buffer in hexadecimal:

buf.toString('hex'); // "48656c6c6f2c20576f726c64"

Node.js supports several encodings for buffers. For example, you can encode a buffer in base 64 for sending an email attachment or basic auth.

buf.toString('base64'); // "SGVsbG8sIFdvcmxk"

'utf8' and 'ascii' are two other useful encodings. Usually you want to use 'utf8' unless you notice the performance difference between the two.

buf.toString('utf8'); // "Hello, World"
buf.toString('ascii'); // "Hello, World"

Unless you specify an encoding, Node.js' fs.readFileSync() function returns a buffer.

fs.readFileSync('./test.txt'); // "<Buffer 48 65 6c 6c 6f 2c 20 57 6f 72 6c 64>"
fs.readFileSync('./test.txt').toString('hex'); // "48656c6c6f2c20576f726c64"
fs.readFileSync('./test.txt').toString('ascii'); // "Hello, World"

Buffers are similar to arrays. For example, you can take the byte at position 0 or iterate over a buffer using a for/of loop.

buf[0] === 0x48; // true
for (const v of buf) { console.log(v); } // "72", "101", ...

Passing a buffer to JSON.stringify() converts the buffer into an object:

// { type: 'Buffer',
//   data: [ 72, 101, 108, 108, 111, 44, 32, 87, 111, 114, 108, 100 ] }
JSON.parse(JSON.stringify(buf));

If you want to send a buffer to the client side, it is often more convenient to convert it to a hex string instead:

Buffer.prototype.toJSON = function() { return this.toString('hex') };
JSON.stringify({ buf: buf }); // '{"buf":"48656c6c6f2c20576f726c64"}'

You can then reconstruct the buffer using another form of the Buffer.from() function:

// "Hello, World"
Buffer.from('48656c6c6f2c20576f726c64', 'hex').toString('utf8');

Storing Buffers in MongoDB

The MongoDB Node.js driver and Mongoose have good support for buffers. Here's how you save a buffer using the MongoDB driver.

const { MongoClient } = require('mongodb');

run();

async function run() {
  const client = await MongoClient.connect('mongodb://localhost:27017/test');
  const db = client.db();

  const doc = {
    buf: Buffer.from('Hello, World')
  };

  await db.collection('Test').insertOne(doc);

  const fromDb = await db.collection('Test').findOne({ _id: doc._id });
  // Binary {
  //  _bsontype: 'Binary',
  //  sub_type: 0,
  //  position: 12,
  //  buffer: <Buffer 48 65 6c 6c 6f 2c 20 57 6f 72 6c 64> }
  console.log(fromDb.buf);
  console.log(fromDb.buf.buffer.toString()); // "Hello, World"
}

MongoDB actually stores buffers in a special Binary class. A MongoDB binary is just a wrapper around a buffer with an additional sub_type property that is useful for UUIDs. For the purposes of this article though, you can ignore the sub_type property and just use the buffer property to get a Node.js buffer.

You can also declare a Mongoose schema type as a buffer, so Mongoose can cast the buffer for you.

const mongoose = require('mongoose');

run();

async function run() {
  await mongoose.connect('mongodb://localhost:27017/test');

  const Model = mongoose.model('Foo', new mongoose.Schema({
    buf: Buffer
  }));

  // Mongoose will automatically call `Buffer.from()` for you
  const doc = await Model.create({ buf: 'Hello, World' });

  const fromDb = await Model.findOne({ _id: doc._id });

  // Unlike the MongoDB driver, Mongoose gives you a buffer back.
  console.log(fromDb.buf.toString('hex')); // 48656c6c6f2c20576f726c64
}

If you specify a path must have type Buffer, Mongoose will handle casting and give you a buffer back when you load the document from the database. The document will be stored as a Binary in MongoDB, but Mongoose will convert it to a buffer for you, and add a couple extra properties like _subtype.

Buffers are especially powerful in MongoDB when combined with bitwise query operators. Since a byte contains 8 bits, you can store whether you're available or not during each hour of a 24 hour interval using 24 bits (3 bytes), one for each hour. For 24 hours this doesn't offer much savings, but if you're looking to store half-hour availability for the next year, MongoDB bitwise query operators and buffers can save you a lot of network overhead.

const mongoose = require('mongoose');

run();

async function run() {
  await mongoose.connect('mongodb://localhost:27017/test');

  const Model = mongoose.model('Schedule', new mongoose.Schema({
    buf: Buffer
  }));

  const doc = await Model.create({
    // The 8th bit in the buffer (counted from the right) is set, rest are 0
    buf: [0b00000000, 0b00000001, 0b00000000]
  });

  console.log(await Model.countDocuments({ buf: { $bitsAllSet: [8] } })); // 1
  console.log(await Model.countDocuments({ buf: { $bitsAllSet: [0] } })); // 0
}

Moving On

Buffers are a fundamental part of Node.js, particularly when it comes to files and sockets. In addition to tasks like file uploads, buffers are useful for bitmapping, particularly when combined with MongoDB bitwise query operators. Next time you run into a buffer in Node.js, take a moment to think deeper about how buffers work.