Buffer

A Buffer is a chunk of memory, just like you would have it in C/C++. You can interpret this memory as an array of integer or floating point numbers of various lengths, or as a binary string. Unlike higher-level data structures like arrays, a buffer is not resizable.

Array in nodeJS is grow-able

In simple words, a buffer is a raw binary data and a binary consist of bits that is just a 0 or a 1.

Every data in Nodejs is stored as raw binary data and everything is stored as 0 and 1 only, so buffers can store any kind of data in Buffers.

  • text file
  • image
  • video
  • an array
  • etc

A buffer is a region of a physical memory storage used to temporarily store data while it is being moved from one place to another. In node, each buffer corresponds to some raw memory allocated outside V8.

They are allocated outside of V8, meaning that they are not managed by V8. While V8 is generally high performance, sometimes it will move data unnecessarily.

Using a Buffer allows you to work around this and work with the memory more directly for higher performance.

The point of a Buffer is to simply store anything as an array of bytes. The reason this is important is because every thing in computing communicates in bytes.

Creating Buffers

There are a few ways to create new buffers:

var buffer = new Buffer(8);

var buffer = new Buffer([ 8, 6, 7, 5, 3, 0, 9]);

var buffer = new Buffer("I'm a string!", "utf-8")

This initializes the buffer to a binary encoding of the first string as specified by the second argument (in this case, utf-8). utf-8 is by far the most common encoding used with node, but Buffer also supports:

  • "ascii" : This encoding is way fast, but is limited to the ascii character set. Moreover, it will convert null characters into spaces, unlike the utf-8 encoding.
  • "ucs2" : A two-byte, little-endian encoding. Can encode a subset of unicode.
  • "base64" : Base64 string encoding.
  • "binary" : This is the "binary string" format mentioned earlier, and is in the process of being deprecated. Avoid its use.

Operations with Buffers

We can few operations using the Buffers in nodejs, i have listed few :

  • Writing to Buffers
  • Reading from Buffers buf.toString(‘utf8’,0,5)
  • Convert Buffer to JSON var json = buf.toJSON(buf);
  • Concatenate Buffers var buffer3 = Buffer.concat([buffer1,buffer2]);
  • Compare Buffers buf.compare(otherBuffer);
  • Copy Buffer buffer1.copy(buffer2);
  • Slice Buffer var buffer2 = buffer1.slice(0,9);
  • Buffer Length buffer.length

Writing Buffers

write() function in buffers, writes the specified value into a buffer, at the given position.

buffer.write(value, start, bytes, encoding);


  • value : Required. The string to insert
  • start : Optional. Where to start writing. Default 0
  • bytes : Optional. How many bytes to write. Default length of buffer minus start position
  • encoding : Optional. The encoding of the value. Default 'utf8'
  • Return Value : Number of octets written. If there is not enough space in the buffer to fit the entire string, it will write a part of the string.
Buffers does not always meant to write to files, we can use buffer for writing to console as well

var buf = Buffer(10);
buf.write('karthiq', 'utf8');

octet-buffer-nodejs From above example, the first argument to buffer.write is the string ("karthiq") to write to the buffer, and the second argument is the string encoding ("utf8").

In this case, second argument is not mandatory as "utf8" is the default encoding. buf.write() has returned 7, i.e. we have written seven bytes of the buffer. The length of the string "karthiq" is also seven.

Octets & Bytes :

All modern computer systems implement a byte as an 8-bit quantity. Octets and bytes are the same from this perspective.

For this reason, some people use the two terms interchangeably. Historically, however, computers have supported bytes having different numbers of bits; octets and bytes mean different things in this context.

Network professionals began using the term octet many years ago to maintain this distinction.

Computer systems engineers often use the term nibble when referring to a 4-bit quantity (half of one octet or byte) rather than calling it a "half octet" (or "quartet," as is common in music). write() function number of octet written in buffer


var buf = new Buffer(140);
len = buf.write("Learning is fun");

console.log("Octets written : "+  len);


What happens when we have text which need more buffer size than what we have created ?
Nodejs will write the part of the string or the value in to the buffer, remaining string will be left out.

For example, I have created the Buffer with size of 10 but i will try to write the text "Learning is fun" which has length of more than 10.


var buf = new Buffer(10);
len = buf.write("Learning is fun");

console.log(buf)
console.log("Octets written : "+  len);

more-than-created-buffer-nodejs

Reading from Buffers in Nodejs

The most common way to read buffers is to use the toString() function, since many buffers contain text:


var buf = new Buffer("Learning is fun");
buf.toString()

reading-buffer-nodejs

Convert Buffer to JSON

Buffers can convert to JSON, we can convert from Buffer to JSON using buf.toJSON(). This method returns a JSON-representation of the Buffer instance.


var buf = new Buffer("Learning is fun");
console.log(buf.toJSON());

buffer-json-nodejs-convert

We can also convert JSON to Buffer

Node.js – To convert JSON to Buffer, first convert JSON object to string JSON.stringify(jsonObj), then use Buffer.from(jsonStr) function to read the JSON string to a buffer.


var bio = '{"name":"karthiq", "site":"chercher tech"}';
 
var jsonObj = JSON.parse(bio);
 
// convert JSON object to String
var jsonStr = JSON.stringify(jsonObj);
 
// read json string to Buffer
const buf = Buffer.from(jsonStr);
 
console.log(buf.length);
console.log(buf.toString());

buffer-json-conversion-nodejs

Concatenate Buffers

The concat() method joins all buffer objects in an array into one buffer object.


Buffer.concat(list[, totalLength])

This is the syntax of the class method which is used to concatenate Node buffers to a single Node Buffer.

list – It is the array List of the Buffer objects which are to be concatenated.
totalLength − It is the total length of the buffers when they are concatenated.

concat() function returns a Buffer instance composed of the concatenated buffers.


var buffer1 = new Buffer('Chercher Tech ');
var buffer2 = new Buffer(' Learning is fun ');
var buffer3 = new Buffer(' -  ');
var buffer4 = Buffer.concat([buffer1, buffer3, buffer2]);
console.log("buffer4 content is : " + buffer4.toString());

concat-append-buffers-nodejs

Compare Buffers

compare() function compares two given buffer objects and returns a number defining their differences


var buffer1 = new Buffer('Chercher Tech ');
var buffer2 = new Buffer(' Learning is fun ');
var diff = Buffer.compare(buffer1, buffer2);
console.log(diff);

  • 0 if they are equal
  • 1 if buf1 is higher than buffer1
  • -1 if buf1 is lower than buffer2
Comparison is happening with the amount/capacity of the Buffer created


var buffer1 = new Buffer(5);
var buffer2 = new Buffer(3);
var diff = Buffer.compare(buffer1, buffer2);
console.log("Buffer1 with buffer2 ->"+ diff);
var diff2 = Buffer.compare(buffer2, buffer1);
console.log("Buffer2 with buffer1 ->"+ diff);

buffer-comparision-nodejs

Copy Buffers in nodejs

copy() function copies data from one buffer object into another buffer object.


								buffer.copy(target, targetStart, sourceStart, sourceEnd);
								

  • target : Required. The array of buffers to copy
  • targetStart : Optional. A number specifying where to begin copying to. Default 0
  • sourceStart : Optional. A number specifying where to begin copying from. Default 0
  • sourceEnd : Optional. A number specifying where to stop copying from. Default end of buffer

var buffer1 = Buffer.from('chercher');
var buffer2 = Buffer.from('1234');

buffer2.copy(buffer1, 2, 0, 4);

console.log(buffer1.toString());

copy-buffer-nodejs We can also copy to empty Buffer


var buff = Buffer('Nodejsera');
var newbuff = Buffer(20);
buff.copy(newbuff);
console.log("Content of newbuff :  " + newbuff.toString());

Slice Buffer

slice() function returns a new buffer which references the same memory as the old, but offset and cropped by the start (defaults to 0) and end (defaults to buffer.length) indexes.

Negative indexes start from the end of the buffer. Modifying the new buffer slice will modify memory in the original buffer!


buffer.slice(start, end);


start Number, Optional, Where to start slicing. Default 0
end Number, Optional, Where to end the slice, default at the very end of the buffer


var base = Buffer.from('chercher tech');
var sub = base.slice(2,6);

console.log(sub.toString());

slice-buffer-nodejs

Buffer Length

The length property is size of the buffer in bytes. Note that this is not necessarily the size of the contents. length refers to the amount of memory allocated for the buffer object. It does not change when the contents of the buffer are changed.


buf = new Buffer(1234);
console.log(buf.length);

buf.write("some string", 0, "ascii");
console.log(buf.length);

length-of-buffer-nodejs While the length property is not immutable, changing the value of length can result in undefined and inconsistent behavior.

Applications that wish to modify the length of a buffer should therefore treat length as read-only and use buf.slice to create a new buffer.


buf = new Buffer(10);
buf.write("abcdefghj", 0, "ascii");
console.log(buf.length); // 10
buf = buf.slice(0,5);
console.log(buf.length); // 5

NodeJS Buffer to Stream

Converting a Buffer into a stream, this is useful when you want to pipe data into an exist stream.


let Duplex = require('stream').Duplex;  
function bufferToStream(buffer) {  
  let stream = new Duplex();
  stream.push(buffer);
  stream.push(null);
  return stream;
}

This operation creates a duplex string and simply writes the buffer to it. The stream can now be used as any write stream normally would be.

NodeJS Buffer to Array

We can convert the Buffer into array using the Array.prototype.slice.call method in Nodejs


var buffer = new Buffer([1,2,3])
var arr = Array.prototype.slice.call(buffer, 0)
console.log(arr)		

NodeJS Buffer to Base64

Buffers can be used for taking a string or piece of data and doing base64 encoding of the result.
For example:


console.log(Buffer.from("Hello World").toString('base64'));

NodeJS Buffer to Images


var fs = require('fs');

// function to encode file data to base64 encoded string
function base64_encode(file) {
    // read binary data
    var bitmap = fs.readFileSync(file);
    // convert binary data to base64 encoded string
    return new Buffer(bitmap).toString('base64');
}

// function to create file from base64 encoded string
function base64_decode(base64str, file) {
    // create buffer object from base64 encoded string, 
	// it is important to tell the constructor 
	// that the string is base64 encoded
    var bitmap = new Buffer(base64str, 'base64');
    // write buffer to file
    fs.writeFileSync(file, bitmap);
    console.log('**** File created from base64 encoded string *****');
}

// convert image to base64 encoded string
var base64str = base64_encode('kitten.jpg');
console.log(base64str);
// convert base64 string back to image 
base64_decode(base64str, 'copy.jpg');		

About Author

Myself KarthiQ, I am the author of this blog, I know ways to write a good article but some how I donot have the skills to make it to reach people, would you like help me to reach more people By sharing this Article in the social media.

Share this Article Facebook
Comment / Suggestion Section
Point our Mistakes and Post Your Suggestions

Recent Addition

new tutorial Selenium Online Training : Our next online training course for Selenium with Java starts from 17th December 2018.

You can attend first 3 classes for free, the total course fee is INR 10,000

The course time would be 8.00 PM(IST) for the first three classes

If you are interested to learn, then you can join the course by sending email to chercher.tech@gmail.com

or Register below


 
Join My Facebook Group
Join Group