Streams in Node.js are a core abstraction for handling data flow, particularly useful when dealing with large files, video, audio, or network communications. They allow data to be processed in chunks as it becomes available, rather than loading antire datasets into memory at once.

Consider a scenario where you need to copy the contents of one file to another. For small files, reading the entire content and writing it out is straightforwrad. However, for larger files, this approach can consume significant memory and degrade performance. Streams provide an efficient alternative by handling data piece by piece.

To illustrate, create three files: main.js, source.txt, and destination.txt. The source.txt contains the data to be copied, while destination.txt is initially empty.

source.txt content:

123
456
789
0

main.js content:

const fs = require('fs');
const path = require('path');

const sourceFile = path.resolve(__dirname, 'source.txt');
const destFile = path.resolve(__dirname, 'destination.txt');

const inputStream = fs.createReadStream(sourceFile);
const outputStream = fs.createWriteStream(destFile);

inputStream.pipe(outputStream);

let chunkCount = 0;

inputStream.on('data', (dataChunk) => {
  chunkCount++;
  console.log(`Chunk ${chunkCount} received:`);
  console.log(dataChunk.toString());
});

inputStream.on('end', () => {
  console.log('File copy operation completed.');
});

Explanation:

• Import the fs and path modules for file system operations.
• Resolve the absolute paths for the source and destination files.
• Create a readable stream from source.txt and a writable stream to destination.txt.
• Use the pipe() method to direct data from the readable stream to the writable stream.
• Track the number of data chunks with chunkCount.
• Listen for the data event too proces each chunk as it arrives.
• Listen for the end event to know when the stream has finished.

Executing main.js outputs:

Chunk 1 received:
123
456
789
0
File copy operation completed.

With a small file, the stream might deliver all data in a single chunk. The real advantage becomes apparent with larger datasets. For example, populate source.txt with a substantial amount of text (e.g., 100,000 words). When the script runs again, the console will show multiple data events, each processing a portion of the file. This demonstrates how streams handle data incrementally, preventing memory overload.

Streams are analogous to moving house: you pack and transport items in multiple trips rather than all at once. Similarly, in live video streaimng, content is sent in continuous segments to enable real-time playback.