1. Event Loop (Phases, Microtasks vs Macrotasks)

The Event Loop is the heart of Node.js's asynchronous execution model. It allows handling multiple tasks efficiently without blocking the main thread

Phases of the Event Loop

1. Timers Phase → Executes setTimeout() and setInterval() callbacks.

2. I/O Callbacks Phase → Handles asynchronous I/O operations (e.g., file read/write).

3. Idle, Prepare Phase → Internal Node.js operations.

4. Poll Phase → Retrieves new I/O events and executes callbacks.

5. Check Phase → Executes setImmediate() callbacks.

6. Close Callbacks Phase → Executes close event handlers (e.g., socket.on('close')).

Microtasks vs Macrotasks

• Microtasks (Higher Priority)

  • Always execute before the next phase of the Event Loop.

  • Includes:

    • process.nextTick()

    • Promises (.then(), .catch())

• Macrotasks (Lower Priority)

  • Executed after the current phase completes.

  • Includes:

    • setTimeout(), setImmediate()

    • I/O operations (file read/write, database queries)

Real-World Example: Handling Millions of Requests in E-commerce (Amazon, Flipkart)

Imagine Flipkart's Big Billion Days Sale, where millions of users are adding items to their cart. The backend needs to handle requests efficiently without blocking the server.

Example Code: Handling Orders in Flipkart

console.log("Start Processing Order");

setTimeout(() => console.log("Checking Stock (setTimeout)"), 0);
setImmediate(() => console.log("Applying Discount (setImmediate)"));

Promise.resolve().then(() => console.log("Payment Processing (Promise)"));
process.nextTick(() => console.log("Cart Validation (nextTick)"));

console.log("End Processing Order");

Expected Output:

Start Processing Order  
End Processing Order  
Cart Validation (nextTick)  
Payment Processing (Promise)  
Checking Stock (setTimeout) / Applying Discount (setImmediate) (Order May Vary)

Why?

• nextTick() and Promise run before macrotasks (setTimeout() and setImmediate()).

• This ensures critical tasks (Cart Validation, Payment) execute before non-critical tasks (Checking Stock, Discounts).

2. Buffers (Creating, Writing, and Reading Buffers)

Node.js Buffers handle binary data, making it useful for working with images, audio files, and video streams.

Real-World Example: Streaming Images in WhatsApp

When you send a photo on WhatsApp, the server must process the image in binary format before storing or transmitting it.

Example Code: Creating and Reading a Buffer

const fs = require("fs");

// Read an image file as a Buffer
fs.readFile("profile.jpg", (err, data) => {
    if (err) throw err;
    console.log("Image Buffer:", data);
});

The image is stored as a Buffer before being sent to the recipient or displayed in WhatsApp.

3. Streams (Readable, Writable, Duplex, and Transform)

Streams process large amounts of data without loading everything into memory, making them useful for video streaming, file processing, and logs.

Real-World Example: Netflix Video Streaming

When you watch a movie on Netflix, the server streams video data in chunks instead of loading the whole file. This reduces memory usage and allows smooth playback.

Example Code: Streaming a Large File (Like Netflix)

const fs = require("fs");
const http = require("http");

http.createServer((req, res) => {
    const stream = fs.createReadStream("movie.mp4"); // Read video file as a stream
    stream.pipe(res); // Send it to the client in chunks
}).listen(3000, () => console.log("Streaming server running"));

🔹 How it works?

• Instead of loading the entire movie into memory, the video streams in chunks, reducing buffering issues in Netflix.

4. Encrypting Files using Streams

Encryption is essential for securing sensitive data, such as storing passwords, transmitting financial data, or encrypting confidential files.

Real-World Example: Secure File Storage in Google Drive

When you upload a document to Google Drive, it is encrypted before storage to prevent unauthorized access.

Example Code: Encrypting a File Before Uploading to Google Drive

const crypto = require("crypto");
const fs = require("fs");

const algorithm = "aes-256-ctr";
const password = "SecureSecretKey"; // Key for encryption

const cipher = crypto.createCipher(algorithm, password);
const input = fs.createReadStream("confidential.txt");
const output = fs.createWriteStream("confidential.enc");

// Encrypt the file before saving it
input.pipe(cipher).pipe(output);

console.log("File Encrypted Successfully!");

How it works?

• The file is read as a stream, encrypted, and stored as confidential.enc.

• This ensures that even if someone steals the file, they cannot read it without the encryption key.

---

Summary of Real-World Use Cases

---

Express.js and API Design - Detailed Explanation with Real-World Examples

This list covers important concepts for designing efficient, secure, and scalable APIs using Express.js. Let's go through each topic in detail with real-world examples.

---

1. Custom Middleware

Middleware functions in Express.js are functions that execute between the request and response cycle. They can be used for logging, authentication, modifying requests, or handling errors.

Real-World Example: Logging User Requests in Uber

In a ride-booking app like Uber, every user request (e.g., booking a ride, canceling a ride) is logged to track activity and improve the user experience.

Example Code: Logging Middleware

const express = require("express");
const app = express();

// Custom middleware to log request details
const requestLogger = (req, res, next) => {
    console.log(`Request Method: ${req.method}, URL: ${req.url}`);
    next(); // Pass control to the next middleware
};

app.use(requestLogger);

app.get("/", (req, res) => {
    res.send("Welcome to Uber API!");
});

app.listen(3000, () => console.log("Server running on port 3000"));

🔹 How it works?

• Each time a user makes a request, it logs the HTTP method and URL, which helps in monitoring API usage.

2. Error Handling (AsyncHandler, Global Error Middleware)

Error handling ensures that the API handles failures gracefully instead of crashing.

Real-World Example: Handling Payment Failures in Amazon

If a payment request fails due to insufficient balance or an invalid card, the API should return a proper error message instead of a system crash.

Example Code: Handling Async Errors

const express = require("express");
const app = express();

// Async error handler wrapper
const asyncHandler = (fn) => (req, res, next) => {
    Promise.resolve(fn(req, res, next)).catch(next);
};

// Route that throws an error
app.get("/payment", asyncHandler(async (req, res) => {
    throw new Error("Payment Failed!");
}));

// Global Error Middleware
app.use((err, req, res, next) => {
    console.error(err.message);
    res.status(500).json({ message: err.message });
});

app.listen(3000, () => console.log("Server running on port 3000"));

🔹 How it works?

• The asyncHandler() function catches errors from async operations.

• The global error handler ensures that API failures return a JSON error response instead of crashing the server.

3. Rate Limiting

Rate limiting restricts the number of requests a user can make in a given period to prevent abuse.

Real-World Example: Preventing API Abuse in Twitter

Twitter limits the number of tweets a user can send per hour to prevent spamming.

Example Code: Applying Rate Limiting

const express = require("express");
const rateLimit = require("express-rate-limit");

const app = express();

// Apply rate limiting: max 5 requests per minute per IP
const limiter = rateLimit({
    windowMs: 1 * 60 * 1000, // 1 minute
    max: 5, // Limit each IP to 5 requests per windowMs
    message: "Too many requests, please try again later.",
});

app.use(limiter);

app.get("/", (req, res) => {
    res.send("Welcome to Twitter API!");
});

app.listen(3000, () => console.log("Server running on port 3000"));

🔹 How it works?

• If a user makes more than 5 requests per minute, they get blocked temporarily.

4. API Versioning Strategies (URL, Header, Content-Type)

API versioning allows introducing new features without breaking old ones.

Real-World Example: API Versioning in Google Maps

Google Maps provides v1, v2, v3 APIs, so older applications still work while new apps can use updated features.

Example Code: API Versioning Using URL

const express = require("express");
const app = express();

// Version 1 API
app.get("/api/v1/products", (req, res) => {
    res.json({ version: "v1", products: ["iPhone X", "Samsung S9"] });
});

// Version 2 API with new products
app.get("/api/v2/products", (req, res) => {
    res.json({ version: "v2", products: ["iPhone 13", "Samsung S22"] });
});

app.listen(3000, () => console.log("Server running on port 3000"));

🔹 How it works?

• Older apps can still use /api/v1/products.

• Newer apps can use /api/v2/products, ensuring backward compatibility.

5. Implementing CORS with Advanced Options

CORS (Cross-Origin Resource Sharing) allows secure API access from different domains.

Real-World Example: Enabling CORS for Stripe Payment Gateway

A frontend app using React (https://shop.com) needs to communicate with the Stripe Payment API (https://stripe.com).

Example Code: Enabling CORS with Advanced Options

const express = require("express");
const cors = require("cors");

const app = express();

// CORS Configuration
const corsOptions = {
    origin: ["https://shop.com", "https://admin.shop.com"], // Allow only these domains
    methods: ["GET", "POST", "DELETE"],
    allowedHeaders: ["Content-Type", "Authorization"],
};

app.use(cors(corsOptions));

app.get("/", (req, res) => {
    res.send("CORS Configured for Stripe API!");
});

app.listen(3000, () => console.log("Server running on port 3000"));

🔹 How it works?

• Only requests from allowed domains (shop.com, admin.shop.com) are accepted.

• Restricts methods to GET, POST, and DELETE for security.

  Summary Table of Concepts & Real-World Examples

  | Concept | Real-World Use Case | | --- | --- | | Custom Middleware | Uber logs all ride booking requests for tracking. | | Error Handling | Amazon API gracefully handles failed payments. | | Rate Limiting | Twitter limits how many tweets can be sent per hour. | | API Versioning | Google Maps provides v1, v2, and v3 APIs for compatibility. | | CORS | Stripe enables secure cross-origin requests for payment processing. |

  Mastering Express.js API Design helps you build scalable, secure, and optimized applications like Amazon, Twitter, Uber, and Google Maps.

Complete Guide to Using Redis with Node.js and Express

Redis is an in-memory data structure store commonly used as a cache, message broker, and real-time database. In this guide, I'll cover everything from setting up Redis to using its advanced features in a Node.js + Express application.

---

📌 1. What is Redis?

Redis (Remote Dictionary Server) is a fast, in-memory key-value store that supports various data structures like strings, lists, sets, hashes, sorted sets, and more.

🔹 Why Use Redis?

✅ Super fast – Data is stored in RAM instead of a disk.
✅ Supports advanced data structures – Lists, Sets, Sorted Sets, and Hashes.
✅ Persistence – Can store data permanently if needed.
✅ Pub/Sub Messaging – Used for real-time communication.
✅ Atomic Operations – Supports increment, decrement, and transactions.

🔹 Where is Redis Used?

📌 Caching: Speeding up database queries in applications like Amazon and Netflix.
📌 Session Storage: Storing user sessions in web applications.
📌 Rate Limiting: Preventing API abuse by setting request limits.
📌 Message Queue: Used in chat applications for real-time messaging.

---

📌 2. Setting Up Redis Locally

To use Redis, you need to install it on your system.

🔹 Install Redis on Windows (Using WSL)

sudo apt update
sudo apt install redis

Start Redis server:

redis-server

🔹 Install Redis on macOS

brew install redis
brew services start redis

🔹 Install Redis on Linux

sudo apt update
sudo apt install redis-server

Verify Redis is running:

redis-cli ping
# Output: PONG

📌 3. Connecting to Redis in Node.js

To use Redis in a Node.js + Express application, install the redis package:

npm install redis

🔹 Example Code: Connecting to Redis

const redis = require("redis");

// Create Redis client
const client = redis.createClient({
    host: "127.0.0.1",
    port: 6379
});

client.on("connect", () => console.log("Connected to Redis"));
client.on("error", (err) => console.error("Redis Error:", err));

// Close the connection when done
process.on("exit", () => {
    client.quit();
});

🔹 How it works?

• The createClient() function creates a connection to the Redis server running on localhost:6379.

• The connect event fires when the connection is successful.

---

📌 4. Basic CRUD Operations (SET, GET, DEL)

CRUD (Create, Read, Update, Delete) operations are fundamental in Redis.

🔹 Storing and Retrieving Data

client.set("user:1", "John Doe", (err, reply) => {
    console.log(reply); // OK
});

client.get("user:1", (err, data) => {
    console.log(data); // John Doe
});

🔹 Deleting a Key

client.del("user:1", (err, response) => {
    console.log(response); // 1 (if deleted)
});

🔹 Where is this used?

• Storing user sessions in Express.js applications.

• Caching API responses to improve performance.

---

📌 5. Atomic Operations (INCR, DECR)

Redis supports atomic operations to modify values without race conditions.

🔹 Incrementing and Decrementing a Counter

client.incr("page_views", (err, count) => {
    console.log("Page Views:", count);
});

client.decr("page_views", (err, count) => {
    console.log("Page Views:", count);
});

🔹 Where is this used?

• Counting user logins or API calls (rate limiting).

• Tracking page views in web applications.

---

📌 6. Redis Data Types

🔹 Strings

client.set("name", "Alice");
client.get("name", (err, data) => console.log(data)); // Alice

📌 Used for caching API responses.

🔹 Lists (FIFO Queue)

client.lpush("tasks", "Task1", "Task2"); // Add to list (left)
client.rpop("tasks", (err, data) => console.log(data)); // Remove from list (right)

📌 Used for message queues (e.g., background jobs).

🔹 Sets (Unique Values)

client.sadd("users", "Alice", "Bob", "Alice");
client.smembers("users", (err, data) => console.log(data)); // ['Alice', 'Bob']

📌 Used for removing duplicates from a dataset.

🔹 Sorted Sets (Leaderboard)

client.zadd("scores", 100, "Alice", 200, "Bob");
client.zrevrange("scores", 0, -1, "WITHSCORES", (err, data) => console.log(data));

📌 Used for real-time leaderboards (e.g., gaming apps).

---

📌 7. Pub/Sub Messaging (Real-time Communication)

Redis can be used as a message broker to send messages between services.

🔹 Example Code: Chat App

Publisher.js

const redis = require("redis");
const publisher = redis.createClient();

publisher.publish("chat", "Hello from User 1");

Subscriber.js

const redis = require("redis");
const subscriber = redis.createClient();

subscriber.subscribe("chat");
subscriber.on("message", (channel, message) => {
    console.log(`Received message: ${message}`);
});

📌 Used for real-time notifications and chat applications.

---

📌 8. Transactions (Atomic Multi-Operations)

client.multi()
    .set("user:100", "Alice")
    .incr("login_count")
    .exec((err, replies) => {
        console.log(replies);
    });

📌 Used for executing multiple Redis commands atomically.

---

📌 9. Pipelining (Faster Performance)

Instead of sending multiple commands individually, pipelining allows sending them in bulk.

const pipeline = client.pipeline();
pipeline.set("name", "Alice");
pipeline.get("name");
pipeline.exec((err, results) => {
    console.log(results);
});

📌 Used for reducing network latency in batch operations.

---

📌 10. Using ioredis (Better Redis Client)

ioredis is a more feature-rich Redis client than the default redis package.

🔹 Install ioredis

npm install ioredis

🔹 Example Code: Using ioredis

const Redis = require("ioredis");
const redis = new Redis();

redis.set("key", "value");
redis.get("key", (err, result) => console.log(result)); // value

📌 ioredis supports advanced features like clustering and Sentinel.

---

📌 Conclusion

Redis is a powerful in-memory database that speeds up web applications by handling caching, real-time data processing, and pub/sub messaging.

🔹 Summary of Use Cases

🚀 Next Steps:
🔹 Integrate Redis caching in your Express.js project
🔹 Use Redis for session management in authentication