Home

Home

Kafka-Penguin

An easy-to-use, lightweight KafkaJS library for message re-processing.

Table of Contents

• About

• Getting Started

• Example

• Strategies

• Contributors

• License

About

Kafka-Penguin is an easy-to-use, lightweight KafkaJS plugin for message re-processing. It provides developers with three strategies for setting up message re-processing: FailFast, Ignore, and Dead Letter Queue.‌

The package allows developers to build event-driven applications with dedicated "fail strategies" modeled after best practices in the field. This in turn allows developers to effectively address bugs in development and deploy more fault-tolerant systems in production.‌

This package is meant to work in conjunction with with KafkaJS. For more information on KafkaJS, check out Getting Started with KafkaJS.‌

Getting Started

Install Kafka-Penguin as an npm module and save it to your package.json file as a dependency:

npm install kafka-penguin

Once installed it can now be referenced by simply calling require('kafka-penguin');‌

Example

All Kafka-Penguin needs is a KafkaJS client to run. Start by passing the client for your preferred strategy and Kafka-Penguin will create bespoke consumers, producers, and admins with built-in functionality to execute the chosen strategy. On the surface, you implement your application exactly as you would with KafkaJS.

/* eslint-disable no-console */
import { FailFast } from 'kafka-penguin';

const exampleClient = require('./clientConfig.ts');

// Set up the preferred strategy with a configured KafkaJS client
const exampleStrategy = new FailFast(2, exampleClient);

// Initialize a producer or consumer from the instance of the strategy
const producer = exampleStrategy.producer();

const message = {
  topic: 'wrong-topic',
  messages: [
    {
      key: 'hello',
      value: 'world',
    },
  ],
};

// Connect, Subscribe, Send, or Run virtually the same as with KafkaJS
producer.connect()
  .then(() => console.log('Connected!'))
  // The chosen strategy executes under the hood, like in this send method
  .then(() => producer.send(message))
  .catch((e: any) => console.log('error: ', e.message));

Strategies

Dive in deeper to any of the strategies for set up, execution, and implementation.

​FailFast ​‌

​Ignore​‌

​Dead Letter Queue​‌

Contributors

​Ausar English @ausarenglish​‌

​Kushal Talele @ktrane1​‌

​Timeo Williams @timeowilliams​‌

​Ziyad El Baz @zelbaz946​‌

License

This product is licensed under the MIT License - see the LICENSE.md file for details.‌

This is an open source product. We are not affiliated nor endorsed by either the Apache Software Foundation or KafkaJS.‌

This product is accelerated by OS Labs.