Apache Pulsar Public Beta Release

Apache Pulsar Public Beta release

It’s a big day for queuing and streaming enthusiasts today. Our engineering team is proud to release Apache Pulsar as an add-on on Clever Cloud in public beta!

Apache Pulsar is a distributed messaging and streaming platform based on a “publish-subscribe” model. Clever Cloud Pulsar add-on provides one Pulsar namespace, with almost all management rights.

Common use cases

  • Replicating data among databases using Pulsar IO is commonly used to distribute change events from databases.
  • Parallel processing and workflows. You can efficiently distribute a large number of tasks among multiple workers (compressing text files, sending email notifications).
  • Data streaming from IoT devices. For example, a residential sensor can stream data to backend servers.
  • Refreshing distributed caches. For example, an application can publish invalidation events to update the IDs of objects that have changed.
  • Real-time event distribution. Events, raw or processed, may be made available to multiple applications across your team and organization for real-time processing.

Biscuit token support

At Clever Cloud, we are working on the Biscuit token, the foundation for our authorization systems. Consequently, we open-sourced our Biscuit token for Pulsar implementation which is directly pluggable to the Pulsar authentication and authorization system.

This way, we can provide a Pulsar namespace with token-based authorizations. A generated access token enables you to operate your Pulsar namespace and its topics directly using the Pulsar Admin REST API.

Using the power of Biscuit tokens, you can also attenuate your Biscuit token and create a new token that is only able to consume a single topic or multiple topics matching a regex. Here is an example with a topic prefix.

The rights management we support is documented here.

Usage

We advise you to use pulsarctl provided by StreamNative. Here is an example to list topics in your namespace:

pulsarctl \
--admin-service-url $ADDON_PULSAR_HTTP_URL \
--auth-params $ADDON_PULSAR_TOKEN \
--auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
namespaces topics $ADDON_PULSAR_TENANT/$ADDON_PULSAR_NAMESPACE

As Biscuit is a token, you can use AuthenticationToken("") provided by clients libraries to authenticate to our clusters without any tweak.

  • C++ client
  • C# client
  • Go client
  • Java client (example)
  • Node.js client
  • Python client
  • Rust client (example)
  • WebSocket client

Storage policies

The Pulsar add-on comes with default retention policies and offload policies.

Retention policies

A freshly created Pulsar add-on has infinite retention policies, you can change it using:

# Example to set retention of namespace to 2 weeks and/or 100 GB
pulsarctl \
--admin-service-url $ADDON_PULSAR_HTTP_URL \
--auth-params $ADDON_PULSAR_TOKEN \
--auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
namespaces set-retention $ADDON_PULSAR_TENANT/$ADDON_PULSAR_NAMESPACE --time 2w --size 100G

Offload policies (cold storage)

Each Pulsar add-on is provided with a hidden Cellar add-on (object-storage add-on) used as a target for offloading data to cold storage. These policies are deactivated by default; you can enable offloading using:

# Example to set offload to run when hot storage is > 10G and put data to Cellar add-on as cold storage
pulsarctl \
--admin-service-url $ADDON_PULSAR_HTTP_URL \
--auth-params $ADDON_PULSAR_TOKEN \
--auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
namespaces set-offload-treshold $ADDON_PULSAR_TENANT/$ADDON_PULSAR_NAMESPACE 10G

Considering the previous retention of 100GB and the offload threshold of 10G. You will always have 10GB in the hot storage (Apache BookKeeper) and 90GB in Cellar.

We will provide in the add-on dashboard the current storage usage of your hot and cold storage.

Pricing

Today, the pricing is in beta, meaning that it is subject to change depending on our client’s usages.
Please try the calculator to simulate the cost of your usage.

Learn more on the Clever Cloud documentation

Blog

À lire également

Clever Cloud and Cloud Temple announce partnership

Cloud Temple, the reference secure cloud provider, and Clever Cloud, the leader in Platform as a Service (PaaS), have announced a strategic partnership to offer a complete, secure and sovereign French cloud solution, with the aim of making Clever Cloud's PaaS available on Cloud Temple's SecNumCloud-qualified infrastructures.
Company Press

Clever Cloud achieves ISO 27001 : 2022 certification

We are proud to announce that Clever Cloud has achieved a major milestone in its ongoing commitment to the security and confidentiality of our customers' data: ISO 27001 : 2022 certification.
Company Press

Clever Cloud structures itself to support its organic growth

In 2023, Clever Cloud has once again made great strides, with a significant increase in its turnover. Having recently passed the 60-strong mark, the company is welcoming new profiles to support its development, and is expanding its Management Committee.
Company Press