On this page

Deploy a fluentd

Overview

Since you deploy microservices on Clever Cloud, you may need some data pipes between your services to:

  • collect data from your PostgreSQL to create Elasticsearch indexes for your website search engine
  • collect application logs to analyze them with Elasticsearch, then archive them with S3
  • collect Apache access logs to analyze them in MongoDB
  • extract data from the database of your PHP/MySQL application to transform then load them in your other node.js/PostgreSQL application
  • and many more …

Fluentd is an open source data collector written in Ruby, which lets you unify the data collection and consumption for a better use and understanding of data.

Create an application on Clever Cloud

via the web console

  1. Create a new app by clicking on the Add an Application button, in the sidebar.
  2. Select a brand new instance (or a repository from GitHub if your account is linked).
  3. Then select Docker in the platform list.
  4. Configure your scaling options.
  5. Enter your application’s name and description and click “Next”. You can also select the region you want (North America or Europe).

Refer to https://www.clever-cloud.com/doc/getting-started/ for more details on application creation via the console.

via the Clever Tools CLI

  1. Make sure you have clever-tools installed locally. Report to https://www.clever-cloud.com/doc/reference/clever-tools/getting_started/
  2. In your code folder, do clever create --type <type> <app-name> --region <zone> --org <org> where type is the type of technology you rely on, app-name the name you want for your application, zone deployment zone (par for Paris and mtl for Montreal), and org the organization ID the application will be created under.

Refer to https://www.clever-cloud.com/doc/reference/clever-tools/create/ for more details on application creation with Clever Tools.

Setting up environment variables on Clever Cloud

With the Clever Cloud console

  1. Go to the Clever Cloud console, and find the app you want to fine tune under it’s organization.
  2. Find the Environment variables menu and select it.
  3. In this menu, you will see a form with VARIABLE_NAME and variable value fields. Fill them with the desired values then select Create.

With the Clever Tools CLI

  1. Make sure you have clever-tools installed locally. Report to https://www.clever-cloud.com/doc/reference/clever-tools/getting_started/
  2. In your code folder, do clever env set <variable-name> <variable-value>

Refer to https://www.clever-cloud.com/doc/reference/reference-environment-variables/ for more details on available environment variables on Clever Cloud. You can of course create custom ones with the interface we just demonstrated, they will be available for your application.

Configure your Fluentd + Docker application

Mandatory configuration

To follow this tutorial, you will need:

  • Ruby >= 2.4.4 (w/ Rubygems)
  • Bundler
  • Docker
  • Git
  • curl
  • a Ruby versions manager

My application does not exists already

Create a fluentd application locally

$ mkdir myFluentd
$ cd myFluentd
$ touch Gemfile Dockerfile go.sh td-agent.conf
$ chmod +x go.sh

Inside Gemfile put the following:

source 'https://rubygems.org'

ruby '2.4.4'

gem 'fluentd'
gem 'fluent-plugin-td'

Then run bundler to install dependencies and generate your Gemfile.lock

$ bundle install

Clever Cloud needs that your application answers on requests made on 0.0.0.0:8080, we’ll use a PORT environment variable for local test purposes (this variable is automatically setup on each application). Inside td-agent.conf put the following:

<source>
  @type monitor_agent
  bind 0.0.0.0
  port "#{ENV['PORT']}"
</source>

Inside go.sh put the following:

#!/bin/sh
bundle exec fluentd --use-v1-config -c td-agent.conf
echo "🌍 Fluentd server started"

Test locally

Start you service

$ PORT=9292 ./go.sh

Verify that it responds to requests

$ curl 0.0.0.0:9292

You can now read My application already exists

Fine tune you application

You can update your configuration with all inputs, filters and outputs you need or check for a community based plugin.

My application already exists

Prepare your application for deployment

Create a Dockerfile at the root of your project and put inside the following (assuming your start script is in go.sh):

FROM ruby:2.4.4
EXPOSE 8080
COPY Gemfile Gemfile.lock td-agent.conf go.sh ./

RUN bundle config --global frozen 1
RUN bundle install
RUN chmod +x go.sh

CMD [ "/go.sh" ]

Create an application on Clever Cloud

via the web console

  1. Create a new app by clicking on the Add an Application button, in the sidebar.
  2. Select a brand new instance (or a repository from GitHub if your account is linked).
  3. Then select Docker in the platform list.
  4. Configure your scaling options.
  5. Enter your application’s name and description and click “Next”. You can also select the region you want (North America or Europe).

Refer to https://www.clever-cloud.com/doc/getting-started/ for more details on application creation via the console.

via the Clever Tools CLI

  1. Make sure you have clever-tools installed locally. Report to https://www.clever-cloud.com/doc/reference/clever-tools/getting_started/
  2. In your code folder, do clever create --type <type> <app-name> --region <zone> --org <org> where type is the type of technology you rely on, app-name the name you want for your application, zone deployment zone (par for Paris and mtl for Montreal), and org the organization ID the application will be created under.

Refer to https://www.clever-cloud.com/doc/reference/clever-tools/create/ for more details on application creation with Clever Tools.

Setting up environment variables on Clever Cloud

With the Clever Cloud console

  1. Go to the Clever Cloud console, and find the app you want to fine tune under it’s organization.
  2. Find the Environment variables menu and select it.
  3. In this menu, you will see a form with VARIABLE_NAME and variable value fields. Fill them with the desired values then select Create.

With the Clever Tools CLI

  1. Make sure you have clever-tools installed locally. Report to https://www.clever-cloud.com/doc/reference/clever-tools/getting_started/
  2. In your code folder, do clever env set <variable-name> <variable-value>

Refer to https://www.clever-cloud.com/doc/reference/reference-environment-variables/ for more details on available environment variables on Clever Cloud. You can of course create custom ones with the interface we just demonstrated, they will be available for your application.

Environment injection

Clever Cloud injects environment variables from your application settings as mentioned in setting up environment variables and is also injecting in your application production environment, those from your linked add-ons.

Git Deployment on Clever Cloud

You need git on your computer to deploy via this tool. Here is the official website of Git to get more information: git-scm.com

Setting up your remotes

  1. The “Information” page of your app gives you your git deployment URL. It looks like this: git+ssh://git@push.clever-cloud.com/<your_app_id>.git. Copy it in your clipboard.

  2. Locally, under your code folder, type in git init to set up a new git repository or skip this step if you already have one.

  3. Add the deploy URL with git remote add <name> <your-git-deployment-url>.

  4. Add your files via git add <files path> and commit them via git commit -m <your commit message>.

  5. Now push your application on Clever Cloud with git push <name> master

Refer to https://www.clever-cloud.com/doc/getting-started/quickstart/ for more details on git deployments.

Linking a database or any other add-on to your application

By linking an application to an add-on, the application has the add-on environment variables in its own environment by default.

On add-on creation

Many add-ons do exist on Clever Cloud, please refer to the full list and check add-ons dedicated pages for full instructions.

During add-on creation, you will see Applications screen, with a list of your applications. You can toggle the button to Link and click next. If you finish the process of add-on creation, the application will automatically be linked to it.

Add-on already exists

In the Clever Cloud console, under the Service Dependencies menu of your application, you can use the Link addons dropdown menu to select the name of the add-on you want to link and use the add button to finish the process.

You can also link another application from the same page in the Clever Cloud console, using the Link applications dropdown menu.

More configuration

Need more configuration? To run a script at the end of your deployment? To add your private SSH key to access private dependencies? Go check the Common configuration page.

You may want to have an advanced usage of your application, in which case we recommend you to read the Administrate documentation section.

If you can’t find something or have a specific need like using a non supported version of a particular software, please reach out to the support.