Machine Learning with Clever Grid

Concepts in Clever Grid

Clever Grid is the machine learning oriented platform by Clever Cloud. It provides a unified GPU ecosystem to run your scripts.

Organization, Application, Add-on

In Clever Grid you create some applications under one organization. Organization is useful to manage roles and team collaboration.

An application is a stateless virtual machine which runs your code.

Additionally to an application, you can link one or more stateful virtual machine called add-on in Clever Grid. That lets you keep your data

Application as Runner, Application as Webapp

In some case, you may just need a one run of a script. To train a model or proceed a huge calculation for example. That the interest of Runner. A Runner runs a script and then go away.

A Webapp keeps the application up and serve via HTTP.

Create an application

Storage on VM

Remember that an application is stateless. To save your data, you need to use an add-on.
  1. Go to https://dashboard.clevergrid.io, create a “brand new application”.
  2. Select the application type :
    • Docker (Runner | Webapp)
    • Python (Runner | Webapp)
    • Java (Runner | Webapp)
  3. Select the size of your instance(s) and the number of nodes
  4. (optional) You can choose a Template for your application
    Template will set up for you a compete ecosystem for a specific application. It provides add-ons and environment variables for you.
    • Jupyter Notebook
    • R Notebook

Create an add-on

  1. Go to https://dashboard.clevergrid.io, Add a “a storage service”.
  2. Select the storage plan you need
  3. Link it to your applications
    Add-ons's credentials will be automatically set up in your application. Yet it can be found on the "Add-on dashboard" tabs of your dashboard by the Clever Cloud console.

Set-up an application

You can pass some parameter to your application by environments variables.

About port listening

If you deploy a Webapp, Clever Grid expects an exposed PORT as environment variable listening the port 8080
  • Go to dashboard.clevergrid.io and select your application
  • Manage your environment variables by the tab "Environment variables"

Python applications

If there is a requirements.txt in the root folder of your application, depedencies will be installed during the deployment.

  • Starting file

    • Clever Grid need to know which file runs to start your application. That can be a python file, declare in the CC_MLPYTHON_RUN_MAIN environment variable or custom bash script in CC_MLPYTHON_START_SCRIPT environment variable.
      It should contain the path of your main file. The path should be relative to the root of your repository.
      For example, “pytorch.py” and “./pytorch.py are both valid. Bash scripts are expected to be executable (via chmod+x)
  • Python version :

Docker application

Refer to the Clever Cloud documentation : https://www.clever-cloud.com/doc/docker/

Java application

Refer to the Clever Cloud documentation : https://www.clever-cloud.com/doc/java/java-jar/

Hooking build steps

Between the build steps and before running the script, you can hook scripts. The hooks are documented here: https://www.clever-cloud.com/doc/clever-cloud-overview/hooks/ E.g. you can have a post build hook to initialize conda and stuff.

Send your code

Your Clever Grid application is now set up to receive your code.

  • Go to your project and add the Clever Grid repository to your repository :

    git remote add clever-grid git+ssh://git@push-clevergrid-clevercloud-customers.services.clever-cloud.com/<APP_ID>
    You can find the repository address on the overview page of your application
  • Push your code

Command line management

In addition to the Clever Grid console, you can manage your add-ons and applications from the command line with Clever Tools.

see : https://www.clever-cloud.com/doc/clever-tools/getting_started/

Billing

Clever Grid application are reports on the Clever Cloud platform to billing purpose

Support

If you are having issues, please let us know support@clever-cloud.com

Edit me on GitHub