Building an extensible HTTP client for GitHub in Java

At Clever Cloud, we use a lot of Java. A large part of our API is written using JEE (aka Java Enterprise Edition). Here is how we implemented the http client we use for our Github integration.

The standard HTTP implementation in JEE is Jersey, so that's what we'll use here.

A bit of background first

Our API is designed in a "micro-services" way. There's a central part and various external components.

Here, we'll be dealing with the central part, which itself is internally designed in a sort-of "micro-services" way. JEE offers a powerful dependency injection mechanism called CDI, which we use to build a lot of small tools we can then use just as we need.

Serialization

When you're building APIs and clients, one of the problematics is about serialization. We made a tool to do just that which you'll see in the code as MapperHelper, namely mh. It's basically a wrapper around jackson's ObjectMapper.

Let's get to work

We'll be working on an internal tool called GithubAPIHelper. We'll not get through the whole of it, but you'll quickly get the concept.

First, some standard JEE boilerplate injecting required tools using CDI.

@Stateless
public class GithubAPIHelper {

   @Inject
   private MapperHelper mh;
   @Inject
   private LogsHelper lh;

   private final static String GITHUB_URL = "https://api.github.com";
}

Now, we want to build an HTTP client using jersey-client, this is pretty straightforward. Btw, we create a method to make the client directly point at Github.

private Client getClient() {
   if (client == null) {
      client = ClientBuilder.newClient(new ClientConfig().property(ClientProperties.FOLLOW_REDIRECTS, true));
   }
   return client;
}

private WebTarget getTarget() {
   return getClient().target(GITHUB_URL);
}

What about authentication? Let's create a new method to get an authenticated request to Github:

private Invocation.Builder getBuilder(WebTarget target, String token) {
   return target.request().header("Authorization", "token " + token);
}

Ok, now that we have those basic pieces in place, we can start requesting for real.

Let's get the basic user profile, for starters

First, we create a basic data structure that we'll use to deserialize the payload from Github. I'll skip most of the available fields here as they're not needed.

public class GithubOwner implements Serializable {

   public int id;
   public String name;
   public String login;
   public String email;
   public String avatar_url;
}

Now, you should all be yelling at me about those public fields. Let me remember you that this data structure is only used for deserializing purpose, it won't ever be used for anything else, so what's the point in complicating things?

Now let's get back to business, we have a way to construct authenticated requests, and a model in which to store the data. So let's do it!

First, let's get to the right endpoint: /user

   getTarget().path("/user");

Now, let's actually issue the request, and get a response

   Response r = getBuilder(getTarget().path("/user"), token).get();

Let's deserialize the data from github

   GithubOwner owner = mh.readValue(r.readEntity(String.class), GithubOwner.class);

And here we are, with some integrity checks added:

   public GithubOwner getSelf(String token) {
      Response r = getBuilder(getTarget().path("/user"), token).get();
      if (r.getStatus() < 300) {
         try {
            GithubOwner owner = mh.readValue(r.readEntity(String.class), GithubOwner.class);
            return owner;
         } catch (IOException e) {
            lh.log(GithubAPIHelper.class.getName(), Level.SEVERE, null, e);
         }
      }
      return null;
   }

Dealing with pagination

Getting the user's profile was quite easy and straightforward, but everything isn't that smooth.

Let's say we now want all the repositories, instinctively, here is what we'd do:

public class GithubRepository implements Serializable {

   public long id;
   public String name;
   public String description;
   public GithubOwner owner;
   @JsonProperty(value = "private")
   public boolean isPrivate;
   public String ssh_url;
   public String git_url;
}
   private TypeSafeList<GithubRepository> getSelfRepositories(String token) {
      TypeSafeList<GithubRepository> repos = new TypeSafeArrayList<>();
      Response r = getBuilder(getTarget().path("/user/repos"), token).get();
      if (r.getStatus() < 300) {
         try {
            List<GithubRepository> ms = mh.readValue(r.readEntity(String.class), new TypeReference<List<GithubRepository>>() {
            });
            repos.addAll(new TypeSafeArrayList<>(ms));
         } catch (IOException e) {
            lh.log(GithubAPIHelper.class.getName(), Level.SEVERE, null, e);
         }
      }
      return repos;
   }

That will actually work, but you won't get all of them, only the first 20 or so. If you want all of them, you'll have to follow the pagination.

First, we want to get the link marked as rel=next in the Link http header.

   private String getNextLink(Response r) {
      String link = r.getHeaderString("Link");   /* <http://foobar>; rel="next", <http://blah/>; rel=last */
      if (link != null) {
         String[] links = link.split(",");
         for (String l : links) {               /* <http://foobar>; rel="next" */
            if (l.contains("rel=\"next\"")) {
               String[] tmp1 = l.split("<", 2);
               if (tmp1.length == 2) {
                  return tmp1[1].split(">", 2)[0];  /* http://foobar */
               }
            }
         }
      }
      return null;
   }

Next, we want to request this url and get the next data to our initial request

   private Response getNextData(Response r, String token) {
      String link = getNextLink(r);
      if (link == null)
         return null;
      return getBuilder(getClient().target(link), token).get();
   }

Now let's put all the pieces together, addind a loop to retrieve all the data

   private TypeSafeList<GithubRepository> getSelfRepositories(String token) {
      TypeSafeList<GithubRepository> repos = new TypeSafeArrayList<>();
      Response r = getBuilder(getTarget().path("/user/repos"), token).get();
      while (r != null) {
         if (r.getStatus() < 300) {
            try {
               List<GithubRepository> ms = mh.readValue(r.readEntity(String.class), newhTypeReference<List<GithubRepository>>() {
               });
               repos.addAll(new TypeSafeArrayList<>(ms));
            } catch (IOException e) {
               lh.log(GithubAPIHelper.class.getName(), Level.SEVERE, null, e);
            }
         }
         r = getNextData(r, token);
      }
      return repos;
   }

And voila, you now have a working Github HTTP client. Once you got these, the other endpoints are really easy.

Blog

À lire également

Materia KV: our easy-to-use serverless key-value database is available to all

Clever Cloud was born out of a desire to make life easier for developers, by providing them automation tools and interfaces, so that they can concentrate on their applications and sites.
Features

Clever Cloud and Vates: Two French champions join forces to create comprehensive and autonomous cloud solutions

Nantes and Grenoble, May 23, 2024 - Clever Cloud and Vates are announcing today a partnership aimed at strengthening the strategic autonomy of the cloud in France and Europe.
Press

The Values and Policies of Personal Information Chair at Institut Mines-Télécom welcomes Clever Cloud among its sponsors

The Institut Mines-Télécom's Values and Policies of Personal Information (VP-IP) Chair conducts research into the issues surrounding the processing of personal data, digital identities and sovereignty, in a context of European regulation.
Press