Building an extensible HTTP client for GitHub in Java

At Clever Cloud, we use a lot of Java. A large part of our API is written using JEE (aka Java Enterprise Edition). Here is how we implemented the http client we use for our Github integration.

The standard HTTP implementation in JEE is Jersey, so that's what we'll use here.

A bit of background first

Our API is designed in a "micro-services" way. There's a central part and various external components.

Here, we'll be dealing with the central part, which itself is internally designed in a sort-of "micro-services" way. JEE offers a powerful dependency injection mechanism called CDI, which we use to build a lot of small tools we can then use just as we need.

Serialization

When you're building APIs and clients, one of the problematics is about serialization. We made a tool to do just that which you'll see in the code as MapperHelper, namely mh. It's basically a wrapper around jackson's ObjectMapper.

Let's get to work

We'll be working on an internal tool called GithubAPIHelper. We'll not get through the whole of it, but you'll quickly get the concept.

First, some standard JEE boilerplate injecting required tools using CDI.

@Stateless
public class GithubAPIHelper {

   @Inject
   private MapperHelper mh;
   @Inject
   private LogsHelper lh;

   private final static String GITHUB_URL = "https://api.github.com";
}

Now, we want to build an HTTP client using jersey-client, this is pretty straightforward. Btw, we create a method to make the client directly point at Github.

private Client getClient() {
   if (client == null) {
      client = ClientBuilder.newClient(new ClientConfig().property(ClientProperties.FOLLOW_REDIRECTS, true));
   }
   return client;
}

private WebTarget getTarget() {
   return getClient().target(GITHUB_URL);
}

What about authentication? Let's create a new method to get an authenticated request to Github:

private Invocation.Builder getBuilder(WebTarget target, String token) {
   return target.request().header("Authorization", "token " + token);
}

Ok, now that we have those basic pieces in place, we can start requesting for real.

Let's get the basic user profile, for starters

First, we create a basic data structure that we'll use to deserialize the payload from Github. I'll skip most of the available fields here as they're not needed.

public class GithubOwner implements Serializable {

   public int id;
   public String name;
   public String login;
   public String email;
   public String avatar_url;
}

Now, you should all be yelling at me about those public fields. Let me remember you that this data structure is only used for deserializing purpose, it won't ever be used for anything else, so what's the point in complicating things?

Now let's get back to business, we have a way to construct authenticated requests, and a model in which to store the data. So let's do it!

First, let's get to the right endpoint: /user

   getTarget().path("/user");

Now, let's actually issue the request, and get a response

   Response r = getBuilder(getTarget().path("/user"), token).get();

Let's deserialize the data from github

   GithubOwner owner = mh.readValue(r.readEntity(String.class), GithubOwner.class);

And here we are, with some integrity checks added:

   public GithubOwner getSelf(String token) {
      Response r = getBuilder(getTarget().path("/user"), token).get();
      if (r.getStatus() < 300) {
         try {
            GithubOwner owner = mh.readValue(r.readEntity(String.class), GithubOwner.class);
            return owner;
         } catch (IOException e) {
            lh.log(GithubAPIHelper.class.getName(), Level.SEVERE, null, e);
         }
      }
      return null;
   }

Dealing with pagination

Getting the user's profile was quite easy and straightforward, but everything isn't that smooth.

Let's say we now want all the repositories, instinctively, here is what we'd do:

public class GithubRepository implements Serializable {

   public long id;
   public String name;
   public String description;
   public GithubOwner owner;
   @JsonProperty(value = "private")
   public boolean isPrivate;
   public String ssh_url;
   public String git_url;
}
   private TypeSafeList<GithubRepository> getSelfRepositories(String token) {
      TypeSafeList<GithubRepository> repos = new TypeSafeArrayList<>();
      Response r = getBuilder(getTarget().path("/user/repos"), token).get();
      if (r.getStatus() < 300) {
         try {
            List<GithubRepository> ms = mh.readValue(r.readEntity(String.class), new TypeReference<List<GithubRepository>>() {
            });
            repos.addAll(new TypeSafeArrayList<>(ms));
         } catch (IOException e) {
            lh.log(GithubAPIHelper.class.getName(), Level.SEVERE, null, e);
         }
      }
      return repos;
   }

That will actually work, but you won't get all of them, only the first 20 or so. If you want all of them, you'll have to follow the pagination.

First, we want to get the link marked as rel=next in the Link http header.

   private String getNextLink(Response r) {
      String link = r.getHeaderString("Link");   /* <http://foobar>; rel="next", <http://blah/>; rel=last */
      if (link != null) {
         String[] links = link.split(",");
         for (String l : links) {               /* <http://foobar>; rel="next" */
            if (l.contains("rel=\"next\"")) {
               String[] tmp1 = l.split("<", 2);
               if (tmp1.length == 2) {
                  return tmp1[1].split(">", 2)[0];  /* http://foobar */
               }
            }
         }
      }
      return null;
   }

Next, we want to request this url and get the next data to our initial request

   private Response getNextData(Response r, String token) {
      String link = getNextLink(r);
      if (link == null)
         return null;
      return getBuilder(getClient().target(link), token).get();
   }

Now let's put all the pieces together, addind a loop to retrieve all the data

   private TypeSafeList<GithubRepository> getSelfRepositories(String token) {
      TypeSafeList<GithubRepository> repos = new TypeSafeArrayList<>();
      Response r = getBuilder(getTarget().path("/user/repos"), token).get();
      while (r != null) {
         if (r.getStatus() < 300) {
            try {
               List<GithubRepository> ms = mh.readValue(r.readEntity(String.class), newhTypeReference<List<GithubRepository>>() {
               });
               repos.addAll(new TypeSafeArrayList<>(ms));
            } catch (IOException e) {
               lh.log(GithubAPIHelper.class.getName(), Level.SEVERE, null, e);
            }
         }
         r = getNextData(r, token);
      }
      return repos;
   }

And voila, you now have a working Github HTTP client. Once you got these, the other endpoints are really easy.

Blog

À lire également

SuperBOL: The COBOL revolution in the Cloud

COBOL, a programming language that is over 60 years old, continues to power a large proportion of the IT systems of the world's major companies, particularly in the financial and insurance sectors.
Features

Clever Cloud welcomes the first startups to the UP Programme

Clever Cloud is proud to announce the arrival of the first five startups selected to join its UP Programme, an initiative dedicated to supporting young technology companies in their growth phase.
Company

A minor update resulted in a cascade of errors: how it went wrong, what we’ve learnt

On Friday, August 2nd, 2024 Clever Cloud’s platform became very unstable, leading to downtime of varying duration and scope, for customers using services on the EU-FR-1 (PAR) region, and remote zones depending on the EU-FR-1 control plane (OVHcloud, Scaleway, and Oracle). Privates and on-premise zones weren’t impacted.
Company Engineering