GitHub GraphQL API (v4) support.

[dmitshur]

This is a long-term tracking/planning issue for GitHub GraphQL API (v4). /cc @willnorris @gmlewis

Today, GitHub has announced that the GitHub GraphQL API is out of the Early Access program. The documentation is thorough, and very helpful at explaining the neccessary concepts and how to get started with using the API. (Note that it requires authentication; it's no longer possible to make unauthenticated API calls.)

Announcement: https://github.com/blog/2359-introducing-github-marketplace-and-more-tools-to-customize-your-workflow#user-content-github-graphql-api
GitHub API Documentation: https://developer.github.com/v4/

GitHub GraphQL API v4 vs REST API v3

The GraphQL API v4 is significantly different from the REST API v3, which this repository currently implements. There's very little in common, so support will be offered in a separate package (and maybe even separate repo).

Implementing this will require investigation and discovery. What does a good Go client library for a GraphQL look like? We don't know yet, we'll need to try and see.

I think the discovery part is best prototyped in separate repositories. That way, no one's blocked on anyone, and different approaches can be tested out by different people. We can coordinate and report on progress here.

Once there's agreement on the best API/solution, we can decide to either:

• Move the package into this repository, perhaps under githubql or v4/github subfolder.
• Keep the package in a separate repository, but link to it from README.

Initial Observations

I've started looking into this, and here are some observations:

• It's very likely the end result package can be entirely generated. The GraphQL schema is entirely introspectable and very structured, and strongly typed. See https://developer.github.com/v4/guides/intro-to-graphql/#discovering-the-graphql-api. Aside from prototyping an initial version, there's really no need to write/maintain the entire package manually.

• It might be the case that one Go type per GraphQL fragment would be a good approach.

• My initial idea (that I'm thinking about) on how a Go client for a GraphQL API can look like is similar to json.Unmarshal API, with the githubql package providing all the needed types to compose queries. For example:

  /*
  The following GraphQL query:
  
  query {
    viewer {
      login
      createdAt
    }
  }
  
  Can look *roughly* like this:
  */
  var query struct {
      Viewer struct {
          Login     githubql.String
          CreatedAt githubql.DateTime
          // other fields you're interested in, expressed in some Go-compatible way
      }
  }
  err := githubqlClient.Do(ctx, &query)
  if err != nil {
      // handle error
  }
  
  fmt.Println("current user:", query.Viewer.Login, query.Viewer.CreatedAt)
  
  // Output:
  // current user: gopher 2016-03-07T23:46:14Z

  However, the devil is in details. Figuring out a way to make this work in a general case for all types of GraphQL queries is the required investigation/discovery effort. Perhaps this is not viable at all, and a completely different approach is better.

• It might turn out that it's not viable to create a helpful Go client package for GraphQL. It might turn out to be easier to just make ad-hoc requests. We will find out.

• Nothing about this is GitHub specific. It's GraphQL specific. A good solution to this will likely apply to any other GraphQL API.

• I've looked at the starwars example in github.com/neelance/graphql-go, a good Go GraphQL package (/cc @neelance). However, that package is for creating GraphQL servers, and the example simply uses GraphiQL for the client.

I'm not an expert on GraphQL, just looking into it now. I might be missing some existing efforts/knowledge about creating Go clients for GraphQL APIs. If so, any insight or links would be appreciated.

[willnorris]

Thanks for looking into this... this has long been in the back of my mind as something we need to do. I do like the idea of using an existing graphql library if possible... I seem to recall finding one, but maybe it was neelance/graphql-go and I didn't realize it was primarily for servers. But at this point, I fully trust you to take this and run with it :)

We'll almost certainly need to move this into a new go package, but that's an implementation detail we can sort out later.

[dmitshur]

I've been working on this during this week, and I believe I've gotten to a point where I have something that looks like a good start and is worth sharing.

First, let me share my experience and insight gained along the way. Then I'll point to the package I've created.

I believe there are 3 different high-level approaches for a Go GraphQL client library:

1. The user provides only the query; the library constructs a new type or uses an existing type to populate response into and returns it.
2. The user provides only the Go type to populate response into; the library constructs the query.
3. The user provides (to the library) both the query, and the type to populate response into.

Option 3 is easiest to implement. It's also the most flexible and safe approach – it's very unlikely to run into issues with dealing with some complicated GraphQL queries. But it puts the most burden on the user, and the usage is more verbose. Also, the user needs to manually keep the query and response type in sync, or face issues.

There is a very direct relationship between the GraphQL query, and the response type. That's the very core of what GraphQL is all about. So I think it would be very unfortunate to make the user provide both the query (a string) and the Go type to populate response into, and not leverage the fact those two things have a tight mapping between each other. But, it might be hard, because we need to find a good way to represent GraphQL queries (fairly complex and expressive language) using the Go type system (fairly rigid, simple).

I've started by thinking/prototyping about APIs that use option 1 or 2, because IMO they lead to best outcome if viable. If not viable, then I think there's no choice but to fall back to 3.

However, what I've got so far (option 2) looks quite promising, and hopefully it scales to all advanced GraphQL queries.

Option 1 (user provides query) vs Option 2 (user provides response type)

Here's what I mean by option 1-style API:

// User passes a GraphQL query as string, library constructs a response and returns it.
var query string = `query {
    viewer {
        login
        createdAt
    }
}`
resp, err := githubqlClient.Do(ctx, query)
if err != nil {
    // handle error
}

// The exact type of resp is unclear. If it's interface{}, then using it will be very hard
// because you'll constantly need to do type assertions... So it has to be predeclared types.
fmt.Println("current user:", resp) // ?

Here's what I mean by option 2-style API:

// User passes a response type, and library constructs a corresponding GraphQL query from it.
var query struct {
    Viewer struct {
        Login     githubql.String
        CreatedAt githubql.DateTime
    }
}
err := githubqlClient.Do(ctx, &query)
if err != nil {
    // handle error
}

// This is very clear, readable and obvious. No surprises. You use what you constructed.
fmt.Println("current user:", query.Viewer.Login, query.Viewer.CreatedAt)

At first, both options looked quite reasonable to me. I wasn't sure which of the two would win in the end. They have tradeoffs.

Option 1 lets people use GraphQL queries directly and easily. It's just a string you pass to the library. This has advantages, especially for complicated GraphQL queries that are hard to represent with Go type system. It also means all GraphQL queries will be possible, even if GraphQL spec changes to support more exotic syntax.

But the downsides are that the response it returns would either have to be a predeclared type that contains all possible fields, and only the ones specified by the query would be populated... Or, it uses reflection and creates a custom type that contains only the needed fields, but then it has no choice but to return that as interface{}, and users would have a nightmare with forced type assertions. It could also return map[string]interface{}, but that's still pretty messy.

Option 2 makes it very clear what the response type will be and what it'll contain - it's exactly what you write. But, you have to express a GraphQL query using Go type, which might not be as obvious or intuitive. There may be some exotic GraphQL query syntax that wouldn't be possible to express.

Going with Option 2 (for now)

I've considered both option 1 and 2, but started to lean more towards option 2. It seemed riskier, but also more promising if it worked out. So, in the next section I'll discuss challenges and solutions of option 2-style API, since that's the more promising approach so far.

GraphQL Arguments

Let's look at a more complicated GraphQL query, where the challenge of option 2 becomes apparent. How to represent this GraphQL query?

query {
    repository(owner: "octocat", name: "Hello-World") {
        description
    }
}

If you just write:

var q struct {
    Repository struct {
        Description githubql.String
    }
}

It's not obvious how to express that repository should have arguments owner "octocat" and name "Hello-World".

At first I attempted to use a special field named Arguments:

var q struct {
    Repository struct {
        Arguments githubql.RepositoryArguments

        Description githubql.String
    }
}
q.Repository.Arguments = githubql.RepositoryArguments{Owner: "octocat", Name: "Hello-World"}

This worked initially for simple queries such as the one above, but proved not to scale well. As soon as you have slices involved, trying to combine a type with values becomes very problematic. Consider:

var q struct {
    Repository struct {
        Issue struct {
            Comments struct {
                Nodes []struct {
                    Author struct {
                        Login githubql.String

                        // How to provide an (size: 72) argument to AvatarURL here?
                        // q.Repository.Issue.Comments.Nodes is an empty slice...
                        AvatarURL graphql.URI

I then came up with the idea of putting arguments into Go's struct field tags, and tried that:

var q struct {
    Repository struct {
        Description githubql.String
    } `graphql:"repository(owner: \"octocat\", name: \"Hello-World\")"`
}

Which makes the above complex query possible, easy even:

var q struct {
    Repository struct {
        Issue struct {
            Comments struct {
                Nodes []struct {
                    Author struct {
                        Login     githubql.String
                        AvatarURL githubql.URI `graphql:"avatarUrl(size: 72)"`

This also lets you take care of various more advanced GraphQL features, such as:

# Aliases.
query {
    helloRepo: repository(owner: "octocat", name: "Hello-World") {
        description
    }
    spoonRepo: repository(owner: "octocat", name: "Spoon-Knife") {
        description
    }
}

// Are possible.
var q struct {
    HelloRepo struct {
        Description githubql.String
    } `graphql:"helloRepo: repository(owner: \"octocat\", name: \"Hello-World\")"`

    SpoonRepo struct {
        Description githubql.String
    } `graphql:"spoonRepo: repository(owner: \"octocat\", name: \"Spoon-Knife\")"`
}

# Directives.
{
    friend @include(if: $withFriend) {
        name
    }
}

// Are possible.
var q struct {
    Friend struct {
        Name githubql.String
    } `graphql:"friend @include(if: $withFriend)"`
}

# Inline fragments.
hero {
    name
    ... on Droid {
        primaryFunction
    }
    ... on Human {
        height
    }
 }

// Should be possible! I haven't tried/tested this code yet, but it seems like it'd work.
type DroidFragment struct {
    PrimaryFunction githubql.String
}
type HumanFragment struct {
    Height githubql.Float
}
var q struct {
    Hero struct {
        Name          githubql.String
        DroidFragment `graphql:"... on Droid"`
        HumanFragment `graphql:"... on Human"`
    }
}

But it still has another problem. Struct field tags are constant and need to be provided at compilation time. What if their values need to be variables?

Luckily, there's a solution. GraphQL supports passing variables. So I came up with this solution:

var q struct {
    Repository struct {
        Description githubql.String
    } `graphql:"repository(owner: $RepositoryOwner, name: $RepositoryName)"`
}
variables := map[string]interface{}{
    "RepositoryOwner": githubql.String(owner),
    "RepositoryName":  githubql.String(name),
}
err := githubqlClient.Do(ctx, &q, variables)

Which in my experience so far (doing medium-sized queries, simple mutations, etc.) has proven to scale and work well.

Conclusion

So in general, the option 2-style API with the graphql struct field tag approach (combined with use of variables) seems to be very promising and I'm not aware of any show-stoppers.

It's also possible (and relatively easy) to create a tool that will automatically convert valid GraphQL queries to equivalent Go code for this client library. I plan to do that later.

One big show-stopper with option 1 (with pre-declared static types) that I'm aware of is... how to support GraphQL queries with aliases? I can't think of a good solution at this time. But I haven't thought very hard about it yet.

Announcing the start of a githubql package

Since the results I have so far seem pretty promising (to me), I've created and published the start of a githubql package that currently implements option 2-style API, as I've described above.

I'd really appreciate feedback on it. It's still very early in its stages (see the roadmap in the Goals section of the README), but I think it's ready for people to start looking at and providing feedback. You can open issues in that repo, or leave replies here.

https://github.com/shurcooL/githubql

Thanks!

[dmitshur]

I've made some progress, creating a generator for enum.go in shurcooL/githubv4#7. In doing so, I discovered a problem of many name collisions, because different enum types share same enum value names:

// IssueState represents the possible states of an issue.
type IssueState string

// The possible states of an issue.
const (
	Open   IssueState = "OPEN"   // An issue that is still open.
	Closed IssueState = "CLOSED" // An issue that has been closed.
)

// PullRequestState represents the possible states of a pull request.
type PullRequestState string

// The possible states of a pull request.
const (
	Open   PullRequestState = "OPEN"   // A pull request that is still open.
	Closed PullRequestState = "CLOSED" // A pull request that has been closed without being merged.
	Merged PullRequestState = "MERGED" // A pull request that has been closed by being merged.
)

// ProjectState represents state of the project; either 'open' or 'closed'.
type ProjectState string

// State of the project; either 'open' or 'closed'.
const (
	Open   ProjectState = "OPEN"   // The project is open.
	Closed ProjectState = "CLOSED" // The project is closed.
)

...

I've been thinking about ways of resolving that, and so far I have 4 different solutions outlined at shurcooL/githubv4#8. If anyone has a good idea for a solution I haven't already considered, please post it there.

[dmitshur]

Some updates.

I've taken a step forward on the above issue shurcooL/githubv4#8 with one of the solutions. Thanks for the input, it was helpful. I'm keeping the issue open for longer to see if anything better comes up.

I had an idea to improve support for basic Go types (e.g., using string instead of githubql.String, but that idea is on hold for now). Tracking it in shurcooL/githubv4#9.

The next big challenge is having a good story for union types. I've prototyped a working solution, but the current version is very heavy on user code. They need to write a custom UnmarshalJSON method for the union struct, which is very unpleasant to have to do every time, and error prone. I wrote up about the situation in detail in shurcooL/githubv4#10. Any suggestions or ideas are very welcome.

[dmitshur]

An update, I've written a generator for input objects, so the entire input.go file is now completely generated. See shurcooL/githubv4@f0510d8.

With that, all of GitHub GraphQL API v4 is supported by githubql! 🎉

Next, I plan to work on resolving shurcooL/githubv4#10 (improving support for unions), because that's the biggest usability issue right now. I have a plan for how to tackle it.

[dmitshur]

Next, I plan to work on resolving shurcooL/githubv4#10 (improving support for unions), because that's the biggest usability issue right now. I have a plan for how to tackle it.

Good news, I've implemented it and found it to work really well. The issue is resolved, githubql supports unions quite nicely now (more details are in the linked issue).

[dmitshur]

Thanks for adding the reference to the githubql repository in a8d73f9, @willnorris! There's a minor typo in the URL that we should fix, as I mentioned in a8d73f9#commitcomment-23660360.

I think that means we can close this issue, since the task of starting a client library for GitHub GraphQL v4 is complete. We can use that repository's issue tracker for tracking issues specific to it. Does that sound good?

Also, I would like to invite you @willnorris and @gmlewis as collaborators, so you have access and ownership over it, if you're interested. Of course, you should still create PRs and merge them after an approved code review, just as we do it here.

[willnorris]

yep, sounds good to me.