Support multiple code hosting services

[alphabetek]

Support multiple public and private repositories (Github or Bitbucket). It would be nice if the user could connect from the webapp to external services (Github, Bitbucket ...), and why not support a feature of workspaces on top of "APIs". This gives a clear visibily while working on APIs belonging to different projects.

[EricWittmann]

Yes! The project will ultimately have a proper backend which will allow us to simultaneously support multiple repositories. It's probably not clear from the roadmap that this is a goal - I will fix that when I get a chance. :)

[EricWittmann]

Also congrats on Issue #100! ;)

[danmikita]

I'm interested in adding Bitbucket and Bitbucket Server support - but after looking at the code and the workflow of the UI, it doesn't seem to have a clear entry point for user credentials. As of now you need to log into Apicurio Studio with your Github credentials, which I could just add the option of Bitbucket OR Github, but to have both there would need to be some kind of user settings page.

On the other hand, I would like to ideally add LDAP support for log in and perhaps have global settings for the bitbucket server. No matter what the solution - I think I need some UX direction first. Thoughts?

[EricWittmann]

Hi @danmikita - contributions for bitbucket would be fantastic. However, I think some core changes are needed before that can happen. The upshot is that we're moving in a direction that is very compatible with what you want to do (login and storage separate) but we're not quite there yet. Some background:

Step 1: We started out with a serverless approach, with NO back-end code and just a UI that directly used the GitHub APIs for login and for storing/retrieving content.
Step 2: The goal was always to have a server back-end, which is almost complete. However, the back-end is still coupled to GitHub, both for login and for storing/retrieving content. But at least now most of the entities and how to access/manipulate them are no longer GitHub specific in the UI layer.
Step 3: Break the coupling between the server back-end and GitHub and decouple user authentication from storage. The idea here is that user authentication will be pluggable so we can support a wide array of options (GitHub, Auth0, KeyCloak, etc). Additionally, we'll provide a way for users to link their Apicurio Studio account to their GitHub account so that we can operate on their behalf. This way they can login with e.g. LDAP but still store data in GH. At that point, we can also support BitBucket and other source control systems.

So I should be releasing a version that does Step 2 this week. After that we can start working on making it to Step 3.

I was going to prioritize OpenAPI 3 support above the Step 3 work described above. But now that I know we have not only community demand but also an offer to help, I'll reverse that. :)

[danmikita]

@EricWittmann That sounds great! I have a whole team of guys over here who are super excited about this tool and are looking forward to contributing in the near future. Could you ping me here when the code base is ready for us to jump in?

[EricWittmann]

Sure will! I'll drop a note here when the time comes.

FYI I just released 0.1.6 about 10 minutes ago... let me know if you find any problems. This is the first version that uses the server back-end instead of going straight to GH. :)

[EricWittmann]

Hey @danmikita - just wanted to drop a quick update in here. I am still working on a bunch of changes to apicurio to make it nice and easy to contribute bitbucket support. There are some things to note. :)

First, I have decided to start using Keycloak for authentication and for account linking (both relevant to this discussion). If you're not familiar with it, have a look here:

http://www.keycloak.org/

KC solves a bunch of problems, and has all sorts of great features for authentication (various social logins, user registration, ldap integration, etc etc). In addition, it supports account linking, which is what I'll be using to allow an Apicurio user to store their APIs in a variety of places (e.g. GitHub, GitLab, and Bitbucket). Keycloak will handle the account linking and it will make OAuth access tokens available. Apicurio will implement code to invoke the respective provider APIs (github, gitlab, etc) in order to store and retrieve data.

Keycloak comes with support for GitHub already, but it doesn't yet have GitLab or Bitbucket. There is a generic OIDC connector that may work, but I haven't tried that yet.

Anyway, once this is all done, if you'd like to contribute support for Bitbucket, it would be done in two places:

1. Implement a Bitbucket "Identity Provider" and contribute it to Keycloak
2. Implement a Bitbucket "connector" (or whatever I end up calling it) in Apicurio

If you were so inclined, you could already start on (1) now. I haven't started looking into exactly how a new Identity Provider can be contributed in Keycloak, but I know the guys who work on that project, so I'm sure it's not too hard. :)

Hopefully the delay hasn't been too long (I was out on vacation for a week). I should have the stuff I'm working on in Apicurio done in another day or two barring any unexpected complications. :)

[danmikita]

@EricWittmann Thank you for such a detailed writeup! I'll take a look at the Keycloak repo and familiarize myself with the codebase and try to get a PR out there in the few days.

Once a month our group has something we call "Innovation Days" where we all work on "pet" projects. I have a group of about 5 or so developers who want to contribute to this project. Our next session is August 28th and 29th. Hopefully the codebase will be ready for us by then!

Thank you so much for this work. It is going to be amazing!

[EricWittmann]

Thanks for the response, @danmikita . Good news: it turns out that the Keycloak guys have already added authentication and account-linking support for both GitLab and Bitbucket. So now the only thing we need to do for Apicurio+Bitbucket is:

1. implement an Apicurio "SourceConnector" that uses the Bitbucket API (e.g. to store API content)
2. relatively minor UI changes to the Settings>Accounts page (this is new) and the Create API page

Should be a piece of cake. :)

[danmikita]

@EricWittmann That is great news! I'll still need to check their Bitbucket implementation to make sure it supports the Enterprise on-site version of Bitbucket. (Bitbucket Server). BTW - do you have any other quick hitter tasks like this for us? I don't think I can keep 5 devs busy for 2 days with this one feature. :)

[EricWittmann]

@danmikita You read my mind - this feature will likely not take long (I hope!). I was planning on picking out some likely tasks from the roadmap and write them up more fully as github Issues in case you guys wanted to tackle them. :) I'll have that done by the end of the week.

A new release of Apicurio Studio with the Account Linking feature should be released today. I had some trouble wrangling with Keycloak on Openshift last week, but it's all resolved now. I'll be doing the release today and updating the documentation.

[EricWittmann]

@danmikita Note that I released a new version a couple of days ago. The new version has the account linking feature. Give it a whirl and let me know if it's hopefully at least somewhat aligned with what you're looking for.

I also created some documentation for setting up a development environment (docs for contributors): https://apicurio-studio.readme.io/v0.1.9/docs/setting-up-a-development-environment

At the moment I'm working on creating/updating some issues from the short-term roadmap. Things that you guys could maybe knock out during your Innovation Days. Hopefully some of them will sound interesting. I'll create a separate parent issue for that so we stop spamming poor @alphabetek with GitHub messages. :)

[EricWittmann]

We now have support for GitLab and Bitbucket. Closing this issue. :)