No internet connection
  1. Home
  2. Ideas

Configurable Github User's Email API to support Github Enterprise

By Phongsiri S @domehuhu2019-03-04 08:43:06.763Z

I've tried to configure the authentication for Github Enterprise using Github OpenAuth. However, it failed where it calls to the /user/emails api from the Github server. Can you please make it be configurable like other endpoints? (e.g. github.authorizationURL and github.accessTokenURL)

  private def loadPublicAndVerifiedEmailAddrs(oauth2AuthInfo: OAuth2Info)
        : Future[(Option[ExternalEmailAddr], Option[ExternalEmailAddr])] = {
    val githubRequest: ws.WSRequest =
      wsClient.url(s"https://api.github.com/user/emails").withHeaders(

Expected:
The url to emails API can be configurable in the play-framework.conf

Actual:
The url is currently hard-coded to https://api.github.com/user/emails

  • 17 replies

There are 17 replies. Estimated reading time: 19 minutes

  1. @KajMagnus marked this topic as Planned 2019-03-04 10:13:46.776Z.
  2. KajMagnus @KajMagnus2019-03-04 10:13:44.037Zreplies todomehuhu:

    Ok I'll fix. As per the docs, https://help.github.com/en/enterprise/2.15/admin/installation/about-the-github-enterprise-server-api,
    GitHub Enterprise has the same API as api.github.com?

    Then it should be enough to make the API origin (i.e. https://api.github.com) configurable? (but not the URL path, /user/emails)

    I have in mind to add this config value:

    github.apiOrigin="https://api.github.com"  # default
    

    What do you think?

    Thanks for including the source code snippet and details.

    Edit: Now I found this: https://stackoverflow.com/a/50612869/694469 and seems GitHub Enterprise also has other endpoints, e.g. https://hostname/api/v3/... — so now I'm thinking I can instead add this config value:

    github.api.userEmails='https://api.github.com/user/emails"
    
    1. D
      Phongsiri S @domehuhu2019-03-04 13:41:22.966Zreplies toKajMagnus:

      Thank you very much!
      Yes my Github also contains /api/ as well. The response of the API also follow the Github V3 spec, So I think should be working.

      1. KajMagnus @KajMagnus2019-03-04 14:34:55.800Zreplies todomehuhu:

        Now I had a look here: https://developer.github.com/enterprise/2.16/v3/#current-version

        The API is accessed from http(s)://[hostname]/api/v3

        And here: https://developer.github.com/enterprise/2.16/v3/users/emails/#list-email-addresses-for-a-user

        GET /user/emails

        So that's the same, as github.com's API, just a different origin and the URL path is prefixed with /api/v3/. So instead of github.apiOrigin or github.api.userEmails, I'll add github.apiURL:

        github.apiURL="https://your.github.server/api/v3"   # defaults to https://api.github.com
        
        1. @KajMagnus marked this topic as Started 2019-03-04 14:57:54.236Z.
        2. D
          Phongsiri S @domehuhu2019-03-05 04:14:01.492Zreplies toKajMagnus:

          The github.apiURL is looking good!. Thank you very much.

          1. KajMagnus @KajMagnus2019-03-07 09:46:58.398Zreplies todomehuhu:

            I just released a new version v0.6.22-85e88ba, with this config value included. @domehuhu your server should auto upgrade this night, before Friday. Please let me know if / how this works. (Not impossible that you'll run into some additional issue, I suppose.)

            1. D
              Phongsiri S @domehuhu2019-03-07 10:56:48.035Zreplies toKajMagnus:

              @KajMagnus Thank you very much. I'm really appreciate it.

              I just upgraded my Talkyard using upgrad-if-needed.sh. Unfortunately, there was still an error. I think it's the different one.

              500 Internal Server Error
              Error when signing in with github: [Silhouette][github] Error retrieving profile information. Error message: Bad credentials, doc URL: Some(https://developer.github.com/v3) [TyEOAUTH0B]
              
              Stack trace:
              com.mohiva.play.silhouette.impl.exceptions.ProfileRetrievalException: [Silhouette][github] Error retrieving profile information. Error message: Bad credentials, doc URL: Some(https://developer.github.com/v3)
              	at com.mohiva.play.silhouette.impl.providers.oauth2.BaseGitHubProvider.$anonfun$buildProfile$1(GitHubProvider.scala:77)
              	at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
              	at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
              	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
              	at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
              	at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
              	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
              	at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
              	at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)
              	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
              	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
              	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
              	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
              	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
              	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
              

              From my investigation, the "apiURL" setting of the GithubProvider need to be configurable via play-framework.yml (I think it's also great idea to support every settings below).
              https://github.com/debiki/talkyard/blob/master/app/debiki/Globals.scala#L385

              https://www.silhouette.rocks/docs/config-oauth2-based-providers

              case class OAuth2Settings(
                authorizationURL: Option[String] = None,
                accessTokenURL: String,
                redirectURL: String,
                apiURL: Option[String] = None,
                clientID: String,
                clientSecret: String,
                scope: Option[String] = None,
                authorizationParams: Map[String, String] = Map.empty,
                accessTokenParams: Map[String, String] = Map.empty,
                customProperties: Map[String, String] = Map.empty)
              

              From the GithubProvider souce code, The default value of apiURL is "https://api.github.com/user?access_token=%s". For my Github, it will be https://mygit.com/api/v3/user?access_token=%s (already tested in my browser).

              There might be something else. But this is what I've go so far.

              Best,

              1. KajMagnus @KajMagnus2019-03-07 16:02:00.221Z2019-03-07 16:08:56.525Zreplies todomehuhu:

                Hi @domehuhu, did you update the play-framework.conf file and add apiURL? Like so?:

                silhouette {
                
                  # Authenticator settings
                  ...
                
                  # Facebook provider
                  ...
                
                  # Google provider
                  ...
                
                  # Twitter provider
                  ...
                
                  # GitHub
                  github.authorizationURL="https://github.com/login/oauth/authorize"   # <—— needs to be changed
                  github.accessTokenURL="https://github.com/login/oauth/access_token"  # <—— this too
                  github.clientID="..."
                  github.clientSecret="..."
                  github.scope="read:user,user:email"
                  github.apiURL="https://mygit.com/api/v3"  # <—————— here
                                                            # instead of the default, https://api.github.com
                
                }
                

                apiURL gets loaded here, from /opt/talkyard/conf/play-framework.conf.

                Maybe you could post the silhouette.github.* section of your config file, with your client id and secret, and your actual hostname, edited out?

                Edit: Oh maybe I accidentally added a config value, with the same name, as an already existing Silhouette config value. I added apiURL and looking at the link you posted: https://www.silhouette.rocks/docs/config-oauth2-based-providers, seems there's also a Silhouette config value with the same name. Ok I'll have a look at this tomorrow probably. Maybe Silhouette expects apiURL = The URL to fetch the profile from the API to include the /user URL path. /Edit

                1. KajMagnus @KajMagnus2019-03-08 06:49:21.806Zreplies todomehuhu:

                  Hi again @domehuhu, sorry for the a-little-bit confused message yesterday. Now I've read your reply + the Silhouette docs in more details (I was short of time yesterday).

                  Ok, so Silhouette already has an apiURL setting, (with the default value here like you mentioned), which one needs to configure. I'll make it configurable then (see 1 below), and remove the setting I added myself — and instead use Silhouette's own apiURL setting to derive it (see 2 below): by dropping-right everything up to and including the rightmost / one gets the API base URL which can be used to construct other requests.

                  1.

                      val githubOAuthSettings: OAuth2Settings Or ErrorMessage = goodOrError {
                        def getGitHub(confValName: String) = getConfValOrThrowDisabled(confValName, "GitHub")
                        OAuth2Settings(
                          authorizationURL = conf.getString("silhouette.github.authorizationURL"),
                          accessTokenURL = getGitHub("silhouette.github.accessTokenURL"),
                          redirectURL = makeRedirectUrl("github"),
                          apiURL = conf.getString("silhouette.github.apiURL"),    //  <————
                          clientID = getGitHub("silhouette.github.clientID"),
                          clientSecret = getGitHub("silhouette.github.clientSecret"),
                          scope = conf.getString("silhouette.github.scope"))
                      }
                  

                  2.

                    // This is the base api url, used to construct requests to the GitHub server.
                    val apiBaseUrl: String = apiUserUrl.dropRightWhile(_ != '/').dropRight(1)
                  
                    // The url to fetch the user's profile, from the GitHub server.
                    // For GitHub.com, it's GitHubProvider.API = "https://api.github.com/user?access_token=%s".
                    // And for GitHub Enterprise, it's "https://own.github/api/v3/user?access_token=%s".
                    // Dropping-right up to and incl the rightmost '/' results in the api base url,
                    // which can be used to construct other requests to GitHub.
                    def apiUserUrl: String = settings.apiURL.getOrElse(
                      com.mohiva.play.silhouette.impl.providers.oauth2.GitHubProvider.API)
                  
                  1. D
                    Phongsiri S @domehuhu2019-03-08 09:27:44.688Zreplies toKajMagnus:

                    Hi @KajMagnus

                    I'm sorry for the late reply. I think it should be working fine.

                    BTW, I wish I could help making the PR but I'm not able to sync submodule http://luajit.org/git/luajit-2.0.git (always get http 403 error). and request to the keyservers (in prod-images.sh) always get timed-out (may be only in my network). Can you please help checking if you have time?

                    Thank you very much for your updates.

                    1. KajMagnus @KajMagnus2019-03-11 12:43:23.704Zreplies todomehuhu:

                      Hi Phongsiri @domehuhu, now I've released the new version, where you can set apiURL="https://mygit.com/api/v3/user?access_token=%s", and that should result in the correct server & url path being used in both the /users and /user/emails requests. Please let me know if / how this works for you?

                      About the submodule, LuaIT: This won't work for you? git clone http://luajit.org/git/luajit-2.0.git — works for me. I suppose it's the-network-where-you-are then, yes. Seems there's a mirror over at GitHub: https://github.com/LuaJIT/LuaJIT; I'll have a closer look and then I'll start using the mirror instead.

                      With prod-images.sh, you mean s/build-prod-images.sh? Keyserver timeouts, that sounds like when you're building the Gulp image (based on Nodejs), or maybe the Web image (with Nginx). Maybe this is also something with the network then. Is it ok if I ask where you're located? I'm wondering if ... could it be some corporate firewalls? or country firewalls? that blocks the servers

                      1. D
                        Phongsiri S @domehuhu2019-03-13 11:18:23.365Zreplies toKajMagnus:

                        @KajMagnus Thank you very much for the update. The Github error has gone. It seems that the OAuth part is working now!

                        But now it failed at spam email checking... my network doesn't allow connection to www.stopforumspam.com and akismet.com. Those get blocked by firewall. I will check with my network admin if they have any solution :(.

                        you're right luajit and keyservers it is also blocked by firewall... May I ask for your help again for the new option to enabling/disabling spam email checking? (it's only available internally in the company so I think it is OK). It's difficult to make PR at my end..

                        1. KajMagnus @KajMagnus2019-03-14 03:55:06.872Zreplies todomehuhu:

                          Ok good to hear that it (seems to) work now.

                          About the spam checks: Yes I'll add config values so you and others can disable stopforumspam and akismet.

                          1. KajMagnus @KajMagnus2019-03-15 08:22:57.678Zreplies todomehuhu:

                            Hi again Phongsiri, now I've released a new version, and you can do this: (and restart the app contaner afterwards)

                            talkyard.spamChecks.enabled=false
                            

                            In /opt/talkyard/conf/play-framework.conf. This'll prevent any requests to Stopforumspam and Akismet.

                            1. D
                              Phongsiri S @domehuhu2019-03-15 11:07:11.570Zreplies toKajMagnus:

                              @KajMagnus
                              Thank you very much! It's working perfectly now.

                              FYI, I have to connect all registered users to the Github login manually by inserting records in the identities table myself. Is there any way to connect Github login from the user profile page?

                              1. KajMagnus @KajMagnus2019-03-17 05:06:06.420Zreplies todomehuhu:

                                Ok I'm glad it works.

                                Currently you cannot connect Github login from the user profile page. However, seems to me I need to add that. (Plus connecting Gmail and Facebook accounts too — to make it more smooth to sign up for Talkyard's hosting.) So I think this'll happen in one or two months.