Timeout errors

[abemedia]

I'm trying to set up a repo with v1.11.0 but keep getting the following error:

{
    "level": "error",
    "rid": "c112o158nids72mvqcn0",
    "github_event_type": "issue_comment",
    "github_delivery_id": "1c27cb00-7db3-11eb-9400-cea1ebb29542",
    "error": "failed to get pull request theappnest/baustelle#1: Get \"https://api.github.com/repos/theappnest/baustelle/pulls/1\": could not refresh installation id 14084299's token: could not get access_tokens from GitHub API for installation ID 14084299: dial tcp 140.82.121.5:443: i/o timeout\ngithub.com/palantir/bulldozer/server/handler.(*IssueComment).Handle\n\t/Users/Adam/go/pkg/mod/github.com/palantir/bulldozer@v1.10.1/server/handler/issue_comment.go:56\ngithub.com/palantir/go-githubapp/githubapp.Dispatch.Execute\n\t/Users/Adam/go/pkg/mod/github.com/palantir/go-githubapp@v0.5.0/githubapp/scheduler.go:56\ngithub.com/palantir/go-githubapp/githubapp.(*scheduler).safeExecute\n\t/Users/Adam/go/pkg/mod/github.com/palantir/go-githubapp@v0.5.0/githubapp/scheduler.go:183\ngithub.com/palantir/go-githubapp/githubapp.QueueAsyncScheduler.func1\n\t/Users/Adam/go/pkg/mod/github.com/palantir/go-githubapp@v0.5.0/githubapp/scheduler.go:257\nruntime.goexit\n\t/usr/local/Cellar/go/1.16/libexec/src/runtime/asm_amd64.s:1371",
    "time": "2021-03-05T13:08:02.586685627Z",
    "message": "Unexpected error handling webhook"
}

Any ideas what could be causing this?

[asvoboda]

Hey @abemedia,

Can you share your config file (without the secrets of course)?

Are you running bulldozer behind any sort of proxy or MITM setup? The follow error suggests that bulldozer can't reach github.com

dial tcp 140.82.121.5:443: i/o timeout

[abemedia]

It's running on Lambda. The weird thing is occasionally it works but more often than not I get the above error or this one:

{
    "level": "error",
    "rid": "c1151nd47s8s72qhf5eg",
    "github_event_type": "pull_request",
    "github_delivery_id": "12e5ca40-7dc9-11eb-893c-7ce1c5dfe6d6",
    "error": "failed to get pull request theappnest/baustelle#11: Get \"https://api.github.com/repos/theappnest/baustelle/pulls/11\": could not refresh installation id 14084299's token: could not get access_tokens from GitHub API for installation ID 14084299: net/http: TLS handshake timeout\ngithub.com/palantir/bulldozer/server/handler.(*PullRequest).Handle\n\t/Users/Adam/go/pkg/mod/github.com/palantir/bulldozer@v1.11.0/server/handler/pull_request.go:61\ngithub.com/palantir/go-githubapp/githubapp.Dispatch.Execute\n\t/Users/Adam/go/pkg/mod/github.com/palantir/go-githubapp@v0.5.0/githubapp/scheduler.go:56\ngithub.com/palantir/go-githubapp/githubapp.(*scheduler).safeExecute\n\t/Users/Adam/go/pkg/mod/github.com/palantir/go-githubapp@v0.5.0/githubapp/scheduler.go:183\ngithub.com/palantir/go-githubapp/githubapp.QueueAsyncScheduler.func1\n\t/Users/Adam/go/pkg/mod/github.com/palantir/go-githubapp@v0.5.0/githubapp/scheduler.go:257\nruntime.goexit\n\t/usr/local/Cellar/go/1.16/libexec/src/runtime/asm_amd64.s:1371",
    "time": "2021-03-05T15:41:31.164081196Z",
    "message": "Unexpected error handling webhook"
}

The config:

github:
  web_url: "https://github.com"
  v3_api_url: "https://api.github.com"
  app:
    integration_id: 123
    webhook_secret: <redacted>
    private_key: <redacted>
options:
  configuration_path: .github/bulldozer.yml

There's no server config as I'm proxying the requests from lambda directly to the underlying goji.Mux.

[abemedia]

Just for context here is how I'm running it on Lambda:

package main

import (
	"context"
	"encoding/json"
	"log"
	"os"
	"reflect"
	"strings"
	"unsafe"

	"github.com/aws/aws-lambda-go/events"
	"github.com/aws/aws-lambda-go/lambda"
	"github.com/awslabs/aws-lambda-go-api-proxy/httpadapter"
	"github.com/kelseyhightower/envconfig"
	"github.com/palantir/bulldozer/server"
	"github.com/palantir/go-baseapp/baseapp"
)

var gojiLambda *httpadapter.HandlerAdapter

func init() {
	c, err := server.ParseConfig(nil)
	if err != nil {
		log.Fatal(err)
	}

	// set config from env vars
	if err = envconfig.Process("bulldozer", c); err != nil {
		log.Fatal(err)
	}

	s, err := server.New(c)
	if err != nil {
		log.Fatal(err)
	}

	// get unexported struct field "base" to access the underlying router
	v := reflect.ValueOf(s).Elem().FieldByName("base").Elem()
	base := (*baseapp.Server)(unsafe.Pointer(v.UnsafeAddr()))

	gojiLambda = httpadapter.New(base.Mux())
}

func handler(ctx context.Context, req events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
	return gojiLambda.ProxyWithContext(context.Background(), req)
}

func main() {
	lambda.Start(handler)
}

[asvoboda]

We would need to dig in a little more to double check whats failing here. It does seem like something is not correctly wired up for the http handler, however I don't have a good environment to test with lambda to be certain.

However, a word of caution: bulldozer makes good use of a request cache to avoid hitting rate limits and other issues when interacting with many events from the Github API. While it might be possible to run the app entirely on Lambda, you might find that the app hits these limits more frequently.

[abemedia]

It seems there's other issues running on Lambda as well. Sometimes you can see the hook coming in in the logs but nothing happens. My assumption was that there's some async stuff going on in the scheduler, rather than directly on the webhook request, and the Lambda gets frozen after the request completes.

[shresthaujjwal]

@abemedia Where you able to get this to work with lambda ?