Gin (Golang) server cannot connect to self-hosted Ory Kratos ports within main.go file: connection refused

[paigen11]

Hello,

I'm attempting to build an authentication service that uses a Gin-powered Go server to interface with a self-hosted Ory Kratos instance and will eventually have a Sveltekit UI for users to register, login, view their accounts, etc.

After going through the docs and quickstart for self-hosted Ory Kratos many times (and running the local version of the quickstart repo and poring over that code as well), I started building my own.

I'm using Docker to build my Gin Go server, Docker Compose to spin up all the necessary pieces: the PostgresDB, the Kratos SQL migration, the MailSlurper SMTP server, and generally trying to do things in a similar fashion to the quickstart, however, when I try to cURL any Ory Kratos command via the API or the kratos-client-go SDK, I get back Get "http://127.0.0.1:4433/<API_URL_HERE>": dial tcp 127.0.0.1:4433: connect: connection refused - it doesn't matter if I'm trying to reach the public Ory port of 4433 or the private Ory port of 4434, either through the app start up or by hitting an exposed API endpoint via my Gin framework (running on port 8080), I get connection refused rain or shine.

I've tried changing the Kratos ServerConfiguration {{URL: "<URL_HERE>"}} to http://127.0.0.1:4433, localhost:4433, host.docker.internal:4433, and app:4433 (the name of the service in the docker compose file) in my main.go file and kratos.yml file to no avail and I'm running out of ideas of what to try next and desperately would like help.

If someone's willing to help, I can also make a copy of the repo available for public consumption as well.

Any help from someone with a better handle on this technology is greatly appreciated.

For reference, here is the Dockerfile and compose.yml file at the root of my project:

FROM golang:1.18-alpine

WORKDIR /app

COPY go.mod go.sum ./

RUN go mod download

COPY . .

RUN go build -o main

CMD ["./main"]

And the compose.yml file:

version: '3'
services:
  kratos-migrate:
    image: oryd/kratos:v1.0.0
    container_name: kratos_migrate
    environment:
      - DSN=postgres://${KRATOS_DB_USER}:${KRATOS_DB_PASSWORD}@${KRATOS_DB_HOST}:${KRATOS_DB_PORT}/${KRATOS_DB_NAME}?sslmode=disable&max_conns=20&max_idle_conns=4
    volumes:
      - ./config/kratos/email-password:/etc/config/kratos
    command: -c /etc/config/kratos/kratos.yml migrate sql -e --yes
    restart: on-failure
    networks:
      - fullstack

  app:
    container_name: auth
    build: .
    ports: 
      - '8080:8080' 
      - '4433:4433' # public port
      - '4434:4434' # admin port
    restart: on-failure
    volumes:
      - api:/usr/src/app/
      - ./config/kratos/email-password:/etc/config/kratos   
    depends_on:
      - kratos-migrate
    networks:
      - fullstack
    env_file:
      - .env

  kratos-postgres:
    image: postgres:latest
    container_name: kratos_db_postgres
    environment:
      - POSTGRES_USER=${KRATOS_DB_USER}
      - POSTGRES_PASSWORD=${KRATOS_DB_PASSWORD}
      - POSTGRES_DB=${KRATOS_DB_NAME}
      - POSTGRES_HOST=${KRATOS_DB_HOST}
    ports:
      - '5433:${KRATOS_DB_PORT}'
    volumes:
      - database_kratos:/var/lib/postgresql/data 
    restart: always
    networks:
      - fullstack 

  mailslurper:
    image: oryd/mailslurper:latest-smtps
    container_name: mailslurper_email
    ports:
      - '4436:4436'
      - '4437:4437'
    networks:
      - fullstack

volumes:
  api:
  database_kratos:

# Networks to be created to facilitate communication between containers
networks:
  fullstack:
    driver: bridge

Here is my identity.schema.json and my kratos.yml file within the config/kratos/email-password/ folder in my project:

{
  "$id": "https://config/kratos/schemas/email-password/identity.schema.json",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Person",
  "type": "object",
  "properties": {
    "traits": {
      "type": "object",
      "properties": {
        "email": {
          "type": "string",
          "format": "email",
          "title": "E-Mail",
          "minLength": 3,
          "ory.sh/kratos": {
            "credentials": {
              "password": {
                "identifier": true
              }
            },
            "verification": {
              "via": "email"
            },
            "recovery": {
              "via": "email"
            }
          }
        },
        "account_uid": {
          "type": "string"
        },
        "full_name": {
          "type": "string"
        },
        "must_verify_email": {
          "type": "boolean"
        }
      },
      "required": [
        "email"
      ],
      "additionalProperties": false
    }
  }
}

And

version: v1.0.0

serve:
  public:
    base_url: http://127.0.0.1:4433/
    cors:
      enabled: true
  admin:
    base_url: http://127.0.0.1:4434/

selfservice:
  default_browser_return_url: http://127.0.0.1:5173/ # the port the Sveltekit project runs on
  allowed_return_urls:
    http://127.0.0.1:5173

methods:
  password:
    enabled: true
  link:
    enabled: true
  oidc:
    enabled: false

flows:
  error:
    ui_url: http://127.0.0.1:5173/error

  settings:
    ui_url: http://127.0.0.1:5173/account-settings
    privileged_session_max_age: 10m
    lifespan: 1h

  recovery:
    enabled: true
    ui_url: http://127.0.0.1:5173/recovery
    use: link
    lifespan: 10m

  verification:
    lifespan: 1h
    ui_url: http://127.0.0.1:5713/verify
    use: code
    enabled: true

  logout:
    after:
      default_browser_return_url: http://127.0.0.1:5173/sign-in

  login:
    ui_url: http://127.0.0.1:5173/sign-in
    lifespan: 10m

  registration:
    lifespan: 10m
    ui_url: http://127.0.0.1:5173/sign-up
    after:
      password:
        hooks:
          - hook: session
          - hook: show_verification_ui
  
log:
  level: debug
  format: json
  leak_sensitive_values: true

# todo how are secrets generated?
secrets:
  cookie:
    - PLEASE-CHANGE-ME-I-AM-VERY-INSECURE
  cipher:
    - 32-LONG-SECRET-NOT-SECURE-AT-ALL

hashers:
  argon2:
    parallelism: 1
    memory: 128KB
    iterations: 2
    salt_length: 16
    key_length: 16

identity:
  default_schema_id: default
  schemas:
    - id: default
      url: file:///etc/config/kratos/identity.schema.json

courier:
  smtp:
    connection_uri: smtps://test:test@mailslurper:1025/?skip_ssl_verify=true

And here is the main.go file where I'm trying to just verify I can connect to the Ory Kratos service (which spins up fine when I run docker compose up --build --force-recreate).

package main

import (
	"context"
	"log"
	"net/http"

	"github.com/gin-gonic/gin"
	kratos "github.com/ory/kratos-client-go"
)

func setupRouter() *gin.Engine {
	// Creates a gin router with default middleware:
	router := gin.Default()

	// set up Kratos Public Client just to make sure it connects
	configuration := kratos.NewConfiguration()
	configuration.Servers = []kratos.ServerConfiguration{
		{
			URL:         "http://127.0.0.1:4433", // Kratos Public API
			Description: "Kratos Public API",
		},
	}
	apiClient := kratos.NewAPIClient(configuration)

	log.Println("Attempt to get health of Ory via Metadata API...")

	health, _, err := apiClient.MetadataApi.IsAlive(context.Background()).Execute()
	if err != nil {
		log.Println("Error when calling `MetadataApi.IsAlive`: ", err)
	}

	log.Println("health response", health)

	router.Run(":8080")
	return router
}

func main() {
	setupRouter()
}

[Nikola-Milovic]

Sorry, if you havent already resolved this, but to confirm. You have kratos running locally on your machine outside of the docker compose (can't see it defined there) and the service app running inside the docker compose thats trying to connect to it.

I am on my phone so please correct me.

If this is the case, trying to connect to 127.0.0.1 will just route to the localhost of the container. The internal.docker.host works if you add the extra_hosts option. (But wouldnt recommend it, for me its always behaving weirdly)

An easy fix should be to add kratos to the docker compose as a service and connect to it via http://kratos:<port>

[paigen11]

The logs when the kratos service starts up seems fine. It reports that the public port is 0.0.0.0:4433 and the admin port is 0.0.0.0:4434.

Sorry in advance for the log traces I'm about to paste in here

{"audience":"application","level":"info","msg":"No tracer configured - skipping tracing setup","service_name":"Ory Kratos","service_version":"v1.0.0","time":"2024-03-01T18:27:29.677186346Z"}
2024-03-01 13:27:29 {"audience":"application","connMaxLifetime":0,"idlePool":4,"level":"debug","msg":"Connecting to SQL Database","pool":20,"service_name":"Ory Kratos","service_version":"v1.0.0","time":"2024-03-01T18:27:29.677281298Z"}
2024-03-01 13:27:30 {"audience":"application","error":{"message":"failed to connect to `host=kratos-postgres user=ory-user database=kratos-db`: dial error (dial tcp 192.168.176.3:5432: connect: connection refused)","stack_trace":"\ngithub.com/ory/kratos/persistence/sql.(*Persister).Ping\n\t/project/persistence/sql/persister.go:146\ngithub.com/ory/kratos/driver.(*RegistryDefault).Init.func1\n\t/project/driver/registry_default.go:624\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/ory/kratos/driver.(*RegistryDefault).Init\n\t/project/driver/registry_default.go:592\ngithub.com/ory/kratos/driver.New\n\t/project/driver/factory.go:24\ngithub.com/ory/kratos/cmd/serve.NewServeCmd.func1\n\t/project/cmd/serve/root.go:26\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992\ngithub.com/ory/kratos/cmd.Execute\n\t/project/cmd/root.go:61\nmain.main\n\t/project/main.go:21\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598"},"level":"warning","msg":"Unable to ping database, retrying.","service_name":"Ory Kratos","service_version":"v1.0.0","time":"2024-03-01T18:27:30.200281817Z"}
2024-03-01 13:27:30 {"audience":"application","connMaxLifetime":0,"idlePool":4,"level":"debug","msg":"Connecting to SQL Database","pool":20,"service_name":"Ory Kratos","service_version":"v1.0.0","time":"2024-03-01T18:27:30.564505796Z"}
2024-03-01 13:27:31 {"audience":"application","error":{"debug":"","message":"Unable to locate the table","reason":"","stack_trace":"\ngithub.com/ory/x/sqlcon.handlePostgres\n\t/go/pkg/mod/github.com/ory/x@v0.0.562/sqlcon/error.go:59\ngithub.com/ory/x/sqlcon.HandleError\n\t/go/pkg/mod/github.com/ory/x@v0.0.562/sqlcon/error.go:78\ngithub.com/ory/x/networkx.(*Manager).Determine\n\t/go/pkg/mod/github.com/ory/x@v0.0.562/networkx/manager.go:46\ngithub.com/ory/kratos/persistence/sql.(*Persister).DetermineNetwork\n\t/project/persistence/sql/persister.go:93\ngithub.com/ory/kratos/driver.(*RegistryDefault).Init.func1\n\t/project/driver/registry_default.go:643\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/ory/kratos/driver.(*RegistryDefault).Init\n\t/project/driver/registry_default.go:592\ngithub.com/ory/kratos/driver.New\n\t/project/driver/factory.go:24\ngithub.com/ory/kratos/cmd/serve.NewServeCmd.func1\n\t/project/cmd/serve/root.go:26\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992\ngithub.com/ory/kratos/cmd.Execute\n\t/project/cmd/root.go:61\nmain.main\n\t/project/main.go:21\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598","status":"Internal Server Error","status_code":500},"level":"warning","msg":"Unable to determine network, retrying.","service_name":"Ory Kratos","service_version":"v1.0.0","time":"2024-03-01T18:27:31.141341879Z"}
2024-03-01 13:27:32 {"audience":"application","connMaxLifetime":0,"idlePool":4,"level":"debug","msg":"Connecting to SQL Database","pool":20,"service_name":"Ory Kratos","service_version":"v1.0.0","time":"2024-03-01T18:27:32.023319533Z"}
2024-03-01 13:27:32 {"audience":"application","level":"warning","msg":"\n\nYOU ARE RUNNING Ory KRATOS IN DEV MODE.\nSECURITY IS DISABLED.\nDON'T DO THIS IN PRODUCTION!\n\n","service_name":"Ory Kratos","service_version":"v1.0.0","time":"2024-03-01T18:27:32.597702754Z"}
2024-03-01 13:27:32 {"audience":"application","level":"info","msg":"Software quality assurance features are enabled. Learn more at: https://www.ory.sh/docs/ecosystem/sqa","service_name":"Ory Kratos","service_version":"v1.0.0","time":"2024-03-01T18:27:32.597870943Z"}
2024-03-01 13:27:32 {"audience":"application","level":"info","msg":"TLS has not been configured for public, skipping","service_name":"Ory Kratos","service_version":"v1.0.0","time":"2024-03-01T18:27:32.598385948Z"}
2024-03-01 13:27:32 {"audience":"application","level":"info","msg":"Starting the public httpd on: 0.0.0.0:4433","service_name":"Ory Kratos","service_version":"v1.0.0","time":"2024-03-01T18:27:32.598468907Z"}
2024-03-01 13:27:32 {"audience":"application","level":"info","msg":"TLS has not been configured for admin, skipping","service_name":"Ory Kratos","service_version":"v1.0.0","time":"2024-03-01T18:27:32.599130794Z"}
2024-03-01 13:27:32 {"audience":"application","level":"info","msg":"Starting the admin httpd on: 0.0.0.0:4434","service_name":"Ory Kratos","service_version":"v1.0.0","time":"2024-03-01T18:27:32.599395314Z"}

And when I try to hit the /identity-schemas API endpoint in my app, the Kratos service logs the following:

{"http_request":{"headers":{"accept":"application/json","accept-encoding":"gzip","user-agent":"OpenAPI-Generator/1.0.0/go"},"host":"kratos:4434","method":"GET","path":"/schemas","query":null,"remote":"192.168.176.8:58410","scheme":"http"},"level":"info","msg":"started handling request","time":"2024-03-01T18:28:03.211694307Z"}
2024-03-01 13:28:03 {"http_request":{"headers":{"accept":"application/json","accept-encoding":"gzip","user-agent":"OpenAPI-Generator/1.0.0/go"},"host":"kratos:4434","method":"GET","path":"/admin/schemas","query":null,"remote":"192.168.176.8:58410","scheme":"http"},"http_response":{"headers":{"content-type":"text/html; charset=utf-8","location":"/admin/schemas"},"size":50,"status":307,"text_status":"Temporary Redirect","took":257287},"level":"info","msg":"completed handling request","time":"2024-03-01T18:28:03.21188425Z"}
2024-03-01 13:28:03 {"http_request":{"headers":{"accept":"application/json","accept-encoding":"gzip","referer":"http://kratos:4434/schemas","user-agent":"OpenAPI-Generator/1.0.0/go"},"host":"kratos:4434","method":"GET","path":"/admin/schemas","query":null,"remote":"192.168.176.8:58410","scheme":"http"},"level":"info","msg":"started handling request","time":"2024-03-01T18:28:03.212793948Z"}
2024-03-01 13:28:03 {"http_request":{"headers":{"accept":"application/json","accept-encoding":"gzip","referer":"http://kratos:4434/schemas","user-agent":"OpenAPI-Generator/1.0.0/go"},"host":"kratos:4434","method":"GET","path":"/admin/schemas","query":null,"remote":"192.168.176.8:58410","scheme":"http"},"http_response":{"headers":{"cache-control":"private, no-cache, no-store, must-revalidate","content-type":"text/html; charset=utf-8","location":"http://127.0.0.1:4433/schemas"},"size":65,"status":307,"text_status":"Temporary Redirect","took":591946},"level":"info","msg":"completed handling request","time":"2024-03-01T18:28:03.213394608Z"}

And if I cURL the Kratos admin endpoint directly, I get a Temporary Redirect href. Does that ring any bells to you?

curl -X GET \
     -H 'Content-Type: application/json' \
     -H 'Accept: application/json' \
     "http://127.0.0.1:4434/admin/schemas"
     
  // response   
<a href="http://127.0.0.1:4433/schemas">Temporary Redirect</a>. 

[Nikola-Milovic]

My only guess is the following, you get a temporary redirect to 127.0.0.1:4433(which cannot be resolved in the app container) and it leads you to the connection refused error.

Can you try instead of hitting the admin api, hit the public api (since its redirected to it anyways). So call directly the public api

[paigen11]

It worked!

It seems very odd that I need to call the public port to access the admin API, but it seems to have worked.

Here's my new Kratos admin client configuration

func AdminKratosClient() kratos.IdentityApi {
	configuration := kratos.NewConfiguration()
	configuration.Servers = []kratos.ServerConfiguration{
		{
			URL:         "http://kratos:4433", // changed port from 4434 to 4433
			Description: "Kratos Admin API",
		},
	}
	apiClient := kratos.NewAPIClient(configuration)
	return apiClient.IdentityApi
}

The logs from the Kratos service

2024-03-01 13:47:00 {"http_request":{"headers":{"accept":"application/json","accept-encoding":"gzip","user-agent":"OpenAPI-Generator/1.0.0/go"},"host":"kratos:4433","method":"GET","path":"/schemas","query":null,"remote":"192.168.208.8:37028","scheme":"http"},"level":"info","msg":"started handling request","time":"2024-03-01T18:47:00.129814707Z"}
2024-03-01 13:47:00 {"http_request":{"headers":{"accept":"application/json","accept-encoding":"gzip","user-agent":"OpenAPI-Generator/1.0.0/go"},"host":"kratos:4433","method":"GET","path":"/schemas","query":null,"remote":"192.168.208.8:37028","scheme":"http"},"http_response":{"headers":{"cache-control":"private, no-cache, no-store, must-revalidate","content-type":"application/json; charset=utf-8","link":"<http://127.0.0.1:4433/schemas?page=0&per_page=1>; rel=\"first\"","set-cookie":["csrf_token_806060ca5bf70dff3caa0e5c860002aade9d470a5a4dce73bcfa7ba10778f481=jPZQOXkIKR7vQ5AAidYki6taWd6rTEh0ywzeadPWV90=; Path=/; Max-Age=31536000; HttpOnly; SameSite=Lax"],"vary":"Origin","x-total-count":"1"},"size":594,"status":200,"text_status":"OK","took":2776315},"level":"info","msg":"completed handling request","time":"2024-03-01T18:47:00.132560184Z"}

And the logs from the app service

2024/03/01 18:47:00 Get identity schemas
2024-03-01 13:47:00 2024/03/01 18:47:00 identity schemas [{0xc000586750 map[$id:https://config/kratos/schemas/email-password/identity.schema.json $schema:http://json-schema.org/draft-07/schema# properties:map[traits:map[additionalProperties:false properties:map[account_uid:map[type:string] email:map[format:email minLength:3 ory.sh/kratos:map[credentials:map[password:map[identifier:true]] recovery:map[via:email] verification:map[via:email]] title:E-Mail type:string] full_name:map[type:string] must_verify_email:map[type:boolean]] required:[email] type:object]] title:Person type:object] map[]}]

[Nikola-Milovic]

Yeah it seems that the admin api is just redirecting some calls to the public api. To avoid calling public api by yourself, and to allow the kratos to redirect you when it sees fit, I believe you can make the following change to your kratos.yml file

version: v1.0.0

serve:
  public:
    base_url: http://kratos:4433/ <- this 
    cors:
      enabled: true
  admin:
    base_url: http://kratos:4434/ <- this

So now when kratos redirects you, instead of 127.0.0.1 it will redirect you to kratos hostname, and your Docker DNS will resolve it correctly. With this change you should be able to keep using admin api

[paigen11]

Ok, I have made the change. Thanks very much for your help. I appreciate it!