Skip to content
This repository has been archived by the owner on Aug 23, 2023. It is now read-only.

PANIC: runtime error: invalid memory address or nil pointer dereference #811

Open
motyla opened this issue Jan 4, 2018 · 9 comments
Open

PANIC: runtime error: invalid memory address or nil pointer dereference #811

motyla opened this issue Jan 4, 2018 · 9 comments
Milestone
vnext

Comments

@motyla
Copy link

motyla commented Jan 4, 2018

version : 0.7.4_633_gd5ca2bcb-1

Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] PANIC: runtime error: invalid memory address or nil pointer dereference
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/panic.go:491 (0x42b342)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/panic.go:63 (0x42a24d)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/signal_unix.go:367 (0x4426cb)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/idx/memory/memory.go:113 (0xa6739c)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/idx/memory/memory.go:807 (0xa65f5a)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/idx/memory/memory.go:807 (0xa65f5a)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/api/cluster.go:96 (0xa3f04f)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/api/routes.go:38 (0xa5d25a)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/asm_amd64.s:510 (0x45a35a)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:434 (0x4b8f85)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:302 (0x4b8563)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0x9ab288)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0x9aab89)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:113 (0x9c7aad)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:104 (0x9c79e5)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/api/middleware/tracer.go:64 (0x9e658f)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/asm_amd64.s:509 (0x45a2da)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:434 (0x4b8f85)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:302 (0x4b8563)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0x9ab288)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0x9aab89)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:113 (0x9c7aad)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:104 (0x9c79e5)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/api/middleware/stats.go:72 (0x9e5d78)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/asm_amd64.s:509 (0x45a2da)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:434 (0x4b8f85)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:302 (0x4b8563)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0x9ab288)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0x9aab89)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:113 (0x9c7aad)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:104 (0x9c79e5)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/raintank/gziper/gzip.go:73 (0xa3c257)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/asm_amd64.s:509 (0x45a2da)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:434 (0x4b8f85)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:302 (0x4b8563)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0x9ab288)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0x9aab89)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:113 (0x9c7aad)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:104 (0x9c79e5)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/recovery.go:161 (0x9d8dba)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/asm_amd64.s:509 (0x45a2da)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:434 (0x4b8f85)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:302 (0x4b8563)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0x9ab288)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0x9aab89)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:113 (0x9c7aad)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:104 (0x9c79e5)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/logger.go:43 (0x9d808b)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/asm_amd64.s:509 (0x45a2da)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:434 (0x4b8f85)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:302 (0x4b8563)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0x9ab288)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0x9aab89)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:113 (0x9c7aad)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/router.go:184 (0x9da0a2)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/router.go:286 (0x9d36eb)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/macaron.go:177 (0x9cbc27)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/net/http/server.go:2619 (0x6e22b3)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/net/http/server.go:1801 (0x6de5cc)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/asm_amd64.s:2337 (0x45cad0)
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] #033[1;32m2018-01-04 11:43:27: Completed /index/find 200 OK in 4.635476ms#033[0m
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] #033[1;36m2018-01-04 11:43:27: Completed /index/find 500 Internal Server Error in 2.983263ms#033[0m
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] #033[1;32m2018-01-04 11:43:27: Completed /index/find 200 OK in 5.357149ms#033[0m
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] #033[1;32m2018-01-04 11:43:27: Completed /index/find 200 OK in 3.039755ms#033[0m
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] #033[1;32m2018-01-04 11:43:27: Completed /index/find 200 OK in 5.15759ms#033[0m
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] #033[1;32m2018-01-04 11:43:27: Completed /index/find 200 OK in 383.664763ms#033[0m
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] 2018-01-04 11:43:27: Started POST /getdata for 10.106.38.183
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] #033[1;32m2018-01-04 11:43:27: Completed /index/find 200 OK in 3.571271ms#033[0m
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] #033[1;32m2018-01-04 11:43:27: Completed /index/find 200 OK in 156.45785ms#033[0m
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] #033[1;32m2018-01-04 11:43:27: Completed /index/find 200 OK in 356.869488ms#033[0m
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] #033[1;32m2018-01-04 11:43:27: Completed /index/find 200 OK in 5.362755ms#033[0m
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] #033[1;32m2018-01-04 11:43:27: Completed /index/find 200 OK in 6.321715ms#033[0m
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] #033[1;32m2018-01-04 11:43:27: Completed /index/find 200 OK in 4.746094ms#033[0m
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] PANIC: runtime error: invalid memory address or nil pointer dereference
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/panic.go:491 (0x42b342)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/panic.go:63 (0x42a24d)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/signal_unix.go:367 (0x4426cb)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/idx/memory/memory.go:113 (0xa6739c)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/idx/memory/memory.go:807 (0xa65f5a)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/idx/memory/memory.go:807 (0xa65f5a)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/api/cluster.go:96 (0xa3f04f)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/api/routes.go:38 (0xa5d25a)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/asm_amd64.s:510 (0x45a35a)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:434 (0x4b8f85)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:302 (0x4b8563)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0x9ab288)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0x9aab89)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:113 (0x9c7aad)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:104 (0x9c79e5)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/api/middleware/tracer.go:64 (0x9e658f)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/asm_amd64.s:509 (0x45a2da)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:434 (0x4b8f85)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:302 (0x4b8563)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0x9ab288)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0x9aab89)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:113 (0x9c7aad)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:104 (0x9c79e5)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/api/middleware/stats.go:72 (0x9e5d78)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/asm_amd64.s:509 (0x45a2da)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:434 (0x4b8f85)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:302 (0x4b8563)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0x9ab288)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0x9aab89)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:113 (0x9c7aad)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:104 (0x9c79e5)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/raintank/gziper/gzip.go:73 (0xa3c257)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/asm_amd64.s:509 (0x45a2da)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:434 (0x4b8f85)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:302 (0x4b8563)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0x9ab288)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0x9aab89)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:113 (0x9c7aad)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:104 (0x9c79e5)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/recovery.go:161 (0x9d8dba)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/asm_amd64.s:509 (0x45a2da)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:434 (0x4b8f85)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:302 (0x4b8563)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0x9ab288)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0x9aab89)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:113 (0x9c7aad)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:104 (0x9c79e5)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/logger.go:43 (0x9d808b)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/asm_amd64.s:509 (0x45a2da)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:434 (0x4b8f85)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/reflect/value.go:302 (0x4b8563)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0x9ab288)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0x9aab89)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:113 (0x9c7aad)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/router.go:184 (0x9da0a2)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/router.go:286 (0x9d36eb)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/macaron.go:177 (0x9cbc27)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/net/http/server.go:2619 (0x6e22b3)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/net/http/server.go:1801 (0x6de5cc)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /usr/local/go/src/runtime/asm_amd64.s:2337 (0x45cad0)
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] #033[1;36m2018-01-04 11:43:27: Completed /index/find 500 Internal Server Error in 4.587958ms#033[0m
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] #033[1;32m2018-01-04 11:43:27: Completed /index/find 200 OK in 6.628582ms#033[0m
Jan  4 11:43:27 metrictank002 metrictank[6702]: [Macaron] #033[1;32m2018-01-04 11:43:27: Completed /index/find 200 OK in 6.727015ms#033[0m
Jan  4 11:43:27 metrictank002 metrictank[6702]: panic: runtime error: invalid memory address or nil pointer dereference
Jan  4 11:43:27 metrictank002 metrictank[6702]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xa6739c]
Jan  4 11:43:27 metrictank002 metrictank[6702]: goroutine 3245986 [running]:
Jan  4 11:43:27 metrictank002 metrictank[6702]: github.com/grafana/metrictank/idx/memory.(*Node).HasChildren(...)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/idx/memory/memory.go:113
Jan  4 11:43:27 metrictank002 metrictank[6702]: github.com/grafana/metrictank/idx/memory.(*MemoryIdx).find(0xc420828ae0, 0x1, 0xcc8421e400, 0x71, 0xcf6f58, 0x12, 0x4, 0x12, 0x4)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/idx/memory/memory.go:912 +0x65c
Jan  4 11:43:27 metrictank002 metrictank[6702]: github.com/grafana/metrictank/idx/memory.(*MemoryIdx).Find(0xc420828ae0, 0x1, 0xcc8421e400, 0x71, 0x5a4dbf00, 0x0, 0x0, 0x0, 0x0, 0x0)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/idx/memory/memory.go:807 +0x10b
Jan  4 11:43:27 metrictank002 metrictank[6702]: github.com/grafana/metrictank/api.(*Server).findSeriesLocal(0xc420905860, 0x1111240, 0xcb12868640, 0x1, 0xc80da5a9d0, 0x1, 0x1, 0x5a4dbf00, 0x0, 0x0, ...)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/api/graphite.go:133 +0x45c
Jan  4 11:43:27 metrictank002 metrictank[6702]: github.com/grafana/metrictank/api.(*Server).findSeries.func1(0xc420905860, 0x1111240, 0xcb12868640, 0x1, 0xc80da5a9d0, 0x1, 0x1, 0x5a4dbf00, 0xc80da5a9e0, 0xc9216ca5a0, ...)
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/api/graphite.go:76 +0x9b
Jan  4 11:43:27 metrictank002 metrictank[6702]: created by github.com/grafana/metrictank/api.(*Server).findSeries
Jan  4 11:43:27 metrictank002 metrictank[6702]: /home/circleci/.go_workspace/src/github.com/grafana/metrictank/api/graphite.go:75 +0x495
@motyla
Copy link
Author

motyla commented Jan 4, 2018

@Dieterbe, seems that this looks the same as old issue where opening root namespace in grafana query editor crash the nodes

@Dieterbe
Copy link
Contributor

Dieterbe commented Jan 5, 2018

for an instance that crashes like this, have you run any specific index related commands such as metric deletes? if not, this looks like there's some corrupt state in your index, specifically some child node is nil.
looking at the code, it could either be the startNode or one of its children, or grandchildren. enabling debug logging would help clarify, or getting an index dump may also reveal the problem.

we could also write a custom utility that reads your index, and analyzes the index around where the problem lies, and that should reveal the problem, but that would be custom development work.

@Dieterbe Dieterbe mentioned this issue Jan 5, 2018
Closed
@motyla
Copy link
Author

motyla commented Jan 6, 2018

thanks @Dieterbe ,
No , we did not ran any index related commands and it's more likely we have a corrupted index.
I think it's related to a bad data ingestion we made due to a misconfiguration at the source.

We will discuss it here this week to see how to handle.

Dieterbe added a commit that referenced this issue Jan 8, 2018
@Dieterbe
add clauses to detect nil Node's in idx.Find
2bf76b5 
helps #770
helps #811
@Dieterbe Dieterbe mentioned this issue Jan 8, 2018
Merged
@Dieterbe
Copy link
Contributor

Dieterbe commented Jan 8, 2018

this looks like the same problem as #770 . i have pushed some extra logging via #812 , please run that code and share the error messages you get.

@motyla
Copy link
Author

motyla commented Jan 8, 2018

@Dieterbe , here is the log output (now in the right place):

2018/01/08 14:27:52 [D] HTTP Render querying metrictank003/index/find for 1:["*"]
2018/01/08 14:27:52 [D] memory-idx: found first pattern sequence at node * pos 0
2018/01/08 14:27:52 [D] memory-idx: starting search at the root node
2018/01/08 14:27:52 [D] memory-idx: found first pattern sequence at node * pos 0
2018/01/08 14:27:52 [D] memory-idx: searching 17 children of  that match *
2018/01/08 14:27:52 [D] memory-idx: Matching all children
2018/01/08 14:27:52 [D] memory-idx: starting search at the root node
2018/01/08 14:27:52 [D] memory-idx: searching 17 children of  that match *
2018/01/08 14:27:52 [D] memory-idx: Matching all children
2018/01/08 14:27:52 [D] memory-idx: reached pattern length. 17 nodes matched
2018/01/08 14:27:52 [D] memory-idx: reached pattern length. 17 nodes matched
2018/01/08 14:27:52 [D] memory-idx: orgId -1 has no metrics indexed.
2018/01/08 14:27:52 [D] memory-idx: orgId -1 has no metrics indexed.
2018/01/08 14:27:52 [D] memory-idx: 17 nodes matching pattern * found
2018/01/08 14:27:52 [D] memory-idx: 17 nodes matching pattern * found
2018/01/08 14:27:52 [D] memory-idx Find: adding to path host archive id=1.3688022064380aeae3228ecdd381adc7 name=host int=60 schemaId=28 aggId=0 lastSave=1515421173
2018/01/08 14:27:52 [D] memory-idx Find: adding to path host archive id=1.3688022064380aeae3228ecdd381adc7 name=host int=60 schemaId=28 aggId=0 lastSave=1515421173
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xa68c35]

goroutine 233965 [running]:
github.com/grafana/metrictank/idx/memory.(*MemoryIdx).Find(0xc420bb74a0, 0x1, 0xc81496f845, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /home/circleci/.go_workspace/src/github.com/grafana/metrictank/idx/memory/memory.go:822 +0x475
github.com/grafana/metrictank/api.(*Server).findSeriesLocal(0xc4202f0460, 0x1116280, 0xcb7267d900, 0x1, 0xc51208c300, 0x1, 0x1, 0x0, 0x0, 0x0, ...)
        /home/circleci/.go_workspace/src/github.com/grafana/metrictank/api/graphite.go:133 +0x432
github.com/grafana/metrictank/api.(*Server).findSeries.func1(0xc4202f0460, 0x1116280, 0xcb7267d900, 0x1, 0xc51208c300, 0x1, 0x1, 0x0, 0xc51208c7a0, 0xcbc8fd0360, ...)
        /home/circleci/.go_workspace/src/github.com/grafana/metrictank/api/graphite.go:76 +0x9b
created by github.com/grafana/metrictank/api.(*Server).findSeries
        /home/circleci/.go_workspace/src/github.com/grafana/metrictank/api/graphite.go:75 +0x44d

@Dieterbe
Copy link
Contributor

Dieterbe commented Jan 24, 2018 •
edited

that log output is not as helpful as what you posted in #812 because this one doesn't show any errors.
but in that ticket, we have the clue i needed:

Jan  9 09:21:16 metrictank004 metrictank[36063]: 2018/01/09 09:21:16 [memory.go:934 find()] [E] memory-idx: grandChild is nil. o
rg=1,patt="*",i=0,pos=0,p="*",path="megaraidsas-metrics"

most likely the problem is those metrics are malformed, as mentioned in the other ticket.
'.megaraidsas-metrics' (starting with a dot)

2 things need to happen:

  1. stop sending the badly formed ones. then, delete the bad entries from the index and restart MT. let me know if you need further assistance
  2. we should not accept these metrics. how did you ingest them? via kafka-mdm or via carbon?

@Dieterbe
Copy link
Contributor

seems like we had this same issue before (#668) and fixed it via #694 but that was specifically carbon, so please let me know if you used carbon or kafka, and in kafka, how do you put them into kafka? where are these metrics being generated and/or transformed?

@motyla
Copy link
Author

motyla commented Jan 24, 2018 •
edited

We push metrics directly to Kafka and also through carbon-relay-ng. While getting the patch from you we used it to prevent crashes and we also started to put safeties where needed. In our code that write directly and with blacklists in carbon.

From what I saw , bad metrics came from both directions

This is my carbon-relay-ng blacklist:

'regex .*\.$',
'regex .*\.\..*',
'regex ^\..*',

@Dieterbe Dieterbe added this to the 0.10.0 milestone May 2, 2018
Dieterbe added a commit to raintank/fakemetrics that referenced this issue Aug 21, 2018
@Dieterbe
mode to send bad metrics (starting with .)
9d7c042 
helps with grafana/metrictank#811
@Dieterbe Dieterbe mentioned this issue Aug 21, 2018
Closed
@Dieterbe
Copy link
Contributor

Dieterbe commented Aug 21, 2018 •
edited

good news. with raintank/fakemetrics#9 i was able to trivially reproduce.
sending metric with name ".foo.bar" and trying to browse in grafana results in the same error:

metrictank_1    | [Macaron] 2018-08-21 13:16:25: Started GET /metrics/find?from=1534835724&query=*&until=1534857444 for 172.18.0.1, 172.18.0.1, 172.18.0.10
metrictank_1    | [Macaron] 2018-08-21 13:16:25: Started POST /index/find for 127.0.0.1
metrictank_1    | [Macaron] 2018-08-21 13:16:25: Completed /index/find 500 Internal Server Error in 445.8µs
metrictank_1    | 2018/08/21 13:16:25 [memory.go:1011 find()] [E] memory-idx: grandChild is nil. org=1,patt="*",i=0,pos=0,p="*",path="foo"
metrictank_1    | [Macaron] 2018-08-21 13:16:25: Completed /metrics/find?from=1534835724&query=*&until=1534857444 200 OK in 1.258231ms
metrictank_1    | 2018/08/21 13:16:25 [asm_amd64.s:2361 goexit()] [E] HTTP Render error querying default/index/find: "500 Internal Server Error"

our options are:

  1. graciously allow: in MT adjust incoming data accordingly.
  2. invalidate in MT, but that also means invalidate in tsdb-gw; in carbon-relay-ng we can strip it when converting from carbon to metrics2.0, this would be ideal, but unrealistic as we can't expect everyone to upgrade their carbon-relay-ng, so MT will just have to adjust the data.

in this process i also found out not only does graphite allow 1 leading dot, it seems to allow an unlimited amount, and will just chomp them all of. it also reduces sequences of dots in the middle of metric keys, so the previous fix for carbon (#694) will have to be improved.

creating a new ticket, #1008 to track this

Dieterbe added a commit to grafana/carbon-relay-ng that referenced this issue Aug 21, 2018
@Dieterbe
consume leading dots when converting from carbon
8f6d9ca 
graphite allows them, but our backends should be more strict

see grafana/metrictank#811
@Dieterbe Dieterbe mentioned this issue Aug 21, 2018
Closed
fkaleo pushed a commit to raintank/fakemetrics that referenced this issue Apr 16, 2019
@Dieterbe @fkaleo
mode to send bad metrics (starting with .)
e7b3c4f 
helps with grafana/metrictank#811
@robert-milan robert-milan mentioned this issue Jul 17, 2019
Merged
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
vnext
Development

No branches or pull requests
2 participants
@Dieterbe @motyla