backupDSLoop exception: mark end of backup operation: driver: bad connection; driver: bad connection

[kalavt]

Describe the bug
mark end of backup operation: driver: bad connection; driver: bad connection

Expected behavior
version: 1.22.1

Screenshots
`
2024-04-02T13:32:11.775Z ERROR utils.misc misc/misc.go:926 Panic detected. Application will crash. {"stack": "goroutine 430 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x5e\ngithub.com/rudderlabs/rudder-server/utils/misc.BugsnagNotify.func1.1()\n\t/rudder-server/utils/misc/misc.go:927 +0x37e\nsync.(*Once).doSlow(0x60f1280?, 0xc002b31e18?)\n\t/usr/local/go/src/sync/once.go:74 +0xbf\nsync.(*Once).Do(...)\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/rudderlabs/rudder-server/utils/misc.BugsnagNotify.func1()\n\t/rudder-server/utils/misc/misc.go:916 +0xbb\npanic({0x32a8f60?, 0xc08b668dc0?})\n\t/usr/local/go/src/runtime/panic.go:914 +0x21f\ngithub.com/rudderlabs/rudder-server/jobsdb.(*Handle).backupDSLoop(0xc00147f880, {0x40ae8d0, 0xc00086d860})\n\t/rudder-server/jobsdb/backup.go:105 +0x345\ngithub.com/rudderlabs/rudder-server/jobsdb.(*Handle).startBackupDSLoop.func1()\n\t/rudder-server/jobsdb/jobsdb.go:1016 +0x1f\ngithub.com/rudderlabs/rudder-server/jobsdb.(*Handle).startBackupDSLoop.WithBugsnag.func2()\n\t/rudder-server/utils/misc/misc.go:945 +0x5c\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/sync@v0.6.0/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 573\n\t/go/pkg/mod/golang.org/x/sync@v0.6.0/errgroup/errgroup.go:75 +0x96\n", "panic": "mark end of backup operation: driver: bad connection; driver: bad connection", "team": "Core", "goRoutines": 1169, "version": "1.22.1", "releaseStage": "development"}
2024-04-02T13:32:11.776Z ERROR utils.misc misc/misc.go:926 goroutine 430 [running]:
github.com/rudderlabs/rudder-go-kit/logger.(*logger).Fatalw(0xc001490a50, {0x38fbe93?, 0x4bf?}, {0xc089ae1500?, 0x1?, 0x4ffbcc?})
/go/pkg/mod/github.com/rudderlabs/rudder-go-kit@v0.23.2/logger/logger.go:332 +0xa5
github.com/rudderlabs/rudder-server/utils/misc.BugsnagNotify.func1.1()
/rudder-server/utils/misc/misc.go:926 +0x5f6
sync.(*Once).doSlow(0x60f1280?, 0xc002b31e18?)
/usr/local/go/src/sync/once.go:74 +0xbf
sync.(*Once).Do(...)
/usr/local/go/src/sync/once.go:65
github.com/rudderlabs/rudder-server/utils/misc.BugsnagNotify.func1()
/rudder-server/utils/misc/misc.go:916 +0xbb
panic({0x32a8f60?, 0xc08b668dc0?})
/usr/local/go/src/runtime/panic.go:914 +0x21f
github.com/rudderlabs/rudder-server/jobsdb.(*Handle).backupDSLoop(0xc00147f880, {0x40ae8d0, 0xc00086d860})
/rudder-server/jobsdb/backup.go:105 +0x345
github.com/rudderlabs/rudder-server/jobsdb.(*Handle).startBackupDSLoop.func1()
/rudder-server/jobsdb/jobsdb.go:1016 +0x1f
github.com/rudderlabs/rudder-server/jobsdb.(*Handle).startBackupDSLoop.WithBugsnag.func2()
/rudder-server/utils/misc/misc.go:945 +0x5c
golang.org/x/sync/errgroup.(*Group).Go.func1()
/go/pkg/mod/golang.org/x/sync@v0.6.0/errgroup/errgroup.go:78 +0x56
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 573
/go/pkg/mod/golang.org/x/sync@v0.6.0/errgroup/errgroup.go:75 +0x96

2024/04/02 13:32:11 notifying bugsnag: mark end of backup operation: driver: bad connection; driver: bad connection
2024/04/02 13:32:11 bugsnag.Notify: bugsnag/payload.deliver: invalid api key: ''
2024/04/02 13:32:11 bugsnag/sessions/publisher.publish invalid API key: ''
panic: mark end of backup operation: driver: bad connection; driver: bad connection [recovered]
panic: mark end of backup operation: driver: bad connection; driver: bad connection [recovered]
panic: mark end of backup operation: driver: bad connection; driver: bad connection

goroutine 430 [running]:
github.com/bugsnag/bugsnag-go/v2.AutoNotify({0xc06e6099f0, 0x3, 0x7f09e72068b8?})
/go/pkg/mod/github.com/bugsnag/bugsnag-go/v2@v2.3.0/bugsnag.go:114 +0x3e8
panic({0x32a8f60?, 0xc08b668dc0?})
/usr/local/go/src/runtime/panic.go:914 +0x21f
github.com/rudderlabs/rudder-server/utils/misc.BugsnagNotify.func1.1()
/rudder-server/utils/misc/misc.go:935 +0x614
sync.(*Once).doSlow(0x60f1280?, 0xc002b31e18?)
/usr/local/go/src/sync/once.go:74 +0xbf
sync.(*Once).Do(...)
/usr/local/go/src/sync/once.go:65
github.com/rudderlabs/rudder-server/utils/misc.BugsnagNotify.func1()
/rudder-server/utils/misc/misc.go:916 +0xbb
panic({0x32a8f60?, 0xc08b668dc0?})
/usr/local/go/src/runtime/panic.go:914 +0x21f
github.com/rudderlabs/rudder-server/jobsdb.(*Handle).backupDSLoop(0xc00147f880, {0x40ae8d0, 0xc00086d860})
/rudder-server/jobsdb/backup.go:105 +0x345
github.com/rudderlabs/rudder-server/jobsdb.(*Handle).startBackupDSLoop.func1()
/rudder-server/jobsdb/jobsdb.go:1016 +0x1f
github.com/rudderlabs/rudder-server/jobsdb.(*Handle).startBackupDSLoop.WithBugsnag.func2()
/rudder-server/utils/misc/misc.go:945 +0x5c
golang.org/x/sync/errgroup.(*Group).Go.func1()
/go/pkg/mod/golang.org/x/sync@v0.6.0/errgroup/errgroup.go:78 +0x56
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 573
/go/pkg/mod/golang.org/x/sync@v0.6.0/errgroup/errgroup.go:75 +0x96
`

[gitcommitshow]

@kalavt can you please share more details? How to reproduce this issue?