Skip to content

Commit

Permalink
Fix possible deadlock in background WAL uploader
Browse files Browse the repository at this point in the history
Stop() of background uploader was acquiring upload scan mutex. Under
this mutex, it was disallowing new uploads and awaits finish of all
spawned uploads. But scan process has to acquire that mutex too.
In various race outcomes, upload could deadlock with Stop().

Now Stop() awaits for finished uploads with the released mutex.
Mutex guards that upload is not started if Stop() in progress.

With current mutex guards, we could relax all atomic operations
with bgUploader.parallelWorkers, but this requires extra refactoring,
now we should just fix the bug.
  • Loading branch information
x4m committed Mar 27, 2019
1 parent 159ee0a commit 7853673
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion bguploader.go
Original file line number Diff line number Diff line change
Expand Up @@ -65,8 +65,9 @@ func (uploader *BgUploader) Stop() {
} // Wait until noone works

uploader.mutex.Lock()
defer uploader.mutex.Unlock()
// We have to do this under mutex to exclude interference with shouldKeepScanning() branch
atomic.StoreInt32(&uploader.maxParallelWorkers, 0) // stop new jobs
uploader.mutex.Unlock()
uploader.running.Wait() // wait again for those how jumped to the closing door
}

Expand Down

0 comments on commit 7853673

Please sign in to comment.