nsqd: add topology region/zone aware message consumption #1301
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I think this is ready for a first round of review @mreiferson @ploxiln (along w/ it's pair nsqio/go-nsq#312 Some items to decide on -
Once this is squared away and we are happy with it i'll follow up with documentation PR's and exposing region/zone in nsqadmin. |
ploxiln
reviewed
Nov 22, 2020
apps/nsqd/options.go
Outdated
| @@ -120,6 +120,8 @@ func nsqdFlagSet(opts *nsqd.Options) *flag.FlagSet { | |||
| flagSet.Var(&lookupdTCPAddrs, "lookupd-tcp-address", "lookupd TCP address (may be given multiple times)") | |||
| flagSet.Duration("http-client-connect-timeout", opts.HTTPClientConnectTimeout, "timeout for HTTP connect") | |||
| flagSet.Duration("http-client-request-timeout", opts.HTTPClientRequestTimeout, "timeout for HTTP request") | |||
| flagSet.String("topology-region", opts.TopologyRegion, "A region represents a larger domain, made up of one or more zones") | |||
| flagSet.String("topology-zone", opts.TopologyZone, "A zone represents a logical failure domain") | |||
There was a problem hiding this comment.
should probably mention something about "for preferring closer consumer"
jehiah
force-pushed
the
topology_aware_msg_delivery_1301
branch
from
f390edc
to
96adf24
Compare
zoemccormick
force-pushed
the
topology_aware_msg_delivery_1301
branch
2 times, most recently
from
91517a4
to
fa8d2b4
Compare
zoemccormick
force-pushed
the
topology_aware_msg_delivery_1301
branch
from
2b90b29
to
42d28f5
Compare
zoemccormick
force-pushed
the
topology_aware_msg_delivery_1301
branch
from
88161cb
to
af1edde
Compare
zoemccormick
force-pushed
the
topology_aware_msg_delivery_1301
branch
from
af1edde
to
4631064
Compare
zoemccormick
force-pushed
the
topology_aware_msg_delivery_1301
branch
2 times, most recently
from
e93d2b5
to
b84cfe2
Compare
zoemccormick
force-pushed
the
topology_aware_msg_delivery_1301
branch
5 times, most recently
from
83e5939
to
d0cd516
Compare
zoemccormick
force-pushed
the
topology_aware_msg_delivery_1301
branch
from
8f51ed1
to
00fcfa9
Compare
|
@jehiah @ploxiln @mreiferson This PR is now officially ready for review - in conjunction with nsqio/go-nsq#312 and nsqio/nsqio.github.io#89. We have written up an experience report based on our observations running these changes for 2 months in our production environment - it can be found here. It also lays out the changes to nsqadmin. Please let me know if you have any other questions! Thanks in advance! |
jehiah
commented
May 21, 2024
Co-authored-by: Jehiah Czebotar <jehiah@gmail.com>
Co-authored-by: Jehiah Czebotar <jehiah@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This updates nsqd to have a
--topology-regionand--topology-zoneconfig argument and prefer sending messages to same-zone and same-region consumers as proposed in #1300