Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Subscription Pairs and Encapsulation #1501

Open
wants to merge 89 commits into
base: master
Choose a base branch
from

Conversation

gbjk
Copy link
Collaborator

@gbjk gbjk commented Mar 11, 2024

  • Move Subscriptions to their own encapsulated store, list, and package
    This work starts the process of disentangling subscriptions from websocket, since a subscription is an exchange concept, and we want the collection of subscriptions separate from each websocket so we can start to separate auth, non-auth, assets, private and multiplexed websockets
  • Move Subscription Pair to Pairs
    For some exchanges we've immediately made full use of this.
    For others we've ported over the existing one-pair method, and will revist it when we tackle subscription configuration
  • Subscription methods all use pointers, and mutexes exist on the store and on the subscriptions to allow this
  • Subscription configuration support for Bitmex and CoinbasePro
  • Fixes and improvements to many exchanges' subscription logic as a driveby
  • Linter: Disable testify.Len
  • Tests: Fix WsAuth turned off by config checking
  • Tests: TestFixtureToDataHandler preserve WS
  • Websocket: Log actual sent message when Verbose

Type of change

  • Non-backwards compatible feature

How has this been tested

  • go test ./... -race
  • golangci-lint run

@gbjk gbjk added the review me This pull request is ready for review label Mar 11, 2024
@gbjk gbjk self-assigned this Mar 11, 2024
Copy link

codecov bot commented Mar 11, 2024

Codecov Report

Attention: Patch coverage is 37.95380% with 752 lines in your changes are missing coverage. Please review.

Project coverage is 36.02%. Comparing base (c967d8a) to head (c45b65e).
Report is 23 commits behind head on master.

Current head c45b65e differs from pull request most recent head 913fb2d

Please upload reports for the commit 913fb2d to get more accurate results.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1501      +/-   ##
==========================================
- Coverage   36.04%   36.02%   -0.03%     
==========================================
  Files         411      415       +4     
  Lines      176906   177039     +133     
==========================================
  Hits        63774    63774              
- Misses     105351   105475     +124     
- Partials     7781     7790       +9     
Files Coverage Δ
currency/pair.go 86.40% <ø> (ø)
currency/pairs.go 98.14% <100.00%> (+0.04%) ⬆️
exchanges/bitfinex/bitfinex_wrapper.go 37.16% <100.00%> (ø)
exchanges/bithumb/bithumb_wrapper.go 29.38% <100.00%> (ø)
exchanges/btcmarkets/btcmarkets.go 24.14% <ø> (ø)
exchanges/exchange.go 77.08% <100.00%> (ø)
exchanges/kucoin/kucoin_wrapper.go 34.59% <100.00%> (+0.05%) ⬆️
exchanges/sharedtestvalues/sharedtestvalues.go 71.31% <100.00%> (-0.66%) ⬇️
exchanges/subscription/keys.go 100.00% <100.00%> (ø)
exchanges/subscription/list.go 100.00% <100.00%> (ø)
... and 39 more

... and 14 files with indirect coverage changes

Copy link
Collaborator

@shazbert shazbert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome update. Just a once over before I test all the changes. 🚀

engine/rpcserver.go Outdated Show resolved Hide resolved
exchanges/okx/okx_websocket.go Outdated Show resolved Hide resolved
exchanges/stream/websocket.go Outdated Show resolved Hide resolved
exchanges/stream/websocket_test.go Outdated Show resolved Hide resolved
exchanges/subscription/store.go Outdated Show resolved Hide resolved
exchanges/subscription/store.go Show resolved Hide resolved
exchanges/subscription/store.go Outdated Show resolved Hide resolved
exchanges/subscription/subscription.go Outdated Show resolved Hide resolved
exchanges/subscription/subscription_test.go Outdated Show resolved Hide resolved
@gbjk gbjk force-pushed the feature/sub_pairs branch 5 times, most recently from 3149dba to d7e5c4c Compare March 21, 2024 05:45
@gbjk gbjk added reconstructing Based on PR feedback, this is currently being reworked and is not to be merged and removed reconstructing Based on PR feedback, this is currently being reworked and is not to be merged labels Apr 8, 2024
@gbjk gbjk force-pushed the feature/sub_pairs branch 2 times, most recently from d2622f0 to 6774e9d Compare April 9, 2024 10:54
@gbjk gbjk requested a review from shazbert April 11, 2024 10:31
Copy link
Collaborator

@gloriousCode gloriousCode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a big PR with many changes so it will take multiple looks. I'm overall happy with the design of it all, along with the expansion of configurable subscriptions

exchanges/bitmex/bitmex_websocket.go Show resolved Hide resolved
currency/pairs_test.go Outdated Show resolved Hide resolved
exchanges/binance/binance_websocket.go Outdated Show resolved Hide resolved
exchanges/bitfinex/bitfinex_websocket.go Outdated Show resolved Hide resolved
exchanges/subscription/subscription.go Outdated Show resolved Hide resolved
exchanges/bybit/bybit_websocket.go Outdated Show resolved Hide resolved
exchanges/coinut/coinut_websocket.go Outdated Show resolved Hide resolved
exchanges/gateio/gateio_ws_delivery_futures.go Outdated Show resolved Hide resolved
exchanges/gateio/gateio_ws_option.go Outdated Show resolved Hide resolved
exchanges/kraken/kraken_websocket.go Outdated Show resolved Hide resolved
@gbjk gbjk mentioned this pull request Apr 14, 2024
5 tasks
@gbjk gbjk force-pushed the feature/sub_pairs branch 2 times, most recently from 73b8699 to c3250d3 Compare April 15, 2024 11:11
@gbjk gbjk requested a review from gloriousCode April 15, 2024 11:14
Copy link
Collaborator

@shazbert shazbert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good stuff this is looking really good. 🚀

currency/pairs.go Show resolved Hide resolved
exchanges/coinbasepro/coinbasepro_websocket.go Outdated Show resolved Hide resolved
exchanges/coinbasepro/coinbasepro_websocket.go Outdated Show resolved Hide resolved
exchanges/kucoin/kucoin_test.go Show resolved Hide resolved
exchanges/kucoin/kucoin_test.go Outdated Show resolved Hide resolved
exchanges/poloniex/poloniex_types.go Show resolved Hide resolved
exchanges/subscription/keys.go Show resolved Hide resolved
exchanges/subscription/keys.go Show resolved Hide resolved
exchanges/subscription/subscription.go Show resolved Hide resolved
exchanges/subscription/subscription.go Show resolved Hide resolved
If the exchange passed in already has a websocket, don't clobber it
instead of errChannelAlreadySubscribed
Given that some subscriptions have multiple pairs, support that as the
standard.
We deliberately use Equal over Len to avoid spamming the contents of large Slices
Copy link
Collaborator

@gloriousCode gloriousCode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feels like there's still heaps more for me to go over and test. So here's some things I've thought about while reviewing

exchanges/bitfinex/bitfinex_websocket.go Outdated Show resolved Hide resolved
exchanges/coinbasepro/coinbasepro_websocket.go Outdated Show resolved Hide resolved
exchanges/coinut/coinut_websocket.go Outdated Show resolved Hide resolved
exchanges/kraken/kraken_websocket.go Outdated Show resolved Hide resolved
exchanges/binance/binance_websocket.go Show resolved Hide resolved
exchanges/stream/websocket.go Outdated Show resolved Hide resolved
@@ -212,11 +214,10 @@ func New(input string) (Item, error) {
return USDCMarginedFutures, nil
case options, "option":
return Options, nil
case all:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is my own personal review comment to say that I have reviewed the manner of all and am satisfied. This is to prevent me from reviewing it out of concern every single time I review this PR

Copy link
Collaborator

@gloriousCode gloriousCode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Things are looking good. My next review run will go over running each exchange to verify that existing functionality is maintained

exchanges/subscription/keys_test.go Show resolved Hide resolved
exchanges/subscription/keys_test.go Show resolved Hide resolved
exchanges/btcmarkets/btcmarkets_websocket.go Outdated Show resolved Hide resolved
exchanges/btcmarkets/btcmarkets_websocket.go Show resolved Hide resolved
exchanges/coinbasepro/coinbasepro_wrapper.go Show resolved Hide resolved
exchanges/poloniex/poloniex_types.go Show resolved Hide resolved
exchanges/subscription/store.go Outdated Show resolved Hide resolved
exchanges/subscription/subscription.go Outdated Show resolved Hide resolved
Comment on lines 106 to 109
if s.Key == nil {
s.Key = DefaultKey{
Channel: s.Channel,
Asset: s.Asset,
Pair: s.Pair,
}
s.Key = &ExactKey{s}
}
return s.Key
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looke like it should have mutex use now

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed bd73186

Comment on lines +68 to +71
if k, ok := key.(MatchableKey); ok {
return k.String()
}
return fmt.Sprintf("%v: %s", key, ExactKey{s}.String())
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a potential race zone

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. Not particularly easy to get around, either.

Fixed 913fb2d

@codecov-commenter
Copy link

codecov-commenter commented May 24, 2024

Codecov Report

Attention: Patch coverage is 37.95380% with 752 lines in your changes are missing coverage. Please review.

Project coverage is 36.02%. Comparing base (c967d8a) to head (c45b65e).
Report is 25 commits behind head on master.

Current head c45b65e differs from pull request most recent head 913fb2d

Please upload reports for the commit 913fb2d to get more accurate results.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1501      +/-   ##
==========================================
- Coverage   36.04%   36.02%   -0.03%     
==========================================
  Files         411      415       +4     
  Lines      176906   177039     +133     
==========================================
  Hits        63774    63774              
- Misses     105351   105475     +124     
- Partials     7781     7790       +9     
Files Coverage Δ
currency/pair.go 86.40% <ø> (ø)
currency/pairs.go 98.14% <100.00%> (+0.04%) ⬆️
exchanges/bitfinex/bitfinex_wrapper.go 37.16% <100.00%> (ø)
exchanges/bithumb/bithumb_wrapper.go 29.38% <100.00%> (ø)
exchanges/btcmarkets/btcmarkets.go 24.14% <ø> (ø)
exchanges/exchange.go 77.08% <100.00%> (ø)
exchanges/kucoin/kucoin_wrapper.go 34.59% <100.00%> (+0.05%) ⬆️
exchanges/sharedtestvalues/sharedtestvalues.go 71.31% <100.00%> (-0.66%) ⬇️
exchanges/subscription/keys.go 100.00% <100.00%> (ø)
exchanges/subscription/list.go 100.00% <100.00%> (ø)
... and 39 more

... and 14 files with indirect coverage changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants