New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Coinbase: Update exchange implementation #1480
base: master
Are you sure you want to change the base?
Coinbase: Update exchange implementation #1480
Conversation
Continual enhance of Coinbase tests The revamp continues Oh jeez the Orderbook part's unfinished don't look Coinbase revamp, Orderbook still unfinished
…otrader into coinbase_api_revamp
V3 done, onto V2 Coinbase revamp nears completion Coinbase revamp nears completion Test commit should fail Coinbase revamp nears completion
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1480 +/- ##
==========================================
+ Coverage 36.01% 36.10% +0.08%
==========================================
Files 411 412 +1
Lines 177760 179015 +1255
==========================================
+ Hits 64025 64636 +611
- Misses 105881 106557 +676
+ Partials 7854 7822 -32
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just basic things, I think we should default to JWT as I cannot generate new keys for this 😭
if err != nil { | ||
return err | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
rm
err = c.Websocket.Conn.SendJSONMessage(req) | ||
if err != nil { | ||
return err | ||
} | ||
c.Websocket.RemoveSubscriptions(channelsToUnsubscribe...) | ||
return nil | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
return c.Websocket.Conn.SendJSONMessage(req)
for i := range data.Changes { | ||
switch data.Changes[i].Side { | ||
case "bid": | ||
bids = append(bids, orderbook.Item{ | ||
Price: data.Changes[i].PriceLevel, | ||
Amount: data.Changes[i].NewQuantity, | ||
}) | ||
case "offer": | ||
asks = append(asks, orderbook.Item{ | ||
Price: data.Changes[i].PriceLevel, | ||
Amount: data.Changes[i].NewQuantity, | ||
}) | ||
default: | ||
return nil, nil, errors.Errorf(errUnknownSide, data.Changes[i].Side) | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
for i := range data.Changes { | |
switch data.Changes[i].Side { | |
case "bid": | |
bids = append(bids, orderbook.Item{ | |
Price: data.Changes[i].PriceLevel, | |
Amount: data.Changes[i].NewQuantity, | |
}) | |
case "offer": | |
asks = append(asks, orderbook.Item{ | |
Price: data.Changes[i].PriceLevel, | |
Amount: data.Changes[i].NewQuantity, | |
}) | |
default: | |
return nil, nil, errors.Errorf(errUnknownSide, data.Changes[i].Side) | |
} | |
} | |
bids = make([]orderbook.Item, 0, len(data.Changes)) | |
asks = make([]orderbook.Item, 0, len(data.Changes)) | |
for i := range data.Changes { | |
change := orderbook.Item{Price: data.Changes[i].PriceLevel, Amount: data.Changes[i].NewQuantity} | |
switch data.Changes[i].Side { | |
case "bid": | |
bids = append(bids, change) | |
case "offer": | |
asks = append(asks, change) | |
default: | |
return nil, nil, errors.Errorf(errUnknownSide, data.Changes[i].Side) | |
} | |
} |
You can pre allocate, slightly wasteful, so leave it up to your discretion.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
After benchmarking, that doesn't seem wasteful, it seems flat out better. Nice.
exchanges/coinbasepro/coinbasepro.go
Outdated
return fee * amount * purchasePrice | ||
p.Values.Set(labelStart, startDate.Format(time.RFC3339)) | ||
p.Values.Set(labelEnd, endDate.Format(time.RFC3339)) | ||
return err |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
return nil
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice work, programmer! Here's some initial feedback on handling errors, authentication issues and test design
@@ -3,751 +3,1135 @@ package coinbasepro | |||
import ( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I feel like this package, filenames etc should be renamed to just coinbase
given that coinbase has dropped it https://www.coinbase.com/blog/hello-advanced-trade-goodbye-coinbase-pro
the "pro" part of things just refers to the "pro" interface on coinbase versus "basic". But we can look at doing this towards the end.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm a tad concerned about missing stuff in that conversion, so I'll leave it for later.
err := c.API.Endpoints.SetDefaultEndpoints(map[exchange.URL]string{ | ||
exchange.RestSpot: coinbaseproSandboxAPIURL, | ||
}) | ||
log.Fatal("failed to set sandbox endpoint", err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please address
log.Fatal("failed to set sandbox endpoint", err) | |
if err != nil { | |
log.Fatal("failed to set sandbox endpoint", err) | |
} |
t.Errorf("Coinbase, GetProducts() Error: %s", err) | ||
} | ||
func TestGetAllAccounts(t *testing.T) { | ||
sharedtestvalues.SkipTestIfCredentialsUnset(t, c) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add t.Parallel()
to all tests. Unless there is a reason you haven't - if so, please tell me so we can work out a way to have parallel tests 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Parallelism wasn't added for tests with outbound requests due to concerns over rate/race issues with sending a boatload of requests to the exchange at once.
Admittedly, I developed that heuristic very early on when working with this codebase, and I haven't properly examined whether it's an actual issue or not. If you don't think it is, I can chuck that into most of these, save for a few websocket tests that may need a linear order.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rate limits will handle outbound request issues. I'm seeing changes in ratelimits.go in this PR so I assume you've handled the API requirements for them. That would address any concerns you have over t.Parallel
. Let me know if you need help with rate limits otherwise
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah that was unfounded. Added that to all non-WS tests.
accounts, err := c.GetAllAccounts(context.Background(), 250, "") | ||
assert.NoError(t, err) | ||
if accounts == nil || len(accounts.Accounts) == 0 { | ||
t.Fatal(errExpectedNonEmpty) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems more appropriate to skip, than to fatal. It is not an error that I, a peasant, have no funds 😭
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
By default, I believe Coinbase should list at least one account associated with an API key; if none come back I think that implies something was wrong with the data. Later, when I do an actual check for funds, I merely skip if there isn't enough.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll reverify after you've addressed the other comments. I have a verified account and API key and was hitting this line (len(accounts.Accounts) was 0). I don't believe I was receiving any errors from the API
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please reverify; if you're still getting that, I'll change it to a skip.
c = &CoinbasePro{} | ||
testCrypto = currency.BTC | ||
testFiat = currency.USD | ||
testPair = currency.NewPairWithDelimiter(testCrypto.String(), testFiat.String(), "-") | ||
) | ||
|
||
// Please supply your APIKeys here for better testing | ||
const ( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you please move this section above the vars?
exchanges/coinbasepro/ratelimit.go
Outdated
case WSRate: | ||
return r.RateLimWS.Wait(ctx) | ||
default: | ||
return errors.Errorf(errUnknownEndpointLimit, f) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We typically format errors like the following:
return errors.Errorf(errUnknownEndpointLimit, f) | |
return fmt.Errorf("%w %v",errUnknownEndpointLimit, f) |
Then, you should be changing errUnknownSide
to the following:
var (
errUnknownSide = errors.New("unknown side")
)
That way, in test TestProcessSnapshotUpdate
you can do the following check:
assert.ErrorIs(t, err, errUnknownSide)
Or, as a complete example:
func TestProcessSnapshotUpdate(t *testing.T) {
t.Parallel()
req := WebsocketOrderbookDataHolder{Changes: []WebsocketOrderbookData{{Side: "fakeside", PriceLevel: 1.1,
NewQuantity: 2.2}}, ProductID: currency.NewBTCUSD()}
err := c.ProcessSnapshot(&req, time.Time{})
assert.ErrorIs(t, err, errUnknownSide)
req.Changes[0].Side = "offer"
err = c.ProcessSnapshot(&req, time.Now())
assert.NoError(t, err)
err = c.ProcessUpdate(&req, time.Now())
assert.NoError(t, err)
}
fmt.Errorf
allows for rich details to be added to errors, which is helpful for the user. In addition, golang allows you to wrap multiple errors together. Wrapping multiple errors allow for good debugging techniques where we can handle scenarios like
if err != nil {
if errors.Is(err, errTheThingBlewUp) {
// do soemthing specific to address the error
} else {
return nil, err
}
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So what I'd like you to do is update all the const err
you have declared in the coinbasepro
package, and change them to errors.New("...")
type errors. Then you can change the error handling in your tests to better parse what's going on, instead of relying on string comparisons via err.Error()
. Make them like my example above
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've changed most of these, but left errIntervalNotSupported (never used sprintf formatting; was used to maintain every return value being a string) and warnSequenceIssue (kept as a string to distinguish this recoverable issue of data loss from a more pressing error, since only string-based functions are used on it, and since strings passed to websocketDataHandler are logged differently from errors).
If you want me to change those too, let me know.
I also took the liberty of changing some code to remove further error.Error calls that were ultimately rooted in the Coinbase code, but didn't previously have associated variables of any kind.
skipTestIfLowOnFunds(t) | ||
resp, err := c.PreviewOrder(context.Background(), testPair.String(), "BUY", "MARKET", "", "", 0, 1, 0, 0, 0, 0, false, | ||
false, false, time.Time{}) | ||
assert.NoError(t, err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am receiving the following:
CoinbasePro unsuccessful HTTP status code: 400 raw response: {"error":"unknown","error_details":"proto: (line 1:78): invalid value for enum type: \"\"","message":"proto: (line 1:78): invalid value for enum type: \"\""}, authenticated request failed
This seems to suggest an invalid request. I'm receiving this with multiple authorised requests. Please let me know if you are passing these tests and I'll look into what's going on for me a bit further before commenting
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, that slipped under my testing. The marginType parameter was added later; when I went back to add it I neglected to remove skipTestIfLowOnFunds. Since I didn't have enough funds to properly run it, that had me never check if margin_type could be sent as an empty string, and my assumption that it could was wrong.
I've fixed that oversight. Now, I fail with the error "account is not available", which I suspect means that I have insufficient funds. Further info would be appreciated!
_, err := c.GetAllPaymentMethods(context.Background()) | ||
// This endpoint appears to currently be broken for all users | ||
if err.Error() != errInternalMessageNil { | ||
t.Error(err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You are making some assumptions that an err
will be returned. You should be checking if the err != nil
. Since I did not receive an error, this test panics:
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x18 pc=0x104889870]
goroutine 259 [running]:
testing.tRunner.func1.2({0x104a44900, 0x104e518d0})
/opt/homebrew/Cellar/go/1.22.1/libexec/src/testing/testing.go:1631 +0x1c4
testing.tRunner.func1()
/opt/homebrew/Cellar/go/1.22.1/libexec/src/testing/testing.go:1634 +0x33c
panic({0x104a44900?, 0x104e518d0?})
/opt/homebrew/Cellar/go/1.22.1/libexec/src/runtime/panic.go:770 +0x124
github.com/thrasher-corp/gocryptotrader/exchanges/coinbasepro.TestGetAllPaymentMethods(0x140003a6820)
/Users/WOW/go/src/github.com/thrasher-corp/gocryptotrader/exchanges/coinbasepro/coinbasepro_test.go:601 +0x60
testing.tRunner(0x140003a6820, 0x104af7e68)
/opt/homebrew/Cellar/go/1.22.1/libexec/src/testing/testing.go:1689 +0xec
created by testing.(*T).Run in goroutine 1
/opt/homebrew/Cellar/go/1.22.1/libexec/src/testing/testing.go:1742 +0x318
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So please update your tests to check for nil
errors before calling err.Error()
. However, along with my other commentary about changing the error strings to errors.New
it will be much easier to handle these issues and you won't need to call err.Error()
. Other exchanges like binance_test.go
will be helpful in that regard such as TestWrapperGetServerTime
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah that was just me implementing a dumb way of being notified when that endpoint stopped returning an error for everyone. Fixed, and I'll try to avoid that sort of thing in the future.
}) | ||
// sendRequest is a helper function which sends a websocket message to the Coinbase server | ||
func (c *CoinbasePro) sendRequest(msgType, channel string, productIDs currency.Pairs) error { | ||
creds, err := c.GetCredentials(context.Background()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe this is causing issues. I do not have credentials set in my config and authenticatedWebsocketApiSupport
is set to false
. So I am receiving the following error when attempting to subscribe:
[ERROR] | WEBSOCKET | 26/03/2024 15:20:05 | CoinbasePro websocket: subscription failure, CoinbasePro REST or Websocket authentication support is not enabled
I do not believe authentication is required to subscribe to a ticker
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is. All websocket endpoints require authentication. That's not explicitly stated, but these three links imply it https://docs.cloud.coinbase.com/advanced-trade-api/docs/auth https://docs.cloud.coinbase.com/advanced-trade-api/docs/ws-auth https://docs.cloud.coinbase.com/advanced-trade-api/docs/ws-channels and removing the signature/key from the request causes the exchange to complain, even for channels like "ticker".
vals.Set("end", strconv.FormatInt(endDate.Unix(), 10)) | ||
} | ||
var resp Ticker | ||
return &resp, c.SendAuthenticatedHTTPRequest(ctx, exchange.RestSpot, http.MethodGet, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is not an authenticated endpoint: https://docs.cloud.coinbase.com/advanced-trade-api/reference/retailbrokerageapi_getmarkettrades
[ERROR] | SYNC | 26/03/2024 15:23:23 | Failed to get REST ticker. Error: CoinbasePro REST or Websocket authentication support is not enabled
return &resp, c.SendAuthenticatedHTTPRequest(ctx, exchange.RestSpot, http.MethodGet, | |
return &resp, c.SendHTTPRequest(ctx, exchange.RestSpot, path, &resp) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please verify other API endpoints and verify that they are not attempting to send requests with authentication when it is not needed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No V3 endpoint is public except for Get Server Time, as per https://docs.cloud.coinbase.com/advanced-trade-api/docs/rest-api-overview
Although that was accidentally left authenticated too; I'll push a fix for that along with the other stuff you brought up soon.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just to add onto this. Could you please implement all unauthenticated endpoints from https://docs.cloud.coinbase.com/exchange/reference
This will allow for basic data retrieval without credentials. Endpoints such as
Instruments
Ticker
Orderbook
Trades
Candles
These will be important to reference in the wrapper functions:
FetchTradablePairs
UpdateTickers
UpdateTicker
UpdateOrderbook
GetRecentTrades
GetHistoricTrades
GetHistoricCandles
GetHistoricCandlesExtended
UpdateOrderExecutionLimits
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for your updates @cranktakular! I have some extra requests to utilise public endpoints. The reason for this is because the Advanced Trading API does not yet have a public API, but is promised to be "soon"™️.
Utilising the existing public trading endpoints for the base API will allow for more utilisation of data without requiring credentials. However, when API credentials are present, it is still better to utilise those endpoints as this is a replacement of "coinbase pro" and not "coinbase"
err = c.Setup(cfg) | ||
if err.Error() != errx7f { | ||
t.Errorf(errExpectMismatch, err, errx7f) | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please update all instances of error checking to utilise err != nil
and errors.Is
.
I would also expect a test to be able to perform the important function of Setup
to also have a scenario where no error occurs
// blockedExchanges are exchanges that are not able to be tested in general | ||
var blockedExchanges = []string{ | ||
"coinbasepro", // coinbasepro API requires authentication for almost every endpoint | ||
} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe you've thought about this already, but worth mentioning in case it hasn't been covered already:
As a design pattern I think it's preferable to find any way to consolidate all information about an exchange onto it's own packages.
Off the top of my head I'm thinking about adding an exported constant using build tags for tests, and then using reflect to check for it.
The things I'm aiming for with that are:
- No cost or boilerplate to exchanges which don't care
- No production runtime cost
- More information about the exchange is consolidated into it's dir, making it easier to find and delete in one go
I'm not suggesting you need to do this, especially in light of blockedCIExchanges
already existing, but wanted to share it in case you want to engage, and for any other work in flight where it's appropriate.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
After incorporating public endpoints from an earlier version of the API, I no longer need that blockedExchanges variable to exist so I've removed it.
Moving towards that sort of design for other similar stuff within those standards seems like a good idea, but I think it's a bit out of scope for my current work.
PR Description
Updating Coinbase to the Advanced Trade and Sign In With Coinbase APIs, as well as a few tiny fixes of other parts of the codebase found along the way.
Type of change
Please delete options that are not relevant and add an
x
in[]
as item is complete.How has this been tested
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration and
also consider improving test coverage whilst working on a certain feature or package.
Checklist
The only tests which seem to be failing (exchanges/kucoin, database/models/postgres, config, and common/file) are parts that I haven't substantively changed, and which seem to be failing on master too.