Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Coinbase: Update exchange implementation #1480

Open
wants to merge 56 commits into
base: master
Choose a base branch
from

Conversation

cranktakular
Copy link
Contributor

@cranktakular cranktakular commented Feb 14, 2024

PR Description

Updating Coinbase to the Advanced Trade and Sign In With Coinbase APIs, as well as a few tiny fixes of other parts of the codebase found along the way.

Type of change

Please delete options that are not relevant and add an x in [] as item is complete.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update

How has this been tested

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration and
also consider improving test coverage whilst working on a certain feature or package.

  • go test ./... -race
  • golangci-lint run

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation and regenerated documentation via the documentation tool
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally and on Github Actions with my changes

The only tests which seem to be failing (exchanges/kucoin, database/models/postgres, config, and common/file) are parts that I haven't substantively changed, and which seem to be failing on master too.

Continual enhance of Coinbase tests

The revamp continues

Oh jeez the Orderbook part's unfinished don't look

Coinbase revamp, Orderbook still unfinished
V3 done, onto V2

Coinbase revamp nears completion

Coinbase revamp nears completion

Test commit should fail

Coinbase revamp nears completion
@thrasher- thrasher- changed the title Coinbase api revamp Coinbase: Update exchange implementation Feb 14, 2024
@thrasher- thrasher- requested a review from a team February 14, 2024 23:40
Copy link

codecov bot commented Feb 27, 2024

Codecov Report

Attention: Patch coverage is 40.48203% with 1457 lines in your changes are missing coverage. Please review.

Project coverage is 36.10%. Comparing base (ece58eb) to head (01ccd01).
Report is 11 commits behind head on master.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1480      +/-   ##
==========================================
+ Coverage   36.01%   36.10%   +0.08%     
==========================================
  Files         411      412       +1     
  Lines      177760   179015    +1255     
==========================================
+ Hits        64025    64636     +611     
- Misses     105881   106557     +676     
+ Partials     7854     7822      -32     
Files Coverage Δ
common/common.go 94.79% <ø> (ø)
config/config.go 86.42% <100.00%> (ø)
currency/code_types.go 0.00% <ø> (ø)
currency/pair.go 84.78% <ø> (ø)
engine/datahistory_manager.go 64.44% <100.00%> (ø)
exchanges/account/account.go 97.36% <100.00%> (ø)
exchanges/asset/asset.go 94.91% <100.00%> (+0.08%) ⬆️
exchanges/exchange.go 76.21% <100.00%> (ø)
exchanges/lbank/lbank.go 28.14% <100.00%> (ø)
exchanges/lbank/lbank_wrapper.go 32.29% <100.00%> (ø)
... and 11 more

... and 17 files with indirect coverage changes

Copy link
Collaborator

@shazbert shazbert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just basic things, I think we should default to JWT as I cannot generate new keys for this 😭

Comment on lines 457 to 459
if err != nil {
return err
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rm

Comment on lines 460 to 465
err = c.Websocket.Conn.SendJSONMessage(req)
if err != nil {
return err
}
c.Websocket.RemoveSubscriptions(channelsToUnsubscribe...)
return nil
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return c.Websocket.Conn.SendJSONMessage(req)

Comment on lines 470 to 485
for i := range data.Changes {
switch data.Changes[i].Side {
case "bid":
bids = append(bids, orderbook.Item{
Price: data.Changes[i].PriceLevel,
Amount: data.Changes[i].NewQuantity,
})
case "offer":
asks = append(asks, orderbook.Item{
Price: data.Changes[i].PriceLevel,
Amount: data.Changes[i].NewQuantity,
})
default:
return nil, nil, errors.Errorf(errUnknownSide, data.Changes[i].Side)
}
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
for i := range data.Changes {
switch data.Changes[i].Side {
case "bid":
bids = append(bids, orderbook.Item{
Price: data.Changes[i].PriceLevel,
Amount: data.Changes[i].NewQuantity,
})
case "offer":
asks = append(asks, orderbook.Item{
Price: data.Changes[i].PriceLevel,
Amount: data.Changes[i].NewQuantity,
})
default:
return nil, nil, errors.Errorf(errUnknownSide, data.Changes[i].Side)
}
}
bids = make([]orderbook.Item, 0, len(data.Changes))
asks = make([]orderbook.Item, 0, len(data.Changes))
for i := range data.Changes {
change := orderbook.Item{Price: data.Changes[i].PriceLevel, Amount: data.Changes[i].NewQuantity}
switch data.Changes[i].Side {
case "bid":
bids = append(bids, change)
case "offer":
asks = append(asks, change)
default:
return nil, nil, errors.Errorf(errUnknownSide, data.Changes[i].Side)
}
}

You can pre allocate, slightly wasteful, so leave it up to your discretion.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After benchmarking, that doesn't seem wasteful, it seems flat out better. Nice.

return fee * amount * purchasePrice
p.Values.Set(labelStart, startDate.Format(time.RFC3339))
p.Values.Set(labelEnd, endDate.Format(time.RFC3339))
return err
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return nil 

Copy link
Collaborator

@gloriousCode gloriousCode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work, programmer! Here's some initial feedback on handling errors, authentication issues and test design

@@ -3,751 +3,1135 @@ package coinbasepro
import (
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like this package, filenames etc should be renamed to just coinbase given that coinbase has dropped it https://www.coinbase.com/blog/hello-advanced-trade-goodbye-coinbase-pro
the "pro" part of things just refers to the "pro" interface on coinbase versus "basic". But we can look at doing this towards the end.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a tad concerned about missing stuff in that conversion, so I'll leave it for later.

err := c.API.Endpoints.SetDefaultEndpoints(map[exchange.URL]string{
exchange.RestSpot: coinbaseproSandboxAPIURL,
})
log.Fatal("failed to set sandbox endpoint", err)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address

Suggested change
log.Fatal("failed to set sandbox endpoint", err)
if err != nil {
log.Fatal("failed to set sandbox endpoint", err)
}

t.Errorf("Coinbase, GetProducts() Error: %s", err)
}
func TestGetAllAccounts(t *testing.T) {
sharedtestvalues.SkipTestIfCredentialsUnset(t, c)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add t.Parallel() to all tests. Unless there is a reason you haven't - if so, please tell me so we can work out a way to have parallel tests 👍

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Parallelism wasn't added for tests with outbound requests due to concerns over rate/race issues with sending a boatload of requests to the exchange at once.

Admittedly, I developed that heuristic very early on when working with this codebase, and I haven't properly examined whether it's an actual issue or not. If you don't think it is, I can chuck that into most of these, save for a few websocket tests that may need a linear order.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rate limits will handle outbound request issues. I'm seeing changes in ratelimits.go in this PR so I assume you've handled the API requirements for them. That would address any concerns you have over t.Parallel. Let me know if you need help with rate limits otherwise

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that was unfounded. Added that to all non-WS tests.

accounts, err := c.GetAllAccounts(context.Background(), 250, "")
assert.NoError(t, err)
if accounts == nil || len(accounts.Accounts) == 0 {
t.Fatal(errExpectedNonEmpty)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems more appropriate to skip, than to fatal. It is not an error that I, a peasant, have no funds 😭

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By default, I believe Coinbase should list at least one account associated with an API key; if none come back I think that implies something was wrong with the data. Later, when I do an actual check for funds, I merely skip if there isn't enough.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll reverify after you've addressed the other comments. I have a verified account and API key and was hitting this line (len(accounts.Accounts) was 0). I don't believe I was receiving any errors from the API

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please reverify; if you're still getting that, I'll change it to a skip.

c = &CoinbasePro{}
testCrypto = currency.BTC
testFiat = currency.USD
testPair = currency.NewPairWithDelimiter(testCrypto.String(), testFiat.String(), "-")
)

// Please supply your APIKeys here for better testing
const (
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please move this section above the vars?

case WSRate:
return r.RateLimWS.Wait(ctx)
default:
return errors.Errorf(errUnknownEndpointLimit, f)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We typically format errors like the following:

Suggested change
return errors.Errorf(errUnknownEndpointLimit, f)
return fmt.Errorf("%w %v",errUnknownEndpointLimit, f)

Then, you should be changing errUnknownSide to the following:

var (
	errUnknownSide = errors.New("unknown side")
)

That way, in test TestProcessSnapshotUpdate you can do the following check:
assert.ErrorIs(t, err, errUnknownSide)

Or, as a complete example:

func TestProcessSnapshotUpdate(t *testing.T) {
	t.Parallel()
	req := WebsocketOrderbookDataHolder{Changes: []WebsocketOrderbookData{{Side: "fakeside", PriceLevel: 1.1,
		NewQuantity: 2.2}}, ProductID: currency.NewBTCUSD()}
	err := c.ProcessSnapshot(&req, time.Time{})
	assert.ErrorIs(t, err, errUnknownSide)

	req.Changes[0].Side = "offer"
	err = c.ProcessSnapshot(&req, time.Now())
	assert.NoError(t, err)

	err = c.ProcessUpdate(&req, time.Now())
	assert.NoError(t, err)
}

fmt.Errorf allows for rich details to be added to errors, which is helpful for the user. In addition, golang allows you to wrap multiple errors together. Wrapping multiple errors allow for good debugging techniques where we can handle scenarios like

if err != nil {
    if  errors.Is(err, errTheThingBlewUp) {
         // do soemthing specific to address the error
    } else {
        return nil, err
    }
}

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So what I'd like you to do is update all the const err you have declared in the coinbasepro package, and change them to errors.New("...") type errors. Then you can change the error handling in your tests to better parse what's going on, instead of relying on string comparisons via err.Error(). Make them like my example above

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've changed most of these, but left errIntervalNotSupported (never used sprintf formatting; was used to maintain every return value being a string) and warnSequenceIssue (kept as a string to distinguish this recoverable issue of data loss from a more pressing error, since only string-based functions are used on it, and since strings passed to websocketDataHandler are logged differently from errors).

If you want me to change those too, let me know.

I also took the liberty of changing some code to remove further error.Error calls that were ultimately rooted in the Coinbase code, but didn't previously have associated variables of any kind.

skipTestIfLowOnFunds(t)
resp, err := c.PreviewOrder(context.Background(), testPair.String(), "BUY", "MARKET", "", "", 0, 1, 0, 0, 0, 0, false,
false, false, time.Time{})
assert.NoError(t, err)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am receiving the following:
CoinbasePro unsuccessful HTTP status code: 400 raw response: {"error":"unknown","error_details":"proto: (line 1:78): invalid value for enum type: \"\"","message":"proto: (line 1:78): invalid value for enum type: \"\""}, authenticated request failed
This seems to suggest an invalid request. I'm receiving this with multiple authorised requests. Please let me know if you are passing these tests and I'll look into what's going on for me a bit further before commenting

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, that slipped under my testing. The marginType parameter was added later; when I went back to add it I neglected to remove skipTestIfLowOnFunds. Since I didn't have enough funds to properly run it, that had me never check if margin_type could be sent as an empty string, and my assumption that it could was wrong.

I've fixed that oversight. Now, I fail with the error "account is not available", which I suspect means that I have insufficient funds. Further info would be appreciated!

Comment on lines 597 to 600
_, err := c.GetAllPaymentMethods(context.Background())
// This endpoint appears to currently be broken for all users
if err.Error() != errInternalMessageNil {
t.Error(err)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are making some assumptions that an err will be returned. You should be checking if the err != nil. Since I did not receive an error, this test panics:

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x18 pc=0x104889870]

goroutine 259 [running]:
testing.tRunner.func1.2({0x104a44900, 0x104e518d0})
	/opt/homebrew/Cellar/go/1.22.1/libexec/src/testing/testing.go:1631 +0x1c4
testing.tRunner.func1()
	/opt/homebrew/Cellar/go/1.22.1/libexec/src/testing/testing.go:1634 +0x33c
panic({0x104a44900?, 0x104e518d0?})
	/opt/homebrew/Cellar/go/1.22.1/libexec/src/runtime/panic.go:770 +0x124
github.com/thrasher-corp/gocryptotrader/exchanges/coinbasepro.TestGetAllPaymentMethods(0x140003a6820)
	/Users/WOW/go/src/github.com/thrasher-corp/gocryptotrader/exchanges/coinbasepro/coinbasepro_test.go:601 +0x60
testing.tRunner(0x140003a6820, 0x104af7e68)
	/opt/homebrew/Cellar/go/1.22.1/libexec/src/testing/testing.go:1689 +0xec
created by testing.(*T).Run in goroutine 1
	/opt/homebrew/Cellar/go/1.22.1/libexec/src/testing/testing.go:1742 +0x318

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So please update your tests to check for nil errors before calling err.Error(). However, along with my other commentary about changing the error strings to errors.New it will be much easier to handle these issues and you won't need to call err.Error(). Other exchanges like binance_test.go will be helpful in that regard such as TestWrapperGetServerTime

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that was just me implementing a dumb way of being notified when that endpoint stopped returning an error for everyone. Fixed, and I'll try to avoid that sort of thing in the future.

})
// sendRequest is a helper function which sends a websocket message to the Coinbase server
func (c *CoinbasePro) sendRequest(msgType, channel string, productIDs currency.Pairs) error {
creds, err := c.GetCredentials(context.Background())
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this is causing issues. I do not have credentials set in my config and authenticatedWebsocketApiSupport is set to false. So I am receiving the following error when attempting to subscribe:
[ERROR] | WEBSOCKET | 26/03/2024 15:20:05 | CoinbasePro websocket: subscription failure, CoinbasePro REST or Websocket authentication support is not enabled

I do not believe authentication is required to subscribe to a ticker

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is. All websocket endpoints require authentication. That's not explicitly stated, but these three links imply it https://docs.cloud.coinbase.com/advanced-trade-api/docs/auth https://docs.cloud.coinbase.com/advanced-trade-api/docs/ws-auth https://docs.cloud.coinbase.com/advanced-trade-api/docs/ws-channels and removing the signature/key from the request causes the exchange to complain, even for channels like "ticker".

vals.Set("end", strconv.FormatInt(endDate.Unix(), 10))
}
var resp Ticker
return &resp, c.SendAuthenticatedHTTPRequest(ctx, exchange.RestSpot, http.MethodGet,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not an authenticated endpoint: https://docs.cloud.coinbase.com/advanced-trade-api/reference/retailbrokerageapi_getmarkettrades
[ERROR] | SYNC | 26/03/2024 15:23:23 | Failed to get REST ticker. Error: CoinbasePro REST or Websocket authentication support is not enabled

Suggested change
return &resp, c.SendAuthenticatedHTTPRequest(ctx, exchange.RestSpot, http.MethodGet,
return &resp, c.SendHTTPRequest(ctx, exchange.RestSpot, path, &resp)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please verify other API endpoints and verify that they are not attempting to send requests with authentication when it is not needed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No V3 endpoint is public except for Get Server Time, as per https://docs.cloud.coinbase.com/advanced-trade-api/docs/rest-api-overview

Although that was accidentally left authenticated too; I'll push a fix for that along with the other stuff you brought up soon.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to add onto this. Could you please implement all unauthenticated endpoints from https://docs.cloud.coinbase.com/exchange/reference
This will allow for basic data retrieval without credentials. Endpoints such as
Instruments
Ticker
Orderbook
Trades
Candles

These will be important to reference in the wrapper functions:
FetchTradablePairs
UpdateTickers
UpdateTicker
UpdateOrderbook
GetRecentTrades
GetHistoricTrades
GetHistoricCandles
GetHistoricCandlesExtended
UpdateOrderExecutionLimits

@gloriousCode gloriousCode added reconstructing Based on PR feedback, this is currently being reworked and is not to be merged and removed review me This pull request is ready for review labels Apr 7, 2024
Copy link
Collaborator

@gloriousCode gloriousCode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your updates @cranktakular! I have some extra requests to utilise public endpoints. The reason for this is because the Advanced Trading API does not yet have a public API, but is promised to be "soon"™️.

Utilising the existing public trading endpoints for the base API will allow for more utilisation of data without requiring credentials. However, when API credentials are present, it is still better to utilise those endpoints as this is a replacement of "coinbase pro" and not "coinbase"

Comment on lines 88 to 91
err = c.Setup(cfg)
if err.Error() != errx7f {
t.Errorf(errExpectMismatch, err, errx7f)
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update all instances of error checking to utilise err != nil and errors.Is.

I would also expect a test to be able to perform the important function of Setup to also have a scenario where no error occurs

exchanges/coinbasepro/coinbasepro_test.go Show resolved Hide resolved
exchanges/coinbasepro/coinbasepro_test.go Show resolved Hide resolved
exchanges/coinbasepro/coinbasepro_test.go Outdated Show resolved Hide resolved
exchanges/coinbasepro/coinbasepro_test.go Outdated Show resolved Hide resolved
exchanges/coinbasepro/coinbasepro_test.go Outdated Show resolved Hide resolved
exchanges/coinbasepro/coinbasepro_test.go Show resolved Hide resolved
Comment on lines 607 to 611
// blockedExchanges are exchanges that are not able to be tested in general
var blockedExchanges = []string{
"coinbasepro", // coinbasepro API requires authentication for almost every endpoint
}

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe you've thought about this already, but worth mentioning in case it hasn't been covered already:
As a design pattern I think it's preferable to find any way to consolidate all information about an exchange onto it's own packages.

Off the top of my head I'm thinking about adding an exported constant using build tags for tests, and then using reflect to check for it.
The things I'm aiming for with that are:

  • No cost or boilerplate to exchanges which don't care
  • No production runtime cost
  • More information about the exchange is consolidated into it's dir, making it easier to find and delete in one go

I'm not suggesting you need to do this, especially in light of blockedCIExchanges already existing, but wanted to share it in case you want to engage, and for any other work in flight where it's appropriate.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After incorporating public endpoints from an earlier version of the API, I no longer need that blockedExchanges variable to exist so I've removed it.

Moving towards that sort of design for other similar stuff within those standards seems like a good idea, but I think it's a bit out of scope for my current work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
rebase/merge of master required reconstructing Based on PR feedback, this is currently being reworked and is not to be merged
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants