exchanges: OkCoin Update #1436
Open
exchanges: OkCoin Update #1436
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
3 tasks
shazbert
requested changes
Dec 28, 2023
exchanges/okcoin/okcoin_types.go
Outdated
| Checksum int64 `json:"checksum"` | ||
| Asks [][4]types.Number `json:"asks"` // [ Price, Quantity, depreciated, number of orders at the price ] | ||
| Bids [][4]types.Number `json:"bids"` // [ Price, Quantity, depreciated, number of orders at the price ] | ||
| Timestamp convert.ExchangeTime `json:"ts"` | ||
| } | ||
|
|
||
| func (a *WebsocketOrderBook) prepareOrderbook() { |
There was a problem hiding this comment.
I don't know if this function is needed. Can you review this?
There was a problem hiding this comment.
Yes. This was to match the order book checksum calculation criteria of OkCoin.
After subscribing to the incremental load push (such as books 400 levels) of Order Book Channel, users first receive the initial full load of market depth. After the incremental load is subsequently received, update the local full load.
If there is the same price, compare the size. If the size is 0, delete this depth data. If the size changes, replace the original data.
If there is no same price, sort by price (bid in descending order, ask in ascending order), and insert the depth information into the full load.
Calculate Checksum
There was a problem hiding this comment.
There was a problem hiding this comment.
So there are a few changes below and the reasonings:
- Removed
prepareOrderbookmethod completely as the 0 size manipulation is being handled ino.Websocket.Orderbook.Update - Removes mutex protection as these updates are currently processed linearly.
- In snapshot we were checking the data incoming for integrity, pushes check to post storage and application checking to ensure that was in stored is true and correct against checksum.
- In updates checking checksum is not pushed on every update, so only fetch in store and do checksum calculation when needed.
- Removed
CalculateChecksumpre-check checksum functionality as it's a double and not needed.
diff --git a/exchanges/okcoin/okcoin_types.go b/exchanges/okcoin/okcoin_types.go
index 73711893c..28035c473 100644
--- a/exchanges/okcoin/okcoin_types.go
+++ b/exchanges/okcoin/okcoin_types.go
@@ -166,37 +166,6 @@ type WebsocketOrderBook struct {
Timestamp convert.ExchangeTime `json:"ts"`
}
-func (a *WebsocketOrderBook) prepareOrderbook() {
- asks := [][4]types.Number{}
- for x := range a.Asks {
- if len(asks) > 0 && asks[len(asks)-1][0].Float64() == a.Asks[x][0].Float64() {
- if a.Asks[x][1].Float64() != 0 {
- if asks[len(asks)-1][1].Float64() > a.Asks[x][1].Float64() {
- asks[len(asks)-1], a.Asks[x] = a.Asks[x], asks[len(asks)-1]
- }
- } else if a.Asks[x][1] == 0 {
- continue
- }
- }
- asks = append(asks, a.Asks[x])
- }
- bids := [][4]types.Number{}
- for x := range a.Bids {
- if len(bids) > 0 && bids[len(bids)-1][0].Float64() == a.Bids[x][0].Float64() {
- if a.Bids[x][1].Float64() != 0 {
- if bids[len(bids)-1][1].Float64() < a.Bids[x][1].Float64() {
- bids[len(bids)-1], a.Bids[x] = a.Bids[x], bids[len(bids)-1]
- }
- } else if a.Bids[x][1] == 0 {
- continue
- }
- }
- bids = append(bids, a.Bids[x])
- }
- a.Asks = asks
- a.Bids = bids
-}
-
// WebsocketDataResponse formats all response data for a websocket event
type WebsocketDataResponse struct {
ID string `json:"id"`
diff --git a/exchanges/okcoin/okcoin_websocket.go b/exchanges/okcoin/okcoin_websocket.go
index aadd2f2f7..9d235d2a4 100644
--- a/exchanges/okcoin/okcoin_websocket.go
+++ b/exchanges/okcoin/okcoin_websocket.go
@@ -10,7 +10,6 @@ import (
"net/http"
"strconv"
"strings"
- "sync"
"time"
"github.com/gorilla/websocket"
@@ -474,11 +473,7 @@ func (o *Okcoin) wsProcessAccount(respRaw []byte) error {
return nil
}
-var (
- // the following variables are used to synchronize and track changes to the orderbook buffer and corresponding instrument ID
- orderbookSnapshotLock sync.Mutex
- orderbookSnapshotMap = map[string]struct{}{}
-)
+var orderbookSnapshotMap = map[string]bool{}
func (o *Okcoin) wsProcessOrderbook(respRaw []byte, obChannel string) error {
var resp WebsocketOrderbookResponse
@@ -491,25 +486,22 @@ func (o *Okcoin) wsProcessOrderbook(respRaw []byte, obChannel string) error {
return err
}
var snapshot bool
- orderbookSnapshotLock.Lock()
- defer orderbookSnapshotLock.Unlock()
if resp.Action == "" {
- _, okay := orderbookSnapshotMap[resp.Arg.InstrumentID]
+ okay := orderbookSnapshotMap[resp.Arg.InstrumentID]
if !okay {
- orderbookSnapshotMap[resp.Arg.InstrumentID] = struct{}{}
+ orderbookSnapshotMap[resp.Arg.InstrumentID] = true
snapshot = true
}
} else {
switch resp.Action {
case "snapshot":
snapshot = true
- orderbookSnapshotMap[resp.Arg.InstrumentID] = struct{}{}
+ orderbookSnapshotMap[resp.Arg.InstrumentID] = true
case "update":
snapshot = false
}
}
if snapshot {
- resp.Data[0].prepareOrderbook()
if len(resp.Data[0].Asks)+len(resp.Data[0].Bids) == 0 {
return nil
}
@@ -529,18 +521,7 @@ func (o *Okcoin) wsProcessOrderbook(respRaw []byte, obChannel string) error {
base.Bids[b].Amount = resp.Data[0].Bids[b][1].Float64()
base.Bids[b].Price = resp.Data[0].Bids[b][0].Float64()
}
- var signedChecksum int32
- signedChecksum, err = o.CalculateChecksum(&resp.Data[0])
- if err != nil {
- return fmt.Errorf("%s channel: Orderbook unable to calculate orderbook checksum: %s", o.Name, err)
- }
- if resp.Data[0].Checksum != 0 && int64(signedChecksum) != resp.Data[0].Checksum {
- return fmt.Errorf("%s channel: Orderbook for %v checksum invalid", o.Name, cp)
- }
- err = base.Process()
- if err != nil {
- return err
- }
+
err = o.Websocket.Orderbook.LoadSnapshot(&base)
if err != nil {
if errors.Is(err, orderbook.ErrOrderbookInvalid) {
@@ -551,8 +532,21 @@ func (o *Okcoin) wsProcessOrderbook(respRaw []byte, obChannel string) error {
}
return err
}
+
+ if resp.Data[0].Checksum != 0 {
+ var check *orderbook.Base
+ check, err = o.Websocket.Orderbook.GetOrderbook(cp, asset.Spot)
+ if err != nil {
+ return err
+ }
+ if checksum := int64(o.CalculateOrderbookUpdateChecksum(check)); checksum != resp.Data[0].Checksum {
+ return fmt.Errorf("%s channel: Orderbook for %v checksum invalid got: %v expecting: %v", o.Name, cp, checksum, resp.Data[0].Checksum)
+ }
+ }
+
return nil
}
+
if len(resp.Data[0].Asks)+len(resp.Data[0].Bids) == 0 {
return nil
}
@@ -580,13 +574,15 @@ func (o *Okcoin) wsProcessOrderbook(respRaw []byte, obChannel string) error {
}
return err
}
- updatedOb, err := o.Websocket.Orderbook.GetOrderbook(cp, asset.Spot)
- if err != nil {
- return err
- }
- checksum := o.CalculateOrderbookUpdateChecksum(updatedOb)
- if resp.Data[0].Checksum != 0 && int64(checksum) != resp.Data[0].Checksum {
- return fmt.Errorf("checksum failed, calculated '%v' received '%v'", checksum, resp.Data)
+
+ if resp.Data[0].Checksum != 0 {
+ check, err := o.Websocket.Orderbook.GetOrderbook(cp, asset.Spot)
+ if err != nil {
+ return err
+ }
+ if checksum := o.CalculateOrderbookUpdateChecksum(check); int64(checksum) != resp.Data[0].Checksum {
+ return fmt.Errorf("checksum failed, calculated '%v' received '%v'", checksum, resp.Data)
+ }
}
return nil
}
@@ -720,32 +716,6 @@ func (o *Okcoin) AppendWsOrderbookItems(entries [][4]types.Number) ([]orderbook.
return items, nil
}
-// CalculateChecksum alternates over the first 25 bid and ask
-// entries from websocket data. The checksum is made up of the price and the
-// quantity with a semicolon (:) deliminating them. This will also work when
-// there are less than 25 entries (for whatever reason)
-// eg Bid:Ask:Bid:Ask:Ask:Ask
-func (o *Okcoin) CalculateChecksum(orderbookData *WebsocketOrderBook) (int32, error) {
- orderbookData.prepareOrderbook()
- var checksum strings.Builder
- for i := 0; i < allowableIterations; i++ {
- if len(orderbookData.Bids)-1 >= i {
- checksum.WriteString(orderbookData.Bids[i][0].String() +
- currency.ColonDelimiter +
- orderbookData.Bids[i][1].String() +
- currency.ColonDelimiter)
- }
- if len(orderbookData.Asks)-1 >= i {
- checksum.WriteString(orderbookData.Asks[i][0].String() +
- currency.ColonDelimiter +
- orderbookData.Asks[i][1].String() +
- currency.ColonDelimiter)
- }
- }
- checksumStr := strings.TrimSuffix(checksum.String(), currency.ColonDelimiter)
- return int32(crc32.ChecksumIEEE([]byte(checksumStr))), nil
-}
-
// CalculateOrderbookUpdateChecksum calculated the orderbook update checksum using currency pair full snapshot.
func (o *Okcoin) CalculateOrderbookUpdateChecksum(orderbookData *orderbook.Base) int32 {
var checksum strings.BuilderPlease apply and test updates.
shazbert
requested changes
Jan 2, 2024
exchanges/okcoin/okcoin_types.go
Outdated
| Checksum int64 `json:"checksum"` | ||
| Asks [][4]types.Number `json:"asks"` // [ Price, Quantity, depreciated, number of orders at the price ] | ||
| Bids [][4]types.Number `json:"bids"` // [ Price, Quantity, depreciated, number of orders at the price ] | ||
| Timestamp convert.ExchangeTime `json:"ts"` | ||
| } | ||
|
|
||
| func (a *WebsocketOrderBook) prepareOrderbook() { |
There was a problem hiding this comment.
So there are a few changes below and the reasonings:
- Removed
prepareOrderbookmethod completely as the 0 size manipulation is being handled ino.Websocket.Orderbook.Update - Removes mutex protection as these updates are currently processed linearly.
- In snapshot we were checking the data incoming for integrity, pushes check to post storage and application checking to ensure that was in stored is true and correct against checksum.
- In updates checking checksum is not pushed on every update, so only fetch in store and do checksum calculation when needed.
- Removed
CalculateChecksumpre-check checksum functionality as it's a double and not needed.
diff --git a/exchanges/okcoin/okcoin_types.go b/exchanges/okcoin/okcoin_types.go
index 73711893c..28035c473 100644
--- a/exchanges/okcoin/okcoin_types.go
+++ b/exchanges/okcoin/okcoin_types.go
@@ -166,37 +166,6 @@ type WebsocketOrderBook struct {
Timestamp convert.ExchangeTime `json:"ts"`
}
-func (a *WebsocketOrderBook) prepareOrderbook() {
- asks := [][4]types.Number{}
- for x := range a.Asks {
- if len(asks) > 0 && asks[len(asks)-1][0].Float64() == a.Asks[x][0].Float64() {
- if a.Asks[x][1].Float64() != 0 {
- if asks[len(asks)-1][1].Float64() > a.Asks[x][1].Float64() {
- asks[len(asks)-1], a.Asks[x] = a.Asks[x], asks[len(asks)-1]
- }
- } else if a.Asks[x][1] == 0 {
- continue
- }
- }
- asks = append(asks, a.Asks[x])
- }
- bids := [][4]types.Number{}
- for x := range a.Bids {
- if len(bids) > 0 && bids[len(bids)-1][0].Float64() == a.Bids[x][0].Float64() {
- if a.Bids[x][1].Float64() != 0 {
- if bids[len(bids)-1][1].Float64() < a.Bids[x][1].Float64() {
- bids[len(bids)-1], a.Bids[x] = a.Bids[x], bids[len(bids)-1]
- }
- } else if a.Bids[x][1] == 0 {
- continue
- }
- }
- bids = append(bids, a.Bids[x])
- }
- a.Asks = asks
- a.Bids = bids
-}
-
// WebsocketDataResponse formats all response data for a websocket event
type WebsocketDataResponse struct {
ID string `json:"id"`
diff --git a/exchanges/okcoin/okcoin_websocket.go b/exchanges/okcoin/okcoin_websocket.go
index aadd2f2f7..9d235d2a4 100644
--- a/exchanges/okcoin/okcoin_websocket.go
+++ b/exchanges/okcoin/okcoin_websocket.go
@@ -10,7 +10,6 @@ import (
"net/http"
"strconv"
"strings"
- "sync"
"time"
"github.com/gorilla/websocket"
@@ -474,11 +473,7 @@ func (o *Okcoin) wsProcessAccount(respRaw []byte) error {
return nil
}
-var (
- // the following variables are used to synchronize and track changes to the orderbook buffer and corresponding instrument ID
- orderbookSnapshotLock sync.Mutex
- orderbookSnapshotMap = map[string]struct{}{}
-)
+var orderbookSnapshotMap = map[string]bool{}
func (o *Okcoin) wsProcessOrderbook(respRaw []byte, obChannel string) error {
var resp WebsocketOrderbookResponse
@@ -491,25 +486,22 @@ func (o *Okcoin) wsProcessOrderbook(respRaw []byte, obChannel string) error {
return err
}
var snapshot bool
- orderbookSnapshotLock.Lock()
- defer orderbookSnapshotLock.Unlock()
if resp.Action == "" {
- _, okay := orderbookSnapshotMap[resp.Arg.InstrumentID]
+ okay := orderbookSnapshotMap[resp.Arg.InstrumentID]
if !okay {
- orderbookSnapshotMap[resp.Arg.InstrumentID] = struct{}{}
+ orderbookSnapshotMap[resp.Arg.InstrumentID] = true
snapshot = true
}
} else {
switch resp.Action {
case "snapshot":
snapshot = true
- orderbookSnapshotMap[resp.Arg.InstrumentID] = struct{}{}
+ orderbookSnapshotMap[resp.Arg.InstrumentID] = true
case "update":
snapshot = false
}
}
if snapshot {
- resp.Data[0].prepareOrderbook()
if len(resp.Data[0].Asks)+len(resp.Data[0].Bids) == 0 {
return nil
}
@@ -529,18 +521,7 @@ func (o *Okcoin) wsProcessOrderbook(respRaw []byte, obChannel string) error {
base.Bids[b].Amount = resp.Data[0].Bids[b][1].Float64()
base.Bids[b].Price = resp.Data[0].Bids[b][0].Float64()
}
- var signedChecksum int32
- signedChecksum, err = o.CalculateChecksum(&resp.Data[0])
- if err != nil {
- return fmt.Errorf("%s channel: Orderbook unable to calculate orderbook checksum: %s", o.Name, err)
- }
- if resp.Data[0].Checksum != 0 && int64(signedChecksum) != resp.Data[0].Checksum {
- return fmt.Errorf("%s channel: Orderbook for %v checksum invalid", o.Name, cp)
- }
- err = base.Process()
- if err != nil {
- return err
- }
+
err = o.Websocket.Orderbook.LoadSnapshot(&base)
if err != nil {
if errors.Is(err, orderbook.ErrOrderbookInvalid) {
@@ -551,8 +532,21 @@ func (o *Okcoin) wsProcessOrderbook(respRaw []byte, obChannel string) error {
}
return err
}
+
+ if resp.Data[0].Checksum != 0 {
+ var check *orderbook.Base
+ check, err = o.Websocket.Orderbook.GetOrderbook(cp, asset.Spot)
+ if err != nil {
+ return err
+ }
+ if checksum := int64(o.CalculateOrderbookUpdateChecksum(check)); checksum != resp.Data[0].Checksum {
+ return fmt.Errorf("%s channel: Orderbook for %v checksum invalid got: %v expecting: %v", o.Name, cp, checksum, resp.Data[0].Checksum)
+ }
+ }
+
return nil
}
+
if len(resp.Data[0].Asks)+len(resp.Data[0].Bids) == 0 {
return nil
}
@@ -580,13 +574,15 @@ func (o *Okcoin) wsProcessOrderbook(respRaw []byte, obChannel string) error {
}
return err
}
- updatedOb, err := o.Websocket.Orderbook.GetOrderbook(cp, asset.Spot)
- if err != nil {
- return err
- }
- checksum := o.CalculateOrderbookUpdateChecksum(updatedOb)
- if resp.Data[0].Checksum != 0 && int64(checksum) != resp.Data[0].Checksum {
- return fmt.Errorf("checksum failed, calculated '%v' received '%v'", checksum, resp.Data)
+
+ if resp.Data[0].Checksum != 0 {
+ check, err := o.Websocket.Orderbook.GetOrderbook(cp, asset.Spot)
+ if err != nil {
+ return err
+ }
+ if checksum := o.CalculateOrderbookUpdateChecksum(check); int64(checksum) != resp.Data[0].Checksum {
+ return fmt.Errorf("checksum failed, calculated '%v' received '%v'", checksum, resp.Data)
+ }
}
return nil
}
@@ -720,32 +716,6 @@ func (o *Okcoin) AppendWsOrderbookItems(entries [][4]types.Number) ([]orderbook.
return items, nil
}
-// CalculateChecksum alternates over the first 25 bid and ask
-// entries from websocket data. The checksum is made up of the price and the
-// quantity with a semicolon (:) deliminating them. This will also work when
-// there are less than 25 entries (for whatever reason)
-// eg Bid:Ask:Bid:Ask:Ask:Ask
-func (o *Okcoin) CalculateChecksum(orderbookData *WebsocketOrderBook) (int32, error) {
- orderbookData.prepareOrderbook()
- var checksum strings.Builder
- for i := 0; i < allowableIterations; i++ {
- if len(orderbookData.Bids)-1 >= i {
- checksum.WriteString(orderbookData.Bids[i][0].String() +
- currency.ColonDelimiter +
- orderbookData.Bids[i][1].String() +
- currency.ColonDelimiter)
- }
- if len(orderbookData.Asks)-1 >= i {
- checksum.WriteString(orderbookData.Asks[i][0].String() +
- currency.ColonDelimiter +
- orderbookData.Asks[i][1].String() +
- currency.ColonDelimiter)
- }
- }
- checksumStr := strings.TrimSuffix(checksum.String(), currency.ColonDelimiter)
- return int32(crc32.ChecksumIEEE([]byte(checksumStr))), nil
-}
-
// CalculateOrderbookUpdateChecksum calculated the orderbook update checksum using currency pair full snapshot.
func (o *Okcoin) CalculateOrderbookUpdateChecksum(orderbookData *orderbook.Base) int32 {
var checksum strings.BuilderPlease apply and test updates.
shazbert
approved these changes
Jan 3, 2024
🙏🏽🫡 |
… into okcoin_update
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1436 +/- ##
========================================
Coverage 35.87% 35.87%
========================================
Files 411 410 -1
Lines 177444 177302 -142
========================================
- Hits 63651 63611 -40
+ Misses 106029 105942 -87
+ Partials 7764 7749 -15
|
thrasher-
requested changes
Jan 16, 2024
There was a problem hiding this comment.
Looks like there's a test issue: https://github.com/thrasher-corp/gocryptotrader/actions/runs/7534671195/job/20516817208#step:9:364
… into okcoin_update
… into okcoin_update
shazbert
added
rebase/merge of master required
and removed
szrc
… into okcoin_update
… into okcoin_update
samuael
force-pushed
the
okcoin_update
branch
from
785ae85
to
7305bcf
Compare
… into okcoin_update
… into okcoin_update
shazbert
requested changes
Feb 22, 2024
There was a problem hiding this comment.
Changes look good just one thing I noticed when running:
[ERROR] | EXCHANGE | 22/02/2024 15:55:53 | LoadExchange Okcoin failed: failed to update tradable pairs: Okcoin unsuccessful HTTP status code: 403 raw response: {"message":"You have been denied access to the station service,Please follow the network access specification！"}
|
I couldn't get this error message whenever I tried. The rate limit value I used is also correct! |
thrasher-
added
blocked
and removed
review me
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Selam Everyone
This PR includes comprehensive updates to types, web-socket handlers, wrapper functions, and some endpoint methods. I have conducted live testing on web socket push data handling including the order book. Cleaned up the codebase by removing the okcoin_convert.go file and replaced timestamp information with 'convert.ExchangeTime' for improved clarity and efficiency.
Fixes # (issue)
Type of change
Please delete options that are not relevant and add an
xin[]as the item is complete.How has this been tested
Checklist