Streaming (WS, Websocket support): Orderbooks, Trades, Balances...

[cklester]

I would like to see a new market property called balances, that would function the same way as symbols. When you call fetchBalances(), then the balances property would get filled with an associative array of currencies with balances (free, used, total, available, etc.).

Balances would also be updated via websocket (when available) or by repeated calls to the API, depending on rate limits.

Ultimately, we could do something like:

kraken.fetchBalances();
let myAvailableBTC = kraken.balance.available['BTC'];
let myTotalBTC = kraken.balance.total['BTC'];

[kroitor]

You will have it, soon ) Thanks for posting!

[kroitor]

We have unified the balance API, as written here, in issue #36 . Now your feature request can be implemented easily ) We will add a dirty balance method that would save the balance on a market instance, first. And when we are ready to release the Websocket implementations, we will make it auto-updating, of course. Keep in touch! )

[cklester]

So, the "dirty balance" is an adjustment of the local ledger to reflect any orders I place? What about orders that fill? Is that also being tracked for the balances?

[kroitor]

Yes, when we add a rate-limited poller and release methods for websocket apis, the balance will reflect everything in real time (as real as it can possibly be). Currently, balances are not reloaded and not cached automatically, you have to do it manually, as we've just finished unifying them (that was a lot of work in itself). But, having them unified, we will soon add auto-adjustment as well. As for order fills, I don't think I get your question entirely, but a balance api usually tells an amount (left) in orders, a free available balance and a total, and there's usually a separate interface for execution details (transactions, trades). It will also be stream-based, basically, all data, including the balance, open orders, partially filled orders and closed orders, all of that must be mutually consistent.

[Helmi]

As this issue is still open and I had a hard time finding anything else about it (except issue #145 which is linked here in closed): Is there anything new on the whole WebSocket topic?

[kroitor]

@Helmi hi! This issue is still open, we are working on it and hope to deliver it soon. We are closing in to v2, as you may see from our versioning, where we will be introducing a major refactoring and many new features ) Our WS implementation will not only stream balances, but also fills, orders, and all that stuff, where available. Stay tuned for more updates! )

[Helmi]

Thanks for the quick headsup. That sounds awesome. Go for it 👍

[alexmipego]

+1
Any word on the version 2.0 release date?

[kroitor]

@alexmipego we are currently working on it intensively to deliver the first version of it asap, hopefully, this month.

[nitzanav]

Thank you for the effort! I am a fan!

@kroitor you wrote:

We are closing in to v2, as you may see from our versioning, where we will be introducing a major refactoring and many new features )

Is there a way to take glimpse on the v2 and WebSocket code and major refactoring? I couldn't find it nowhere.

Where are the versioning is managed? I couldn't find it.

[kroitor]

@nitzanav

Is there a way to take glimpse on the v2 and WebSocket code and major refactoring? I couldn't find it nowhere.

The refactoring is happening right now in the main branch (there's been a lot of changes since then), and the work on websockets is also on its way, but there's no single place that holds all of it right now, the work is done in forks mostly... Here's is an example of it: #751

Where are the versioning is managed? I couldn't find it.

Version single-sourcing is performed upon each build on Travis CI from package.json with the vss.js script. More about it: https://github.com/ccxt/ccxt/blob/master/CONTRIBUTING.md.

[nitzanav]

Thank you for the information and for this amazing project!

[rubobaquero]

@kroitor Is it going to be available in the three languages as usual?

[kroitor]

@rubobaquero yes, that's the whole point, if we could throw a implementation in just one single language into it, adding support for ws wouldn't take so long ) The main difficulty is to unify them all.

[GoChartingAdmin]

Any timeline updates on websocket implementation.

[kroitor]

@GoChartingAdmin yes, we are in the process of merging this PR: #751, so, expect to have first implementations of a generic ws wrapper in the coming days.

[fredfortier]

@kroitor I haven't completed my EtherDelta implementation yet. Does this PR allow for websocket implementation with all basic operations? Is it complete enough for me to give a stab at it?

[kroitor]

@fredfortier unfortunately, not yet... It is a first approach, and it is

• not unified (exchange-specific)
• not portable (language-specific)

We are still working on the unification of WS (this is the hardest part of it). If you want to dive deeper into the history of websockets in ccxt, here's an overview of it (contains several other implementations in JS, that are closer to a unified state): https://gist.github.com/kroitor/7dce1d23a10937ab8c07a5451f17ccf2

[kornrunner]

For PHP we can create a wrapper around some library, eg: https://github.com/amphp/websocket
Thing is, it would be available only for PHP 7+ (we can do it as a suggestion / optional thing)

[TimNZ]

I don't want to seem ungrateful or entitled because this is an awesome project, but you have been promising a WS enabled release for a while.

Perhaps you should create a Github project so it can be tracked more easily so others can contribute, and do a raw write up of simple documentation of the proposed API so we can comment on it for feedback when it is 'cheaper' to change?

I know very well pain of over-architecting things and not getting feedback early.

From what I've read what you are doing sounds quite ambitious and maybe a v3 when we are waiting for a v1.

[kroitor]

I don't want to seem ungrateful or entitled because this is an awesome project, but you have been promising a WS enabled release for a while.

We are doing the core work on the base code to deliver it asap.

Perhaps you should create a Github project so it can be tracked more easily so others can contribute, and do a raw write up of simple documentation of the proposed API so we can comment on it for feedback when it is 'cheaper' to change?

That is exactly as we think, we are going to release a subset of exchanges first, so that the community could pick up from there, but we need to have it passing our current tests at the very least, before we can release it. Hope for your understanding.

I know very well pain of over-architecting things and not getting feedback early.

We would love to get the early feedback, that is the reason for doing the preparation work now in the first place.

[TimNZ]

Awesome!

Since you are talking about passing tests and I see some PR for WS, that must mean you have settled on an implementation, and the API for it and usage?

Perhaps it would be a good idea to quickly knock out a draft section in the Manual marked WIP so we can view and provide feedback on the API and usage (if needed)?

Better for us to tell you now if we think we see problems.

[lfern]

@firepol I think I have fixed all python errors you reported in websockets-multiple branch., Please check it out

[WoK-E]

@kroitor Do you have any info on licensing and fee schedule that you can share before Pro is released?

[kroitor]

@WoK-E we have posted on this matter here:

• Streaming (WS, Websocket support): Orderbooks, Trades, Balances... #56 (comment)
• Streaming (WS, Websocket support): Orderbooks, Trades, Balances... #56 (comment)

Will update you on the pricing grid before the release as soon as we have it (our goal is to roll out a good and flexible tech design for the users, first and foremost).

[eckozen84]

@kroitor will CCXT pro have proper handling of diff orderbook messages that arrive out of sequence?

EDIT: Ignore this question. I just saw your post above. Thank you.

[npomfret]

@frodoe7 you can always implement web sockets yourself for one or two exchanges while you wait to see if paying for CCXT pro is worth paying for. I've been doing just that while I wait for it to be released and it's really not so difficult. I've got public (orderbooks), and private (my trades and orders) ws-feeds working for 6 exchanges and I can't say that any were particularly difficult to do.

[n01ukn0w]

Any word on when CCXT Pro will be released? Jan 1, 2020 is approaching quickly!

[kroitor]

@n01ukn0w we are now closer than ever to getting it finally released, and we will announce beta-testing shortly.

[lookis]

@n01ukn0w we are now closer than ever to getting it finally released, and we will announce beta-testing shortly.

So exciting, I am pay much attention on this issue for month, looking forward to try the Pro at the very first time.

[radiant-alex]

Would be nice to have snippets of usages for various JS frameworks, specially for Angular/Typescript.
If it can handle things like Nativescript, would be awesome! Though from my recent tests ws npm library does not play well with NS. Wondering what is used for streaming under the hood?

[KlausGlueckert]

@kroitor : so excited as well and thanks a lot for all the work!!! a few questions up front:

• will there be a Python wrapper from the begining?
• will ccxt pro handle order book sequences automatically ? (no manual connect, reconnect logic etc)
• will ccxt pro automatically handle current orderbooks in memory? some exchanges only provide REST API orderbook snapshots so you have to build your own logic to take the REST snapshots and update them with the websocket orderbook updates :(
• if you need any input or feedback, do let me know, i have quite recently integrated shrimpy.io, and it just doesnt solve all the problems a developer has, ccxt pro could be so much better than everything available atm

[kroitor]

@KlausGlueckert

will there be a Python wrapper from the begining?

Yep.

will ccxt pro handle order book sequences automatically ? (no manual connect, reconnect logic etc)

Yes, and the other types of data feeds as well.

will ccxt pro automatically handle current orderbooks in memory? some exchanges only provide REST API orderbook snapshots so you have to build your own logic to take the REST snapshots and update them with the websocket orderbook updates :(

Yes, that is also handled.

if you need any input or feedback, do let me know, i have quite recently integrated shrimpy.io, and it just doesnt solve all the problems a developer has, ccxt pro could be so much better than everything available atm

Thx a lot for your support, will let you know if anything!

[juso]

In case release of "ccxt pro" will move into 2020, please say so!

We are determined to not let it move into 2020 and now we are pretty sure that it won't. From here it may take 2 to 4 weeks to roll out the initial version if nothing else pops up. But, anyways, it will be released before the end of this year.

But if "ccxt pro" is around the corner i don't feel like implementing "ccxws" just for a week or two and then reimplement "ccxt pro".

It is around the corner and we are working very hard to deliver it as soon as we can, seriously. We can't tell the exact date, since we may be able to deliver sooner than that. However, it will be released immediately when ready. If you still want the exact date, assume "before the end of 2019".

@kroitor - how does the release look like now? I assume it not gonna happen before 2020, so what is the next target? Better keep ppl up-to-date even with moving target than having no updates at all - in this case one feels nothing gonna happen at all since your priorities might have changed to something else

[kroitor]

@juso the work on it is very actively moving to the release (it's a matter of days now). We already have a working transpileable repo, and we are going to invite beta-testers in the beginning of January. The lack of updates does not mean that our priorities have changed, it's the opposite – it is still our topmost priority and we're so occupied with it that we can't post updates too frequently, other than "almost there". Another aspect to this is that we want to release a stable working product. Our current progress is 85-90%. We will post more news and will disclose the pricing grid in the coming days.

[WoK-E]

@kroitor where/how will you be posting any news on CCXT Pro, and/or invitations to beta testers? Will it be in this thread?

[kroitor]

@WoK-E we will open a standalone announcement thread, and will post links here as well to notify everybody in this issue.

[kroitor]

Happy New Year to everyone! Just published a small update/intermediary roadmap for Q1 here: #6332. We are doing everything we can to roll it out this month.

[HoDaDor]

We have unified the balance API, as written here, in issue #36 . Now your feature request can be implemented easily ) We will add a dirty balance method that would save the balance on a market instance, first. And when we are ready to release the Websocket implementations, we will make it auto-updating, of course. Keep in touch! )

Was scrolling through after seeing the update about ccxt pro plans and couldn't help but notice this/your reply to someone asking about polling the api in intervals? And I'l just like to point out that Binance has a websocket endpoint for trade and account data. Just need to subscribe to the "userData" stream: https://github.com/binance-exchange/binance-official-api-docs/blob/master/user-data-stream.md

Also, in case anyone is interested, I've been using this for my binance websocket needs, hope you guys find it as helpful as I have: https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/

[kroitor]

@HoDaDor this is exactly what CCXT Pro is all about, using the WebSockets for streaming: #6332 (comment) (today)

[kroitor]

Hello, dear CCXT users!

Today we bring you the good news.

We have released CCXT Pro with support for WebSockets!
Please, see the announcement here: #6543

Thank you to all who have been waiting so patiently for it.
As always, your feedback is appreciated a lot!

Finally, we are closing this issue )

Yours, CCXT Dev Team