Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] BTC/LTC/ZEC jobmaker 重启后可能长时间未就绪 | 关于代码中btc Kafka消费偏移量的疑问 #450

Open
rocqina opened this issue Jun 9, 2020 · 3 comments
Open

[Bug] BTC/LTC/ZEC jobmaker 重启后可能长时间未就绪 | 关于代码中btc Kafka消费偏移量的疑问 #450

rocqina opened this issue Jun 9, 2020 · 3 comments
Labels
bug fixed need data Data is required to prove that the modification has no harmful effects need more feedback

Comments

@rocqina
Copy link

rocqina commented Jun 9, 2020 •
edited

请教一个问题

https://github.com/btccom/btcpool/blob/75e57300ed9486ea1889e7bece18efd523233b2f/src/bitcoin/JobMakerBitcoin.cc#L105

为什么设置偏移量是1，每次jobmaker重启的时候从位置1开始读取，导致了大量的无效gbt被消费。

如果设置成-1，每次读取最新的消息，为什么不这样做
@YihaoPeng
Copy link

应该是写错了，我会改成 RD_KAFKA_OFFSET_TAIL(1)

@YihaoPeng YihaoPeng reopened this Jun 9, 2020
@YihaoPeng
Copy link

已经尝试在 f312ceb 中修复

@YihaoPeng YihaoPeng added fixed need more feedback need data Data is required to prove that the modification has no harmful effects and removed TODO labels Jun 9, 2020
@YihaoPeng
Copy link

YihaoPeng commented Jun 9, 2020 •
edited

如果采用受影响的版本，jobmaker重启的时候应该需要很多时间。

受影响的版本：从 2019-07-17（ 34094cb ）至今。

偏移1应该会导致重头消费所有相关topic，导致 jobmaker 重启后需要大量时间才能开始发送新任务。
所以要注意，如果需要重启 jobmaker，应该注意它是否发送 job。如果发送不及时，应该升级到修复该问题的版本。

@YihaoPeng YihaoPeng pinned this issue Jun 9, 2020
@YihaoPeng YihaoPeng changed the title 关于代码中btc Kafka消费偏移量的疑问 [Bug] BTC/LTC/ZEC jobmaker 重启后可能长时间未就绪 | 关于代码中btc Kafka消费偏移量的疑问 Jun 9, 2020
oyangjian pushed a commit to oyangjian/btcpool that referenced this issue Jun 21, 2020
merge from btcpool master.
5d44e08 
Squashed commit of the following:

commit f312ceb
Author: SwimmingTiger <hu60.cn@gmail.com>
Date:   Tue Jun 9 09:52:51 2020 +0800

    fix btccom#450: Unreasonable kafka init offset in jobmaker

commit 7936ce3
Author: SwimmingTiger <hu60.cn@gmail.com>
Date:   Tue Jun 9 09:14:16 2020 +0800

    statshttpd: fix uninitialized variable modifyHoursFlag_

commit 75e5730
Author: EthanHee <249792645@qq.com>
Date:   Tue Jun 9 07:56:58 2020 +0800

    fix bugs: got a huge share_stale number (btccom#448)

commit 0cb505b
Author: EthanHee <249792645@qq.com>
Date:   Tue Jun 9 07:51:07 2020 +0800

    fix bch LightMsg bug (btccom#449)

    * fix bch LightMsg bug

    * fix bug

commit a9970f6
Author: SwimmingTiger <hu60.cn@gmail.com>
Date:   Tue Jun 9 07:21:04 2020 +0800

    add docker file for tools/worker_update

commit 32eef3d
Author: SwimmingTiger <hu60.cn@gmail.com>
Date:   Fri May 15 17:05:17 2020 +0800

    sserver: Remove unnecessary variable StratumMiner :: isGrandPoolClient_

commit faa84f5
Author: SwimmingTiger <hu60.cn@gmail.com>
Date:   Fri May 15 16:52:02 2020 +0800

    remove cmake option "USE_LOCAL_SHARE_BITCOIN"; format the code

commit 21167e2
Author: SwimmingTiger <hu60.cn@gmail.com>
Date:   Fri May 15 16:41:55 2020 +0800

    sserver feature "subres": Fix the issue that the calculation method of submitIndex is inconsistent with BTCAgent.

commit bf5234c
Author: SwimmingTiger <hu60.cn@gmail.com>
Date:   Fri May 15 15:55:45 2020 +0800

    add TEST(StratumSession, LocalShareBitcoin) back

commit 6386573
Author: jasper <jasper.li@bitmain.com>
Date:   Mon May 11 23:34:47 2020 -0700

    Using golang to implement orphan block monitoring script

commit 0e757b5
Author: SwimmingTiger <hu60.cn@gmail.com>
Date:   Sat Apr 25 17:29:43 2020 +0800

    sserver: log chain name when finding blocks

commit 18bf00e
Author: jasper.li <jasper.li@bitmain.com>
Date:   Mon Apr 13 16:53:21 2020 +0800

    renamed updatereward to ckb_orphan_monitor

commit 7379e29
Author: jasper <jasper.li@bitmain.com>
Date:   Wed Apr 8 19:23:12 2020 -0700

    add ltc orphan monitor

commit 2c73c98
Author: SwimmingTiger <hu60.cn@gmail.com>
Date:   Wed Apr 1 00:56:32 2020 +0800

    BTC autotest docker：fix slparser crash

commit 55eff22
Merge: 34e16d8 f891aa4
Author: YihaoPeng <yihao.peng@bitmain.com>
Date:   Wed May 6 16:04:59 2020 +0800

    Merge pull request btccom#439 from zhizhubeiwang/dev202005041753-grandpool-localshare

    add grand pool support

commit f891aa4
Author: xianxiongwang <171279316@qq.com>
Date:   Mon May 4 18:20:47 2020 +0800

    different coin different localshare

commit bc6fc5d
Author: xianxiongwang <171279316@qq.com>
Date:   Tue Apr 28 11:58:44 2020 +0800

    add grand pool support

commit 34e16d8
Author: YihaoPeng <yihao.peng@bitmain.com>
Date:   Thu Apr 23 18:44:52 2020 +0800

    Update INSTALL-BTCPool.md

commit c84484b
Author: YihaoPeng <yihao.peng@bitmain.com>
Date:   Thu Apr 23 18:42:00 2020 +0800

    INSTALL-BTCPool.md: update supported OS

commit 984e8e8
Author: SwimmingTiger <hu60.cn@gmail.com>
Date:   Fri Mar 27 17:10:04 2020 +0800

    format code...

commit 4c7861a
Author: SwimmingTiger <hu60.cn@gmail.com>
Date:   Fri Mar 27 17:07:05 2020 +0800

    remove unused param userCoinbaseInfo

commit fd76b3c
Author: SwimmingTiger <hu60.cn@gmail.com>
Date:   Fri Mar 27 16:39:20 2020 +0800

    remove SuperBitcoin, fix build issue for BCH/BSV

commit fa0705c
Author: SwimmingTiger <hu60.cn@gmail.com>
Date:   Mon Mar 23 06:48:52 2020 +0800

    sserver: Ignore unauthenticated users when switching chain

commit d301d5a
Author: SwimmingTiger <hu60.cn@gmail.com>
Date:   Mon Mar 23 04:02:06 2020 +0800

    jobmaker: change field names of subpool zk updating

commit 9926315
Author: SwimmingTiger <hu60.cn@gmail.com>
Date:   Mon Mar 23 03:37:42 2020 +0800

    jobmaker: Error message will be acked when subpool update fails

commit 4424121
Author: SwimmingTiger <hu60.cn@gmail.com>
Date:   Mon Mar 23 03:13:20 2020 +0800

    jobmaker: Coinbase information and payment address of the subpool can be updated through zookeeper

commit 15cf4f2
Author: SwimmingTiger <hu60.cn@gmail.com>
Date:   Mon Mar 23 06:20:24 2020 +0800

    fixed ckb reward

commit b10ef41
Author: SwimmingTiger <hu60.cn@gmail.com>
Date:   Wed Mar 18 04:48:29 2020 +0800

    sserver: update code & test case of "single_user_chain"

commit ab55948
Author: SwimmingTiger <hu60.cn@gmail.com>
Date:   Tue Mar 17 19:18:37 2020 +0800

    sserver: add a new option "single_user_chain", the entire sserver performs chain switching according to one user's selection.

commit 303a8f8
Author: jasper <jasper.li@bitmain.com>
Date:   Tue Mar 17 01:29:35 2020 -0700

    amend etc block reward

commit 0fdae56
Author: SwimmingTiger <hu60.cn@gmail.com>
Date:   Mon Mar 16 00:40:45 2020 +0800

    sserver: fix sserver crash if no IPs bind on a NIC; record hostname and IPs to the zk lock.

commit 42d9feb
Author: SwimmingTiger <hu60.cn@gmail.com>
Date:   Wed Mar 11 20:27:43 2020 +0800

    update cfg template: add sserver.proxy_protocol

commit e71a11b
Author: SwimmingTiger <hu60.cn@gmail.com>
Date:   Mon Mar 2 16:39:39 2020 +0800

    Increase the length of `mining_workers`.`worker_name` to 50 bytes and add an index on it.

    * Prevent truncation of worker_name in single-user mode.
    * Indexing can speed up worker_name prefix lookups.

commit abe3824
Author: SwimmingTiger <hu60.cn@gmail.com>
Date:   Mon Mar 2 17:03:24 2020 +0800

    Revert "extern workername's length to 30"

    This reverts commit a994fc6ef0daa40150bdfff7f7300ff92aeca22b.

commit f4af14b
Author: jasper <jasper.li@bitmain.com>
Date:   Sun Mar 1 04:43:12 2020 -0800

    extern workername's length to 30

commit 7e53b7f
Author: SwimmingTiger <hu60.cn@gmail.com>
Date:   Fri Feb 28 13:09:33 2020 +0800

    Add a new feature: Send submission response to BTCAgent.

commit 260fa63
Author: SwimmingTiger <hu60.cn@gmail.com>
Date:   Thu Feb 20 00:55:40 2020 +0800

    update cfg generator, add subpool section

commit bb0c49e
Author: SwimmingTiger <hu60.cn@gmail.com>
Date:   Thu Feb 20 00:51:38 2020 +0800

    bitcoin blkmaker: Write the block info of a sub-pool to `found_blocks_<subpool_name>` table.

commit 0b5e9f3
Author: SwimmingTiger <hu60.cn@gmail.com>
Date:   Wed Feb 19 23:57:46 2020 +0800

    bitcoin sserver: add ext_user_id to subpool shares to help data analysis.

commit 4e6a07f
Author: SwimmingTiger <hu60.cn@gmail.com>
Date:   Wed Feb 19 23:38:35 2020 +0800

    bitcoin jobmaker/sserver: Add a sub-pool mode.

    A sub-pool share jobs with the main pool, but with different coinbase information and addresses.

commit 822bf87
Author: SwimmingTiger <hu60.cn@gmail.com>
Date:   Wed Feb 19 14:58:00 2020 +0800

    update sserver.cfg: add section log and management

commit 78d70ba
Author: jasper <jasper.li@bitmain.com>
Date:   Sun Feb 9 21:15:15 2020 -0800

    bitcoin-jobmaker add log the reason of update job
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug fixed need data Data is required to prove that the modification has no harmful effects need more feedback
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@rocqina @YihaoPeng and others