curio: feat: break trees task into TreeD(prefetch) and TreeRC #11895

LexLuthr · 2024-04-18T12:25:41Z

Related Issues

Proposed Changes

TreesTask is now broken into 2 Tasks namely TreeD and TreeRC
The storage reservation with some free space is performed in TreeD
The storage in TypeDetails now take an additional variable (int) MinFreeStoragePercentage.
The sealing pipeline is functioning as normal for CC and Deal sectors

Additional Info

Checklist

Before you mark the PR ready for review, please make sure that:

Commits have a clear commit message.
PR title is in the form of of <PR type>: <area>: <change being made>
- example: fix: mempool: Introduce a cache for valid signatures
- PR type: fix, feat, build, chore, ci, docs, perf, refactor, revert, style, test
- area, e.g. api, chain, state, market, mempool, multisig, networking, paych, proving, sealing, wallet, deps
If the PR affects users (e.g., new feature, bug fix, system requirements change), update the CHANGELOG.md and add details to the UNRELEASED section.
New features have usage guidelines and / or documentation updates in
- Lotus Documentation
- Discussion Tutorials
Tests exist for new functionality or change in behavior
CI is green

LexLuthr · 2024-04-18T19:24:34Z

@magik6k I was tested the new Tasks and they are working fine. Can you please review the logic once?

magik6k

Some issues to address, but after that seems like this approach should work great!

curiosrc/seal/task_treed.go

storage/paths/local.go

curiosrc/seal/task_treed.go

curiosrc/ffi/sdr_funcs.go

curiosrc/seal/task_treed.go

curiosrc/ffi/sdr_funcs.go

curiosrc/seal/task_treerc.go

LexLuthr · 2024-04-21T15:43:16Z

Sealing pipeline test after last commit

magik6k

Looks almost landable now, just a few comments to address (really mostly the one about release calling)

I'm going to give this a try on my cluster.

curiosrc/seal/task_treed.go

LexLuthr · 2024-04-29T15:33:52Z

@snadrus This is ready for your review.

curiosrc/ffi/sdr_funcs.go

curiosrc/seal/task_treerc.go

curiosrc/seal/task_treed.go

magik6k

Looks great! Also appears to work fine on my cluster.

* Fixing dead links (#11907) * ci: ci: create gh workflow that updates sorted pr checks (#11861) * ci: create gh workflow that updates sorted pr checks * ci: use grouped_by_result template for pr checks sticky comment * chore: apply pr review suggestion * Avoid cfg lookup on chain remove since unenabled splitstore delete is noop anyway (#11916) Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com> * Fix mismatched method names in comments (#11913) Signed-off-by: forcedebug <forcedebug@outlook.com> * release: v1.26.3 (#11908) (#11915) (#11922) * deps: update dependencies to address migration memory bloat to address memory concerns during a heavy migration Ref: filecoin-project/go-state-types#260 Ref: whyrusleeping/cbor-gen#96 Ref: filecoin-project/go-amt-ipld#90 * release: prep v1.26.3 patch Prep v1.26.3 patch release: - Update changelog, version and make gen + make docsgen-cli * deps: update cbor-gen to tagged version deps: update cbor-gen to tagged version * deps: update go-state-types to tagged version deps: update go-state-types to tagged version v0.13.2 * chore: deps: update go-state-types to v0.13.3 Fixes a panic when we have fewer than 1k proposals. --------- Co-authored-by: Phi-rjan <orjan.roren@gmail.com> Co-authored-by: Rod Vagg <rod@vagg.org> Co-authored-by: Steven Allen <steven@stebalien.com> * Refactor `LookupID*` APIs in `StateManager` and `StateTree` The naming of `LookupID` can cause confusion when resolving actor IDs vs ID addresses. To avoid this: * Refactor `StateTree` `LookupID` to `LookupIDAddress`, because it returns ID address. * Refactor `StateManager` `LookupID` to `LookupIDAddress` because it also returns ID address via a chain call to `StateTree`. * Introduce a new API `StateManager` dedicated to resolving address to actor ID, called `LookupID` which returns `abi.ActorID`. For context, see: * #11723 (comment) * Add v13 support to invariants-checker (#11931) Add v13 support to invariants-checker * chore: docs: nv-skeleton documentation (#11065) * nv-skeleton documentation Add a tutorial for how one can create a nv-skeleton in Lotus * Add footnote for `Add migration` step Add footnote for `Add migration` step * Indent migration-code Indent migration-code to make it show properly as a footnote. * Add ref-fvm and filecoin-ffi checklist Add ref-fvm and filecoin-ffi checklist * Add Filecoin-FFI steps Add Filecoin-FFI steps * Add step to params_butterfly.go Add step to params_butterfly.go * Fix typo Fix typo * Add links to reference PRs Add links to reference PRs * Update ref-fvm list Update ref-fvm list * feat: curio: add StorageInit api (#11918) * feat: add StorageInit api * remove unused variables * fix gen check * feat: curio: simpler reservation release logic (#11900) * simpler release logic * oops, plus simpler * simpler * fix NewLine (#11893) * fix(events): check for sync-in-progress (#11932) * feat(events): adjust indexes in event index db to match query patterns Introduces a v4 migration that just adjusts indexes. Copies some improvements from #11723 Closes: #11909 * fix(pipeline): should return if error occurred when get network version (#11902) * fix(events): correct log msg for v4 events index db migration * chore: remove duplicate words in strings and comments * fix(events): register events index db migration v4 * fix: curio seal: Failed commit retry strategy (#11870) * ffi: improved-error-handling * curio seal: Failed commit retry strategy * use master ffi * mod tidy * fix: curio: Update pgx imports, fix db_storage alloc * feat: curioweb: Improve task_history indexes (#11911) * mod tidy * Event index should be unique for tipsets (#11952) * event index should be unique for tipsets * fix formatting * migrate to version 5 * chore: bump build version in master (#11946) * Bump version Bump version in master branch in preperation for cutting v1.27.0-rc1 * chore: bump build-version chore: bump build-version * feat: curioweb: Show piece info on the sector page (#11955) * curio: feat: break trees task into TreeD(prefetch) and TreeRC (#11895) * break trees task * fix TreeD reservation * fix nil pointer err * apply suggestions * fix allocate file types * fix dbIndex inserts * set resource, move release func * refactor func(), update memory * remove extra release --------- Signed-off-by: forcedebug <forcedebug@outlook.com> Co-authored-by: parthshah1 <parthhshah171@gmail.com> Co-authored-by: Piotr Galar <piotr.galar@gmail.com> Co-authored-by: ZenGround0 <5515260+ZenGround0@users.noreply.github.com> Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com> Co-authored-by: forcedebug <167591285+forcedebug@users.noreply.github.com> Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com> Co-authored-by: Phi-rjan <orjan.roren@gmail.com> Co-authored-by: Rod Vagg <rod@vagg.org> Co-authored-by: Steven Allen <steven@stebalien.com> Co-authored-by: Masih H. Derkani <m@derkani.org> Co-authored-by: Lee <octalgah@gmail.com> Co-authored-by: Andrew Jackson (Ajax) <snadrus@gmail.com> Co-authored-by: beck <34204218+beck-8@users.noreply.github.com> Co-authored-by: 0x5459 <0x5459@protonmail.com> Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com> Co-authored-by: Łukasz Magiera <magik6k@gmail.com> Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>

* fix: curio: Update pgx imports, fix db_storage alloc * feat: curioweb: Improve task_history indexes (#11911) * 1 * relatable * add and delete layer * chore: curio: merge master (#11956) * Fixing dead links (#11907) * ci: ci: create gh workflow that updates sorted pr checks (#11861) * ci: create gh workflow that updates sorted pr checks * ci: use grouped_by_result template for pr checks sticky comment * chore: apply pr review suggestion * Avoid cfg lookup on chain remove since unenabled splitstore delete is noop anyway (#11916) Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com> * Fix mismatched method names in comments (#11913) Signed-off-by: forcedebug <forcedebug@outlook.com> * release: v1.26.3 (#11908) (#11915) (#11922) * deps: update dependencies to address migration memory bloat to address memory concerns during a heavy migration Ref: filecoin-project/go-state-types#260 Ref: whyrusleeping/cbor-gen#96 Ref: filecoin-project/go-amt-ipld#90 * release: prep v1.26.3 patch Prep v1.26.3 patch release: - Update changelog, version and make gen + make docsgen-cli * deps: update cbor-gen to tagged version deps: update cbor-gen to tagged version * deps: update go-state-types to tagged version deps: update go-state-types to tagged version v0.13.2 * chore: deps: update go-state-types to v0.13.3 Fixes a panic when we have fewer than 1k proposals. --------- Co-authored-by: Phi-rjan <orjan.roren@gmail.com> Co-authored-by: Rod Vagg <rod@vagg.org> Co-authored-by: Steven Allen <steven@stebalien.com> * Refactor `LookupID*` APIs in `StateManager` and `StateTree` The naming of `LookupID` can cause confusion when resolving actor IDs vs ID addresses. To avoid this: * Refactor `StateTree` `LookupID` to `LookupIDAddress`, because it returns ID address. * Refactor `StateManager` `LookupID` to `LookupIDAddress` because it also returns ID address via a chain call to `StateTree`. * Introduce a new API `StateManager` dedicated to resolving address to actor ID, called `LookupID` which returns `abi.ActorID`. For context, see: * #11723 (comment) * Add v13 support to invariants-checker (#11931) Add v13 support to invariants-checker * chore: docs: nv-skeleton documentation (#11065) * nv-skeleton documentation Add a tutorial for how one can create a nv-skeleton in Lotus * Add footnote for `Add migration` step Add footnote for `Add migration` step * Indent migration-code Indent migration-code to make it show properly as a footnote. * Add ref-fvm and filecoin-ffi checklist Add ref-fvm and filecoin-ffi checklist * Add Filecoin-FFI steps Add Filecoin-FFI steps * Add step to params_butterfly.go Add step to params_butterfly.go * Fix typo Fix typo * Add links to reference PRs Add links to reference PRs * Update ref-fvm list Update ref-fvm list * feat: curio: add StorageInit api (#11918) * feat: add StorageInit api * remove unused variables * fix gen check * feat: curio: simpler reservation release logic (#11900) * simpler release logic * oops, plus simpler * simpler * fix NewLine (#11893) * fix(events): check for sync-in-progress (#11932) * feat(events): adjust indexes in event index db to match query patterns Introduces a v4 migration that just adjusts indexes. Copies some improvements from #11723 Closes: #11909 * fix(pipeline): should return if error occurred when get network version (#11902) * fix(events): correct log msg for v4 events index db migration * chore: remove duplicate words in strings and comments * fix(events): register events index db migration v4 * fix: curio seal: Failed commit retry strategy (#11870) * ffi: improved-error-handling * curio seal: Failed commit retry strategy * use master ffi * mod tidy * fix: curio: Update pgx imports, fix db_storage alloc * feat: curioweb: Improve task_history indexes (#11911) * mod tidy * Event index should be unique for tipsets (#11952) * event index should be unique for tipsets * fix formatting * migrate to version 5 * chore: bump build version in master (#11946) * Bump version Bump version in master branch in preperation for cutting v1.27.0-rc1 * chore: bump build-version chore: bump build-version * feat: curioweb: Show piece info on the sector page (#11955) * curio: feat: break trees task into TreeD(prefetch) and TreeRC (#11895) * break trees task * fix TreeD reservation * fix nil pointer err * apply suggestions * fix allocate file types * fix dbIndex inserts * set resource, move release func * refactor func(), update memory * remove extra release --------- Signed-off-by: forcedebug <forcedebug@outlook.com> Co-authored-by: parthshah1 <parthhshah171@gmail.com> Co-authored-by: Piotr Galar <piotr.galar@gmail.com> Co-authored-by: ZenGround0 <5515260+ZenGround0@users.noreply.github.com> Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com> Co-authored-by: forcedebug <167591285+forcedebug@users.noreply.github.com> Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com> Co-authored-by: Phi-rjan <orjan.roren@gmail.com> Co-authored-by: Rod Vagg <rod@vagg.org> Co-authored-by: Steven Allen <steven@stebalien.com> Co-authored-by: Masih H. Derkani <m@derkani.org> Co-authored-by: Lee <octalgah@gmail.com> Co-authored-by: Andrew Jackson (Ajax) <snadrus@gmail.com> Co-authored-by: beck <34204218+beck-8@users.noreply.github.com> Co-authored-by: 0x5459 <0x5459@protonmail.com> Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com> Co-authored-by: Łukasz Magiera <magik6k@gmail.com> Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com> * linter oops * gen cleanup * fix * named returns are confusing --------- Signed-off-by: forcedebug <forcedebug@outlook.com> Co-authored-by: Łukasz Magiera <magik6k@gmail.com> Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com> Co-authored-by: LexLuthr <88259624+LexLuthr@users.noreply.github.com> Co-authored-by: parthshah1 <parthhshah171@gmail.com> Co-authored-by: Piotr Galar <piotr.galar@gmail.com> Co-authored-by: ZenGround0 <5515260+ZenGround0@users.noreply.github.com> Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com> Co-authored-by: forcedebug <167591285+forcedebug@users.noreply.github.com> Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com> Co-authored-by: Phi-rjan <orjan.roren@gmail.com> Co-authored-by: Rod Vagg <rod@vagg.org> Co-authored-by: Steven Allen <steven@stebalien.com> Co-authored-by: Masih H. Derkani <m@derkani.org> Co-authored-by: Lee <octalgah@gmail.com> Co-authored-by: beck <34204218+beck-8@users.noreply.github.com> Co-authored-by: 0x5459 <0x5459@protonmail.com> Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>

break trees task

7b96924

LexLuthr requested a review from magik6k April 18, 2024 12:25

fix TreeD reservation

c3925ad

LexLuthr mentioned this pull request Apr 18, 2024

[Epic] Curio Beta #11754

Open

19 tasks

fix nil pointer err

8382a20

magik6k reviewed Apr 19, 2024

View reviewed changes

apply suggestions

feabcdb

LexLuthr marked this pull request as draft April 21, 2024 15:30

fix allocate file types

6ff467b

LexLuthr marked this pull request as ready for review April 21, 2024 15:43

fix dbIndex inserts

f7fc8ac

LexLuthr requested a review from snadrus April 22, 2024 16:35

magik6k reviewed Apr 29, 2024

View reviewed changes

curiosrc/seal/task_treed.go Show resolved Hide resolved

curiosrc/seal/task_treed.go Outdated Show resolved Hide resolved

curiosrc/seal/task_treed.go Outdated Show resolved Hide resolved

curiosrc/seal/task_treed.go Outdated Show resolved Hide resolved

LexLuthr added 2 commits April 29, 2024 19:31

Merge branch 'master' into feat/break-treestask

8fea86e

set resource, move release func

1e1d305

LexLuthr force-pushed the feat/break-treestask branch from a10d1ef to 1e1d305 Compare April 29, 2024 14:35

snadrus requested changes Apr 29, 2024

View reviewed changes

curiosrc/ffi/sdr_funcs.go Outdated Show resolved Hide resolved

curiosrc/ffi/sdr_funcs.go Outdated Show resolved Hide resolved

curiosrc/seal/task_treerc.go Outdated Show resolved Hide resolved

refactor func(), update memory

ccc704c

LexLuthr requested a review from snadrus April 30, 2024 06:50

magik6k reviewed Apr 30, 2024

View reviewed changes

curiosrc/seal/task_treed.go Outdated Show resolved Hide resolved

remove extra release

d4d92cf

LexLuthr force-pushed the feat/break-treestask branch from e7da516 to d4d92cf Compare April 30, 2024 17:49

Merge branch 'master' into feat/break-treestask

1621623

magik6k approved these changes May 2, 2024

View reviewed changes

magik6k enabled auto-merge (squash) May 2, 2024 09:11

magik6k disabled auto-merge May 2, 2024 09:14

magik6k merged commit 00edad4 into master May 2, 2024
186 checks passed

magik6k deleted the feat/break-treestask branch May 2, 2024 09:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

curio: feat: break trees task into TreeD(prefetch) and TreeRC #11895

curio: feat: break trees task into TreeD(prefetch) and TreeRC #11895

LexLuthr commented Apr 18, 2024 •

edited by snadrus

LexLuthr commented Apr 18, 2024

magik6k left a comment

LexLuthr commented Apr 21, 2024

magik6k left a comment

LexLuthr commented Apr 29, 2024

magik6k left a comment

curio: feat: break trees task into TreeD(prefetch) and TreeRC #11895

curio: feat: break trees task into TreeD(prefetch) and TreeRC #11895

Conversation

LexLuthr commented Apr 18, 2024 • edited by snadrus

Related Issues

Proposed Changes

Additional Info

Checklist

LexLuthr commented Apr 18, 2024

magik6k left a comment

Choose a reason for hiding this comment

LexLuthr commented Apr 21, 2024

magik6k left a comment

Choose a reason for hiding this comment

LexLuthr commented Apr 29, 2024

magik6k left a comment

Choose a reason for hiding this comment

LexLuthr commented Apr 18, 2024 •

edited by snadrus