Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: curio: allow multiple pieces per sector #11935

Open
wants to merge 23 commits into
base: master
Choose a base branch
from

Conversation

LexLuthr
Copy link
Contributor

@LexLuthr LexLuthr commented Apr 26, 2024

Related Issues

Proposed Changes

  1. Multiple piece per sector in market and precommit
  2. All open sectors are coordinated from DB
  3. Config changes for MaxDealWaitTime
  4. DDO deals in market
  5. DDO deals in precommit
  6. Updated backpressure
  7. Seal command
  • Test multiple deals in devnet
  • Test seal command
  • Test Curio restart with Open Deal sectors
  • Test Boost restarts while AP()
  • Test Curio restarts while AP()
  • Test DDO deal
  • Happy path test without deals

Additional Info

Checklist

Before you mark the PR ready for review, please make sure that:

  • Commits have a clear commit message.
  • PR title is in the form of of <PR type>: <area>: <change being made>
    • example: fix: mempool: Introduce a cache for valid signatures
    • PR type: fix, feat, build, chore, ci, docs, perf, refactor, revert, style, test
    • area, e.g. api, chain, state, market, mempool, multisig, networking, paych, proving, sealing, wallet, deps
  • If the PR affects users (e.g., new feature, bug fix, system requirements change), update the CHANGELOG.md and add details to the UNRELEASED section.
  • New features have usage guidelines and / or documentation updates in
  • Tests exist for new functionality or change in behavior
  • CI is green

@LexLuthr LexLuthr requested a review from snadrus April 29, 2024 15:29
@LexLuthr
Copy link
Contributor Author

@snadrus This is WIP, I will be moving everything to DB.

@LexLuthr LexLuthr requested a review from magik6k April 30, 2024 13:26
@LexLuthr LexLuthr marked this pull request as ready for review April 30, 2024 19:32
@LexLuthr LexLuthr changed the base branch from release/curio-beta to master May 8, 2024 07:09
@LexLuthr LexLuthr marked this pull request as draft May 8, 2024 11:34
@LexLuthr
Copy link
Contributor Author

LexLuthr commented May 8, 2024

Converting back to draft as I need to add new tables to avoid sectors_sdr_initial_pieces_sp_id_fkey and do the relevant code changes. I will also try to account for snap-deals.

node/config/def.go Outdated Show resolved Hide resolved
@LexLuthr LexLuthr marked this pull request as ready for review May 13, 2024 07:49
@LexLuthr LexLuthr requested a review from magik6k May 14, 2024 10:36
Copy link
Contributor

@magik6k magik6k left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an absolutely amazing chunk of work!

Some easy to address comments, but nothing fundamentally wrong. My main concert is around how we assign pieces to sectors - where we should be taking TermMin/Max bounds of the piece we're inserting vs expirations of pieces already assigned to a given sector (also possible that I've missed this bit of logic, this PR is big).

In lotus-miner this is done in - https://github.com/filecoin-project/lotus/blob/master/storage/pipeline/sealing.go#L155

cmd/curio/market.go Outdated Show resolved Hide resolved
node/config/types.go Outdated Show resolved Hide resolved
lib/harmony/harmonydb/sql/20240508-open-deal-sectors.sql Outdated Show resolved Hide resolved
lib/harmony/harmonydb/sql/20240508-open-deal-sectors.sql Outdated Show resolved Hide resolved
cmd/lotus-shed/miner.go Show resolved Hide resolved
curiosrc/market/lmrpc/lmrpc.go Outdated Show resolved Hide resolved
curiosrc/market/lmrpc/lmrpc.go Outdated Show resolved Hide resolved
curiosrc/market/deal_ingest.go Show resolved Hide resolved
curiosrc/web/api/sector/sector.go Outdated Show resolved Hide resolved
itests/curio_test.go Outdated Show resolved Hide resolved
Copy link

github-actions bot commented May 21, 2024

All checks have passed

Copy link
Contributor

@magik6k magik6k left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, as much as it is possible to review PRs this big.

We'll probably find some issues here when we start running this code, so the sooner we do that the better.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants