Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PseudoKV migration acceleration #144

Open
MeijeSibbel opened this issue Oct 22, 2020 · 0 comments
Open

PseudoKV migration acceleration #144

MeijeSibbel opened this issue Oct 22, 2020 · 0 comments

Comments

@MeijeSibbel
Copy link

##Motivation
Faster migration optimizes contract formation as contracts can be re-used to storage data faster. Furthermore if migration is faster less contracts will have to be formed which is advantageous both in terms of financial and in terms of scalability.

Current behavior:

The velocity of data migration < new data uploaded --> New contract set will have to be formed to continue uploading.

Currently migration is slow as can be seen on the snippet below from one of our gateways:

2020-10-22T12:03:14.494Z        DEBUG   grpc/client.go:104      pulled a storage class  {"path": "/ddos1-2/file.ezpp", "request_id": "e86c8e14d667e0f0903f2fb08b4750f2591f2fdc", "class": {"gatewayID":"a74982c8-67a1-4dae-a366-f082c85bc5d0:67e1a599-f654-4bb0-af52-2958768811dd","userID":"a74982c8-67a1-4dae-a366-f082c85bc5d0","bucket":"ddos1-2","className":"STANDARD"}, "active_contract": "6f38b62b-5017-4f1d-9097-7229819e5bea", "reserved_contract": "0528065b-7a62-4d80-8feb-c5fde1875c65", "migrating_contracts": ["6d7d6d2e-c789-40ad-98cd-7b14ece377cb", "c5689dfd-f047-41dc-8c3c-c81aaf871119", "8878808b-1b2a-40ac-a5c6-5ff8a6bf391c", "f1111543-3ce6-4646-a171-7d38da78233c", "a1ef4748-7fe6-4821-a3b7-dba0dfb5fe68", "07ba7094-6119-4caa-8f54-dc9e4a5fe175", "d9e0b28d-bb6a-4b8b-8c43-92ea63d6a9d9", "e608807d-9d07-4639-a1ed-cd7370ace921", "1c91412b-c9df-440a-a857-b3ef2c3805ec", "2a65461e-f8e7-4669-a7e6-3904ddcdaf40", "7b7c4a68-0b83-4cb9-be43-6977373a0db6", "33c25ce3-aa50-455b-9809-3d0eb9997d86", "602815dd-804e-4ba6-840a-e2b5f567e692", "169b6e19-c352-4c42-8329-e84ca9d3b87c", "b3770c9b-9a96-4596-b4eb-306bc5d156e8", "8ba48fc3-f13c-4444-a769-2d9804eb1b29", "25bc62e1-72f0-46fa-b0dc-2fb767655116", "164edbc8-656c-4d15-bb24-6a5e8902d4bd", "4aecceee-329a-4d6e-83f7-64876d6eec75"], "waiting_contracts": [], "full_contracts": ["5e64aea0-fa89-418a-9a44-24c3b099c0ed", "5c585ef8-ab51-4f25-85bf-6c2041d62c78", "4d6821c2-e57b-4284-b7b8-05cfc5a2687f", "578bd561-0b99-47f7-a518-01a10c27b5a8", "a44abcbb-df3e-4f57-b27e-b6ab87a7c5b5"]}

Many contract sets are stuck in the migration state (19 contract sets ~ 19*40 = 760 contracts).

If migration is carried out from a resource deficient or low upload throughput host, migration can take a very long time to complete. The impact of this grows as more data is stored.

Expected behavior

Migrate from the fastest host(s) with the highest upload throughput to minimize migration time. As overdrive will be optimizing for slow hosts during upload, it would be beneficial to have something similar for migration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant