Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: recover from panics in LoadSectorStates #1115

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

fix: recover from panics in LoadSectorStates #1115

wants to merge 2 commits into from

Conversation

frrist
Copy link
Member

@frrist frrist commented Jan 19, 2023

birdychang
birdychang approved these changes Jan 19, 2023
View reviewed changes
Copy link
Contributor

@birdychang birdychang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@davidgasquez
Copy link

I'm having some issues with this PR. This is the lily.log.

{"level":"info","ts":"2023-01-22T12:48:26.438Z","logger":"lily/index/manager","caller":"integrated/manager.go:125","msg":"task success","height":"2399050","reporter":"walk_1674391696","task":"verified_registry_verif
ier","status":"OK","duration":9.576490392}
{"level":"error","ts":"2023-01-22T12:48:26.536Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01944708","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f01944708.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change
 type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"info","ts":"2023-01-22T12:48:26.766Z","logger":"lily/index/processor","caller":"processor/state.go:362","msg":"processor ended","task":"miner_sector_post","height":"2399050","reporter":"walk_1674391696","d
uration":9.904742493}
{"level":"info","ts":"2023-01-22T12:48:26.766Z","logger":"lily/index/manager","caller":"integrated/manager.go:125","msg":"task success","height":"2399050","reporter":"walk_1674391696","task":"miner_sector_post","sta
tus":"OK","duration":9.904740094}
{"level":"error","ts":"2023-01-22T12:48:27.338Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01983523","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: context canceled"}
{"level":"error","ts":"2023-01-22T12:48:27.395Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01915730","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading current sector states for miner f01915730.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change 
type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:27.544Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01283930","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f01283930.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change
 type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:27.591Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01931245","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading current sector states for miner f01931245.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change 
type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:27.827Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f014409","code":"fil
/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f014409.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change typ
e modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"info","ts":"2023-01-22T12:48:27.835Z","logger":"lily/index/processor","caller":"processor/state.go:362","msg":"processor ended","task":"data_cap_balance","height":"2399050","reporter":"walk_1674391696","du
ration":10.973147105}
{"level":"info","ts":"2023-01-22T12:48:27.835Z","logger":"lily/index/manager","caller":"integrated/manager.go:125","msg":"task success","height":"2399050","reporter":"walk_1674391696","task":"data_cap_balance","stat
us":"OK","duration":10.973145203}
{"level":"error","ts":"2023-01-22T12:48:28.536Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01907880","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading current sector states for miner f01907880.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change 
type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:28.565Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f0396473","code":"fi
l/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f0396473.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change t
ype modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:29.248Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01915033","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f01915033.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change
 type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:29.248Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01915033","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: context canceled"}
{"level":"error","ts":"2023-01-22T12:48:29.248Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01915033","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: context canceled"}
{"level":"error","ts":"2023-01-22T12:48:29.410Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f0126037","code":"fi
l/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f0126037.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change t
ype modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:29.493Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01983523","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f01983523.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change
 type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:30.319Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f0215497","code":"fi
l/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading current sector states for miner f0215497.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change ty
pe modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:30.414Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01964132","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading current sector states for miner f01964132.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change 
type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:30.496Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01172955","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f01172955.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change
 type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:30.496Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01172955","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: context canceled"}
{"level":"error","ts":"2023-01-22T12:48:30.496Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01172955","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: context canceled"}
{"level":"error","ts":"2023-01-22T12:48:30.794Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f0883206","code":"fi
l/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading current sector states for miner f0883206.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change ty
pe modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:30.813Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f010088","code":"fil
/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading current sector states for miner f010088.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change type
 modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:31.342Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01157027","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f01157027.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change
 type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:31.527Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01790067","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading current sector states for miner f01790067.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change 
type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:31.527Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01790067","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: context canceled"}
{"level":"error","ts":"2023-01-22T12:48:31.527Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01790067","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: context canceled"}
{"level":"error","ts":"2023-01-22T12:48:31.683Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01157018","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f01157018.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change
 type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:31.864Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01154901","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: context canceled"}
{"level":"error","ts":"2023-01-22T12:48:31.864Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01154901","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: context canceled"}
{"level":"error","ts":"2023-01-22T12:48:31.867Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01154901","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading current sector states for miner f01154901.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change 
type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:32.016Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f01858258","code":"f
il/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading current sector states for miner f01858258.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change 
type modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:32.077Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f0413563","code":"fi
l/9/storageminer","error":"failed to extract parsed actor state: diffing sector states loading current sector states for miner f0413563.(bafk2bzacedyux5hlrildwutvvjdcsvjtwsoc5xnqdjl73ouiukgklekeuyfl4) with change ty
pe modify at height 2399050: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"error","ts":"2023-01-22T12:48:32.077Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":2399050,"actor":"fil/9/storageminer","address":"f0413563","code":"fi
l/9/storageminer","error":"failed to extract parsed actor state: context canceled"}

The daemon is kept running but the walk is not completing. The list command outputs this:

[
        {
                "ID": 1,
                "Name": "walk_1674391696",
                "Type": "walk",
                "Error": "",
                "Tasks": [
                        "block_header",
                        "block_parent",
                        "drand_block_entrie",
                        "data_cap_balance",
                        "miner_beneficiary",
                        "miner_sector_deal",
                        "miner_sector_infos_v7",
                        "miner_sector_infos",
                        "miner_sector_post",
                        "miner_pre_commit_info",
                        "miner_sector_event",
                        "miner_current_deadline_info",
                        "miner_fee_debt",
                        "miner_locked_fund",
                        "miner_info",
                        "market_deal_proposal",
                        "market_deal_state",
                        "message",
                        "block_message",
                        "receipt",
                        "message_gas_economy",
                        "parsed_message",
                        "internal_messages",
                        "internal_parsed_messages",
                        "vm_messages",
                        "multisig_transaction",
                        "chain_power",
                        "power_actor_claim",
                        "chain_reward",
                        "actor",
                        "actor_state",
                        "id_addresses",
                        "derived_gas_outputs",
                        "chain_economics",
                        "chain_consensus",
                        "multisig_approvals",
                        "verified_registry_verifier",
                        "verified_registry_verified_client"
                ],
                "Running": true,
                "RestartOnFailure": false,
                "RestartOnCompletion": false,
                "RestartDelay": 0,
                "Params": {
                        "maxHeight": "2399050",
                        "minHeight": "2396162",
                        "storage": "CSV",
                        "window": "0s"
                },
                "StartedAt": "2023-01-22T12:48:16.859521259Z",
                "EndedAt": "0001-01-01T00:00:00Z"
        }
]

@frrist
Copy link
Member Author

frrist commented Jan 23, 2023

@davidgasquez is it failing for all epochs or just some?

@davidgasquez
Copy link

I think is just some (will check tomorrow when I'm back). That said, it happens in most of them and should be easy to reproduce in any recent partial archive.

@davidgasquez
Copy link

I've used historical-exports_snapshot_100800_103682_1666950560.car.zst locally and was able to reproduce the issue by walking any range of epochs.

The visor_processing_report.csv is full (many rows per epoch) of lines like this one:

100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceca3b27wyrbg6y5rowz3lik74gq7f3eiknem7cb5jscyfvujfaymy"",""Address"":""f018469"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f018469.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedreddfw7c5l3tlxxfrgsc34g7vjja4nb64zyvlyhmd6xnd7226t2"",""Address"":""f07125"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07125.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceb6p62a5v3hyogiv37qs4k5w7ayyobdxsiiuszurh4ctdp2csh3q4"",""Address"":""f021755"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f021755.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaced33gej7xvwo6aqzvxfreteetezkciuok276yv3bc5ag53zrhfhyq"",""Address"":""f017484"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f017484.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecfusprzbkxhnlu6463rfw5t76ryqtuncgtsje6dalwcu2td3qktu"",""Address"":""f04748"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04748.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecwgrhywq4h2dqmkxfcbxpt4v3d6lmlpfvcipzbw3wmnsztmlblhk"",""Address"":""f07673"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07673.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebmzdvcnagkwq3vq43xee2ebxm4dcabegpot2ew7pg6uvdoxgqycc"",""Address"":""f04417"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04417.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceb2n2l5ksc5jblp5ukvgxzcs7ringfjp2jwcoov2bp2sl335uejzw"",""Address"":""f06974"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f06974.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebrpyuhnvrnutgzqa4garkwgqmqkwadfq2iflesc2tjohebxijr5m"",""Address"":""f015772"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f015772.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebzurbfrykvuzvh2ksrg5i3etnp4ry42gkzje2sqa3z3hqxenrbim"",""Address"":""f017314"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f017314.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceapm44p4ul5vwla27v2vt2vb2secfittiumtick2shn3ozx2its7g"",""Address"":""f02159"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f02159.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedypbk2cchvsq6wwyzagpoeo2cu7bmh4p5vnlnj5skjer7criisag"",""Address"":""f07543"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f07543.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceausnjjei62ihlpc73jdojtg3teqcf47nnpvo5rfrjuj5oikgw67k"",""Address"":""f07608"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f07608.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceausnjjei62ihlpc73jdojtg3teqcf47nnpvo5rfrjuj5oikgw67k"",""Address"":""f07089"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07089.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceasjwnxgnrcswzduuu667ewl4wkge5dhktgoudoiqqlslavlmei6w"",""Address"":""f012056"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f012056.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacea2k6pvhwp6b6qubzd2irseteyepcp2vw7gfcfcqtqhwqibrz65ge"",""Address"":""f06469"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f06469.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceausnjjei62ihlpc73jdojtg3teqcf47nnpvo5rfrjuj5oikgw67k"",""Address"":""f07634"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07634.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebaaeo6fus4t3h2hajf4ljbfqhzznsw5ixk3ehvru7exebo32smya"",""Address"":""f08895"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f08895.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedfmo4t7bgc5bjnaqeag2zf5ba6eap3fg4qg42vzb3rgxj53xpc4a"",""Address"":""f03778"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f03778.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceabr32exarjf4frpmz7nnytjdgd4mdqtvpswhj3nmf7g3dzx62mwq"",""Address"":""f011606"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f011606.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceb66z6dgo32u5xbju2s2fi7pftaq7aqayi4htuiiyst3u6fzdek62"",""Address"":""f014682"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f014682.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaced5c7gjsiqrzriopbb5uv7vbvodxixlbynxittd65zf7bel36krri"",""Address"":""f018422"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f018422.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedqqqf2jnzdxh6ie3juo5hxx4ufervorjs7qhqeil6tylsm55rzye"",""Address"":""f04519"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04519.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceaxh66opmtlekt6mcpsovx6zdi2buyx7bfjs7flwq2onlyukwpnay"",""Address"":""f020146"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f020146.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceckmqxgee3y5x4pn3p2rxnbvq2ocgpfdbqp242n6wmfvk3z25lr7a"",""Address"":""f05065"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f05065.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceaqc6ip4evgj7irwfdruthwxdep2f6unh4qh73qul5n54e5kadcea"",""Address"":""f06883"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f06883.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceawrzb5rtgbeq25ljbibi3adk6zhocot3mvdvelmmytuaoplsihby"",""Address"":""f04297"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04297.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebtw3vey4cxsjma7lxan5zejo73huu43bs2sqcrbndswqaiiaj77u"",""Address"":""f018790"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f018790.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebik77jtdpzbreekpl6w3mpozed7poila5su5vusa3w5aaakkb2io"",""Address"":""f05766"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f05766.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebcecqsgmq4hcybquaoskottlgrghe7nhnfe45cbqojhhrkebyaau"",""Address"":""f05509"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f05509.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebnycpqatkwhny46aww7azwvfxsdl2qbuqf4kyv7m2ljtbv6l27bw"",""Address"":""f019317"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f019317.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacec67coauakuxgkv4b5s3kjdq5ckd5t4p37fe5dqhwix65yxgrztmg"",""Address"":""f010346"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f010346.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceckmqxgee3y5x4pn3p2rxnbvq2ocgpfdbqp242n6wmfvk3z25lr7a"",""Address"":""f07042"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07042.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceagdury54uijonwrzvasnkbhrs5pf2bggnqaciskt736jnlnmbmec"",""Address"":""f015748"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f015748.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceagymfhc5tv4ulcthjdtnla7z54tpprqhdbtflzvjsixe3ehkfzek"",""Address"":""f015771"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f015771.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacearowuwpzp7zjbfzzxfadhlygozwpkd7d7oc7x3rw76dmuzbzr32e"",""Address"":""f04688"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04688.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacec6xmpyail2rlpy5vis6j3l7dnnwt2bdar2nnwrlys43wqebppj66"",""Address"":""f07594"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f07594.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceavmvddz2e2nwoja36nqgmaewx7jgirdcwsiefwgxowjvnxyhfaug"",""Address"":""f07467"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07467.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceb5d56vudzrpcaf7pgi2wacfo4wacg6lokyce55qokaxt7ww6k43k"",""Address"":""f012067"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f012067.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacea2k6pvhwp6b6qubzd2irseteyepcp2vw7gfcfcqtqhwqibrz65ge"",""Address"":""f06716"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f06716.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebrkycr6iso7ujfmg3xc4qpjghzri7ah57sce7gqpq3eeojaht4li"",""Address"":""f06963"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f06963.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecfusprzbkxhnlu6463rfw5t76ryqtuncgtsje6dalwcu2td3qktu"",""Address"":""f06340"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f06340.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacectsnumtiverwj7owv4cme5wlfmw5bmeg5f23ulsqeyy3htwk5vp6"",""Address"":""f010224"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f010224.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedpgyirsxcvo7tbbasttkssmpphvtnunvuc7kngf6aogxxjrpd5v2"",""Address"":""f012509"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f012509.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebzhytzk2v2x4q3gx5yxi2xkiiqcv2cxobvv4nygwrwsz4o27t2ew"",""Address"":""f013057"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f013057.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecfusprzbkxhnlu6463rfw5t76ryqtuncgtsje6dalwcu2td3qktu"",""Address"":""f04676"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04676.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecbfijzx7bqhl6yutgw6f3pryfrmgkryugeuwfa7vuio7bba65dmi"",""Address"":""f04155"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f04155.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedct2ex4tpioukkbrkwc6c7lnkrvnlullz4yyytg7befolvyoq5u2"",""Address"":""f04903"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04903.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceaszfxuxlipsi7qlmbtimsdekei6stgxl5gcma3dzx3tfg4jfcmqy"",""Address"":""f04588"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04588.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacea4pwqlr32w4mvo56dkwnpmozlzn3t4sn4yds3mzoaqg3wertd2sg"",""Address"":""f014419"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f014419.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceadzscpxothvbyw5r65q7ih63jp3mrjl67truiv72atuehx6r6sro"",""Address"":""f01652"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f01652.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedtgbbumq7y4o5vefp7pjke23l7bfu5akyfv7znffzry3vchbebpq"",""Address"":""f011712"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f011712.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceck4zeazi3f22spyd2trzqov3rbe6ltedpadjvvexaxpimq7tyupc"",""Address"":""f05681"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f05681.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedgqzrqcutottoas4f3wnwqoxs52wrnj625xkn75seizuuy6wxlpo"",""Address"":""f04257"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f04257.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebik77jtdpzbreekpl6w3mpozed7poila5su5vusa3w5aaakkb2io"",""Address"":""f05374"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f05374.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebcecqsgmq4hcybquaoskottlgrghe7nhnfe45cbqojhhrkebyaau"",""Address"":""f07625"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f07625.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecj5jx6ft77ycxhlp2jayys35ouuyohyui3n6e2men6v32fvoyyes"",""Address"":""f05131"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f05131.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecwgrhywq4h2dqmkxfcbxpt4v3d6lmlpfvcipzbw3wmnsztmlblhk"",""Address"":""f06393"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f06393.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedeahyjfo4cb6puhvasijej4bzlkx2b3c5rovhzyhmj42yvnqez7m"",""Address"":""f08993"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f08993.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceaksxc4fdbhmi2qbxj7eslfjzut5wwyurpjwqbg5lmoqoszrsprm2"",""Address"":""f012588"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f012588.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceac5uhbv4h5wpc5ozyjhklusau2h64wljibwqmtv6ckcot5mk3jg2"",""Address"":""f014561"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f014561.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceavizu74t5bsazc4rk4tne2al5xsn3i4hb5aop4jgy5jby3zlc26s"",""Address"":""f014749"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f014749.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceas5567stmr7v4kgqpre6a5izgw744u3zt433hqcu7xdiqvcn3meo"",""Address"":""f012445"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f012445.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebc35qfrxz22e6doc7awjlxpmbbmq3nqi22evlp3446qtpqpkj7ws"",""Address"":""f016519"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f016519.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedd535k7kwm7kfg3y6c32fdhwb4zuf7ur55jt44dxx6ozfu753rhe"",""Address"":""f012022"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f012022.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceause7762ca2nbs4xkhcy6zql63qbjhaovmfx3jndgnmy2netwhbg"",""Address"":""f07705"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f07705.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",

There is also a bunch of empty files:

-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-miner_fee_debts.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-miner_infos.csv
-rw-r--r-- 1 root  root  449K Jan 24 16:28 arch0124-2020-09-28-miner_locked_funds.csv
-rw-r--r-- 1 root  root  1.6M Jan 24 16:28 arch0124-2020-09-28-miner_pre_commit_infos.csv
-rw-r--r-- 1 root  root  6.9K Jan 24 16:28 arch0124-2020-09-28-miner_sector_deals.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-miner_sector_events.csv
-rw-r--r-- 1 root  root  1.8M Jan 24 16:28 arch0124-2020-09-28-miner_sector_infos.csv
-rw-r--r-- 1 root  root   19M Jan 24 16:28 arch0124-2020-09-28-miner_sector_posts.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-multisig_approvals.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-multisig_transactions.csv
-rw-r--r-- 1 root  root   26M Jan 24 16:28 arch0124-2020-09-28-parsed_messages.csv
-rw-r--r-- 1 root  root  250K Jan 24 16:28 arch0124-2020-09-28-power_actor_claims.csv
-rw-r--r-- 1 root  root  2.5M Jan 24 16:28 arch0124-2020-09-28-receipts.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-verified_registry_verified_clients.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-verified_registry_verifiers.csv

Finally, in lily.log, we can see some websocket: close 1000 (normal) errors.

{"level":"error","ts":"2023-01-24T15:27:57.608Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":100904,"actor":"fil/1/storageminer","address":"f02303","code":"fil/1/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f02303.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100904: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"debug","ts":"2023-01-24T15:27:57.798Z","logger":"lily/tasks/miner","caller":"miner/sector_events.go:109","msg":"diff sectors","miner":"f02731","duration":3.888790994}
{"level":"error","ts":"2023-01-24T15:27:57.798Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":100904,"actor":"fil/1/storageminer","address":"f02731","code":"fil/1/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f02731.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100904: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"debug","ts":"2023-01-24T15:27:57.808Z","logger":"rpc","caller":"go-jsonrpc@v0.1.8/websocket.go:624","msg":"websocket error","error":"websocket: close 1000 (normal)"}
{"level":"debug","ts":"2023-01-24T15:27:57.999Z","logger":"watchdog","caller":"go-watchdog@v1.3.0/watermarks.go:32","msg":"watermark policy: evaluating; utilization: 10881658880/33573044224 (used/limit)"}

That said, the walk seems to continue without problems after these websocket errors.

1 similar comment
@davidgasquez
Copy link

I've used historical-exports_snapshot_100800_103682_1666950560.car.zst locally and was able to reproduce the issue by walking any range of epochs.

The visor_processing_report.csv is full (many rows per epoch) of lines like this one:

100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceca3b27wyrbg6y5rowz3lik74gq7f3eiknem7cb5jscyfvujfaymy"",""Address"":""f018469"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f018469.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedreddfw7c5l3tlxxfrgsc34g7vjja4nb64zyvlyhmd6xnd7226t2"",""Address"":""f07125"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07125.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceb6p62a5v3hyogiv37qs4k5w7ayyobdxsiiuszurh4ctdp2csh3q4"",""Address"":""f021755"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f021755.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaced33gej7xvwo6aqzvxfreteetezkciuok276yv3bc5ag53zrhfhyq"",""Address"":""f017484"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f017484.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecfusprzbkxhnlu6463rfw5t76ryqtuncgtsje6dalwcu2td3qktu"",""Address"":""f04748"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04748.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecwgrhywq4h2dqmkxfcbxpt4v3d6lmlpfvcipzbw3wmnsztmlblhk"",""Address"":""f07673"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07673.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebmzdvcnagkwq3vq43xee2ebxm4dcabegpot2ew7pg6uvdoxgqycc"",""Address"":""f04417"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04417.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceb2n2l5ksc5jblp5ukvgxzcs7ringfjp2jwcoov2bp2sl335uejzw"",""Address"":""f06974"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f06974.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebrpyuhnvrnutgzqa4garkwgqmqkwadfq2iflesc2tjohebxijr5m"",""Address"":""f015772"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f015772.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebzurbfrykvuzvh2ksrg5i3etnp4ry42gkzje2sqa3z3hqxenrbim"",""Address"":""f017314"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f017314.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceapm44p4ul5vwla27v2vt2vb2secfittiumtick2shn3ozx2its7g"",""Address"":""f02159"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f02159.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedypbk2cchvsq6wwyzagpoeo2cu7bmh4p5vnlnj5skjer7criisag"",""Address"":""f07543"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f07543.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceausnjjei62ihlpc73jdojtg3teqcf47nnpvo5rfrjuj5oikgw67k"",""Address"":""f07608"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f07608.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceausnjjei62ihlpc73jdojtg3teqcf47nnpvo5rfrjuj5oikgw67k"",""Address"":""f07089"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07089.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceasjwnxgnrcswzduuu667ewl4wkge5dhktgoudoiqqlslavlmei6w"",""Address"":""f012056"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f012056.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacea2k6pvhwp6b6qubzd2irseteyepcp2vw7gfcfcqtqhwqibrz65ge"",""Address"":""f06469"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f06469.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceausnjjei62ihlpc73jdojtg3teqcf47nnpvo5rfrjuj5oikgw67k"",""Address"":""f07634"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07634.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebaaeo6fus4t3h2hajf4ljbfqhzznsw5ixk3ehvru7exebo32smya"",""Address"":""f08895"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f08895.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedfmo4t7bgc5bjnaqeag2zf5ba6eap3fg4qg42vzb3rgxj53xpc4a"",""Address"":""f03778"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f03778.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceabr32exarjf4frpmz7nnytjdgd4mdqtvpswhj3nmf7g3dzx62mwq"",""Address"":""f011606"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f011606.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceb66z6dgo32u5xbju2s2fi7pftaq7aqayi4htuiiyst3u6fzdek62"",""Address"":""f014682"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f014682.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaced5c7gjsiqrzriopbb5uv7vbvodxixlbynxittd65zf7bel36krri"",""Address"":""f018422"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f018422.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedqqqf2jnzdxh6ie3juo5hxx4ufervorjs7qhqeil6tylsm55rzye"",""Address"":""f04519"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04519.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceaxh66opmtlekt6mcpsovx6zdi2buyx7bfjs7flwq2onlyukwpnay"",""Address"":""f020146"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f020146.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceckmqxgee3y5x4pn3p2rxnbvq2ocgpfdbqp242n6wmfvk3z25lr7a"",""Address"":""f05065"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f05065.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceaqc6ip4evgj7irwfdruthwxdep2f6unh4qh73qul5n54e5kadcea"",""Address"":""f06883"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f06883.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceawrzb5rtgbeq25ljbibi3adk6zhocot3mvdvelmmytuaoplsihby"",""Address"":""f04297"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04297.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebtw3vey4cxsjma7lxan5zejo73huu43bs2sqcrbndswqaiiaj77u"",""Address"":""f018790"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f018790.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebik77jtdpzbreekpl6w3mpozed7poila5su5vusa3w5aaakkb2io"",""Address"":""f05766"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f05766.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebcecqsgmq4hcybquaoskottlgrghe7nhnfe45cbqojhhrkebyaau"",""Address"":""f05509"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f05509.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebnycpqatkwhny46aww7azwvfxsdl2qbuqf4kyv7m2ljtbv6l27bw"",""Address"":""f019317"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f019317.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacec67coauakuxgkv4b5s3kjdq5ckd5t4p37fe5dqhwix65yxgrztmg"",""Address"":""f010346"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f010346.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceckmqxgee3y5x4pn3p2rxnbvq2ocgpfdbqp242n6wmfvk3z25lr7a"",""Address"":""f07042"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07042.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceagdury54uijonwrzvasnkbhrs5pf2bggnqaciskt736jnlnmbmec"",""Address"":""f015748"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f015748.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceagymfhc5tv4ulcthjdtnla7z54tpprqhdbtflzvjsixe3ehkfzek"",""Address"":""f015771"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f015771.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacearowuwpzp7zjbfzzxfadhlygozwpkd7d7oc7x3rw76dmuzbzr32e"",""Address"":""f04688"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04688.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacec6xmpyail2rlpy5vis6j3l7dnnwt2bdar2nnwrlys43wqebppj66"",""Address"":""f07594"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f07594.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceavmvddz2e2nwoja36nqgmaewx7jgirdcwsiefwgxowjvnxyhfaug"",""Address"":""f07467"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f07467.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceb5d56vudzrpcaf7pgi2wacfo4wacg6lokyce55qokaxt7ww6k43k"",""Address"":""f012067"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f012067.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacea2k6pvhwp6b6qubzd2irseteyepcp2vw7gfcfcqtqhwqibrz65ge"",""Address"":""f06716"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f06716.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebrkycr6iso7ujfmg3xc4qpjghzri7ah57sce7gqpq3eeojaht4li"",""Address"":""f06963"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f06963.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecfusprzbkxhnlu6463rfw5t76ryqtuncgtsje6dalwcu2td3qktu"",""Address"":""f06340"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f06340.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacectsnumtiverwj7owv4cme5wlfmw5bmeg5f23ulsqeyy3htwk5vp6"",""Address"":""f010224"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f010224.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedpgyirsxcvo7tbbasttkssmpphvtnunvuc7kngf6aogxxjrpd5v2"",""Address"":""f012509"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f012509.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebzhytzk2v2x4q3gx5yxi2xkiiqcv2cxobvv4nygwrwsz4o27t2ew"",""Address"":""f013057"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f013057.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecfusprzbkxhnlu6463rfw5t76ryqtuncgtsje6dalwcu2td3qktu"",""Address"":""f04676"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04676.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecbfijzx7bqhl6yutgw6f3pryfrmgkryugeuwfa7vuio7bba65dmi"",""Address"":""f04155"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f04155.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedct2ex4tpioukkbrkwc6c7lnkrvnlullz4yyytg7befolvyoq5u2"",""Address"":""f04903"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04903.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceaszfxuxlipsi7qlmbtimsdekei6stgxl5gcma3dzx3tfg4jfcmqy"",""Address"":""f04588"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f04588.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacea4pwqlr32w4mvo56dkwnpmozlzn3t4sn4yds3mzoaqg3wertd2sg"",""Address"":""f014419"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f014419.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceadzscpxothvbyw5r65q7ih63jp3mrjl67truiv72atuehx6r6sro"",""Address"":""f01652"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f01652.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedtgbbumq7y4o5vefp7pjke23l7bfu5akyfv7znffzry3vchbebpq"",""Address"":""f011712"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f011712.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceck4zeazi3f22spyd2trzqov3rbe6ltedpadjvvexaxpimq7tyupc"",""Address"":""f05681"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f05681.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedgqzrqcutottoas4f3wnwqoxs52wrnj625xkn75seizuuy6wxlpo"",""Address"":""f04257"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f04257.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebik77jtdpzbreekpl6w3mpozed7poila5su5vusa3w5aaakkb2io"",""Address"":""f05374"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f05374.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebcecqsgmq4hcybquaoskottlgrghe7nhnfe45cbqojhhrkebyaau"",""Address"":""f07625"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f07625.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecj5jx6ft77ycxhlp2jayys35ouuyohyui3n6e2men6v32fvoyyes"",""Address"":""f05131"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f05131.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacecwgrhywq4h2dqmkxfcbxpt4v3d6lmlpfvcipzbw3wmnsztmlblhk"",""Address"":""f06393"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f06393.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedeahyjfo4cb6puhvasijej4bzlkx2b3c5rovhzyhmj42yvnqez7m"",""Address"":""f08993"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f08993.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceaksxc4fdbhmi2qbxj7eslfjzut5wwyurpjwqbg5lmoqoszrsprm2"",""Address"":""f012588"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f012588.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceac5uhbv4h5wpc5ozyjhklusau2h64wljibwqmtv6ckcot5mk3jg2"",""Address"":""f014561"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f014561.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceavizu74t5bsazc4rk4tne2al5xsn3i4hb5aop4jgy5jby3zlc26s"",""Address"":""f014749"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f014749.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceas5567stmr7v4kgqpre6a5izgw744u3zt433hqcu7xdiqvcn3meo"",""Address"":""f012445"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f012445.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacebc35qfrxz22e6doc7awjlxpmbbmq3nqi22evlp3446qtpqpkj7ws"",""Address"":""f016519"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f016519.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzacedd535k7kwm7kfg3y6c32fdhwb4zuf7ur55jt44dxx6ozfu753rhe"",""Address"":""f012022"",""Error"":""failed to extract parsed actor state: diffing sector states loading previous sector states for miner f012022.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",""Head"":""bafy2bzaceause7762ca2nbs4xkhcy6zql63qbjhaovmfx3jndgnmy2netwhbg"",""Address"":""f07705"",""Error"":""failed to extract parsed actor state: diffing sector states loading current sector states for miner f07705.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100947: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference""},{""Code"":""bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza"",""Name"":""fil/1/storageminer"",

There is also a bunch of empty files:

-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-miner_fee_debts.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-miner_infos.csv
-rw-r--r-- 1 root  root  449K Jan 24 16:28 arch0124-2020-09-28-miner_locked_funds.csv
-rw-r--r-- 1 root  root  1.6M Jan 24 16:28 arch0124-2020-09-28-miner_pre_commit_infos.csv
-rw-r--r-- 1 root  root  6.9K Jan 24 16:28 arch0124-2020-09-28-miner_sector_deals.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-miner_sector_events.csv
-rw-r--r-- 1 root  root  1.8M Jan 24 16:28 arch0124-2020-09-28-miner_sector_infos.csv
-rw-r--r-- 1 root  root   19M Jan 24 16:28 arch0124-2020-09-28-miner_sector_posts.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-multisig_approvals.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-multisig_transactions.csv
-rw-r--r-- 1 root  root   26M Jan 24 16:28 arch0124-2020-09-28-parsed_messages.csv
-rw-r--r-- 1 root  root  250K Jan 24 16:28 arch0124-2020-09-28-power_actor_claims.csv
-rw-r--r-- 1 root  root  2.5M Jan 24 16:28 arch0124-2020-09-28-receipts.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-verified_registry_verified_clients.csv
-rw-r--r-- 1 root  root     0 Jan 24 16:29 arch0124-2020-09-28-verified_registry_verifiers.csv

Finally, in lily.log, we can see some websocket: close 1000 (normal) errors.

{"level":"error","ts":"2023-01-24T15:27:57.608Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":100904,"actor":"fil/1/storageminer","address":"f02303","code":"fil/1/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f02303.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100904: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"debug","ts":"2023-01-24T15:27:57.798Z","logger":"lily/tasks/miner","caller":"miner/sector_events.go:109","msg":"diff sectors","miner":"f02731","duration":3.888790994}
{"level":"error","ts":"2023-01-24T15:27:57.798Z","logger":"lily/tasks","caller":"actorstate/task.go:82","msg":"actor returned with error","height":100904,"actor":"fil/1/storageminer","address":"f02731","code":"fil/1/storageminer","error":"failed to extract parsed actor state: diffing sector states loading previous sector states for miner f02731.(bafkqaetgnfwc6mjpon2g64tbm5sw22lomvza) with change type unknown at height 100904: LoadSectorState panic: runtime error: invalid memory address or nil pointer dereference"}
{"level":"debug","ts":"2023-01-24T15:27:57.808Z","logger":"rpc","caller":"go-jsonrpc@v0.1.8/websocket.go:624","msg":"websocket error","error":"websocket: close 1000 (normal)"}
{"level":"debug","ts":"2023-01-24T15:27:57.999Z","logger":"watchdog","caller":"go-watchdog@v1.3.0/watermarks.go:32","msg":"watermark policy: evaluating; utilization: 10881658880/33573044224 (used/limit)"}

That said, the walk seems to continue without problems after these websocket errors.

@davidgasquez
Copy link

Tested things locally and seems its working fine. I've got some errors but I think they're unrelated with this PR:

$ cat lily.log | grep "error" | jq .
{"level":"error","ts":"2023-01-25T08:53:19.389Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacebarb7znoskcwvl6up2avtcu44keoi2c7cfiat7knyvgwqy3iahak"}
{"level":"error","ts":"2023-01-25T08:53:19.389Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacecn3bzbox5h4x3viqwtgwvsz6glfmgm4tkxq2bvqdgddzzu3gx3ws"}
{"level":"error","ts":"2023-01-25T08:53:19.389Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacebdm3jeqsoa74de7qthlxvcjjkvp62yxri7auv7ggwyemfllwntgi"}
{"level":"error","ts":"2023-01-25T08:53:19.389Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacebrgzpswiwdqs5wvmw6zfrsvrdgeqpeozsbkhwuwgsswcfsdrdxto"}
{"level":"error","ts":"2023-01-25T08:53:19.389Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacea7lqx55xxzdsp7use33hgeeh33tyry4m6zg3dsin6lznopk4yq3w"}
{"level":"error","ts":"2023-01-25T08:53:19.389Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacebby6ag7qk743wti5bp6azdcffptrw5ws3u5gwa5c6kiepjt5ayqs"}
{"level":"error","ts":"2023-01-25T08:53:19.389Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacecifg623mszaimdfbooxpj54etykqjy7sjec3xd2ycoflqn5akke2"}
{"level":"error","ts":"2023-01-25T08:53:19.389Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacea4624zeqqzdrdwoe33ejxxz32zq5fr5ysmbuu3vofskgmegkx3qc"}
{"level":"error","ts":"2023-01-25T08:53:19.390Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzaceaoez3byuzqkblaa2gfj4zi2ubh74zj7m7jipkatbpjzoxdofmqvu"}
{"level":"error","ts":"2023-01-25T08:53:19.390Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacebmd7l7vevx56qkh2ehnugkwtmkysf4ryx3si4xwknanneqiybmhu"}
{"level":"error","ts":"2023-01-25T08:53:19.390Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzaceabtnahyz77s6kzhccls6mjfv43y4snibrn4zrbhfirltwvh6wa3u"}
{"level":"error","ts":"2023-01-25T08:53:19.390Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacedjqujbo32cqfkdw3zvhqlswo4lhom4b5v6jnvd54oy3aps5jznle"}
{"level":"error","ts":"2023-01-25T08:53:19.390Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacea5wktkeap5ur4czmrbx4xpwp4edfihrxm2n25adnclo3v6nakl4g"}
{"level":"error","ts":"2023-01-25T08:53:19.390Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzaceddzlla5vs4jxknncyoexprtm2hcn3xz37mrkfzjxtc2yiiepmd66"}
{"level":"error","ts":"2023-01-25T08:53:19.390Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacebnisrg7tjtzd5rwnakvwabhjg7j7m4rgoseae3fopl3cjjsgxbvk"}
{"level":"error","ts":"2023-01-25T08:53:19.390Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacecqx2oxwfbq2iwmkk4l2xvc6kajc6lndly57b6p7vxkojhzrcwgfa"}
{"level":"error","ts":"2023-01-25T08:53:19.390Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacechwhzxskk5cw4qvbaxcyolku6tu6q6prag4kqzlccbzmu7rmvkci"}
{"level":"error","ts":"2023-01-25T08:53:19.391Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzaceciazen2ggajc573g256ill54lk2jf6b6o6jvig2ezxbncseord22"}
{"level":"error","ts":"2023-01-25T08:53:19.391Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzaceb4v2dpsfp7skdoscjdcfzaxc5isxtz43doan4xowqrkmurcwpm44"}
{"level":"error","ts":"2023-01-25T08:53:19.391Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacediy2l4ycnctl6esyuwtid5kjdg42k77c5uunxw52nksxqooumue2"}
{"level":"error","ts":"2023-01-25T08:53:19.392Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacediy2l4ycnctl6esyuwtid5kjdg42k77c5uunxw52nksxqooumue2"}
{"level":"error","ts":"2023-01-25T08:53:19.393Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacebwmgedad75xmpjbpdfkjndeppcoed3b6rjmmyta7dnkaet6g5732"}
{"level":"error","ts":"2023-01-25T08:53:19.393Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacecm5axfyvqzehha6sp3sme6ripsgo7t6bchgt3ulybai3vz4afsik"}
{"level":"error","ts":"2023-01-25T08:53:19.393Z","logger":"statetree","caller":"state/statetree.go:283","msg":"failed to load state tree: failed to load hamt node: ipld: could not find bafy2bzacecrfir5yaawy7sdgmksusi7woxhheqmtej7x7hiz3e5mxccwk3j7u"}

Will send some jobs and keep an eye on things.

@davidgasquez
Copy link

Heads up that I just got the first websocket issue. Seems it might be still happening. I can't dig much for now though. Will leave a proper report when I'm back next week.

@hsanjuan
Copy link
Contributor

@frrist do you have any ideas about why it might be panicking in the first place?

@frrist
Copy link
Member Author

frrist commented Jan 26, 2023

@hsanjuan current theory is filecoin-project/filet#22 (comment)

@hsanjuan
Copy link
Contributor

The "maybe there are blocks missing in a datastore" doesn't fit the randomness that seems to affect this issue.

@frrist
Copy link
Member Author

frrist commented Feb 3, 2023

@davidgasquez can you point me to the github issue for the websocket bug?

@davidgasquez
Copy link

@davidgasquez can you point me to the github issue for the websocket bug?

There is one higher level in Filet and another in Lotus.

@birdychang
Copy link
Contributor

@frrist should we merge?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@birdychang birdychang birdychang approved these changes

@davidgasquez davidgasquez Awaiting requested review from davidgasquez

@kasteph kasteph Awaiting requested review from kasteph

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@frrist @davidgasquez @hsanjuan @birdychang