Extendend whitelist options #1644
Conversation
Cmdv
force-pushed
the
1600-extend-whitelist
branch
from
b1ff17f
to
de26071
Compare
Cmdv
force-pushed
the
1600-extend-whitelist
branch
3 times, most recently
from
cf3fcab
to
cc0f1a7
Compare
Cmdv
requested review from
sgillespie and
kderme
and removed request for
sgillespie
cardano-db-sync/src/Cardano/DbSync/Era/Universal/Insert/Other.hs
Outdated
Show resolved
Hide resolved
Cmdv
force-pushed
the
1600-extend-whitelist
branch
from
cc0f1a7
to
ffd68e0
Compare
Cmdv
force-pushed
the
1600-extend-whitelist
branch
from
72083ce
to
ea4514c
Compare
Cmdv
force-pushed
the
1600-extend-whitelist
branch
from
ea4514c
to
78418bf
Compare
|
Run into following error message with executable built from this branch, on
and it is hanging like that for quite some time, even though I used Longer log: |
|
Another question would be - should we have validator for stake addresses in config whitelisting ? I used this and no errors on |
what version of cardano-node are you using at the moment? I can't even get things to start syncing locally. I can see in the code where the issue is happening. Will try to get sanchonet working today 👍
The problem with that is if you add the stake address before starting a fresh sync, there is no way to validate if it's existence as it's not been yet created. |
@Cmdv not sure if this is right - it seemed to work with this version. |
|
I am using... node: db-sync: Network: Network Configs: wget https://book.play.dev.cardano.org/environments/sanchonet/config.json \
https://book.play.dev.cardano.org/environments/sanchonet/db-sync-config.json \
https://book.play.dev.cardano.org/environments/sanchonet/submit-api-config.json \
https://book.play.dev.cardano.org/environments/sanchonet/topology.json \
https://book.play.dev.cardano.org/environments/sanchonet/byron-genesis.json \
https://book.play.dev.cardano.org/environments/sanchonet/shelley-genesis.json \
https://book.play.dev.cardano.org/environments/sanchonet/alonzo-genesis.json \
https://book.play.dev.cardano.org/environments/sanchonet/conway-genesis.jsondb-sync config: "insert_options": {
"tx_out": {
"value": "enable"
},
"ledger": "enable",
"shelley": {
"enable": true,
"stake_addresses": ["e083d27f03b197750ee41b7dca02c806718164b934b0fbfa747349e8f2"]
},
"multi_asset": {
"enable": true
},
"metadata": {
"enable": true
},
"plutus": {
"enable": true
},
"governance": "enable",
"offchain_pool_data": "enable",
"json_type": "text"
},Logs: So If I am reading it correctly - it breaks on querying / inserting first record in
Query on database synced without whitelisting: sancho=# select * from gov_action_proposal limit 3;
id | tx_id | index | prev_gov_action_proposal | deposit | return_address | expiration | voting_anchor_id | type | description | param_proposal | ratified_epoch | enacted_epoch | dropped_epoch | expired_epoch
----+-------+-------+--------------------------+------------+----------------+------------+------------------+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+----------------+---------------+---------------+---------------
1 | 1878 | 0 | | 1000000000 | 1024 | 251 | 1 | TreasuryWithdrawals | {"tag": "TreasuryWithdrawals", "contents": [[[{"network": "Testnet", "credential": {"keyHash": "248f556b733c3ef24899ae0609d3796198d5470192304c4894dd85cb"}}, 10000000000000]], null]} | | | | |
2 | 1948 | 0 | | 1000000000 | 1024 | 252 | 2 | InfoAction | {"tag": "InfoAction"} | | | | |
3 | 4116 | 0 | | 1000000000 | 1044 | 253 | 1 | TreasuryWithdrawals | {"tag": "TreasuryWithdrawals", "contents": [[[{"network": "Testnet", "credential": {"keyHash": "248f556b733c3ef24899ae0609d3796198d5470192304c4894dd85cb"}}, 51000000]], null]} | | | | |
(3 rows) |
|
OK, I think I got it something more. So basically adding sancho=# select * from gov_action_proposal limit 3;
id | tx_id | index | prev_gov_action_proposal | deposit | return_address | expiration | voting_anchor_id | type | description | param_proposal | ratified_epoch | enacted_epoch | dropped_epoch | expired_epoch
----+-------+-------+--------------------------+------------+----------------+------------+------------------+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+----------------+---------------+---------------+---------------
1 | 1878 | 0 | | 1000000000 | 1024 | 251 | 1 | TreasuryWithdrawals | {"tag": "TreasuryWithdrawals", "contents": [[[{"network": "Testnet", "credential": {"keyHash": "248f556b733c3ef24899ae0609d3796198d5470192304c4894dd85cb"}}, 10000000000000]], null]} | | | | | in this case sancho=# select * from stake_address where id=1024;
id | hash_raw | view | script_hash
------+--------------------------------------------------------------+------------------------------------------------------------------+-------------
1024 | \xe0192a8eb77a6f405d8196dd96db13950c48300a0ff8b6f42e7901a2ef | stake_test1uqvj4r4h0fh5qhvpjmwedkcnj5xysvq2plutdapw0yq69mccrcnd7 |
(1 row)to "stake_addresses": ["e083d27f03b197750ee41b7dca02c806718164b934b0fbfa747349e8f2","e0192a8eb77a6f405d8196dd96db13950c48300a0ff8b6f42e7901a2ef"]Moved error past epoch So if stake address for |
Cmdv
force-pushed
the
1600-extend-whitelist
branch
from
2772d9f
to
efa38c2
Compare
|
@Cmdv With latest version on this branch I am left stuck at I see that tables listed below are empty while the non whitelisted version of DB have records for them ( the common thing is that they have select * from constitution;
id | gov_action_proposal_id | voting_anchor_id | script_hash
----+------------------------+------------------+-------------
(0 rows)
select * from new_committee_info;
id | gov_action_proposal_id | quorum_numerator | quorum_denominator
----+------------------------+------------------+--------------------
(0 rows)
select * from new_committee;
id | gov_action_proposal_id | deleted_members | added_members | quorum_numerator | quorum_denominator
----+------------------------+-----------------+---------------+------------------+--------------------
(0 rows)
select * from new_committee_member;
id | gov_action_proposal_id | committee_hash_id | expiration_epoch
----+------------------------+-------------------+------------------
(0 rows)
select * from voting_procedure;
id | tx_id | index | gov_action_proposal_id | voter_role | drep_voter | pool_voter | vote | voting_anchor_id | committee_voter
----+-------+-------+------------------------+------------+------------+------------+------+------------------+-----------------
(0 rows)Also do not see records in those tables ( contrary to non whitelisted DB version ) EDIT: "shelley": {
"enable": true,
"stake_addresses": ["e083d27f03b197750ee41b7dca02c806718164b934b0fbfa747349e8f2"]
},sancho_regular=# select * from stake_address where id=5;
id | hash_raw | view | script_hash
----+--------------------------------------------------------------+------------------------------------------------------------------+-------------
5 | \xe083d27f03b197750ee41b7dca02c806718164b934b0fbfa747349e8f2 | stake_test1uzpaylcrkxth2rhyrd7u5qkgqeccze9exjc0h7n5wdy73usd0pxpl |
(1 row)
It seems like there might be also issue with sancho_regular=# select * from tx_out where stake_address_id=5;
id | tx_id | index | address | address_has_script | payment_cred | stake_address_id | value | data_hash | inline_datum_id | reference_script_id
----+-------+-------+---------+--------------------+--------------+------------------+-------+-----------+-----------------+---------------------
(0 rows)
sancho_regular=# select count(*) from tx_out where stake_address_id IS NOT NULL;
count
-------
6210
(1 row)and on whitelisted by sancho_test_config_shelley_whitelist=# select count(*) from tx_out;
count
--------
103980
(1 row)
sancho_test_config_shelley_whitelist=# select count(*) from tx_out where stake_address_id IS NOT NULL;
count
-------
0
(1 row)So we should get rid off all those records with NULL |
|
This is related to Assuming that config below is correct: "insert_options": {
"tx_out": {
"value": "enable"
},
"ledger": "enable",
"shelley": {
"enable": true
},
"multi_asset": {
"enable": true
},
"metadata": {
"enable": true
},
"plututs": {
"enable": true,
"script_hashes":["34250edd1e9836f5378702fbf9416b709bc140e04f668cc355208518","67f33146617a5e61936081db3b2117cbf59bd2123748f58ac9678656", "bccf2a430c016bc960fbf31b02694011cd399d20da8882aac9d33611", "1b539b68e5389bdc3fe511c7694583e3755f58d4b38aef50384f4349"]
},
"governance": "enable",
"offchain_pool_data": "enable",
"json_type": "text"
},I am seeing entries in sancho_test_config_shelley_whitelist_plutus=# SELECT id, tx_id, hash, type, json
FROM script
WHERE hash NOT IN (
'\x34250edd1e9836f5378702fbf9416b709bc140e04f668cc355208518', -- hash for 'timelock' - 1st entry in script_hashes
'\x67f33146617a5e61936081db3b2117cbf59bd2123748f58ac9678656', -- hash for 'plutusV1' - 2nd entry in script_hashes
'\xbccf2a430c016bc960fbf31b02694011cd399d20da8882aac9d33611', -- hash for 'plutusV2' - 3rd entry in script_hashes
'\x1b539b68e5389bdc3fe511c7694583e3755f58d4b38aef50384f4349' -- hash for 'plutusV3' - 4th entry in script_hashes
)
ORDER BY type, id;
id | tx_id | hash | type | json
----+-------+------------------------------------------------------------+----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
6 | 17132 | \xeeb006239ad2562556b736c3990f20968fcf89bb6d4f7292aaa0c30a | timelock | {"type": "atLeast", "scripts": [{"type": "sig", "keyHash": "ff750767ac2496a659b8325bf7064f92d969672d931ce0bc0290f6ba"}, {"type": "sig", "keyHash": "b7b9594ef50295739c95144238654413221df009f388cbf9b8ca73b7"}, {"type": "sig", "keyHash": "0253cc2bc1ed8176c675f454dd730fae5bfaa147b73924bde70d786a"}], "required": 2}
24 | 28937 | \x6b0261483fa1fe471c6f941c2a917d75164cbd96a93a9e42f62efd1c | timelock | {"type": "sig", "keyHash": "6c6fe0b26f5bfa67ece9221683ea4e7ad19bdf14db8a9fae7b294b5e"}
9 | 19924 | \x56b0f0b597150e619c76bed60683f3b1e42d7bc0685ed951b882bfc5 | plutusV2 |
10 | 19924 | \x86bff95ba20e9d1d1b34899a56d86bbacc9fed999260b27dcc92d128 | plutusV2 |
11 | 19927 | \x0689163fa0280c34d6e3d78657e2b06980be7ac77854f0dd5201bf05 | plutusV2 |
12 | 21241 | \x8275a6621fdb65e60c117fddd5f162d43adf6364c2645cd865bf5482 | plutusV2 |
14 | 24872 | \x04b8c921d534b5debae418ae679a83ec5e628b8e680af1a241788790 | plutusV2 |
... |
|
@ArturWieczorek thanks a lot for the very thorough reports 🙏
I wonder if it has something with stopping and restarting as I got to 301 last week and that was the tip but now we're on 306. So this morning I restarted and like you it was just not moving forward. I can't fully tell it's it's node not providing more data or something is going on on our side. Will investigate more.
So it's doesn't just whitelist against script hash, it also checks payment creds. Which was the existing functionality. I'm unsure as to the reasoning but it's functionality was kept. New code here |
There was a problem hiding this comment.
I think for multiassets and plutus, there needs to be a better distinction on the checks that happen before trying to insert anything (this checks if there is a whitelist) and the check that happens during inserting (this checks if the feauture is completely disabled). I left some comments that may help
| -- first check the config option | ||
| case ioPlutus iopts of | ||
| PlutusEnable -> True | ||
| PlutusDisable -> False |
There was a problem hiding this comment.
Similarly to isMAPoliciesInWhitelist, I think this should be True.
| case ioPlutus iopts of | ||
| PlutusEnable -> True | ||
| PlutusDisable -> False | ||
| PlutusScripts plutusWhitelist -> toShort scriptHash `elem` plutusWhitelist |
There was a problem hiding this comment.
I think this should be just True. Whiletelisting is checked previoiusly in isPlutusScriptHashesInWhitelist. It also probably needs to be renamed to isPlutusNotDisabled.
There was a problem hiding this comment.
ok I'll check, but isPlutusNotDisabled is a double negative which in english makes it confusing
There was a problem hiding this comment.
@Cmdv thinking again, the current implementation may be correct. I thought the intented behaviout was different, but since the behaviour is open to interpretation, it's fine.
There was a problem hiding this comment.
So let's say we have a whitelist for a specific script A. Some tx contains script A in an output and script B in another output. The tx passes the validation check, but do we want to insert B? I guess both answers are correct as long as we are consinstent.
| } | ||
| ReaderT SqlBackend m (Maybe DB.ScriptId) | ||
| insertScript syncEnv txId script = | ||
| if isSimplePlutusScriptHashInWhitelist syncEnv $ Generic.txScriptHash script |
There was a problem hiding this comment.
I think this check should change. See the comment at isSimplePlutusScriptHashInWhitelist for more.
Cmdv
force-pushed
the
1600-extend-whitelist
branch
from
ff85921
to
81bd0d7
Compare
Signed-off-by: Cmdv <vincent@cmdv.me>
Cmdv
force-pushed
the
1600-extend-whitelist
branch
from
81bd0d7
to
ec10eac
Compare
Cmdv
force-pushed
the
1600-extend-whitelist
branch
from
ec10eac
to
31b0fcd
Compare
Description
This PR fixes #1600 adding more whitelists to the configs
The most complex whitelist was the Shelley Stake Address as it stored in the table
stake_address.The functionality is if the stake address doesn't match in the whitelist on any of the inserts into the following tables, then we don't insert at all:
Checklist
fourmoluon version 0.10.1.0 (which can be run withscripts/fourmolize.sh)Migrations
If there is a breaking change, especially a big one, please add a justification here. Please elaborate
more what the migration achieves, what it cannot achieve or why a migration is not possible.