You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Table Formats encapsulate write actions into an Optimistic Transaction. Various processes could try to commit the info to the Transaction Log, but only one would succeed, making the others retry the operation.
When the user wants to write any extra metadata about the process or to do any checks on the data, it has to wait until the transaction is successfully over, thus leading to delays and possible inconsistencies.
The idea is to add commit hooks that allow to run custom code before or after calling the Delta Lake txn.commit() method.
The text was updated successfully, but these errors were encountered:
If you could provide any other necessary info regarding this enhancement would be great @Jiaweihu08@cugni
osopardo1
changed the title
Add PreCommit hooks to commit extra information within the same transaction
Add Commit Hooks to write extra information within the same transaction
Apr 23, 2024
/** * A hook that can be run before a commit*/traitQbeastPreCommitHook {
/** * The name of the hook*/valname:StringtypePreCommitHookResponse=Map[String, String]
/** * Run the hook with the given actions * * @paramactions * the actions to run the hook with*/defrun(actions: Seq[Action]):PreCommitHookResponse
}
Users can extend that interface and add their code to run pre-commit checks or actions. Those actions need to return a Map[String, String] of information they want to write on the Commit Log about the execution of the hook.
Registering a PreCommitHook
In terms of usage from the DataFrame API, one user may call the hook as:
In that way, we will identify each hook by its name on the options.
Output of a PreCommitHook
Each implementation of a PreCommitHook must return a Map[String, String] with any useful information to register in the Commit Log. Qbeast will add the prefix registered in the options to the result map keys, and it will save it on the tags variable in the CommitInfo.
Table Formats encapsulate write actions into an Optimistic Transaction. Various processes could try to commit the info to the Transaction Log, but only one would succeed, making the others retry the operation.
When the user wants to write any extra metadata about the process or to do any checks on the data, it has to wait until the transaction is successfully over, thus leading to delays and possible inconsistencies.
The idea is to add commit hooks that allow to run custom code before or after calling the Delta Lake
txn.commit()
method.The text was updated successfully, but these errors were encountered: