Checkpointing causes dataChange=False

[echai58]

Environment

Delta-rs version: 0.16.3

Binding: Python

---

Bug

What happened:
Before checkpointing, table.get_add_actions(flatten=True).to_pandas looks like:

After checkpointing, the same call looks like:

which shows everything is unchanged, except data_change goes from True to False.

Examining the checkpoint parquet file, I see the following entry for add:

{'path': '1-16c20083-ac6d-4f35-bf38-6860c3fc9abc-0.parquet',
 'size': 711.0,
 'modificationTime': 1711565377761.0,
 'dataChange': False,
 'stats': '{"numRecords": 2, "minValues": {"k": "a", "v": "1"}, "maxValues": {"k": "b", "v": "2"}, "nullCount": {"k": 0, "v": 0}}',
 'partitionValues': [],
 'tags': [],
 'deletionVector': None,
 'stats_parsed': {'numRecords': 2.0,
  'minValues': {'k': 'a', 'v': '1'},
  'maxValues': {'k': 'b', 'v': '2'},
  'nullCount': {'k': 0.0, 'v': 0.0}}}

while the original commit file shows:

{"path":"1-16c20083-ac6d-4f35-bf38-6860c3fc9abc-0.parquet",
"partitionValues":{},
"size":711,
"modificationTime":1711565377761,
"dataChange":true,
"stats":"{\"numRecords\": 2, \"minValues\": {\"k\": \"a\", \"v\": \"1\"}, \"maxValues\": {\"k\": \"b\", \"v\": \"2\"}, \"nullCount\": {\"k\": 0, \"v\": 0}}",
"tags":null,
"deletionVector":null,
"baseRowId":null,
"defaultRowCommitVersion":null,
"clusteringProvider":null}

What you expected to happen:
Unless I'm misunderstanding what checkpoint does, I believe it is a bug that dataChange is being set to False, instead of preserving the value from the original commit file.

How to reproduce it:

import pyarrow as pa
from deltalake import DeltaTable, write_deltalake

schema = pa.schema(
    [
        ("k", pa.string()),
        ("v", pa.string()),
    ]
)

DeltaTable.create("test", schema=schema)

dt = DeltaTable("test")
write_deltalake(dt, pd.DataFrame.from_dict({"k": ["a", "b"], "v": ["1", "2"]}), mode="append")

dt = DeltaTable("test")
add_actions = dt.get_add_actions(flatten=True)
print(add_actions.to_pandas())

dt = DeltaTable("test")
add_actions = dt.get_add_actions(flatten=True)
print(add_actions.to_pandas())

[echai58]

Looking at the checkpoint code, I see this line:

delta-rs/crates/core/src/protocol/checkpoints.rs

Line 364 in 5b404e2

v["add"]["dataChange"] = Value::Bool(false);

Am I misunderstanding what checkpoint does? Why is it being set to False?

[ion-elgreco]

@echai58 change seems to be done 3 years ago ^^, but reading the protocol it mentions data_change=false if files were already present in a table, so I guess that's why in this context.

You could check what the behavior is of delta-spark and if you see the data_change being preserved as-is?

[echai58]

@echai58 change seems to be done 3 years ago ^^, but reading the protocol it mentions data_change=false if files were already present in a table, so I guess that's why in this context.

You could check what the behavior is of delta-spark and if you see the data_change being preserved as-is?

Yeah, in my eyes a checkpoint isn't really a transaction because it doesn't generate a commit file, so i feel it should preserve it. But yeah, I'll look into what the delta-spark behavior is, good call.

[ion-elgreco]

@rtyler @roeap do you know why it's set to False perhaps?

[echai58]

@ion-elgreco Seems like checkpointing with delta-spark also sets dataChange to False. 🤷‍♂️ I guess that's the desired behavior.

I brought this up because I have a use case where I'm interested in figuring out which partitions were edited in the past n days, so I know which partitions to selectively z-order, so I don't have to z-order the entire table. I was relying on the add_actions table with the dataChange and modificationTime columns to determine this, but because of checkpointing, this seems to not work - do you have any alternatives?