You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I had an asset with a weekly partitions definitions with start date set to 1900-01-01. It had all partitions ranging back to 1900-01-01 was backfilled or marked as backfilled. When i changed it to have daily partitions instead with a start date of 2024-04-20 and enabled auto-materialization(it was enabled previously as well), dagster started requesting materializations of partitions before 2024-04-20. I also had set "all_partitions" to "False" in my cron-schedule but it seems it still tries to materialize all missing partitions.
I was forced to disable the rule "materialize_on_missing" to prevent it from doing that.
Furthermore I also got the following GraphQL-error when accessing the "Automation"-page of the specific asset:
Operation name: GetEvaluationsQuery
Message: 'NoneType' object has no attribute 'start'
Path: ["assetConditionEvaluationRecordsOrError"]
Locations: [{"line":19,"column":3}]
Stack Trace:
File "/usr/local/lib/python3.10/site-packages/graphql/execution/execute.py", line 521, in execute_field
result = resolve_fn(source, info, **args)
File "/usr/local/lib/python3.10/site-packages/dagster_graphql/schema/roots/query.py", line 1157, in resolve_assetConditionEvaluationRecordsOrError
return fetch_asset_condition_evaluation_records_for_asset_key(
File "/usr/local/lib/python3.10/site-packages/dagster_graphql/implementation/fetch_asset_condition_evaluations.py", line 109, in fetch_asset_condition_evaluation_records_for_asset_key
return _get_graphene_records_from_evaluations(
File "/usr/local/lib/python3.10/site-packages/dagster_graphql/implementation/fetch_asset_condition_evaluations.py", line 58, in _get_graphene_records_from_evaluations
records=[
File "/usr/local/lib/python3.10/site-packages/dagster_graphql/implementation/fetch_asset_condition_evaluations.py", line 59, in <listcomp>
GrapheneAssetConditionEvaluationRecord(
File "/usr/local/lib/python3.10/site-packages/dagster_graphql/schema/asset_condition_evaluations.py", line 307, in __init__
evaluation=GrapheneAssetConditionEvaluation(
File "/usr/local/lib/python3.10/site-packages/dagster_graphql/schema/asset_condition_evaluations.py", line 253, in __init__
evaluationNodes = [
File "/usr/local/lib/python3.10/site-packages/dagster_graphql/schema/asset_condition_evaluations.py", line 254, in <listcomp>
GraphenePartitionedAssetConditionEvaluationNode(evaluation, partitions_def)
File "/usr/local/lib/python3.10/site-packages/dagster_graphql/schema/asset_condition_evaluations.py", line 159, in __init__
trueSubset=GrapheneAssetSubset(evaluation.true_subset),
File "/usr/local/lib/python3.10/site-packages/dagster_graphql/schema/asset_condition_evaluations.py", line 75, in __init__
subsetValue=GrapheneAssetSubsetValue(asset_subset.subset_value),
File "/usr/local/lib/python3.10/site-packages/dagster_graphql/schema/asset_condition_evaluations.py", line 49, in __init__
for start, end in value.get_partition_key_ranges(value.partitions_def)
File "/usr/local/lib/python3.10/site-packages/dagster/_core/definitions/time_window_partitions.py", line 1731, in get_partition_key_ranges
return [
File "/usr/local/lib/python3.10/site-packages/dagster/_core/definitions/time_window_partitions.py", line 1734, in <listcomp>
).get_partition_key_range_for_time_window(window)
File "/usr/local/lib/python3.10/site-packages/dagster/_core/definitions/time_window_partitions.py", line 745, in get_partition_key_range_for_time_window
cast(TimeWindow, self.get_prev_partition_window(time_window.end)).start.timestamp()
That it would only try to materialize the latest partition, and not try to catchup. Also not materialize partitions that doesn't exist.
How to reproduce?
use auto-materialize
have a weekly schedule and let it materialize
Change to a daily partitions definition with a later start date which means the partition time window does not include all of the earlier materializations.
Deployment type
Dagster Helm chart
Deployment details
No response
Additional information
No response
Message from the maintainers
Impacted by this issue? Give it a 👍! We factor engagement into prioritization.
The text was updated successfully, but these errors were encountered:
Dagster version
1.7.3
What's the issue?
I had an asset with a weekly partitions definitions with start date set to 1900-01-01. It had all partitions ranging back to 1900-01-01 was backfilled or marked as backfilled. When i changed it to have daily partitions instead with a start date of 2024-04-20 and enabled auto-materialization(it was enabled previously as well), dagster started requesting materializations of partitions before 2024-04-20. I also had set "all_partitions" to "False" in my cron-schedule but it seems it still tries to materialize all missing partitions.
I was forced to disable the rule "materialize_on_missing" to prevent it from doing that.
Furthermore I also got the following GraphQL-error when accessing the "Automation"-page of the specific asset:
This might be related to this github issue: Auto-materialization daemon re-runs all partitions if time window is extended to the past
What did you expect to happen?
That it would only try to materialize the latest partition, and not try to catchup. Also not materialize partitions that doesn't exist.
How to reproduce?
Deployment type
Dagster Helm chart
Deployment details
No response
Additional information
No response
Message from the maintainers
Impacted by this issue? Give it a 👍! We factor engagement into prioritization.
The text was updated successfully, but these errors were encountered: