Skip to content

Push cache - #16

Merged
EstebanMontandon merged 7 commits into
mainfrom
push_cache
Aug 26, 2026
Merged

EstebanMontandon merged 7 commits into
mainfrom
push_cache

Conversation

@EstebanMontandon

Copy link
Copy Markdown
Collaborator

Hola, I added an optional push caching to DHIS2Pusher.

The class DHIS2PushCache (push_cache.py), tracks datapoints already pushed to DHIS2 (per period, on disk as parquet) so that the next run skip datapoints unchanged since the last successful push.

Changes:

  • New DHIS2PushCache class: filter_new() to skip unchanged rows before pushing, mark_pushed() to record what was actually pushed. A datapoint is identified by dx/period/org_unit/category_option_combo/attribute_option_combo; any value difference (including to/from None) counts as changed.

  • Optional cache_path param in DHIS2Pusher (default None, fully backward compatible). When set, push_data() filters via the cache before classification, and marks datapoints as pushed afterward — excluding any DHIS2 rejected (_remove_rejected_points()), so failed pushes get retried next run instead of being wrongly treated as done.

  • Simplified summary["rejected_datapoints"] from a nested {"index", "datapoint"} structure to a flat list of the rejected payloads (the old "index" was chunk-local and not usable for matching).

  • Extensive test coverage for DHIS2PushCache (load/update/dedup/null-handling edge cases) and for the DHIS2Pusher integration (cache skip on repeat push, cache persisted to disk, rejected datapoints excluded from cache).

  • README updated with the new cache_path option.

@EstebanMontandon

Copy link
Copy Markdown
Collaborator Author

pin ruff to use v0.15.22, if this version changes, any new rules cause annoying ruff errors.
The current errors in ruff can be ignored in this PR, as they're produced by the experimental classes :
*org_unit_aligner
*dataset_completion

@lgarridobsq lgarridobsq left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hola! Just a couple of things -- very cool feature! make sure to advertise it to the team when you realease it ;) (i think you need to up the version?)

Comment thread d2d_development/d2d_development/push_cache.py Outdated
Comment thread d2d_development/d2d_development/push.py
@EstebanMontandon
EstebanMontandon merged commit cce18aa into main Aug 26, 2026
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants