Repository navigation
Translate the new-format custom constraints, with relaxation and expansion-limit constraints - #139
Conversation
Each RHS value becomes one constraint instance per investment period (and per timeslice where tagged), with date_from resolved as the value active at the start of each period. LHS terms keep generator and battery names even though those components aren't translated yet — pypsa_build skips-and-logs missing components, and the logging silences itself once generator translation lands. The generators/batteries parameters are accepted now as the future name-mapping hook. Constraint-relaxation expansion options become dummy generators on bus_for_custom_constraint_gens, with their LHS terms filtered per period by build_year so a generator built for 2040 can't relax a 2030 constraint instance — an improvement over the old single-constraint formulation, which couldn't express this. Extendable links and relaxation generators get "_expansion_limit" constraints capping total p_nom at the option's allowed expansion; the suffix keeps their names clear of the parent constraints'. Duplicate input rows and RHS rows without LHS pairs raise rather than collapse silently; the "load" term type maps to Load/p, which pypsa_build logs as unimplemented and skips, matching the old path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
#126 changed the expansion tables to allow blank key cells as wildcards, changed _prepare_expansion_costs to take the enabled element ids, and made expansion links carry forward/reverse per unit of max(forward, reverse). The cherry-picked module still read unresolved options, capped paths at their forward capacity and imported a helper #126 deleted. The expansion-limit cap is now max(forward, reverse), re-resolved for the paths that have expansion links with the network module's own helpers, and constraint relaxations resolve their options and costs against the constraints in the model so blank-id defaults apply to every constraint. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests.
... and 10 files with indirect coverage changes 🚀 New features to boost your workflow:
|
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…raint docstrings _keep_rows_for_enabled_elements is shared by the network and constraints translators, but "enabled elements" only described the network caller — the constraints side passes constraint_ids. Renamed to _keep_rows_for_expansion_ids and reworded the docstrings so each caller's selection is described where the decision is made. The constraint helpers' I/O examples now show full tables, including the blank-cell wildcard forms the options and costs accept. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A relaxation generator's p_nom was always subtracted from the parent constraint's LHS, which loosens a "<=" but tightens a ">=". The sign now follows the constraint's direction (-1.0 for "<=", +1.0 for ">=") so building relaxation capacity always loosens the constraint; an option on an "=" is rejected, since no single-signed term can loosen an equality. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QXjsJKX7ANSJxVyJbszLYW
… periods The translator raised on duplicate rows, missing directions, unpaired constraints and relaxed equalities -- all properties of the input tables, so they're now declared as schema rules the translator trusts without re-checking. What's genuinely per-run remains: after date_from resolution a (constraint, period) can end up with terms on one side only, so those periods are dropped from both sides with an INFO line each way. Relaxation-generator terms derive from the post-drop RHS so they can't re-create one-sided periods. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QXjsJKX7ANSJxVyJbszLYW
…nerators Every option has a cost in every investment period (the costs schema's coverage rule), so joining options to costs gives exactly one generator per option and period. The test fixture violated that rule -- SWQLD1 had a 2026 cost only -- which hid the second per-period generator from every relaxation expectation; it now carries schema-valid costs and the tests exercise both SWQLD1_exp_2026 and SWQLD1_exp_2028. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QXjsJKX7ANSJxVyJbszLYW
Terms naming components not in the model were described as skipped with a log line downstream -- a silent per-term drop. The custom_constraints_lhs schema now ties each term_type's variable_name to the table it must exist in, so the translator and pypsa_build take the names as given. The templater TODO points at the rule as the end state for its IASR ID lookup. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QXjsJKX7ANSJxVyJbszLYW
A blank timeslice isn't "binds at every snapshot": it's the constraint's fallback row, binding only at the snapshots none of its named-timeslice rows cover, matching the path-limits convention. Resolving timeslices to snapshots stays pypsa_build's job; this module passes them through. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QXjsJKX7ANSJxVyJbszLYW
Set-membership probes like "SWQLD1_exp_2026 not in variable_name" pass for many wrong outputs. The relaxation-disabled, late-date_from and missing-link tests now compare whole expected frames, and the negative log case covers every INFO drop line the module can emit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QXjsJKX7ANSJxVyJbszLYW
allowed_expansion rode through the relaxation-generator pipeline without any step using it, only to be read off at the end. The orchestrator now passes the resolved options' caps directly to _create_expansion_limit_constraints, the rez_transmission_expansion gate moves into _resolve_relaxation_options where the decision is made, and _finalise_generators disappears. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QXjsJKX7ANSJxVyJbszLYW
The module docstring and the orchestrator's docstring had become two accounts of the same pipeline, with fixes having to land in both. The orchestrator docstring now carries the lot -- the translation steps, the relaxation and expansion-limit behaviour, the schema rules the pipeline trusts without re-checking, and the blank-timeslice fallback semantics -- next to the I/O example, and the module docstring is deleted. Accuracy fixes folded in along the way: the one-sided period drop names both of its causes (date_from coverage and model scope), the flow-path cap is stated as max(forward, reverse), link_flow expansion is described as replacing the path_id term with per-link terms, and the rez_transmission_expansion gate, per-option-per-period generators and annuitised costs are now stated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QXjsJKX7ANSJxVyJbszLYW
The side-by-side table pairs and the compressed links one-liner in the constraints I/O examples saved vertical space at the cost of scanning -- every input table now sits sequentially in the same tabular format as the rest of the examples. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QXjsJKX7ANSJxVyJbszLYW
…trator Dropping a term whose component isn't in the model left the constraint applied with a weakened LHS -- plausible-looking and silently wrong. The translator now raises when any LHS term fails to resolve to a model component (load terms, which pypsa_build doesn't implement, raise too), and resolves every term through an isp_name -> name mapping supplied by new generators and storage inputs, so new entrant units expand to their per-build-year components the same way link terms expand over a path's links. The generator/storage frame contract (isp_name + name, mirroring links) is assumed ahead of the translators that will produce it, for review by the team building that side. The orchestrator is renamed _translate_custom_constraints (the constraints aren't network-specific any more) and split into three block producers -- translate the user-authored tables, create the constraint relaxations, create the expansion limits -- assembled and finalised once, so future producers of constraint blocks (e.g. new entrant build limits) have a seam to append into before relaxation resolves. The reusable pieces' docstrings now state their source-agnostic contracts rather than relaxation-specific vocabulary. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QXjsJKX7ANSJxVyJbszLYW
The blanket raise on load terms made the real constraint set untranslatable -- the baseline CNSW-SNW South GPG constraint carries date_from-varying load terms. Demand is exogenous (p_set on the Load components pypsa_build attaches per demand node), so a load term is a data term, not a variable term: it now resolves through the standard isp_name -> name mapping to the "load_<bus>" component with attribute p_set, ready for pypsa_build to fold coefficient x demand into the constraint. pypsa_build owes that implementation and should raise, not skip, until it lands. Validation is against a new demand_nodes input rather than all buses, because REZ buses carry no demand and which buses do follows the regional granularity -- a load term for a sub-region the granularity aggregates away raises through the same not-in-model check as link terms. Also labels the missing-component error's tuple structure as (constraint_id, variable_name). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QXjsJKX7ANSJxVyJbszLYW
…l_name The mapping's inherited isp_name/name columns said where the data came from, not what the mapping means, so the constraints module now names both sides for their role. The custom_constraints_lhs schema also sets out the convention those names imply: a variable_name refers to a whole element - a new entrant across all its build years, a path across its existing and expansion links - never a per-build-year component. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0182yKiTKCwGoVPs5hToavKn
…trings Condensed column=value examples made the reader reconstruct the table shape in their head, so the I/O Example convention now requires full tables (with abridged column/row sets for otherwise large ones) and the constraints module's remaining condensed examples follow it. The expansion example also gains a new entrant term, showing the per-build-year fan-out next to the link and existing-unit cases. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0182yKiTKCwGoVPs5hToavKn
…nches Coverage showed four untested paths: the all-empty assembly branch, calendar year period starts, the unmapped-term_type drift guard, and the duplicate RHS name check. The last needed the one realistic collision - a relaxable constraint sharing its ID with an expandable path, giving both the same expansion-limit cap row - which the new test pins down. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0182yKiTKCwGoVPs5hToavKn
A component's p_nom is a single horizon-wide variable, so a term on a not-yet-built or already-retired component would let capacity from outside a period alter that period's constraint (dispatch terms were harmless, since PyPSA zeroes inactive dispatch, but made the output misleading). Terms are now kept for exactly the periods PyPSA activates the component - build_year <= period < build_year + lifetime, with the retirement year derived from the lifetime the translated tables already carry - so the constraints can't disagree with the model's own activity mask. Also pins the schema-legal edge cases the suite didn't cover: blank-timeslice RHS fallbacks, values dated exactly on a period start, mid-horizon coefficient supersession, generator_capacity terms, new-entrant storage expansion, wildcard relaxation options, and populated constraints alongside empty expansion tables. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DBECPsKWwUCxESCcU2Ykxk
The forward-quantisation semantics - date_from, build_year and lifetime all rounding onto the investment-period sequence the way PyPSA's multi-investment-period activity mask does - were described on the top-level orchestrator, away from the function that implements them and duplicated across its step bullets. State the rule once on _translate_constraint_tables, collapse the bullets that restated it, and leave the orchestrator a pointer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DBECPsKWwUCxESCcU2Ykxk
EllieKallmier
left a comment
EllieKallmier
left a comment
There was a problem hiding this comment.
Hey Nick! Took me a minute to go through all this and wrap my head around it all but I think I've got there, and this seems pretty solid to me. A few comments, some questions for my sake, and some extremely minor line-wrapping suggestions left throughout.
Probably the biggest functional piece that I'm not 100% sure about is the wild-card filling for the no-date_from rows - there's a comment in place but I'm not sure how clear my question actually is there lol. The summary q/uncertainty from me I guess is just about whether it's relying on an expectation of input data that if there's at least one row for a given group that has a date_from value, any constraint values that should be applied to periods after that defined one will also be defined with a date_from value.
To be clear though - I don't think that's a problem, just not sure whether that's the full intent/expectation, and if so it might be helpful to have that clarified in place?
Otherwise - I don't have any problems with this, and it feels like a relatively easy flow to edit over time if we need to. I'd be interested to hear your reservations cause I think you have a better sense for the bigger picture surrounding this chunk of work and seemed like you weren't totally sold, but yeah I'm happy with this for the moment :)
| tables currently also matches DER/CER rows that ISPyPSA may not template. | ||
| The custom_constraints_lhs schema's variable_names_resolve_by_term_type rule | ||
| declares that end state; note the new-entrant templater renames units to | ||
| "<geo_id> <technology>", which the lookup here will need to follow. |
There was a problem hiding this comment.
with the update from #144 now merged in this isn't the case anymore - hoping that makes things a little easier. Still though names can be changed for non-subregion granularity so yah worth keeping an eye on
There was a problem hiding this comment.
Pardon, I think this was a stale comment anyway as my plan is to not template custom constraints for non-sub-regional granularities. A follow up PR will address the rest of this todo so the constarint templating uses the templated generator and storage tables rather than the raw ones.
| @@ -0,0 +1,1261 @@ | |||
| import logging | |||
There was a problem hiding this comment.
not an action item, just a style kinda question - no module-level docstring for this one? I don't necessarily have a deep attachment to having them just thinking about internal style consistency (how much do we really care? not too much atm I think) and wondering whether you have strong feelings either way
There was a problem hiding this comment.
with addition that the orchestrator func docstrings in this module essentially play the same role
There was a problem hiding this comment.
Yeah, that was my thought, the func doc string seemed to cover it. I guess at the moment I've been making judgement calls and adding module level docstrings if I think there is too much worth documenting to fit into the orchestrator docstring.
| SWQLD1 qld_peak_demand 2500 2035 # 2032 value held from period start | ||
| """ | ||
| table = table.copy() | ||
| table["date_from"] = pd.to_datetime(table["date_from"]).fillna(pd.Timestamp.min) |
There was a problem hiding this comment.
to confirm: because there's no 'date_to' equivalent value for these tables, the 'resolved' active rows just extend on forever from the date_from value unless/until a row with a later date_from value appears (for the same group columns)?
Just thinking through the wild-card thing and how it can't be used here to implicitly fill out rows that start after any 'real' date_from values (that have the same grouping values ofc). I'm struggling to put my question into clear words atm so this might be a bit loose - but am I understanding that right? That it's always a case of 'latest value wins', and a no-date_from row can never win that unless it's the only row for that group?
There was a problem hiding this comment.
I've updated custom_constraints_lhs.yaml and custom_constraints_rhs.yaml to document this better. As we chatted about date_from are joined on investment_period in an as_of fashion, so they apply untill overwritten by a more recent on. If date_from is blank, those values would just apply before any date_from values became active. So the wildcarding works as for other tables, just with merge_as_of logic.
| lhs without its NQ1 2028 term (no RHS row that period; logged) | ||
| rhs without its SWQLD1 2026 row (no LHS terms that period; logged) |
There was a problem hiding this comment.
It hink this could still be written out as tables, even though it is fairly self-explanatory. Just cause the returned tables would be pretty small but just immediately descriptive
| applied. | ||
| variable_name uses IASR IDs for generators and batteries, path_ids for links | ||
| and sub-region codes for loads. The variable_names_resolve_by_term_type rule | ||
| ties each to its component table, so the translator and pypsa_build take the |
There was a problem hiding this comment.
Is this (custom validation rule) meaningfully different in function to the _raise_on_terms_not_in_model guard in constraints.py? My understanding of the difference between them atm is that the raise func is checking for existence in the model anywhere, and this validation rule is checking for presence specifically linked to the term type (pypsa component?). Unless I'm mixing something up here?
Not any issue on my end, just parsing out where the different validation/checking/guards are coming in. Developing the mental flow chart :)
There was a problem hiding this comment.
Good question. Firstly, that comment is outdated/incorrect because, as you point out, we are checking _raise_on_terms_not_in_model, I'll update the comment.
I kept _raise_on_terms_not_in_model because it operates a bit further down the pipeline so it would catch the case of a generator or storage which was in the ispypsa tables but got dropped in the translator step, or similar cases. Basically, I was erring defensive and wanted to make sure no other part of the translator had silently remove expected variables from custom constraints.
| def test_wildcard_relaxation_option_and_cost_apply_to_every_constraint( | ||
| csv_str_to_df, sample_model_config | ||
| ): | ||
| """A blank expansion_id relaxation option (and a blank expansion_id, blank |
There was a problem hiding this comment.
This is I think a fairly self-indulgent (non-blocking) note purely based on my apparent inability to hold these connections in my head - I would find this kind of test in particular much easier to read (+edit/update/extend/etc) with a few brief comments just pointing out how the inputs connect/lead to the expected outputs.
sub-note - do you need to have the ispypsa_tables = _constraint_tables(csv_str_to_df) line here if all of those tables are written over in subsequent lines?
There was a problem hiding this comment.
That's a fair comment, a couple of comments would probably go a long way. I'll add a few and maybe update the claude.md
I also added to the comments for:
- test_constraint_with_no_lhs_terms_dropped_and_logged
- test_relaxation_option_for_constraint_not_in_model_is_dropped
- test_relaxation_on_greater_equal_constraint_adds_capacity_to_lhsI'v
That's right, ispypsa_tables = _constraint_tables(csv_str_to_df), isn't needed. Probably a clearer read if its removed.
Co-authored-by: EllieKallmier <61219730+EllieKallmier@users.noreply.github.com>
Co-authored-by: EllieKallmier <61219730+EllieKallmier@users.noreply.github.com>
for more information, see https://pre-commit.ci
…schemas The date_from descriptions left readers to infer that a dated value has no end date, that rows only supersede each other within their group, and that a group without an undated row is absent from earlier periods. The term_type description still said load terms are skipped, which no longer matches the translator, and called storage_output discharge rather than net dispatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The note that new-entrant renaming must be followed was stale after #144, and only ever applied at coarser granularities, where custom constraints are not templated. The existing-unit templated tables the TODO waited on now exist, so name the target tables directly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The I/O example described its returned frames in prose, against the convention that DataFrame examples are always shown as full tables. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The source note claimed the translator and pypsa_build take variable names as given, but the translator now raises on terms whose component the configured model doesn't build. Say what each check covers instead, with the REZ connection path case the schema rule can't see. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Several tests depend on rows that live only in the _constraint_tables baseline, so a reader couldn't see why the expected output follows. Add docstrings and comments tying inputs to outputs, and drop a helper call whose tables were all overwritten. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tests whose expected output depends on combining inputs, or on rows hidden in a shared baseline helper, were hard to follow without comments. Ask for brief comments that name the specific rows and values, so they explain the result rather than gesture at it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Translate the new-format custom constraints, with relaxation and expansion-limit constraints