Skip to content

Improved cleaning of grids objects to avoid leaked data #1720

Description

@dylannelson

Proposed new feature or change:

While diagnosing #1444 and developing #1667, we noticed an increasing scope. It started with being concerned about a specific instance of time leaking onto a grid object, causing downstream issues while using subset.
After finding a fix for this scenario, we decided we wanted to make a more robust grid object cleaning function, that can clean grid objects in a way that avoids other potentially damaging situations like this. Doing so we found other interactions with the IO, like #1717 that showed we need to further investigate making rules around cleaning, as there is a surprisingly unique range of incoming data variables, with characteristics that are uncommon, but comparable to what we imagine "leaked" data to look like.
A few main goals:

  1. Reduce the amount of potential incoming grid data variables
    • Based on what we don't need or don't expect to have after a grid file is read in
  2. Reduce the amount of potential incoming coordinates
    • Based on what we don't need or don't expect to have after a grid file is read in, and if they are 0 dimensional (dims=() or "scalar")
  3. Make a method that can run without needing model/format specific solutions
  4. Ensure that no models have their unique attributes removed if they are needed
  5. Ensure that the function is placed within the file read process so that it doesn't disrupt other existing parts

Added notes:

  • We are interested in keeping at a minimum our existing GRID_DIMS:
['n_edge', 'n_face', 'n_max_edge_edges', 'n_max_edge_faces', 
'n_max_edge_nodes', 'n_max_face_edges', 'n_max_face_faces', 
'n_max_face_nodes', 'n_max_node_edges', 'n_max_node_faces', 
'n_node', 'two']
  • We want cases like we've seen, and cases we've not seen, to be covered by this fix (in these examples, time should be assumed to be any variable we don't want/need):
    • time present as a dimension coordinate (1-D, time(time), ndim == 1)
    • time present as a scalar (0-dimensional) coordinate (dims == (), ndim == 0)
    • time present as a dimension without a coordinate (a bare time dimension carrying data variables, no time coordinate)
    • Meaningful dimensions (dimensions within DIM_NAMES) that should be there, but aren't, due to it's unique IO process

These are some of the instances we've seen and want to keep them all in mind when finding a solution. This may be blocked until we understand #1717 better.

Metadata

Metadata

Assignees

Labels

new featureNew user-facing functionality

Type

No type

Projects

Status
📚 Backlog

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions