Gemini Changes by moonraker595 · Pull Request #249 · ral-facilities/operationsgateway-api

moonraker595 · 2026-04-13T14:07:50Z

This PR:

Adds string as a new data type:
Introduces string as a supported data type.
Allows string for shotnum field:
Updates the shotnum field to accept both int and string values.
Maintains backward compatibility with existing numeric shot numbers.
Add an optional type field to shotnum:
Introduces an optional type attribute to the model for shotnum datatype.
If type is not provided, the system infers the type based on the value (str or int).
Support sub-second timestamp ingestion:
Extends ingestion logic to handle timestamps with sub-second precision.
Enhancements to range converter:
Updates range conversion logic to correctly handle mixed shotnum types (int and string).
Clean up:
Adds a new lock file for dependency updates (which is the bulk of the reported line changes)

…turn strings

… ids) to short or long

…ding-strings-to-model

codecov · 2026-04-14T13:34:19Z

Codecov Report

❌ Patch coverage is 90.17544% with 28 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.47%. Comparing base (9381719) to head (5675c30).

Files with missing lines	Patch %	Lines
...nsgateway_api/src/records/ingestion/hdf_handler.py	81.25%	4 Missing and 5 partials ⚠️
...perationsgateway_api/src/records/export_handler.py	76.47%	3 Missing and 5 partials ⚠️
operationsgateway_api/src/records/record.py	45.45%	3 Missing and 3 partials ⚠️
operationsgateway_api/src/routes/auth.py	66.66%	2 Missing and 1 partial ⚠️
...gateway_api/src/records/ingestion/record_checks.py	0.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #249      +/-   ##
==========================================
- Coverage   96.88%   96.47%   -0.41%     
==========================================
  Files          88       88              
  Lines        4944     5164     +220     
  Branches      439      488      +49     
==========================================
+ Hits         4790     4982     +192     
- Misses        103      119      +16     
- Partials       51       63      +12

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

moonraker595 · 2026-09-10T16:48:22Z

I've beefed up the test coverage. Fair warning; the last of the auth ones and all of the user ones were AI generated via co-pilot. I think it's in a much better place now, I'll ping it back over to you.

moonraker595 · 2026-09-10T16:52:21Z

oh one thing I haven't actually done yet is test all the export stuff (post-merge) via Postman. I'll do that first thing tomorrow.

moonraker595 · 2026-09-11T10:34:28Z

oh one thing I haven't actually done yet is test all the export stuff (post-merge) via Postman. I'll do that first thing tomorrow.

I've checked this manually, and all works as expected. Just didn't want to rely solely on the automated test is all.

… channel processing from scalar and string CSV handling.

moonraker595 · 2026-09-11T13:59:11Z

ebc0b76: noticed a logging issue in which string and scaler values were falling through the if statement, this meant 1000's of duplicated logging lines. This just makes things more explicit.

patrick-austin

Had a look over it again and could only find one (very) minor formatting change I'd definitely make (test_get_crosshair_intensity.py).

The rest are things that I noticed and wanted to check you're happy with, but in principle I don't think necessarily need to change if you are. As far as I could see the codecov report is OK given that we won't have the Gemini suite contributing to the coverage (I didn't consider on my last review until you pointed it out). I've left some thoughts about that as well but think it would be fine to let it lie (either for this PR and pick up as a separate issue, or for the foreseeable future).

One final thing - I was asking Kevin what bits of the API code would be covered by the "data integrity" testing he's been doing and if he knew what version/commit the Gemini instance is running, since I wasn't sure if any changes made to this branch would affect things for him. I think the answer was essentially "no" since he wasn't using export, and I guess at some point we'd have to deploy the latest version of the API to the Gemini instance anyway so it would get tested in pre-prod whenever that happens.

Co-authored-by: patrick-austin <61705287+patrick-austin@users.noreply.github.com>

moonraker595 · 2026-09-15T11:05:02Z

Thanks, have made the appropriate changes.

With regards to Kevin's testing; As you mentioned, he is only testing the data, not the version/functionality of OG.
The version deployed to prod (there is no pre-prod) is commit 09f295d58d7e270758650a3556acf28d903ad115 which is from July 23rd and holds most of the UO functionality used for testing with Louise. Once this is commited into main i need to come up with a release plan which will entail:
1st: Gemini and test
2nd: Epac pre-prod and test, then prod
3rd: Clara, using it as a new base for pre-prod and prod (there's nothing on prod at the mo, so it needs setting up)

patrick-austin

Changes look good - just had one comment about the custom validator for the user office config. I think it's fine as is but was wondering if an optional "sub-model" would let us get the same functionality without having to write/test a custom validator? If there's a reason I'm missing why that wouldn't work, or you've already considered it and prefer it this way then I'm happy to approve, but thought it was worth raising for consideration at least.

patrick-austin · 2026-09-16T08:12:31Z

+    @model_validator(mode="after")
+    def check_user_office_settings(self) -> "AuthConfig":
+        """
+        When the User Office integration is enabled, both the API key and the users
+        service URL are needed to contact it
+        """
+        if self.user_office_integration:
+            missing = [
+                name
+                for name in ("user_office_api_key", "user_office_users_service_url")
+                if not getattr(self, name)
+            ]
+            if missing:
+                raise ValueError(
+                    f"{', '.join(missing)} must be set when "
+                    "'user_office_integration' is enabled",
+                )
+
+        return self


An alternative way of doing this could be to define an optional UserOfficeConfig that is optional in the main config, but with both the API key and url being required within the model itself. That would mean we don't need custom validation since Pydantic will allow nothing to be provided (i.e. user_office_config=None) or both, but if only one is provided then `UserOfficeConfig is missing a required field and it will fail.

Done. There are a few files changed, but they're mostly single-line changes. I had forgotten we could check on the empty tag, and we do this elsewhere, so this is a lot neater. Good catch.

moonraker595 · 2026-09-16T08:27:48Z

I initially assumed a sub-model would require moving the User Office settings out of the auth section, but I suppose it can be an optional nested model within AuthConfig, so a sub sub model. So the config would look like:

auth:
  refresh_token_validity_days: ...
  fedid_server_url: ....
  fedid_server_ldap_realm: ....

  user_office:
    api_key: ...
    users_service_url: ...

So the user office section is optional, and other deployments just won't have. Sound ok?

moonraker595 and others added 18 commits October 31, 2025 14:09

Adding string to supported list

1e0ca2e

Adding string to model list

e555f71

Adding string to shotnum types

2443fcd

Adding string to channel dtypes

7f67c99

Adding optional unit field for strings

f0ffa6f

Adding standard string channel checks

33de644

reverting shotnum to int

7b7ccd5

making shotnum string

4b040f5

seperated timestamp formats

765b12a

added millisecond accuracy to hdf checker and file path

2e41c4f

Now using shot type for filtering and changes models to accept and re…

e21e67e

…turn strings

fixinf tests for datetime list

aac1bbf

updating ci config to use short timestamps

0fe77b6

Added config handler for new congig item to set timestamps (or record…

2a268bf

… ids) to short or long

Now checking and parsing timestamps

476c814

Updated error message

ac28b35

Updated tests to account for new shotnum type of string

bca2901

Merge remote-tracking branch 'origin/Adding-strings-to-model' into Ad…

cfe986d

…ding-strings-to-model

moonraker595 changed the title ~~Adding strings to model~~ Adding strings and timestamps Apr 14, 2026

moonraker595 and others added 10 commits April 14, 2026 13:47

Linting

fad12f6

bumping poetry version

36fb186

bumping versions

f976422

bumping versions

5a9e99a

Merge branch 'main' into Adding-strings-to-model

8472876

regenerating lock file

330b9b5

regenerating lock file

fdb3cec

regenerating lock file

2fccbed

regenerating lock file

ccfd000

regenerating lock file

d7d396c

moonraker595 requested a review from patrick-austin September 10, 2026 16:48

moonraker595 added 2 commits September 11, 2026 13:11

Refactorrd scalar and string CSV export handling. Separated ZIP-based…

ebc0b76

… channel processing from scalar and string CSV handling.

Updated doc string

41c5eb3

patrick-austin requested changes Sep 14, 2026

View reviewed changes

moonraker595 and others added 5 commits September 15, 2026 10:35

Update test/endpoints/test_get_crosshair_intensity.py

8a1b875

Co-authored-by: patrick-austin <61705287+patrick-austin@users.noreply.github.com>

Updated minio version

b9d4d68

User Office functionality now sits behind a global flag

ddda61c

replaced type with isinstance

01eb50b

removing raise_for_status

7c46ddb

moonraker595 requested a review from patrick-austin September 15, 2026 11:06

moonraker595 added 2 commits September 15, 2026 11:22

adding tests for user office flag

059ac2f

adding tests for user office flag

fae66d7

patrick-austin reviewed Sep 16, 2026

View reviewed changes

moonraker595 added 7 commits September 16, 2026 11:03

Sub model for user office functionality

79df85e

Getting values from new model

dba0a8f

Adding new sub model for user office functionality

e292537

Now checking if new OU model is present

876179a

Changing tests to deal with new UO sub model

9ff3be3

Using UO model for checking

7927013

linting

5675c30

moonraker595 requested a review from patrick-austin September 16, 2026 11:48

patrick-austin approved these changes Sep 16, 2026

View reviewed changes

moonraker595 merged commit 57c808f into main Sep 16, 2026
7 checks passed

moonraker595 deleted the Adding-strings-to-model branch September 16, 2026 14:07

Conversation

moonraker595 commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

moonraker595 commented Sep 10, 2026

Uh oh!

moonraker595 commented Sep 10, 2026

Uh oh!

moonraker595 commented Sep 11, 2026

Uh oh!

moonraker595 commented Sep 11, 2026

Uh oh!

patrick-austin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

moonraker595 commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

patrick-austin left a comment

Choose a reason for hiding this comment

Uh oh!

patrick-austin Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

moonraker595 Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

moonraker595 commented Sep 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

moonraker595 commented Apr 13, 2026 •

edited

Loading

codecov Bot commented Apr 14, 2026 •

edited

Loading

moonraker595 commented Sep 15, 2026 •

edited

Loading