OperationsGateway API

This API is built using FastAPI to work with MongoDB and ECHO S3. It allows CRUD access to the data stored as part of the OperationsGateway project for the EPAC facility.

A separate repository exists to manage the deployment on pre-production and production machines using the Ansible framework.

The following instructions will detail how to get a development instance up and running. They can run on a command line and are tailored for a new rocky-9-nogui virtual machine with SSH access.

Prerequisites

1. Local development on Windows or Mac can be challenging because they lack the Linux libraries required by the API. Since the API is designed to run in a Python 3.11 environment on a Rocky 9 Linux machine, it’s best to use a Rocky 9 development VM to closely mirror the production setup.
2. You'll need a developer bucket in echo to use the test script to prefill echo, and a local database, with data.
3. One of the dependencies used in this API (epac-data-sim) is a private repository, so the appropriate permissions and SSH keys need to be set up. Guidance for setting up SSH keys for the Rocky 9 VM can be found here.

Environment Setup

Python

See commands below to install the development tools for specific operating systems. Others will need the same libraries, but exact names may differ.

Rocky 9

sudo dnf install "@Development Tools" python3.11 python3.11-pip python3.11-setuptools python3.11-devel openldap-devel swig gcc gcc-c++ openssl-devel cmake3 libuv libuuid-devel git

Debian 12

sudo agt-get update
sudo apt-get install build-essential curl python3.11 python3.11-venv python3.11-dev python3-pip libldap2-dev libsasl2-dev cmake uuid-dev git

Clone the codebase

git clone https://github.com/ral-facilities/operationsgateway-api.git

Poetry

Poetry is used to manage the dependencies of this API.

# Install poetry
curl -sSL https://install.python-poetry.org | python3 -
# Change the directory to where the poetry .toml file is
cd operationsgateway-api
# Set poetry to use python 3.11
poetry env use python3.11
# Install the dependencies
poetry install
# If you don't need the simulated data, or don't have access to the epac repo...
poetry install --without simulated-data

MongoDB:

Create the following file:

sudo vi /etc/yum.repos.d/mongodb-org-8.0.repo

Add:

[mongodb-org-8.0]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/9/mongodb-org/8.0/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://pgp.mongodb.com/server-8.0.asc

Then:

# Install MongoDB
sudo yum install mongodb-org
# Start it
sudo systemctl start mongod
# Enable it to start on boot
sudo systemctl enable mongod

For instructions on installation on specific operating systems:

https://www.mongodb.com/docs/manual/administration/install-on-linux/

MongoDB Indexes:

The following Indexes are uses and need to be set up on local, dev & prod databases.

Using mongosh, select the database: use opsgateway

Then run the following command:

# Multiple users can have the same session name, but a user can't have two sessions with the same name.
db.sessions.createIndex(
  { username: 1, name: 1 },
  { unique: true }
);

Authentication

The authentication system uses JSON Web Tokens (JWTs), which require a private and public key pair to encrypt and decrypt the tokens.

The keys can be generated by navigating to a directory and typing the command:

# When prompted for the location path, enter `id_rsa` to create the keys in the current directory rather than in your home directory.
# Press enter twice when prompted for the password so as not to set one.
ssh-keygen -b 2048 -t rsa

Then, edit the private_key_path and public_key_path settings in the auth section of the config.yml file to reflect the location where these keys have been created.

API Configuration

In operationsgateway_api/, there are several example configuration files which need to be reviewed and renamed, removing the .example part of the filename.

• config.yml.example
• logging.ini.example
• maintenance.json.example
• scheduled_maintenance.json.example

Backup

If using the tape backup feature (enabled by the backup section in config.yml) then authentication for XRootD will need to be configured. XRootD uses environment variables to define the credentials used to authenticate to the server. These can either be explicitly defined in the session as environment variables or in config.yml, which case this will be set as an environment variable on start up. Note that files containing keys need to be only readable/writable by the user, so it may be necessary to manually configure this:

chmod 600 /path/to/.keytab
export XrdSecPROTOCOL=sss
export XrdSecSSSKT=/path/to/.keytab

Test Data

A script has been created to set up the API with some test data and test users. This script also comes with a configuration file that needs to be reviewed and renamed at util/realistic_data/config.yml.example. Note that the example comes pre-populated with example values for epac and gemini style test data, affecting the source and destination for the test data. The ingest will run automatically as part of pytest via the fixtures (see Testing), but it can also be run explicitly using Poetry:

poetry run python util/realistic_data/ingest_echo_data.py

More details about this script can be found in docs/developer/test_data.md

Starting the API

poetry run python operationsgateway_api/src/main.py

Assuming the default configuration, the API will exist on 127.0.0.1:8000.

The port forwarding service on most IDEs will allow you to access the API on the remote VM through your local web browser. Then, you can visit /docs in a browser, which will give an OpenAPI interface detailing each of the endpoints.

Alternatively, you can use the Postman requests in this repo. This gives a more user-friendly way of interacting with the API. Certain environments and scripts have been set up in Postman to help with using the API. For example, before any call is made to the API, the /login call can be used to get and store a JWT, which will be used automatically in the auth header of following calls to the API.

Testing

While initially developed for a single facility (EPAC), the API is now designed to support multiple facilities with specific anticipated behaviour. This behaviour is controlled by configuration settings. These settings should be applied both during ingest of test data and while the tests are run. For ease, there is a "session" scoped fixture which ingests the test data, but only if the database is empty. This means when developing locally, the first execution of the tests may hang for around 4 minutes while ingestion occurs, but subsequent executions will start testing immediately.

By default, the "usual" location of the configuration at operationsgateway_api/config.yml will apply (as it does when running the API locally). However, pytest can be configured to override these settings using the pytest-env plugin.

pytest.ini

To support test suites for EPAC and Gemini style data, test/pytest_epac.ini.example and test/pytest_gemini.ini.example should be copied to test/pytest_epac.ini and test/pytest_gemini.ini respectively. While the majority of the settings can then be used as is, it will also be necessary to choose bucket name(s) (i.e. replace yourname) and create a bucket and Mongo database with the corresponding name(s) before the tests will run. As mentioned above, it is not necessary to manually ingest the data if the database starts empty.

Note that because of how Pydantic parses environment variables, settings in the env section of pytest.ini will always take precedence over the config.yml. If one of the env settings is commented out (;), or removed, then whatever setting is in config.yml will be used as the next highest priority source. Conversely, if a new field is added the env, this will then override the corresponding setting in config.yml. There are therefore multiple ways to make config changes for the tests, and the appropriate method to use will depend on the circumstances.

nox

Like the DataGateway API, this repository contains a Nox file (noxfile.py), which exists at the root level of this repository.

To install Nox, use the following command:

python3.11 -m pip install --user --upgrade nox

To run a specific Nox session, use the following:

nox -s [SESSION NAME]

The following Nox sessions have been created:

• black - This uses Black to format Python code in a pre-defined style.
• lint - This uses flake8 with a number of additional plugins to lint the code to keep it Pythonic.
• safety - This uses safety to check the dependencies (pulled directly from Poetry) for any known vulnerabilities.
• tests - This uses pytest to execute the automated tests in test/. Note that this runs without any additional configuration, so will use whatever values are in the locally defined config plus the default pytest.ini overrides.
• tests_epac - This uses pytest to execute the automated tests in test/. Note that this expects pytest_epac.ini to be defined with appropriate settings (i.e. APP__USE_SUB_SECOND_TIMESTAMPS=False)
• tests_gemini - This uses pytest to execute the automated tests in test/. Note that this expects pytest_gemini.ini to be defined with appropriate settings (i.e. APP__USE_SUB_SECOND_TIMESTAMPS=True)