Add pandas 3 support

[bryan-harter]

In https://github.com/IMAP-Science-Operations-Center/imap_processing/pull/3382/changes, there was a workaround to deal with a special type of pandas data. Is that something that should be handled in cdflib?

Maybe it is just the case that we try to convert everything to numpy if we don't receive a numpy array?

[leowerneck]

@bryan-harter Upon further investigation, this appears to be a bad interaction between Pandas' inferred string dtype and xarray's representation of that dtype, which appears in certain 2025.x.x versions of xarray. One conservative option would be to require xarray>=2026.1, since official Pandas 3 compatibility was introduced in xarray 2026.1.0. This would ensure compatibility with pandas>=2,<4 without modifying cdflib.

Requiring xarray>=2025.6.1 may be sufficient to address this particular failure, but I haven't tested it exhaustively.

Tests

Consider this minimal example script minimal.py:

from pathlib import Path

import pandas as pd
import xarray as xr
from cdflib.xarray import xarray_to_cdf

pdv = pd.__version__
xrv = xr.__version__

labels = pd.Series(["alpha", "beta"])
dataset = xr.Dataset({"label": ("record", labels)})
print(f"Original label dtype is: {dataset['label'].dtype}")

outfile = Path("tmp.cdf")

try:
    xarray_to_cdf(dataset, outfile, istp=False)
    print(f"xarray_to_cdf SUCCESS: Pandas v{pdv}; xarray v{xrv}")
except Exception as error:
    print(f"Direct call failed with {type(error).__name__}: {error}")
    print(f"xarray_to_cdf FAILED: Pandas v{pdv}; xarray v{xrv}")

outfile.unlink(missing_ok=True)

Starting with Pandas 2, if I run:

uv run --no-project --with 'cdflib==1.3.2' --with 'pandas>=2,<3' --with 'xarray==2025.1.2' minimal.py

I get:

Original label dtype is: object
xarray_to_cdf SUCCESS: Pandas v2.3.3; xarray v2025.1.2

The same thing happens if I run:

uv run --no-project --with 'cdflib==1.3.2' --with 'pandas>=2,<3' --with 'xarray==2026.1.0' minimal.py

I get:

Original label dtype is: object
xarray_to_cdf SUCCESS: Pandas v2.3.3; xarray v2026.1.0

When I switch to Pandas 3 and run:

uv run --no-project --with 'cdflib==1.3.2' --with 'pandas>=3,<4' --with 'xarray==2025.1.2' minimal.py

I get:

Original label dtype is: str
NOT SUPPORTED: Data in variable label has data type of str.
NOT SUPPORTED: Data in variable label has data type of str.
Direct call failed with ValueError: 'values' must be a NumPy array, not list
xarray_to_cdf FAILED: Pandas v3.0.5; xarray v2025.1.2

But if I run:

uv run --no-project --with 'cdflib==1.3.2' --with 'pandas>=3,<4' --with 'xarray==2026.1.0' minimal.py

I get:

Original label dtype is: object
xarray_to_cdf SUCCESS: Pandas v3.0.5; xarray v2026.1.0

[bryan-harter]

Interesting, ok I think it is fine to require 2026.1.0 going forward!

If you'd be willing to test it, you can run tests with real CDF files with "pytest --remote-data". It can take ~10 minutes to run through all the CDF files, but at least it ensures that everything still works with real CDFs.

[leowerneck]

On it!

[leowerneck]

@bryan-harter Could you please take a look at #343 (and the corresponding PR #344) when you get a chance? Running the command you suggested locally resulted in errors. Then I noticed GitHub Actions' "Remote Data Tests" are failing since June 22, so I decided to investigate a bit further.