Skip to content

[Feature]: Support for array-typed fields in extraction configuration #4

Description

@damienien

Problem to Solve

The current extraction engine only supports scalar values (STRING, DECIMAL) as targetType in searchFields. There is no way to extract fields whose value is an array of objects, even when the number of elements is unknown at extraction time.

Proposed Solution

The extraction configuration should support a new targetType such as ARRAY or OBJECT_ARRAY, allowing the extractor to capture all elements of a typed array and store them as a JSON array in search_data.

Example proposed config:

{
  "fieldName": "materialsProvenance",
  "targetType": "ARRAY"
}

With a corresponding strategy entry (e.g. in unknownOntology):

"materialsProvenance": {
  "typeHints": ["Material"],
  "collectAll": true
}

The resulting search_data entry would be:

"materialsProvenance": [
  { "materialName": "...", "originCountry": "...", "concentration": 100 },
  { "materialName": "...", "originCountry": "...", "concentration": 100 }
]

Priority

High (blocking for my use case)

Use Case

When extracting data from a Battery Digital Product Passport, some fields could be an array of objects:
example:

"dpp:materialsProvenance": [
  {
    "@type": "dpp:Material",
    "dpp:materialName": "...",
    "dpp:originCountry": "...",
    "dpp:certifiedContent": true,
    "dpp:materialType": "...",
    "dpp:concentration": 100
  },
  {
    "@type": "dpp:Material",
    "dpp:materialName": "...",
    "dpp:originCountry": "...",
    "dpp:certifiedContent": false,
    "dpp:materialType": "...",
    "dpp:concentration": 100
  }
]

The number of elements in the array is unknown at configuration time and can vary per DPP. There is currently no targetType or extraction strategy that can handle this structure.

Workaround

Currently the only workaround is to serialize the array as a concatenated STRING at the provider level, which loses structure and makes filtering/rendering on the client side much harder.

Technical Notes (optional)

- Extractor version: `1.0.0-pgsql-oidc`
- DPP format: JSON-LD with `@context` referencing UNTP/DPPO/BattINFO vocabularies
- Strategy affected: `unknownOntology` (and likely `noOntology`)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions