Collect metadata for other online databases

[fccoelho]

Currently the metadata folder only has metadata for SINAN, we need to collect and make available metadata for other databases.

To implement for other databases we can follow the pattern used for SINAN.

[EasonYang-666]

Hi! I’d like to contribute to this issue. In the current 2.x codebase, the legacy pysus/data/metadata directory does not appear to be used at runtime; dataset metadata is now handled in the API modules. Could you clarify the intended 2.x location/format for this work, and which database would be the best first target?

[fccoelho]

@luabida can you help here?

[luabida]

Hello @EasonYang-666. In the v2 of PySUS, the files in the metadata directory are planned to be transcribed into the s3 catalog. The most important information we need from the metadata is the description of the columns present in the files, so we can create a correlation between shared and specific columns from the datasets. These definitions are not centralized nor normalized in the DATASUS, they may appear in different file formats, such as PDFs, JSONs, CSVs, XMLs in the DATASUS FTP or API.

Any reference you can find that describes that field, or even a methodological description, are welcome. Feel free to save it in the format that fits you the best, but we recommend it to be saved in formats that can be easily managed, such as JSON or CSV, because it will be later parsed into a duckdb file that can be accessed using SQL, more specifically with these specs: https://github.com/AlertaDengue/PySUS/blob/main/pysus/api/ducklake/catalog/orm/columns.py#L9

[Karthikeyadusi]

Hi! I'd like to contribute to this issue. Is there a specific DATASUS database you'd recommend as the first target, and should I follow the existing SINAN metadata structure or the newer v2/S3 catalog format?

[Tanvinaik09]

Hi! I'd like to work on this issue as my first contribution to PySUS. I plan to follow the existing SINAN metadata pattern and add metadata for another database while keeping the current structure consistent. Is this issue still available?