HabitatMech

Knowledge base of microbial habitats and environments, harmonized from the four source vocabularies that describe where microbes are found and grounded in ENVO / UBERON / FOODON / BTO.

HabitatMech is the habitat counterpart of TraitMech (traits), CultureMech (growth media), MediaIngredientMech (ingredients), and CommunityMech (communities), and follows the curation pattern established by dismech: one YAML per entity, ontology-grounded, evidence-backed, schema-validated, curated incrementally.

Browse the corpus online → — every record, browsable by category, plus the ENVO term requests this project is asking the ontology community for.

The problem it solves

The same habitat has different names depending on who is describing it:

Source	How it names marine sediment
JGI GOLD	Environmental > Aquatic > Marine > Sediment (5-level ecosystem path)
BacDive	Marine-sediment (flat isolation-source label)
PREGO	ENVO:00002113 (ontology CURIE)
Madin et al.	ENVO:00002113 (ontology CURIE, literature-curated)

Each becomes a source concept. Every source concept is resolved to an identifier — an ontology CURIE where one is defensible, otherwise a minted, content-hashed habitatmech: CURIE — and source concepts resolving to the same identifier merge into one HabitatRecord carrying all their attestations.

That merge is the product. data/habitats/terrestrial/soil.yaml is one record grounded in ENVO:00001998 that retains each source's assertion volume, Madin's independent literature curation, and kg-microbe's environmental parameters rather than flattening them into one unsupported claim.

Current corpus

3,206 habitat records are currently committed.

Category	Records		Grounding	Records
HOST_ASSOCIATED	1,649		EXACT	1,060
ENGINEERED	487		UNGROUNDED	953
AQUATIC	468		NARROW	985
TERRESTRIAL	353		NOT_APPLICABLE	102
OTHER	178		CLOSE	93
FOOD	61		BROAD	13
AIR	7
CLINICAL	3

176 records are attested by at least two sources; 16 are attested by four or more.

686 records (21.4%) are REVIEWED; the remaining 2,520 are SEEDED.

Run just report for the detailed live report. Class-level decisions deliberately do not count as reviewed; see Curation and docs/HARMONIZATION.md.

Quick start

Development uses Python 3.13 via .python-version; CI selects the same minor explicitly and runs the full quality gate once. The package compatibility floor remains declared in pyproject.toml.

just install                        # uv sync --extra dev
just report                         # corpus stats: grounding, categories, backlog
just validate-all                   # closed-mode schema validation of every record
just verify-corpus                  # check data/habitats/ is what data/raw/ produces
just test                           # unit + corpus-integrity tests
just render                         # regenerate the site under pages/

Re-seeding is only needed when the upstream data changes:

just extract-inventory-dry          # what extraction would produce (no writes)
just extract-inventory              # refresh data/raw/ from a kg-microbe checkout
just seed                           # dry-run: harmonization report, no writes
just seed-canary ENVO:00001998      # write ONE record and check it, first
just seed-apply --force             # rewrite the corpus
just seed-apply --force --prune     # ...and clean up files left by a category move

just extract-inventory is the only step that needs a local kg-microbe checkout; point KG_MICROBE_ROOT or conf/sources.yaml at it. The derived inventories in data/raw/ are committed, so seeding, validation, and tests run without it. MANIFEST.yaml covers kg-microbe extraction; GOLD_MANIFEST.yaml covers the later GOLD bulk-export and API inventories.

Filenames are pinned by data/habitats/PATHS.tsv (identifier → slug), so a re-seed never renames an existing record just because the corpus grew around it. To rename a record, edit its slug there and re-seed — never rename the file directly, or the seeder will recreate it under the pinned name and two files will claim one identifier.

Schema

src/habitatmech/schema/habitatmech.yaml defines HabitatRecord, one per YAML file:

• Identity — identifier, label, definition, definition_source, synonyms, parent_habitats, xrefs, habitat_category.
• source_attestations — the harmonization layer. One entry per upstream vocabulary, with source_id, source_label, GOLD's full source_path, the mapping_predicate relating it to this record, and assertion_count paired with assertion_unit (GOLD counts organisms, BacDive strains, PREGO taxa — the numbers are not summable across sources).
• environmental_parameters — physicochemical bands (salinity, pH, temperature, pressure, water availability, and their variability).
• characteristic_taxa — associated taxa with counts and scores.
• causal_graphs — the "Mech" half: evidence-backed mechanism graphs linking a habitat's conditions to the adaptations they select for. Unlike the seeded descriptive fields, every causal edge must carry a citation.
• grounding_status and mapping_status, kept deliberately separate: the first is how well the identifier fits, the second is curatorial review. A record can be REVIEWED and still honestly UNGROUNDED.
• discussions / datasets from the shared mech_shared module, vendored byte-identical across the Mech repos.

How grounding works

PREGO concepts are already ENVO/BTO CURIEs and ground to themselves.

BacDive concepts go through kg-microbe's curated isolation_source_to_ontology.tsv, which has a row for all 162 sources. An empty target is an upstream curator's deliberate refusal to ground, and is honoured as UNGROUNDED rather than re-guessed. A target in a non-habitat ontology is kept as an xref and marked NOT_APPLICABLE — "Acidic" maps to PATO:0001429, and a quality is a property of a habitat, not a habitat.

GOLD has no upstream mapping table, so its concepts are matched lexically against the vendored ontology labels and synonyms, in this order:

1. the composed label from the last two path levels ("marine sediment") — an exact hit is EXACT, because the path context is included;
2. the leaf label alone, when this path is the shallowest one ending in that leaf. Ten GOLD paths end in "Soil"; the depth-3 Environmental > Terrestrial > Soil is what ENVO means by soil, so it claims ENVO:00001998 and the rest do not;
3. the leaf label when another path claims it, or when several tie at the shallowest depth — the concept gets a minted identifier, NARROW grounding, and the matched term as a parent. This is the anti-conflation rule: ...Marine > Sediment and ...Freshwater > Sediment are different habitats, and grounding both to ENVO:00002007 would merge their attestations into one record;
4. the isolation-source mapping table, keyed on the leaf label;
5. nothing — minted identifier, UNGROUNDED.

just report ranks the ungrounded records by upstream assertion volume and splits them into curator-confirmed term requests and undecided backlog. Those rankings change as the corpus is curated, so the live report is their source of truth rather than a copied list here.

Curation

Records are generated, and just verify-corpus gates that they reproduce exactly from data/raw/ — so curation is never a hand-edit to a record, which the next re-seed would silently revert. Decisions live in curation/decisions.tsv, which the seeder reads as an input. A curation pass is therefore a small reviewable diff in one file, every decision carries its curator, date and reason, and the corpus stays reproducible.

Five decisions are available, each keyed on the minted identifier of one source concept (a content hash of the GOLD path or BacDive id, so it survives an upstream refresh):

Decision	Meaning
GROUND	Redirect this source concept onto an ontology term. It merges with anything else resolving there.
NOT_APPLICABLE	Not a habitat at all — a disease, quality, process or procedure. Keeps its minted id so it stays citable. A host is a habitat; its taxon term is an xref, not the habitat identity.
CONFIRM_UNGROUNDED	A real habitat with no term that fits. May name a nearest-broader term, attached as a parent rather than adopted as identity. This is the ENVO term-request list.
GROUND_AS_PARENT	Narrower than a term: keep the minted identity, record the term as a parent, mark the grounding NARROW. The curated form of the ambiguous-leaf rule.
REVIEW	The curator checked the seeder's own answer and endorsed it.

Each decision also carries a review_depth: ITEM means this concept was examined against its source path and candidate terms; CLASS means it was decided as a member of a mechanically-defined group (for instance "no term in the vendored slice matches this label by any search route"). Only ITEM decisions promote a record to REVIEWED — without that distinction a bulk sweep would report the corpus as reviewed when nobody had read a line of it. A grounding can never be CLASS depth: asserting an equivalence about one concept is always a per-item judgement.

The host/anatomy distinction, MIxS triad roles, and evidence requirements are documented in docs/CURATION.md.

Every target is verified at seed time. A GROUND must name both the CURIE and the label it expects, and the seed fails unless the term exists in the vendored ontology slice and its label matches. An invented term ID cannot pass, and neither can a real ID paired with the wrong concept — which is the failure mode that matters when a curation pass is LLM-assisted.

A merged record is REVIEWED only when every source concept feeding it has been decided. A multi-source record is not checked until every contributing source concept has been looked at.

just worklist                 # the backlog, ranked, with candidate terms
just report                   # term requests vs undecided, and the numbers below

Known limitations

These are real and unfixed; see the issue tracker.

• Most records are unreviewed. SEEDED records have plausible but unverified lexical matches; just report gives the live reviewed and seeded counts. Every multi-source record has been read, and so has every upstream mapping the label check cannot see. What remains splits by the method each part needs: class-swept concepts are individually readable and ranked by volume in just worklist; the EXACT cohort was sampled rather than read, at 0 of 40 wrong (95% CI 0–8.8%), because reading them all is 40-plus judgements per defect; and the NARROW cohort has no screen, because a parent legitimately has a different label from its child and a label-difference screen flags 155 records of which a fair sample was 12 out of 12 correct.
• ENVO has no host-clade environment terms. "Mammals: Human", "Birds", "Fish", "Insects" and the rest are real habitats with only ENVO:01001002 animal-associated environment above them. They are deliberately not grounded there: every host clade would merge onto one record and the host distinction is the entire content. They are the highest-value term requests.
• PREGO's taxon ranking is weak, though measurably not arbitrary. For soil all 8,715 taxa score between 4.000 and 4.007. Cross-checking PREGO's top-25 against two independent routes to taxa gives 2.48x enrichment over chance against BacDive and 2.22x against Madin, so the ordering carries signal, and the alternatives #8 proposed have 4 distinct values against the score's 2,869. Every taxon therefore carries rank and candidate_pool so the claim states its own strength, and entries corroborated across sources are listed first. Seeded taxa still mean "reported from", not "characteristic of" — is_characteristic remains a separate curator-set flag. See docs/HARMONIZATION.md.
• Environmental parameters remain sparse. Compound upstream rows (sediment_marine_cold = ENVO sediment + PATO cold) are skipped rather than misattributed to a term that denotes only one component.
• PCO is not vendored (kg-microbe does not ship it). PO now is, parsed from po.owl.
• OTHER remains a curation bucket for records whose category the ENVO-anchor heuristic cannot infer.

Layout

HabitatMech/
├── conf/sources.yaml                     # where kg-microbe lives
├── data/
│   ├── raw/                              # inventories + provenance manifests
│   └── habitats/
│       ├── PATHS.tsv                     # identifier -> slug, pins filenames
│       └── <category>/<slug>.yaml        # generated HabitatRecords
├── history/                              # append-only curation-session records
├── src/habitatmech/
│   ├── seed.py                             # harmonization library + CLI
│   ├── extract.py                          # inventory extraction library + CLI
│   ├── report.py                           # corpus reporting library + CLI
│   ├── proposals.py                        # conservative decision proposals
│   ├── sampling.py                         # reproducible curation sampling
│   ├── schema/habitatmech.yaml           # LinkML schema
│   ├── schema/mech_shared.yaml           # vendored, sha-pinned shared module
│   ├── schema/history.yaml               # vendored schema for history/ records
│   ├── validation/write_validated.py     # write-time closed-schema gate
│   └── curate/curation_event.py          # append-only audit trail helper
├── scripts/
│   ├── extract_source_inventory.py       # compatibility CLI wrapper
│   ├── seed_from_sources.py              # compatibility CLI wrapper
│   ├── validate_strict.py                # closed-mode corpus validation
│   └── habitat_report.py                 # corpus stats and curation backlog
└── tests/

Sources

• GOLD — JGI Genomes OnLine Database ecosystem classification
• BacDive — DSMZ BacDive isolation sources
• PREGO — PREGO habitat-organism associations
• Madin et al. — prokaryotic phenotypic trait compilation, literature-curated at species level; its isolation_source column is the habitat vocabulary
• ENVO / UBERON / FOODON / BTO — via kg-microbe, which supplies all of the above in harmonized KGX form and contributes the curated isolation-source mapping table

Contributing

See the native merge queue guide for PR checks, queue validation, and recovery when a queued change fails.

License

CC0-1.0. See LICENSE.