[Tracking] SearchSpace distillation coverage and knowledge consolidation #91

Activity

1. Dong1017

   added

   kind:trackingTracking umbrella that follows child workTracking umbrella that follows child work

   area:knowledgeKnowledge line: packs, skill library, knowledge admissionKnowledge line: packs, skill library, knowledge admission

   on Oct 22, 2026
2. Dong1017 commented on Oct 22, 2026

   

   Dong1017

   on Oct 22, 2026 – with ChatGPT Codex Connector

   CollaboratorAuthor

   More actions

   Suggested execution order

   Prioritize bounded gaps with existing source evidence and canonical owners before descriptor-only intake. These are recommendations, not blanket execution or paid-evaluation authorization.

   Track	Suggested order	Why
   Incremental distillation	1. fastsafetensors TP transfer → 2. awesome-ascend-skills host KV / HCCL → 3. DLRover persistence / replica transport → 4. AKG bounded compiler mechanism → 5. afd-plugin / vllm-rlt preflight	Start with identified gaps and reusable owners; unpinned descriptor-only candidates require source selection before extraction
   Knowledge consolidation	1. llama.cpp → 2. vLLM-Ascend → 3. Diffusers → 4. SGLang → 5. vLLM / vLLM-Omni → 6. TensorRT-LLM / PyTorch	Reconcile multi-pin applicability and navigation; this is an overlap-review order, not a claim that all pairs conflict
   Paper recovery	Recover/check the recorded missing archive first; then prioritize PagedAttention / Sarathi-Serve → DistServe / Mooncake → ZeRO-Infinity / FlexGen according to the selected task	Avoid redoing existing extraction; exact versions and readsets must be established before crediting coverage

   Link with #88 without another registry

   flowchart LR
       A["Tracker gap or consolidation"] --> B["Bounded delivery PR"]
       B --> C["Skill Eval plan"]
       C --> D["Authorized run and evidence"]
       D --> E["Update tracker row"]
   

   Loading

   • This tracker answers what source/scope and where. Skill Eval v1.1: automated evaluation for skill additions and updates #88 delivers tools answering what changed in task behavior.
   • New skill: absent/new comparison. Updated skill or behavioral consolidation: old/new comparison, retaining meaningful old-version regression cases. Pure index/link/provenance changes: deterministic integrity checks; no mandatory paid agent run.
   • Start the first joint pilot with one bounded llama.cpp consolidation only if overlap review identifies an actual skill-content change. Do not invent a rewrite to manufacture an eval demonstration.
   • Until Skill Eval v1.1: automated evaluation for skill additions and updates #88's comparison/PR automation ships, keep evaluation manual or mark not evaluated/uncovered. Offline Fake-client tests validate the evaluation tool, not skill efficacy.
   • Update the existing row with delivery/eval evidence and remaining limits. A landed extraction can remain candidate/unvalidated; neither “merged” nor “no observed regression” proves added benefit.
3. david6666666

   closed this as completedin #114on Oct 23, 2026
4. Dong1017

   reopened this on Oct 23, 2026
5. Dong1017 commented on Oct 23, 2026

   

   Dong1017

   on Oct 23, 2026 – with ChatGPT Codex Connector

   CollaboratorAuthor

   More actions

   Reopened at the owner's request. This is a living SearchSpace coverage/consolidation tracker, not the acceptance issue for a single distillation PR. A merged scoped delivery updates its relevant row; it does not complete the whole tracker. Please reference this issue with Refs #91 / Related to #91, rather than closing keywords. Pinning is for visibility, not protection against closure.
6. Dong1017

   pinned this issue on Oct 23, 2026
7. hsliuustc0106

   added a commit that references this issue on Oct 24, 2026

   docs: clarify vLLM-Ascend guidance versions and responsibilities (#130)

   ...

   e6bedfa
8. Dong1017

   added sub-issues on Oct 25, 2026

   • [Task] Distill Flare preparation contracts and session-state lifetimes #148
   • [Task] Distill LoopX goal authority and quota decision boundaries #149
9. Dong1017 commented on Oct 27, 2026

   

   Dong1017

   on Oct 27, 2026

   CollaboratorAuthor

   More actions

   Skill → coverage reverse view, Round 1

   Baseline: main 6b13cf00ccf2e98fcb3144eda69e2442bcb1d652 (2026-10-27).
   Delivery: #194.

   Counting unit. One row per registered skill ID (60 at that revision). The
   21 other SKILL.md documents — five engine entrypoints, their fifteen
   sub-skills, and workflows/inference-engine-research — are listed in their own
   section with a kind and excluded from every count. The historical 41-directory
   figure at the end of the SearchSpace Matrix stays a historical record and is
   not a coverage denominator.

   Scope this round. ID, canonical path, backend, declared task from the
   SKILL.md frontmatter, status/validation/readsets from the sidecar, SearchSpace
   resource layer and lifecycle stage, Reference Matrix section and source,
   and evaluation selection from frozen plans. Anything not read is unknown;
   mechanism, prerequisites and explicit non-goals wait for Round 2.

   Current snapshot, not a capability claim. 60 registered; 47 recorded on a
   resource layer and 24 on two or more; 55 in the Reference Matrix; 55 with a
   sidecar; 2 selected by a frozen plan (inference-benchmark-design,
   dense-performance-evidence in skill-eval-v1-20261021). Thirteen have no
   resource-layer row, five have neither a sidecar nor a Reference Matrix row, and
   58 have no evaluation linkage. Recording the absence is the point; it does not
   assert a gap in the underlying knowledge.

   Mechanics. tools/build_skill_coverage.py is the only writer and
   --check diffs a fresh build against the committed file, so drift is
   detectable and the view stays a projection rather than a third maintained
   matrix. It records an input digest over the files it reads instead of a commit
   SHA, so an unrelated commit does not force regeneration.

   No skill body, sidecar, matrix or maturity was changed; no evaluation, GPU or
   acceptance-rule work was done. Round 2 batches will be chosen from the recorded
   gaps, starting with the five workflow sidecars.

   Refs #91
10. Dong1017

    added 4 commits that reference this issue on Oct 27, 2026

    docs: fix coverage view portability, guidance mappings and state test

    ...

    3e08b15

    docs: regenerate the coverage view on current main

    ...

    54dfee5

    docs: make the coverage digest order stable across platforms

    ...

    b49679e

    test: build the expected digest instead of comparing two calls

    ...

    df5d79f
11. 2 remaining items

    Load more

12. Dong1017 commented on Oct 28, 2026

    

    Dong1017

    on Oct 28, 2026 – with ChatGPT Codex Connector

    CollaboratorAuthor

    More actions

    Skill-side reverse view delivered by #194, merged as 70adb9f0a963756d040100a25cc708787efa249e: generated coverage view. It projects existing registry/body/metadata/matrix/evaluation-plan inputs, including #199, rather than creating a new source of truth.

    Incremental review and Ready four jobs passed. Snapshot: 65 registered skills / 95 SKILL.md documents; inventory coverage is not efficacy or maturity promotion. New relevant inputs require regeneration; unrelated main commits do not.

    flowchart LR
     A[Existing source owners and matrices] --> B[Generated skill-side view]
     B --> C[Human navigation and explicit gaps]
     C -. no efficacy promotion .-> D[Separate evaluation and acceptance]
    

    Loading

    #91 remains the living tracker; this closes neither its ongoing intake nor the uncovered knowledge/evaluation gaps.
13. Dong1017 commented on Oct 28, 2026

    

    Dong1017

    on Oct 28, 2026

    CollaboratorAuthor

    More actions

    Round 2 scope and plan

    Round 1 (merged in #194, generator tools/build_skill_coverage.py) built the
    skeleton: every registered id has a row, each row repeats what its owner
    already states, and every field the inventory could not answer says unknown.
    Round 2 fills content into that skeleton. It does not add a second
    authority, a dashboard, or an eval platform.

    All counts below were read at upstream/main = 6e19e680 on 2026-10-28.

    Counting unit

    Registered skill ids only: 67 in src/inferrsi/data/skill_library/skill-index.json.

    Measure	Value
    Registered ids	67
    frameworks / hardware / methods / repositories / workflows	3 / 22 / 18 / 18 / 6
    common / ascend / cuda	49 / 14 / 4
    metadata.json sidecar present	62
    Sidecar with a non-empty applicability	22 of 62
    Unregistered SKILL.md documents	33 (sub-skills, engine entrypoints, research router)

    The 33 unregistered documents stay excluded. One engine entrypoint routes to
    sub-skills below it; counting both would inflate coverage without adding a
    procedure. This is the same boundary Round 1 set.

    The four content gaps Round 2 closes

    Gap	Scope	What it actually is
    no-sidecar	5	cite-check-downstream-pr, claim-contract, competitor-scan, draft-acceptance-issue, rsi-evidence ship SKILL.md and no metadata.json. Their view row reads unknown · unknown · 0 readset(s).
    Empty applicability	40	22 of 62 sidecars declare one. The other 40 render as blank conditions, so a reader cannot tell "applies everywhere" from "not yet analyzed".
    Generator-owned recommendations	67	All 67 records carry the same two sentences, emitted at tools/build_skill_index.py:131-134. No sidecar contributes. The field is currently a constant, not a recommendation.
    no-SearchSpace-layer	13	Two genuinely look layered (ascend-video-diffusion-optimization, audit-recurrent-kv-finalization); the rest are repository-level, method or process skills.

    no-Reference-Matrix-row (6) overlaps five of these and needs no separate pass.
    no-eval (65) belongs to Round 3 and is out of scope here.

    Contract decisions this round must settle

    These are the parts that need agreement before batch work, because they change
    what every later batch writes:

    1. Workflow sidecar shape. The 12-field record was designed for hardware
       experience. applicability.hardware, prerequisites and backend scoping are
       near-meaningless for cite-check-downstream-pr. Round 2 opens with a
       workflow-specific shape covering input contract, produced artifact, failure
       modes and the point at which the workflow refuses, then writes the 5 sidecars
       against it.
    2. applicability is descriptive, never evidential. Existing shapes in use
       are scope, hardware, prerequisites (ascend-video-diffusion-optimization),
       and scope alone (the frameworks entries). No schema is enforced today, so
       Round 2 fixes the allowed keys and their meaning rather than letting 40 files
       drift into 40 dialects. Filling it states the conditions a procedure assumes.
       It does not raise verification.locally_verified, which stays false, and it
       does not touch status or empirical_validation (currently 62 candidate,
       59 not_run, 3 partial_cpu_only, 5 unset).
    3. Generator-owned fields. recommendations is constant and four of the
       limitations entries are generator-owned with sidecar entries appended. Round 2
       decides whether a sidecar may supply its own recommendations and how the two
       combine. Leaving it as is would mean the view claims a field it does not
       really carry.
    4. not-applicable versus blank. For a skill that genuinely has no hardware
       resource layer, the view must say so. Blank stays reserved for "not analyzed",
       or the next reader re-derives this analysis from scratch. Each of the 13 gets a
       per-skill decision from its body, not a category default.

    Batches

    One PR per batch, in this order. Later batches reuse the shape fixed by batch 1.

    Batch	Content	Files touched
    1	Workflow sidecar shape, plus the 5 workflow sidecars	5 new metadata.json, any generator change for decision 3
    2	applicability + recommendations for methods/	18 sidecars
    3	applicability + recommendations for hardware/	22 sidecars
    4	applicability + recommendations for repositories/ and frameworks/	21 sidecars
    5	Layer decisions for the 13 no-SearchSpace-layer skills	matrix rows or an explicit not-applicable marker

    Batches 2 through 5 are independent of each other and could be reordered. Batch 1
    is not: it settles decisions 1 and 3.

    Acceptance per batch

    • python tools/build_skill_index.py --check and
      python tools/build_skill_coverage.py --check both pass, and
      docs/skill-coverage.md is regenerated and committed with its sources in the
      same commit. The Round 1 contract already requires this; a batch that edits a
      sidecar without regenerating fails CI.
    • No verification, status, empirical_validation or sources field changes
      in a batch that only fills applicability. A reviewer can check this as a
      mechanical diff filter, not by re-reading 18 files.
    • Every filled condition cites where it came from: the skill body, a pinned
      source, or an owner-local case. A condition with no origin is a guess and is
      refused in review.
    • Negative and missing results are kept. If a procedure has no known working
      configuration, the field says that instead of staying blank.

    Explicitly not in Round 2

    • No eval result, no acceptance verdict, no promotion. no-eval stays on 65 rows.
    • No new registry, matrix, dashboard or scoring model.
    • No edit to data/repo_packs/inferrsi/knowledge/. That tree is generated; the
      sources are skill_library/.
    • No change to the 33 unregistered documents' status.

    Open questions for review

    1. Is a workflow-specific sidecar shape acceptable, or should the existing 12-field
       record be reused with applicability reinterpreted for process skills?
    2. Should recommendations stay generator-owned across all skills, or become a
       sidecar field that the generator only defaults when absent?
    3. Batch 5 changes matrix rows, which are hand-maintained. Is a matrix row the
       right home for the layer decision, or should the not-applicable statement
       live in the sidecar so the matrix keeps only real layer coverage?

    Round 1 delivery stays as merged. This comment is the Round 2 plan; batch 1 opens
    only after the three questions above are settled.
14. Dong1017 commented on Oct 28, 2026

    

    Dong1017

    on Oct 28, 2026 – with ChatGPT Codex Connector

    CollaboratorAuthor

    More actions

    Reviewed Round 2 plan. Direction looks sound; these are the recommended contract choices before batch 1:

    1. Reuse the existing sidecar envelope, adding typed workflow input/output/failure/refusal conditions. Do not create a parallel record system or invent hardware applicability. Include the existing npu-skill-factory sidecar in compatibility checks, or explicitly defer its content update.
    2. Allow source-backed sidecar recommendations, with the existing fallback only when absent. Define explicit-empty behavior and string-list validation. Preserve generated safety/evidence limitations. These recommendations are procedure advice, separate from Skill Eval: actionable, revision-bound review guidance for skill and knowledge PRs #215’s evidence-derived PR guidance.
    3. Keep real layer mappings in the SearchSpace Matrix; put reasoned not-applicable in the sidecar. Render unknown/N/A/covered/supplementary distinctly; contradictions must be visible and N/A must not count as coverage.

    Please correct three acceptance details:

    • Index generation requires --source-revision naming committed sources. Say no hand edits to generated knowledge; regenerate and commit it, rather than no changes there. Generated provenance refresh is not evidence promotion.
    • The current coverage view does not collect applicability/recommendations; those appear in discovery abstracts. Name which existing projection will expose the new conditions. Regeneration alone does not add those fields to the reverse view.
    • Freeze each batch’s selected IDs, not the historical totals. Current main has 71 registered IDs/44 unregistered documents/66 sidecars. Explicitly disposition new arrivals; “no eval work” remains valid, but “no-eval stays exactly 65” does not.

    Add focused tests for absent/empty/invalid fields, recommendation fallback, preserved validation fields, and N/A versus unknown/real mappings. No new registry, efficacy requirement, paid run or ownership change is needed. #215 remains the evaluation-guidance owner; #91 stays open as the living tracker.
15. Dong1017

    added a commit that references this issue on Oct 29, 2026

    docs: settle the workflow sidecar shape and let sidecars supply recom…

    ...

    55fbfb4
16. Dong1017 commented on Oct 29, 2026

    

    Dong1017

    on Oct 29, 2026 – with ChatGPT Codex Connector

    CollaboratorAuthor

    More actions

    Track the bounded SkillEval feedback follow-up in #246: adapter-aware retained answer/component differences, concrete strict-failure reasons and visible coverage limits in the actual github-actions PR comment. Prefer one implementation PR, then verify PR133's retained result through the legitimate publisher path without new paid model execution. This does not take over Round 2 metadata/#243, introduce a new task family, or close this ongoing coverage tracker. #235 remains completed within its original pilot scope.
17. Dong1017

    added 7 commits that reference this issue on Oct 29, 2026

    docs: settle the workflow sidecar shape and let sidecars supply recom…

    ...

    8e86c67

    docs: settle the workflow sidecar shape and let sidecars supply recom…

    ...

    417c677

    docs: settle the workflow sidecar shape and let sidecars supply recom…

    ...

    dff0f25

    docs: settle the workflow sidecar shape and let sidecars supply recom…

    ...

    94e7249

    docs: settle the workflow sidecar shape and let sidecars supply recom…

    ...

    3c05727

    docs: settle the workflow sidecar shape and let sidecars supply recom…

    ...

    5076062

    docs: settle the workflow sidecar shape and let sidecars supply recom…

    ...

    401ee8a
18. Dong1017 commented on Oct 30, 2026

    

    Dong1017

    on Oct 30, 2026

    Last edited by Dong1017

    CollaboratorAuthor

    More actions

    Batch 2 scope and sidecar semantics

    Updated 2026-10-30 after review. The open question in the earlier text is
    settled: extending batch 1's four typed keys to methods/ is accepted, on the
    condition that each field keeps a stated meaning rather than becoming a free
    text slot. The section on per-key semantics below is new; the frozen ID
    selection and acceptance rules are unchanged.

    Batch 1 merged as 401ee8a2 (#243) and settled the sidecar shape on the six
    registered workflows/ skills. Batch 2 fills applicability and
    recommendations for methods/, reusing the same four typed keys.

    Frozen ID selection

    Read at main = 401ee8a2: 77 registered ids, 41 with an absent or empty
    applicability. methods/ holds 20 registered ids. Three already declare one
    (performance-ceiling-analysis, repository-evolution-analysis,
    video-diffusion-optimization) and are not touched. Batch 2 is the remaining 17:

    assess-serving-preemption, assess-split-backward-schedule,
    audit-fixed-workload-claim, calibrate-serving-simulation,
    checkpoint-recovery-consistency, choose-parallel-training-plan,
    dense-performance-evidence, evaluate-elastic-serving-placement,
    inference-benchmark-design, inference-optimization-casebook,
    kernel-writing-guide, optimization-search-policy,
    plan-activation-residency, skill-evolution-playbook,
    training-input-pipeline, training-resource-budget,
    weight-loading-optimization.

    Why the typed keys fit methods/

    These 17 bodies are already written in that shape. Each names the evidence it
    requires before it will answer, what it returns, the conditions that make its
    conclusion unsafe, and the point at which it withholds a recommendation. The
    section headings at this revision:

    Skill	Input	Output	Refusal
    assess-serving-preemption	"Evidence required for a candidate"	"Decision output"	"Withhold a preemption recommendation if the apparent gain can be explained by different kernels, unmatched parallelism, or an incomparable latency metric"
    assess-split-backward-schedule	"Required evidence"	"Output, checks and stopping point"	"reasons to withhold an enablement recommendation"
    calibrate-serving-simulation	"Inputs and prerequisites"	"Output"	"Failure / stop conditions"
    plan-activation-residency	"Evidence needed"	"Decision procedure"	"Verification proposal and stopping point"
    audit-fixed-workload-claim	"Recover the declared decision"	"Return an auditable conclusion"	"Do not replace missing evidence with a pass"

    Writing the four keys projects text that already exists.

    Field semantics each sidecar must keep

    Key	States	Must not
    scope	What the procedure applies to	Become a capability or efficacy claim
    prerequisites	What must already hold before use: access, tooling, authorization	Restate the input contract
    input_contract	What the caller supplies for the procedure to run	Enumerate the procedure's own steps
    produced_artifact	What the procedure emits, at the evidence level its body supports	Upgrade a proposal into a validated configuration or a measured result
    failure_modes	Conditions under which the procedure yields a wrong or unusable result	Restate limitations
    refusal_point	The point at which the body states it withholds a conclusion	Invent a refusal the body does not state
    hardware	A device dependency the body establishes	Be inferred from a source's hardware

    Two of these carry most of the drift risk across 17 files.

    produced_artifact is the sharper one. Almost every skill in this batch emits a
    proposal, a shortlist or an audited conclusion, explicitly not a validated
    configuration: plan-activation-residency returns "a preferred candidate,
    simpler fallback, resource ledger"; choose-parallel-training-plan returns "the
    best supported candidate and a simpler fallback". A sidecar that wrote
    "an optimized parallel plan" would promote the skill in the one field a reader
    scans first. The declared artifact stays at the body's evidence level.

    failure_modes is distinct from the existing limitations field, which is not
    owner-only: tools/build_skill_index.py emits four generator-owned sentences
    about indexing, authorization and backend categorization, then appends the
    sidecar's own entries. limitations records what the record does not
    establish; failure_modes records how the procedure goes wrong in use. A
    sidecar repeating "Source-derived candidate; no local efficacy validation" under
    failure_modes states nothing new.

    Cross-cutting, unchanged from batch 1: these fields are descriptive, never
    evidential. Filling them raises no verification.locally_verified, and changes
    no status or empirical_validation. recommendations is procedure advice,
    separate from limitations and from #215's evidence-derived PR guidance.

    No mechanical key allowlist in this batch

    The natural way to hold a vocabulary is a closed key set in skill_catalog.
    Batch 2 does not add one. The library currently uses nine distinct
    applicability keys: the seven above, plus architectures and
    excluded_mechanisms in hardware/cuda/reason-hopper-kernels. A closed set
    would either reject that existing sidecar or grandfather it, which is a
    compatibility decision of its own and not a side effect of filling methods/.
    Batches 3 and 4 cover hardware/, repositories/ and frameworks/ and will
    show whether those two keys generalize. Proposed: revisit a closed key set at
    batch 4, when the real vocabulary is known.

    Until then the semantics live in docs/knowledge-base.md under field ownership,
    and batch 1's shape validation still rejects malformed values.

    What this changes in the view

    tools/build_skill_coverage.py renders every key other than scope,
    hardware and prerequisites in the Conditions column. Extending the keys to
    methods/ populates that column for 17 more rows and clears no-applicability
    for them. No new column, generator field or gap type is added; batch 1's
    rendering is reused unchanged.

    Acceptance

    Both --check runs pass with generated output regenerated in the same commit as
    its sources. No verification, status, empirical_validation or sources
    change: the diff in those 17 sidecars touches applicability and
    recommendations only, which is checkable as a mechanical filter. Every
    condition cites its origin in the skill body, a pinned source or an owner-local
    case; a condition with no origin is refused in review. Negative and missing
    results are kept.

    Batch 1's test_applicability_is_descriptive_and_never_promotes_validation
    iterates every sidecar in the library, so it covers these 17 once they are
    filled. Meaning itself is not mechanically checkable; the per-key table above is
    the review standard, not a validator.

    No evaluation, verdict or promotion; no-eval stays in Round 3.
19. Dong1017 commented on Oct 30, 2026

    

    Dong1017

    on Oct 30, 2026 – with ChatGPT Codex Connector

    CollaboratorAuthor

    More actions

    Scoped technical review of Batch 2 question, against main 1147c5bb41724826b64ad13cebc8a4dce5cc278b:

    Recommendation: reuse the four keys for method procedures where their bodies support them, alongside scope/prerequisites—not instead of them. The current applicability validator already accepts string/string-list conditions without a workflows-only restriction. For example, assess-serving-preemption explicitly specifies required evidence, decision output and withholding/stop conditions. No parallel schema or generator behavior change is needed for that projection.

    Two clarifications keep the proposed batch bounded:

    • Conditions is an inventory of key names, not their values. Coverage rendering lists input_contract, …, refusal_point and advice count; the generated catalog preserves the actual applicability values. Clearing no-applicability records a declaration, not validation that every required condition is complete or enforced.
    • Keep per-skill source/section pointers for each extracted condition. Do not turn proposed rules into established configurations, add a new refusal policy, or force all four keys where the body provides no basis. Review the 17 selected IDs individually; this spot-check is not blanket content approval.

    The existing acceptance corrections in 6470996080 still apply, including committed-source --source-revision, regenerated projections and unchanged maturity/evidence fields. Preserve the three excluded owners. Also coordinate training-resource-budget with the already-open #257 source slice: if it lands first, extract against that updated body rather than overwrite its provenance/prerequisites.

    This is a technical recommendation on the proposed extension, not a scope-acceptance decision, task claim, or authorization to begin another implementation. Existing owners and the living tracker remain unchanged.
20. Dong1017

    added a commit that references this issue on Oct 30, 2026

    docs(knowledge): integrate historical MindSpeed provenance (#257)

    ...

    436c950