Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
2f94a24
fix: expand Gerrit channel URLs in health score v2 maintainer and dev…
joanagmaia Aug 12, 2026
b92adea
fix: add FINAL to repositories and integrations in channels nodes (IN…
joanagmaia Aug 12, 2026
f6a5f72
fix: apply Gerrit channel expansion to raw inputs snapshot pipe (IN-1…
joanagmaia Aug 12, 2026
4332ec7
style: apply tb fmt formatting to health score v2 pipes (IN-1229)
joanagmaia Aug 12, 2026
9a83f6d
refactor: centralize Gerrit channel expansion in repos_channels_ds (I…
joanagmaia Aug 12, 2026
96743f9
fix: remove changelog-style descriptions and fix channels schedule (I…
joanagmaia Aug 12, 2026
92b8107
fix: source allRepos from repositories directly in maintainer pipe (I…
joanagmaia Aug 12, 2026
26a4bf2
fix: route lifecycle pipe joins through repos_channels_ds (IN-1229)
joanagmaia Aug 13, 2026
1801997
fix: exclude archived (v2) and excluded repos across health score pip…
joanagmaia Aug 13, 2026
72ad628
fix: exclude archived + excluded repos in v2 security, aggregator, an…
joanagmaia Aug 13, 2026
37eef68
Merge branch 'main' into worktree-fix+IN-gerrit-health-score-v2-chann…
joanagmaia Aug 13, 2026
fc52778
fix: add FINAL to repositories and vulnerabilities reads in lifecycle…
joanagmaia Aug 13, 2026
14eb2e5
fix: include archived repos in aggregator and signal detail pipes (IN…
joanagmaia Aug 13, 2026
849feff
fix: add FINAL to repositories reads in development and impact pipes …
joanagmaia Aug 13, 2026
2e4f977
fix: formatting
joanagmaia Aug 13, 2026
fa03845
Merge branch 'main' into worktree-fix+IN-gerrit-health-score-v2-chann…
joanagmaia Aug 13, 2026
119708d
Merge branch 'main' into worktree-fix+IN-gerrit-health-score-v2-chann…
joanagmaia Aug 14, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions services/libs/tinybird/datasources/repos_channels_ds.datasource
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
DESCRIPTION >
- Materialized channel expansion for all repositories. Populated by `repos_channels_copy.pipe`
(daily, before health score pipes run).
- Each non-Gerrit repo produces one row (channel = url). Each Gerrit repo produces two rows:
the plain URL and the /q/project: variant used by Gerrit activity ingestion.
- `isGerrit` and `isExcluded` are denormalized from `repositories`/`integrations` so health
score pipes can classify repos without re-joining those tables.
- Sorting key is `channel` because all consumers join on `channel`; `repoUrl` lookups are
full-scans but the table is small enough that this is fine.

SCHEMA >
`repoUrl` String,
`channel` String,
`isGerrit` UInt8,
`isExcluded` UInt8

ENGINE MergeTree
ENGINE_SORTING_KEY channel
7 changes: 6 additions & 1 deletion services/libs/tinybird/pipes/health_score_v2.pipe
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,12 @@ SQL >
) AS coveredCategoryWeight,
l.lifecycleLabelV2 AS lifecycleLabelV2,
i.impactScoreRaw AS impactScoreRaw
FROM (SELECT DISTINCT url AS repoUrl FROM repositories WHERE deletedAt IS NULL) AS base
FROM
(
SELECT DISTINCT url AS repoUrl
FROM repositories FINAL
WHERE deletedAt IS NULL AND excluded = false
) AS base
Comment thread
joanagmaia marked this conversation as resolved.
LEFT JOIN health_score_v2_maintainer_ds AS m ON m.repoUrl = base.repoUrl
LEFT JOIN health_score_v2_security_ds AS s ON s.repoUrl = base.repoUrl
LEFT JOIN health_score_v2_development_ds AS d ON d.repoUrl = base.repoUrl
Expand Down
55 changes: 36 additions & 19 deletions services/libs/tinybird/pipes/health_score_v2_development.pipe
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,9 @@ DESCRIPTION >
(most repos) having lastCommitAt from GitHub enrichment — the fallback covers the vast majority.
- Split out of health_score_v2.pipe into its own copy pipe (2026-07-22) — see
health_score_v2_maintainer.pipe description for why.
- All joins against activityRelations, pull_requests_analyzed, and issues_analyzed route through
`repos_channels_ds` (populated by repos_channels_copy.pipe), which expands each Gerrit repo URL
to both its plain URL and /q/project: channel variant.

NODE health_score_v2_development_calc
SQL >
Expand Down Expand Up @@ -147,7 +150,12 @@ SQL >
FROM
(
SELECT base.url AS repoUrl, rc.lastCommitAt AS lastCommitAt
FROM (SELECT DISTINCT url FROM repositories WHERE deletedAt IS NULL) AS base
FROM
(
SELECT DISTINCT url
FROM repositories FINAL
WHERE deletedAt IS NULL AND archived = false AND excluded = false
) AS base
LEFT JOIN
(
SELECT url, argMax(lastCommitAt, updatedAt) AS lastCommitAt
Expand Down Expand Up @@ -192,37 +200,46 @@ SQL >
LEFT JOIN
(
SELECT
channel AS repoUrl,
countIf(timestamp > now() - INTERVAL 6 MONTH) AS commitsLast6m
FROM activityRelations_deduplicated_cleaned_bucket_union
WHERE type = 'authored-commit'
GROUP BY channel
ch.repoUrl AS repoUrl,
countIf(ar.timestamp > now() - INTERVAL 6 MONTH) AS commitsLast6m
FROM activityRelations_deduplicated_cleaned_bucket_union ar
INNER JOIN repos_channels_ds ch ON ar.channel = ch.channel
WHERE ar.type = 'authored-commit'
GROUP BY ch.repoUrl
) AS c
ON c.repoUrl = rp.repoUrl
LEFT JOIN
(
SELECT
channel AS repoUrl,
countIf(closedAt IS NOT NULL) AS closed12m,
ch.repoUrl AS repoUrl,
countIf(ia.closedAt IS NOT NULL) AS closed12m,
count() AS opened12m,
quantileIf(0.5)
(closedInSeconds, closedAt > now() - INTERVAL 12 MONTH) AS medianCloseS
FROM issues_analyzed
WHERE openedAt > now() - INTERVAL 12 MONTH
GROUP BY channel
(
ia.closedInSeconds, ia.closedAt > now() - INTERVAL 12 MONTH
) AS medianCloseS
FROM issues_analyzed ia
INNER JOIN repos_channels_ds ch ON ia.channel = ch.channel
WHERE ia.openedAt > now() - INTERVAL 12 MONTH
GROUP BY ch.repoUrl
) AS ist
ON ist.repoUrl = rp.repoUrl
LEFT JOIN
(
SELECT
channel AS repoUrl,
countIf(mergedAt IS NOT NULL) AS merged12m,
countIf(closedAt IS NOT NULL AND mergedAt IS NULL) AS closedUnmerged12m,
ch.repoUrl AS repoUrl,
countIf(pra.mergedAt IS NOT NULL) AS merged12m,
countIf(
pra.closedAt IS NOT NULL AND pra.mergedAt IS NULL
) AS closedUnmerged12m,
quantileIf(0.5)
(mergedInSeconds, mergedAt > now() - INTERVAL 12 MONTH) AS medianMergeS
FROM pull_requests_analyzed
WHERE openedAt > now() - INTERVAL 12 MONTH
GROUP BY channel
(
pra.mergedInSeconds, pra.mergedAt > now() - INTERVAL 12 MONTH
) AS medianMergeS
FROM pull_requests_analyzed pra
INNER JOIN repos_channels_ds ch ON pra.channel = ch.channel
WHERE pra.openedAt > now() - INTERVAL 12 MONTH
GROUP BY ch.repoUrl
) AS prs
ON prs.repoUrl = rp.repoUrl
)
Expand Down
5 changes: 4 additions & 1 deletion services/libs/tinybird/pipes/health_score_v2_impact.pipe
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,10 @@ DESCRIPTION >
NODE health_score_v2_impact_calc
SQL >
SELECT base.url AS repoUrl, max(toFloat64OrNull(pk.impact)) * 100 AS impactScoreRaw
FROM (SELECT DISTINCT url FROM repositories WHERE deletedAt IS NULL) AS base
FROM
(
SELECT DISTINCT url FROM repositories FINAL WHERE deletedAt IS NULL AND excluded = false
) AS base
INNER JOIN repos r ON r.url = base.url
INNER JOIN packageRepos pr ON pr.repoId = r.id
INNER JOIN ossPackages_enriched_ds pk ON pk.id = pr.packageId
Expand Down
77 changes: 45 additions & 32 deletions services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,12 @@ SQL >
FROM
(
SELECT base.url AS url, base.archived AS archived, rc.lastCommitAt AS lastCommitAt
FROM (SELECT DISTINCT url, archived FROM repositories WHERE deletedAt IS NULL) AS base
FROM
(
SELECT DISTINCT url, archived
FROM repositories FINAL
WHERE deletedAt IS NULL AND excluded = false
) AS base
LEFT JOIN
(
SELECT url, argMax(lastCommitAt, updatedAt) AS lastCommitAt FROM repos GROUP BY url
Expand All @@ -115,68 +120,76 @@ SQL >
LEFT JOIN
(
SELECT
channel AS repoUrl,
countIf(timestamp > now() - INTERVAL 6 MONTH) AS commitsLast6m,
ch.repoUrl AS repoUrl,
countIf(ar.timestamp > now() - INTERVAL 6 MONTH) AS commitsLast6m,
countIf(
timestamp <= now() - INTERVAL 6 MONTH AND timestamp > now() - INTERVAL 12 MONTH
ar.timestamp <= now() - INTERVAL 6 MONTH
AND ar.timestamp > now() - INTERVAL 12 MONTH
) AS commitsPrior6m
FROM activityRelations_deduplicated_cleaned_bucket_union
WHERE type = 'authored-commit'
GROUP BY channel
FROM activityRelations_deduplicated_cleaned_bucket_union ar
INNER JOIN repos_channels_ds ch ON ar.channel = ch.channel
WHERE ar.type = 'authored-commit'
GROUP BY ch.repoUrl
) AS c
ON c.repoUrl = r.url
LEFT JOIN
(
SELECT channel AS repoUrl, countIf(closedAt IS NULL) AS issuesOpenNow
FROM issues_analyzed
GROUP BY channel
SELECT ch.repoUrl AS repoUrl, countIf(ia.closedAt IS NULL) AS issuesOpenNow
FROM issues_analyzed ia
INNER JOIN repos_channels_ds ch ON ia.channel = ch.channel
GROUP BY ch.repoUrl
) AS n
ON n.repoUrl = r.url
LEFT JOIN
(
SELECT
channel AS repoUrl,
countIf(openedAt > now() - INTERVAL 6 MONTH) AS issuesOpenedLast6m,
ch.repoUrl AS repoUrl,
countIf(ia.openedAt > now() - INTERVAL 6 MONTH) AS issuesOpenedLast6m,
countIf(
openedAt <= now() - INTERVAL 6 MONTH AND openedAt > now() - INTERVAL 12 MONTH
ia.openedAt <= now() - INTERVAL 6 MONTH AND ia.openedAt > now() - INTERVAL 12 MONTH
) AS issuesOpenedPrior6m,
countIf(openedAt > now() - INTERVAL 18 MONTH) AS issuesInWindow18m,
countIf(ia.openedAt > now() - INTERVAL 18 MONTH) AS issuesInWindow18m,
countIf(
openedAt > now() - INTERVAL 18 MONTH
AND respondedInSeconds IS NULL
AND closedAt IS NULL
AND openedAt <= now() - INTERVAL 90 DAY
ia.openedAt > now() - INTERVAL 18 MONTH
AND ia.respondedInSeconds IS NULL
AND ia.closedAt IS NULL
AND ia.openedAt <= now() - INTERVAL 90 DAY
) AS unansweredAged90d
FROM issues_analyzed
GROUP BY channel
FROM issues_analyzed ia
INNER JOIN repos_channels_ds ch ON ia.channel = ch.channel
GROUP BY ch.repoUrl
) AS w
ON w.repoUrl = r.url
LEFT JOIN
(
SELECT channel AS repoUrl, countIf(openedAt > now() - INTERVAL 18 MONTH) AS prsInWindow18m
FROM pull_requests_analyzed
GROUP BY channel
SELECT
ch.repoUrl AS repoUrl,
countIf(pra.openedAt > now() - INTERVAL 18 MONTH) AS prsInWindow18m
FROM pull_requests_analyzed pra
INNER JOIN repos_channels_ds ch ON pra.channel = ch.channel
GROUP BY ch.repoUrl
) AS p
ON p.repoUrl = r.url
LEFT JOIN
(
SELECT
channel AS repoUrl,
ch.repoUrl AS repoUrl,
countIf(
openedAt > now() - INTERVAL 18 MONTH
AND reviewedAt IS NULL
AND approvedAt IS NULL
AND closedAt IS NULL
AND openedAt <= now() - INTERVAL 90 DAY
pra.openedAt > now() - INTERVAL 18 MONTH
AND pra.reviewedAt IS NULL
AND pra.approvedAt IS NULL
AND pra.closedAt IS NULL
AND pra.openedAt <= now() - INTERVAL 90 DAY
) AS unansweredAged90d
FROM pull_requests_analyzed
GROUP BY channel
FROM pull_requests_analyzed pra
INNER JOIN repos_channels_ds ch ON pra.channel = ch.channel
GROUP BY ch.repoUrl
) AS pw
ON pw.repoUrl = r.url
LEFT JOIN
(
SELECT repoUrl, countIf(status = 'OPEN' AND severity = 'CRITICAL') AS openCriticals
FROM vulnerabilities
FROM vulnerabilities FINAL
GROUP BY repoUrl
) AS v
ON v.repoUrl = r.url
Expand Down
Loading
Loading