Skip to content

Add tooling to send email to ci_failure email list on ci failure #351

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
stumbo merged 2 commits into from
Oct 15, 2026
Merged

Add tooling to send email to ci_failure email list on ci failure #351

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
940c3df
Add tooling to send email to ci_failure email list on ci failure
stumbo Oct 8, 2026
a71b196
Merge remote-tracking branch 'origin/main' into bs35_new_bib_type_tra…
stumbo Oct 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
121 changes: 116 additions & 5 deletions .github/workflows/gh-pages.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@
# Interlisp/Interlisp.staging (skips on fork PRs or when
# the STAGING_APP_ID variable is unset)
# deploy - deploy the built site to GitHub Pages
# notify - email ci_failures@interlisp.org on any failure or
# build warnings (notify-only; warnings still deploy)
#
# The workflow is executed either on a push or via scheduled run times. When
# started at a scheduled run time we only do a deploy if the cached bibliography
Expand Down Expand Up @@ -159,11 +161,12 @@ jobs:
repo: Interlisp/Interlisp.staging
ref: main
token: ${{ steps.app-token.outputs.token }}
inputs: '{
"pr_number": "${{ github.event.pull_request.number }}",
"pr_sha": "${{ github.event.pull_request.head.sha }}",
"hugo_version": "${{ env.HUGO_VERSION }}"
}'
inputs: >-
{
"pr_number": "${{ github.event.pull_request.number }}",
"pr_sha": "${{ github.event.pull_request.head.sha }}",
"hugo_version": "${{ env.HUGO_VERSION }}"
}

# ----------------------------------------------------------------------------
# Deploy the built site to GitHub Pages.
Expand All @@ -182,3 +185,111 @@ jobs:
- name: Deploy to GitHub Pages
id: deployment
uses: actions/deploy-pages@v5

# ----------------------------------------------------------------------------
# Notify ci_failures@interlisp.org when the workflow fails or produces
# warnings. GitHub does not notify on workflow failure by default, so
# without this job a broken nightly build can go undetected for days.
#
# Triggers on:
# - any job failure (validate-docs, hugo-version, build, preview, deploy)
# - warnings surfaced by the shared build workflow
# (needs.build.outputs.has-warnings == 'true'; see
# Interlisp/shared-workflows build-site.yml proposal)
#
# Warnings are notify-only: the build still passes and deploys.
#
# Required secrets (set in repo or org Settings > Secrets > Actions):
# MAIL_SERVER e.g. smtp.gmail.com
# MAIL_USERNAME SMTP login (also used as default From address)
# MAIL_PASSWORD SMTP password (Gmail: use an App password)
# Optional vars (Settings > Variables > Actions):
# MAIL_PORT default 587 (STARTTLS)
# MAIL_FROM default is MAIL_USERNAME
# e.g. "Interlisp CI <ci@interlisp.org>"
# MAIL_TO default ci_failures@interlisp.org
#
notify:
if: always() && (needs.validate-docs.result == 'failure' || needs.hugo-version.result == 'failure' || needs.build.result == 'failure' || needs.preview.result == 'failure' || needs.deploy.result == 'failure' || needs.build.outputs.has-warnings == 'true')
runs-on: ubuntu-latest
needs: [validate-docs, hugo-version, build, preview, deploy]
steps:
- name: Check mail secrets configured
id: mail-check
env:
MAIL_SERVER: ${{ secrets.MAIL_SERVER }}
MAIL_USERNAME: ${{ secrets.MAIL_USERNAME }}
MAIL_PASSWORD: ${{ secrets.MAIL_PASSWORD }}
run: |
if [[ -z "$MAIL_SERVER" || -z "$MAIL_USERNAME" || -z "$MAIL_PASSWORD" ]]; then
echo "::warning::Mail secrets (MAIL_SERVER/MAIL_USERNAME/MAIL_PASSWORD) not configured; skipping email to ci_failures@interlisp.org"
echo "configured=false" >> "$GITHUB_OUTPUT"
else
echo "configured=true" >> "$GITHUB_OUTPUT"
fi

- name: Send failure email
if: steps.mail-check.outputs.configured == 'true' && (needs.validate-docs.result == 'failure' || needs.hugo-version.result == 'failure' || needs.build.result == 'failure' || needs.preview.result == 'failure' || needs.deploy.result == 'failure')
uses: dawidd6/action-send-mail@v22
with:
server_address: ${{ secrets.MAIL_SERVER }}
server_port: ${{ vars.MAIL_PORT || '587' }}
secure: false
require_tls: true
username: ${{ secrets.MAIL_USERNAME }}
password: ${{ secrets.MAIL_PASSWORD }}
subject: "[${{ github.repository }} CI] FAILED: ${{ github.workflow }} run ${{ github.run_number }} (${{ github.event_name }}, ${{ github.ref_name }})"
to: ${{ vars.MAIL_TO || 'ci_failures@interlisp.org' }}
from: ${{ vars.MAIL_FROM || secrets.MAIL_USERNAME }}
body: |
CI workflow FAILED for ${{ github.repository }}.

Workflow: ${{ github.workflow }}
Event: ${{ github.event_name }}
Ref: ${{ github.ref }} (${{ github.ref_name }})
SHA: ${{ github.sha }}
Actor: ${{ github.actor }}
Run: https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}

Job results:
validate-docs: ${{ needs.validate-docs.result }}
hugo-version: ${{ needs.hugo-version.result }}
build: ${{ needs.build.result }} (skipped=${{ needs.build.outputs.skipped }}, has-warnings=${{ needs.build.outputs.has-warnings }})
preview: ${{ needs.preview.result }}
deploy: ${{ needs.deploy.result }}

Please review the failed run logs at the URL above.

- name: Send warnings email
if: steps.mail-check.outputs.configured == 'true' && needs.validate-docs.result != 'failure' && needs.hugo-version.result != 'failure' && needs.build.result != 'failure' && needs.preview.result != 'failure' && needs.deploy.result != 'failure' && needs.build.outputs.has-warnings == 'true'
uses: dawidd6/action-send-mail@v3
with:
server_address: ${{ secrets.MAIL_SERVER }}
server_port: ${{ vars.MAIL_PORT || '587' }}
secure: false
require_tls: true
username: ${{ secrets.MAIL_USERNAME }}
password: ${{ secrets.MAIL_PASSWORD }}
subject: "[${{ github.repository }} CI] WARNINGS: ${{ github.workflow }} run ${{ github.run_number }} (${{ github.event_name }}, ${{ github.ref_name }})"
to: ${{ vars.MAIL_TO || 'ci_failures@interlisp.org' }}
from: ${{ vars.MAIL_FROM || secrets.MAIL_USERNAME }}
body: |
CI workflow completed with WARNINGS for ${{ github.repository }} (build passed, deploy continued).

Workflow: ${{ github.workflow }}
Event: ${{ github.event_name }}
Ref: ${{ github.ref }} (${{ github.ref_name }})
SHA: ${{ github.sha }}
Actor: ${{ github.actor }}
Run: https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}

Warnings summary: ${{ needs.build.outputs.warnings-summary }}

Job results:
validate-docs: ${{ needs.validate-docs.result }}
hugo-version: ${{ needs.hugo-version.result }}
build: ${{ needs.build.result }} (skipped=${{ needs.build.outputs.skipped }}, has-warnings=${{ needs.build.outputs.has-warnings }})
preview: ${{ needs.preview.result }}
deploy: ${{ needs.deploy.result }}

Please review the warnings in the run logs at the URL above.
54 changes: 51 additions & 3 deletions .github/workflows/preview-cleanup.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,54 @@ jobs:
repo: Interlisp/Interlisp.staging
ref: main
token: ${{ steps.app-token.outputs.token }}
inputs: '{
"pr_number": "${{ github.event.pull_request.number }}"
}'
inputs: '{"pr_number":"${{ github.event.pull_request.number }}"}'

# Notify ci_failures@interlisp.org if preview cleanup fails. Uses the
# same MAIL_* secrets/vars as gh-pages.yml; skips gracefully when they
# are not configured.
notify:
if: always() && needs.remove-preview.result == 'failure'
runs-on: ubuntu-latest
needs: [remove-preview]
steps:
- name: Check mail secrets configured
id: mail-check
env:
MAIL_SERVER: ${{ secrets.MAIL_SERVER }}
MAIL_USERNAME: ${{ secrets.MAIL_USERNAME }}
MAIL_PASSWORD: ${{ secrets.MAIL_PASSWORD }}
run: |
if [[ -z "$MAIL_SERVER" || -z "$MAIL_USERNAME" || -z "$MAIL_PASSWORD" ]]; then
echo "::warning::Mail secrets (MAIL_SERVER/MAIL_USERNAME/MAIL_PASSWORD) not configured; skipping email to ci_failures@interlisp.org"
echo "configured=false" >> "$GITHUB_OUTPUT"
else
echo "configured=true" >> "$GITHUB_OUTPUT"
fi

- name: Send failure email
if: steps.mail-check.outputs.configured == 'true'
uses: dawidd6/action-send-mail@v22
with:
server_address: ${{ secrets.MAIL_SERVER }}
server_port: ${{ vars.MAIL_PORT || '587' }}
secure: false
require_tls: true
username: ${{ secrets.MAIL_USERNAME }}
password: ${{ secrets.MAIL_PASSWORD }}
subject: "[${{ github.repository }} CI] FAILED: ${{ github.workflow }} run ${{ github.run_number }} (PR #${{ github.event.pull_request.number }})"
to: ${{ vars.MAIL_TO || 'ci_failures@interlisp.org' }}
from: ${{ vars.MAIL_FROM || secrets.MAIL_USERNAME }}
body: |
Preview cleanup FAILED for ${{ github.repository }}.

Workflow: ${{ github.workflow }}
PR: #${{ github.event.pull_request.number }}
Ref: ${{ github.ref }} (${{ github.ref_name }})
SHA: ${{ github.sha }}
Actor: ${{ github.actor }}
Run: https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}

Job results:
remove-preview: ${{ needs.remove-preview.result }}

Please review the failed run logs at the URL above.
69 changes: 62 additions & 7 deletions scripts/update_bibliography.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
#!/usr/bin/env bash

set -e
# Fail fast and loudly so CI does not pass silently on errors.
# -e: exit on any command failure
# -u: exit on use of an unset variable
# -o pipefail: a failed curl/jq in a pipeline fails the whole pipeline
# (without pipefail, e.g. `curl ... | jq ...` hides curl failures)
set -euo pipefail

function usage () {
echo -e "\nUpdate Bibliography for interlisp.org from Zotero\n"
Expand Down Expand Up @@ -63,7 +68,16 @@ function showInfo() {
# The $1 is the level for this message ($showInfoLevel must be >= this)
# The $2 is the message string
if [[ $showInfoLevel -ge $1 ]] ; then
echo "$2"
# Level 2 messages are warnings. When running under GitHub Actions,
# also emit them as ::warning:: annotations so they are visible in
# the workflow UI (and can trigger the ci_failures@interlisp.org
# notification) instead of hiding in the log. Warnings are
# notify-only: they never fail the build.
if [[ "$1" -eq 2 && "${GITHUB_ACTIONS:-}" == "true" ]] ; then
echo "::warning::$2"
else
echo "$2"
fi
fi
}

Expand Down Expand Up @@ -103,7 +117,7 @@ function add_items_from_collection () {
local limit=100
while :; do
showInfo 10 "curl of $collection_key at start $start"
local this_page=$(curl -s "https://api.zotero.org/groups/$GROUP_ID/collections/$collection_key/items?include=data,csljson&start=$start&limit=$limit&v=3")
local this_page=$(curl -fsSL "https://api.zotero.org/groups/$GROUP_ID/collections/$collection_key/items?include=data,csljson&start=$start&limit=$limit&v=3")
local item_count=$(jq '. | length' <<< "$this_page")
showInfo 10 "curl returned $item_count items"
if [[ $item_count > 0 ]] ; then
Expand All @@ -124,7 +138,7 @@ function add_items_from_collection () {
local numCollections=$(jq -r '.numCollections' <<< "$collection_item")
if [[ $numCollections -gt 0 ]] ; then
showInfo 5 "Getting subcollections of $collection_key: $collection_name"
local subcollections=$(curl -s "https://api.zotero.org/groups/$GROUP_ID/collections/$collection_key/collections" | jq -r 'map(.data+.meta)')
local subcollections=$(curl -fsSL "https://api.zotero.org/groups/$GROUP_ID/collections/$collection_key/collections" | jq -r 'map(.data+.meta)')
collections=$(jq -s 'add' <<< "$collections$subcollections")

# Recurse into subcollections
Expand Down Expand Up @@ -215,7 +229,7 @@ fi
root_collection="FSK5IX4F"
if [[ $# -eq 0 ]] ; then
# Initialize with the root collection.
collections=$(curl -s "https://api.zotero.org/groups/$GROUP_ID/collections/$root_collection" | jq -s -r 'map(.data+.meta)')
collections=$(curl -fsSL "https://api.zotero.org/groups/$GROUP_ID/collections/$root_collection" | jq -s -r 'map(.data+.meta)')
add_items_from_collection $root_collection
if $collectionsFiles ; then
echo "$collections" > collections.json
Expand All @@ -224,7 +238,7 @@ if [[ $# -eq 0 ]] ; then

while read neededUrl; do
showInfo 10 "curl of $neededUrl"
item=$(curl -s "$neededUrl?include=data,csljson&v=3")
item=$(curl -fsSL "$neededUrl?include=data,csljson&v=3")
showInfo 10 "curl returned"
items=$(jq -s 'add' <<< "[$item]$items")
showInfo 10 "items updated"
Expand All @@ -246,7 +260,7 @@ fi
showInfo 1 "Got $(jq '. | length' <<< "$items") items"

# Piece-wise processing for debugging:
items=$(jq 'include "./bib-fns";walk(if type == "object" then removeEmptyKeys end)' <<< $items)
items=$(jq 'include "./bib-fns";walk(if type == "object" then removeEmptyKeys end)' <<< "$items")
showInfo 8 "Remove all empty fields and the keys."
if $debugFiles ; then
dfn=$(debugFileName "removeEmptyKeys" $dfn)
Expand Down Expand Up @@ -426,7 +440,48 @@ export BIBITEMS_DIR
# Ensure target directories exist
mkdir -p "$BIBLIOGRAPHY_DIR" "$BIBITEMS_DIR"

# Run bibSplit.pl, capturing its stderr. bibSplit.pl emits
# `Warning: unhandled type ...` lines for Zotero types it does not know how
# to render. Those warnings previously sat silently in bibSplit.err while
# CI stayed green. Now:
# - a non-zero bibSplit.pl exit is a hard ERROR (fails the build, which
# triggers the ci_failures@interlisp.org failure email), and
# - a non-empty bibSplit.err is surfaced as ::warning:: annotations (and
# in the job summary) but does NOT fail the build (notify-only).
: > bibSplit.err
set +e
./bibSplit.pl bibliography-items-by-line.json 2>bibSplit.err
bibSplitStatus=$?
set -e
if [[ $bibSplitStatus -ne 0 ]]; then
echo "::error::bibSplit.pl failed with exit code $bibSplitStatus" >&2
cat bibSplit.err >&2 || true
exit $bibSplitStatus
fi
if [[ -s bibSplit.err ]]; then
bibWarnCount=$(wc -l < bibSplit.err | tr -d ' ')
bibWarnSummary=$(sort bibSplit.err | uniq -c | sort -rn | head -n 10)
if [[ "${GITHUB_ACTIONS:-}" == "true" ]]; then
echo "::warning::$bibWarnCount bibSplit.pl warnings (see bibSplit.err; first occurrence per message):"
# Emit one annotation per distinct warning to avoid flooding the log
# with hundreds of duplicate lines, then dump the full file.
echo "$bibWarnSummary" | while IFS= read -r line; do
echo "::warning::bibSplit: $line"
done
if [[ -n "${GITHUB_STEP_SUMMARY:-}" ]]; then
{
echo "### Bibliography warnings"
echo ""
echo "$bibWarnCount warnings from bibSplit.pl:"
echo '```'
echo "$bibWarnSummary"
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
fi
else
echo "Warn: $bibWarnCount bibSplit.pl warnings (see bibSplit.err)" >&2
fi
fi
# Cleanup (uncomment once working)
# rm bibliography-items-by-line.json

Expand Down
Loading