Skip to content

Stop safe mode containers on cancel and cap concurrent runs - #31

Merged
daniel-larraz merged 19 commits into
kind2-mc:mainfrom
Jabe03:safe-mode-container-cleanup
Oct 30, 2026
Merged

daniel-larraz merged 19 commits into
kind2-mc:mainfrom
Jabe03:safe-mode-container-cleanup

Conversation

@Jabe03

@Jabe03 Jabe03 commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Stop safe mode containers on cancel and cap concurrent runs

Follow-up to #30. Needed by kind2-mc/kind2-language-server#22, which cancels out-of-date parses: without this, cancelling a safe mode run stops the docker client but can leave the container running.

Changes

  • Shared docker run command: getDockerRunCommand() builds the resource limits and the Kind 2/solver mounts once, for checks, the interpreter and checkAvailable().
  • --init: the container runs an init process as PID 1, so signals reach Kind 2 and orphaned solver processes are reaped.
  • Killing containers: each safe mode run gets a unique --name (kind2-api-<uuid>). If the run is cancelled or the process is still alive when it ends, the API runs docker kill on that container before destroying the client process. Destroying the client alone doesn't reliably stop the container (for example on Windows).
  • Concurrent run cap: a semaphore shared by every Kind2Api instance limits how many safe mode runs happen at once (default 2, set with the new static setSafeModeMaxRuns(int)). Waiting runs still respond to their monitor's cancellation.
  • checkAvailable() in safe mode now runs Kind 2 inside the container, and waits for the process instead of reading exitValue() before it has exited.
  • --no_include is only passed when safe mode is on (it used to be passed as --no_include false when safe mode was explicitly off).
  • The two interpret overloads now share one runInterpreter method.
  • After review: the docker kill timeout now applies; finished runs get a short grace period instead of being killed; setSafeModeMaxRuns resizes the limit in place, so it also applies to waiting runs; slots are held with try-with-resources; the interpreter stops waiting for a slot after 60 s; one helper builds the Kind 2 command prefix; the interpreter reads stderr and reports it when no trace comes back; programs are sent as UTF-8.
  • After the second review: the interpreter reports stderr only when stdout has no usable output, and wraps parse errors in Kind2Exception; checkAvailable takes a run slot, names its container and times out after 30 s; after a kill, the container is also removed with docker rm -f once the client has exited (covers a cancel that lands while Docker is still creating it); execute(String)/execute(Program) stop waiting for a slot after 60 s; cleanup clears the interrupt flag while it waits; safeMode is read once per run; one slot-holding pattern; ApiUtil.readAll decodes UTF-8.

Outside safe mode the command line is unchanged.

Testing

  • ./gradlew clean build passes.
  • Second review: each fix tested with real containers and the fake kind2 (plus a docker wrapper that delays kill or splits run into create + start), reproducing the problem on the previous head where possible: Created container left after an early cancel (2/2 before, 0 after), slot released 2 s before docker kill ended on an interrupted thread, harmless stderr hiding the result, garbled non-ASCII paths, checkAvailable hanging on a stuck docker run.
  • With real containers and a statically linked fake kind2 (sleeps, then prints a trace): finished checks and interpreter runs call no docker kill, a cancelled check is killed with no container left, raising and lowering the limit while runs wait takes effect right away, and the interpreter gives up after 60 s when no slot is free.
  • Cancelling a safe mode run through Kind2Api (setSafeMode(true)) left no running containers and no leftover z3 processes in 4 of 4 tries.

🤖 Generated with Claude Code

@Jabe03
Jabe03 marked this pull request as draft October 16, 2026 17:06

@Jabe03 Jabe03 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of the safe mode run lifecycle. I'll address each comment with its own commit and reply here.

Not raised here: a cancel that arrives before Docker creates the container (pending issue 3 in the safe mode notes), since how to handle it hasn't been decided yet.

Jabe03 and others added 8 commits October 16, 2026 12:18
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When Kind 2's output ends, the docker client is usually still waiting for
the container's exit status, so checking isAlive() right away made almost
every finished run call docker kill.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replacing the semaphore left callers that were already waiting on the old
one, so a lower limit was not enforced and a higher one did not wake them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
acquireSafeModeRun now returns a closeable slot (a no-op one outside safe
mode) and null only when cancelled, so callers can't forget the release.
It also checks the monitor once more after acquiring, so a run cancelled
just as a slot frees up does not start a container.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The interpreter has no monitor, so a call made while the run limit was
reached blocked the caller's thread until a long check finished. It now
throws a Kind2Exception instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
checkAvailable, checks and the interpreter now share getKind2Command, so
checkAvailable runs the same command that execute does. The generated
command lines are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
stderr was never read, so docker errors in safe mode were lost, and enough
stderr output blocked Kind 2 and hung the run. It is now read on its own
thread, and if no trace comes back, interpret throws a Kind2Exception with
it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It was encoded with the platform's default charset while the output is
read as UTF-8, so non-ASCII programs broke on Windows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Jabe03
Jabe03 marked this pull request as ready for review October 23, 2026 17:01

@daniel-larraz daniel-larraz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of the safe mode run management and interpreter error reporting. I checked each finding against the code; gson behavior was tested against 2.8.6.

The two worth fixing first are on the new interpreter error-reporting path: harmless stderr now replaces the empty result (L407), and parse failures escape as raw gson exceptions (L398). The rest are lower severity: cleanup/race edge cases, and a few smaller consistency and robustness points.

🤖 Generated with Claude Code

Jabe03 and others added 9 commits October 30, 2026 02:58
Only JsonIOException (empty stdout) was caught, and it was rethrown as is.
Non-JSON stdout threw an uncaught JsonSyntaxException that skipped the
stderr report. Both are JsonParseExceptions, which are now caught and
wrapped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Any stderr text made a run without a trace throw, so a harmless docker
warning (such as the missing swap limit support) replaced a Kind 2 error
reported as JSON on stdout. stderr is now used only when stdout is empty
or cannot be parsed; otherwise a run without a trace returns "" as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…onitor

execute(String) and execute(Program) use a monitor that never cancels, so
once every run slot was taken they waited forever. Like interpret, they
now throw a Kind2Exception after SAFE_MODE_SLOT_WAIT_SECONDS. Calls that
pass their own monitor still wait until it is cancelled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With the flag set, the waits in exitsOnItsOwn and killContainer threw at
once, so the run slot was released before docker kill had stopped the
container, and a hung docker kill was never destroyed. A shared stopRun
now clears the flag while it stops the run and restores it afterwards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docker kill misses a container that docker is still creating. If the
client then dies before starting it, the container stays in the Created
state, which --rm never removes; if the client starts it first, it runs
with nobody tracking it. After a kill, stopRun now waits for the client
to exit and runs docker rm -f on the container.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In safe mode, checkAvailable started a container with no name, no run
slot and no timeout, so a hung docker blocked it forever with nothing to
kill. It now names its container, takes a run slot, gives up after 30 s,
and stops the container like any other run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
newContainerName is now the only place a run reads safeMode. The run
slot, the command and the Kind 2 options all follow from whether the run
has a container name, so a concurrent setSafeMode can no longer produce
a docker run without a name, or a native run that holds a slot.
getOptions() keeps its behavior and delegates to getOptions(boolean).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
acquireSafeModeRun now returns RunSlot.CANCELLED instead of null when the
monitor is cancelled while waiting, so callKind2 acquires its slot in the
try-with-resources header like the other callers and checks isCancelled()
inside it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
readAll turned each byte into a char, which decodes it as Latin-1, so
non-ASCII text in docker and Kind 2 messages (such as a host path quoted
in a docker error) came out garbled. It now decodes the whole stream as
UTF-8, like the rest of the process I/O.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@daniel-larraz
daniel-larraz merged commit 34e79a6 into kind2-mc:main Oct 30, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants