[Bug] EUD disconnects never recorded: close_connection() publishes before its own guard, and the publish races os._exit()

[agnticco]

Environment

• OpenTAKServer: 1.7.13
• Python: 3.11.2 (Debian bookworm, Raspberry Pi 5, aarch64)
• pika: 1.3.2
• Database: PostgreSQL
• Client: ATAK-CIV 5.8.0.5, TLS on 8089

Symptom

euds.last_status stays "Connected" indefinitely after a client disconnects. The handler logs the disconnect, but the database is never updated and the queue bindings are not released.

Observed live — eud_handler_ssl.log recorded the disconnect while the database still disagreed, with zero established sockets on 8089:

[2026-09-20 00:09:28,199] - eud_handler[1420065] - EudHandler - close_connection - 191 - INFO - <EUD_IP> disconnected

 callsign | last_status |     last_event_time
----------+-------------+-------------------------
 <EUD>    | Connected   | 2026-09-20 00:09:13.236

There are two separate defects in EudHandler.close_connection(), and the first hides the second.

Defect 1 — the publish runs ahead of the guard written for it

def close_connection(self):
    self.logger.info("{} disconnected".format(self.client_address[0]))

    self.rabbit_channel.basic_publish(          # <-- unguarded
        exchange="cot_parser",
        body=json.dumps({
            "uid": self.uid,
            "cot": None,
            "disconnected": True,
            "user_id": self.user.id if self.user else None,
        }),
        ...
    )

    self.unbind_rabbitmq_queues()

    if (
        self.rabbit_channel                      # <-- the check the publish needed
        and not self.rabbit_channel.is_closing
        and not self.rabbit_channel.is_closed
    ):
        self.rabbit_channel.close()

self.rabbit_channel is initialised to None in setup() and only assigned later by the async on_connection_open callback. A close before RabbitMQ is ready — or while the channel is closing — raises on the basic_publish line (AttributeError on None, or ChannelWrongStateError), and everything after it is skipped: unbind_rabbitmq_queues() never runs, and the channel is never closed. The guard sitting ~16 lines below is exactly the check the publish itself needs.

Same line, second hazard: self.user is loaded inside with self.app.app_context(): in setup(). By the time close_connection() runs, that context has exited, so self.user.id can raise DetachedInstanceError.

Defect 2 — even when the publish succeeds, the frame is usually discarded

After fixing (1) the publish no longer raises — and the disconnect is still not recorded.

cot_parser.send_disconnect_cot() logs unconditionally on its first line:

def send_disconnect_cot(self, uid: str, user_id: str):
    self.logger.warning("Sending disconnect cot to {}".format(uid))

That line never appears, despite the cot_parser queue showing a live consumer and zero backlog. The message is never delivered.

basic_publish is called on the handler thread, while the pika SelectConnection ioloop runs in self.iothread (started in setup()). socketserver.ForkingMixIn.process_request() calls os._exit() immediately after finish_request() returns, so the buffered frame is discarded before the ioloop ever writes it to the socket. pika raises nothing either way.

This makes the behaviour timing-dependent — identical disconnects occasionally record and usually don't, which matches the intermittency reported elsewhere.

Reproduction

1. Connect an EUD over TLS (8089) and let it authenticate
2. Disconnect the client
3. select uid, last_status from euds; → still Connected
4. eud_handler_ssl.log shows ... disconnected; cot_parser.log shows no corresponding Sending disconnect cot to ...

Impact

• euds.last_status is unreliable. Anything built on it — the web UI's connected-client list, /api/eud, api.py's online_euds query — reports devices as connected indefinitely after they are gone. One device here showed Connected for over 24 hours with no socket.
• Queue bindings leak whenever defect 1 fires.

Suggested fix

1. Move the publish inside the existing channel guard and wrap it, so teardown runs regardless.
2. Capture user_id defensively, outside the publish call.
3. Do not depend on the async publish for state that must be durable. Either publish via connection.add_callback_threadsafe() and let the ioloop flush before returning, or write last_status directly in close_connection() — the handler already has self.app, db, EUD and update imported.

Applied locally and verified; the record now flips within ~40ms of close:

00:48:42  sockets=1  Connected
00:49:06  sockets=0  Disconnected
   LOG  close_connection - <EUD_IP> disconnected
   LOG  Marked <EUD_UID> disconnected

Happy to open a PR if the approach looks right — particularly whether you would prefer the threadsafe-publish route over a direct write, since the direct write skips cot_parser's disconnect-CoT broadcast to other clients when the message is lost.

Possibly related

#191, #206 and #209 describe similar disconnect-time failures, but all reference client_controller.py, which no longer exists in 1.7.13. This report is against the current EudHandler.py / ForkingTCPServer path.