crash on disconnect

[ericweiner]

Maybe I bumped the USB cable?

LVP_Logbumped wire.zip

[ericweiner]

Triage from LVP_Logbumped.wire.zip

Disconnect timestamp: 16:07:05.805 (cable bump, single instant, motor + camera both hit at once).

Observable signatures

Source	Pre-bump	At-bump
Camera (Pylon)	err_code=3791651346 'Payload data has been discarded' (transient, recoverable, ~30 events over 50 min)	err_code=433 'A device which does not exist was specified' -- 100+ events in 3s
Motor (serial)	normal target_status reads	TARGET_RZ -> TIMEOUT (0.0ms) (port gone), then target_pos(Z) failed: int() argument must be a string... not 'NoneType' (driver got None from a dead port)

Failure cascade

16:07:05.805 PylonImageGrab    err_code=433 device-gone (first occurrence)
16:07:05.805 autofocus_thread  TARGET_RZ -> TIMEOUT
16:07:05.806 autofocus_thread  target_pos(Z) failed -- None response
16:07:05.806 autofocus_thread  move_rel_pos(Z): cannot read position, skipping move
16:07:05.807 motion-monitor    MotorBoard.get_limit_status(Z) inactive
16:07:05.807 motion-monitor    get_target_status(Z) failed -- STATUS_R returned None
                               Traceback:
                                 _lumascope.py:4811 get_target_status
                                 motorboard.py:1107 -> raise ValueError("STATUS_R returned None")
16:07:05.807+ PylonImageGrab   err_code=433 x 100+ consecutive (log ends 16:07:08)

gui_interactions.log last entry is 15:24:15 (the Logs Zipped popup OK from the earlier TSR session). No user GUI action logged between the bump and end-of-log, so app state at end-of-log is ambiguous: crashed / frozen / user killed.

Code findings

1. Rule 29 violation -- motorboard.py:1107 raises untyped ValueError. Hardware-disconnect should raise a typed exception from modules/exceptions.py (e.g. HardwareError) so the API layer can classify and route to notifications.error per Rule 14. The untyped ValueError leaks past the _lumascope.py:4811 try block as a stack trace into the motion-monitor thread without notifying the user.

2. Rule 14 / Rule 40 violation -- no user notification fired. The protocol/AF threads kept attempting operations on dead hardware. The existing 30s are_all_connected() check that protocol-failure classification (ce209c9) routes through would catch this -- but it fires every 30s; the cascade is 100+ events in <2s. Fast-cascade detection is a separate code path.

3. Pylon err_code=433 is observable + distinct from 'Payload data has been discarded'. Today both fall under the same warning wording. Per Rule 40 the "silent-stuck" / "hardware-disconnect" distinction needs to surface to the user. The cleanup: classify Pylon error codes inside pyloncamera.py so 433 (device-gone) routes to a typed HardwareError while 3791651346 (payload discarded) stays a transient warning.

Disposition

Fix slots into project_hw_disconnect_detection.md (planned, not shipped) and is the fast-cascade variant of the protocol-failure classification work (ce209c9). Queued on 4.0.0-wave7-decomp after Wave 7 Phase 2c lands. NOT bench-blocking -- steady-state disconnect detection already works at 30s cadence via ce209c9; this issue is the <2s cascade variant.

[ericweiner]

Two surgical fixes shipped on 4.0.0-wave7-decomp. Both target patterns visible in the LVP_Logbumped.wire cascade from this issue.

Site 1 -- 6769830 fix(motorboard): target_status raises typed HardwareError on disconnect

drivers/motorboard.py::target_status() raised an untyped ValueError("STATUS_R returned None") when the firmware response was None (timeout / disconnect). The API layer caught it as a generic Exception and could not classify it for a Rule-14 notification; the cascade visible in lumaviewpro.log at 16:07:05.807 (the _motion_monitor traceback into motorboard.py:1107 -> raise ValueError(...)) is exactly this leak path.

Fix: replaced with HardwareError, matching the existing pattern for home() / zhome() / thome() in the same file. Updated docstring Raises: section. Also corrected a misleading log line ("MotorBoard.get_limit_status(axis) inactive" -- the function is target_status, not get_limit_status) per Rule 20.

1 regression test (test_target_status_raises_hardware_error_on_none) asserts HardwareError fires on simulated serial timeout, mirroring the test_home_no_flag_on_none pattern.

Site 2 -- f3a5253 fix(pyloncamera): classify err_code=433 as fast-path device disconnect

OnImageGrabbed today routes all err_codes other than _PYLON_ERR_BUFFER_CANCELED (0xE2000102) and _PYLON_ERR_PAYLOAD_DISCARDED (0xE2050012) into the generic fallback: WARNING log + _record_failure() + only auto-disconnect after MAX_CONSECUTIVE_FAILURES (128 frames =~4.3s at 30fps). Your bench cascade shows 100+ err_code=433 events in <2s -- that's 3 seconds of WARNING spam before the slow-path disconnect surfaces.

Added _PYLON_ERR_DEVICE_NOT_FOUND = 433 as a third classification branch:

• log once at ERROR (real disconnect, user-actionable);
• call _mark_disconnected() immediately so the API-layer Rule-14 notification fires off the disconnect flag in 1 frame instead of 128;
• stop grabbing immediately so the SDK doesn't keep firing callbacks at the wedged handle;
• skip _record_failure -- physical removal has its own signal (the disconnect flag); the consecutive-failure counter is for transport degradation, not single-event removals.

5 structural regression tests in tests/test_audit_fixes.py::TestPylonDeviceNotFoundClassification (mirroring TestPylonPayloadDiscardedClassification): constant value matches bench-witnessed 433, source comment explains the cascade-rate WHY, OnImageGrabbed contains the elif branch with the correct outcome name, branch calls _mark_disconnected, branch skips _record_failure.

Still open (separate concerns, tracked in TODO):

• The fast-cascade detection in MotionAPI / motion-monitor (Rule 14 notification firing off the typed exception now that it's typed) is the next layer in this stack; site 1 unblocked it but the notification wiring is separate.
• Pre-existing Rule 16 sibling: drivers/serialboard.py::_open_serial:110 raises untyped ValueError("No port found for {label}") on hardware-absent connect-time. Same pattern as site 1; different concern (connect-time vs runtime); follow-up cleanup commit pending.

Bench retest from 4.0.0-wave7-decomp 8bec51d (or later): bump the cable mid-grab; expect the WARNING spam to be replaced by a single [CAM Class ] Camera device not found (USB disconnect / device removed) err_code=433 ... ERROR line followed by the existing disconnect-notification path. Motion-side STATUS_R failures during the same event should now surface as HardwareError in the log instead of raw ValueError stack traces.

[ericweiner]

Both surgical fixes are in beta12 (`4.0.0-beta @ f64073c`):

• `6769830` motorboard `target_status` raises typed `HardwareError` on disconnect (was untyped ValueError leaking into motion-monitor traceback).
• `f3a5253` Pylon `err_code=433` (device-not-found) routed to fast-path disconnect: single ERROR line + immediate `_mark_disconnected` instead of 100+ WARNING spam over 3 seconds.

Test asks (only if you can deliberately bump a cable -- this isn't a daily-use repro):

1. Cold-start LVP, get to live preview (any layer, LED on).
2. Pull the camera USB cable (or motor cable if you want to test that path) while live preview is running.
3. Expected after fix:
   • Camera disconnect: ONE log line at ERROR severity (`[CAM Class] Camera device not found (USB disconnect / device removed) err_code=433...`) followed by a user notification ("Camera disconnected" or similar via Rule 14). No 100+ "Payload data has been discarded" WARNING spam.
   • Motor disconnect: `HardwareError` traceback in motion-monitor (not raw `ValueError`); user notification fires.
4. Reconnect the cable, restart LVP, verify it re-connects cleanly.
5. Attach `lumaviewpro_errors.log` showing what surfaced during the disconnect window.

If you can't physically bump cables, no action needed -- next field issue with a disconnect will validate.

Still open / separate concern: the fast-cascade Rule-14 notification from MotionAPI when HardwareError fires on motion-monitor (the typed exception is in place; the notification wiring is a separate follow-up).

[ericweiner]

Fix shipped -- ready for bench verification

Root cause: Cable-bump disconnects (motor or camera) produced unhelpful tracebacks and sometimes hung the protocol thread. Two separate code paths: motorboard raised raw OSError that the protocol runner didn't classify; pyloncamera waited the full grab-timeout before recognizing the SDK-disconnect signal.

Fix: drivers/motorboard.py now raises typed HardwareError on serial-port disappearance, routed through Rule 29 error propagation so the protocol runner classifies it as fatal. drivers/pyloncamera.py fast-paths Pylon err_code=433 (device removed) to skip the grab-timeout wait.

SHA: 6769830 + f3a5253 on 4.0.0-beta (current tip 4b972e9, 4.0.0-beta14 build).

Regression test: tests/test_serial_safety.py + tests/test_audit_fixes.py cover both paths.

Retest signal:

1. Mid-protocol, deliberately yank the motor USB cable. Expected: a clean error popup naming the motor disconnect; protocol thread aborts cleanly (no hang).
2. Mid-protocol, deliberately yank the camera USB cable. Expected: error popup within ~1 s (not after a 5+ s grab timeout); protocol aborts cleanly.
3. Reconnect, restart LVP, verify normal startup.

EtalumaSupport, please download the latest beta build and verify the fix. Then either close this issue if fully resolved, or add a note describing what is still happening.

[ericweiner]

is this better?

LVP_Log.zip

(read from bottom up, there was some other stuff I was testing earlier)