[agentserver] Process-isolated hard execution cap for task timeouts (spawn + fork)

[Nathandrake229]

Summary

Adds a hard execution cap for per-turn task timeouts in azure-ai-agentserver-core (durable/resilient tasks). When a per-turn timeout elapses and the handler ignores the cooperative cancel, the framework can now force-stop it after a bounded grace by running the handler in an isolated child process the timeout watchdog can SIGKILL — without disrupting the main container or co-located tasks, and without the recovery system resurrecting the force-stopped turn.

Draft / WIP. This branch is rebased onto latest main, which recently landed a breaking change to TaskContext (ctx.metadata moved off TaskContext). The isolation code + test handlers were written and validated against the pre-change base, so the isolation tests currently fail on this rebased branch at ctx.metadata. Follow-up: adapt _build_child_context / _build_worker_snapshot / IsolationBridge + test handlers to main's new metadata model. The design and mechanism are otherwise complete and were validated end-to-end on the prior base (see Testing).

Design

Full design doc included: sdk/agentserver/azure-ai-agentserver-core/docs/task-timeout-isolation-design.md (common core + backend-specific internals for spawn vs fork).

Key points:

• Opt-in, default off (AGENTSERVER_TASK_ISOLATION); zero overhead when off — _run_handler short-circuits to await fn(ctx).
• Only authoritative stop mechanism for arbitrary native Python is a separate OS process + SIGKILL; in-process injectors are non-authoritative/unsafe (documented, with the full options-evaluated table).
• Tiered watchdog: cooperative cancel (ctx.timeout_exceeded → ctx.cancel, persist timeout_cancelled_at marker) → grace (AGENTSERVER_TASK_TIMEOUT_HARDCAP_GRACE_SECONDS, default 1h) → SIGKILL → _WorkerKilled → CancelledError → existing finalization (one-shot delete / multi-turn drain-or-suspend).
• Durable terminal-outcome IPC contract (MSG_RESULT/SUSPEND/EXIT_FOR_RECOVERY/ERROR + metadata); parent stays sole store/lease writer; marker + derived backstop make recovery finalize (not re-run) a timed-out turn.

Backends (differ only in worker internals)

• Spawn (create_subprocess_exec, portable default): blank interpreter, re-imports app, std-pipe transport.
• Fork (multiprocessing fork, Linux-only, AGENTSERVER_TASK_WORKER_FORK): inherits the imported app (COW, ~ms start, no re-import); _after_fork_child sanitization (set_task_manager(None), close inherited peer fd, fresh loop); socketpair transport; reuses all shared worker code via a _ForkProcAdapter.
• Per-chain reuse + idle-TTL (AGENTSERVER_TASK_WORKER_REUSE, multi-turn only): a spawn optimization (import once/chain); fork barely needs it.

All backends share the same protocol, IsolatedRun/PersistentWorker, bidirectional ctx bridge, terminal contract, and recovery — selected by a single OS-branch at worker creation.

Testing (on the pre-rebase base)

• Full tests/tasks suite green with isolation off (654 passed) — default path unchanged.
• Isolation suite (spawn one-shot / multi-turn / reuse) green; fork suite (one-shot + multi-turn drain/suspend + reuse same-pid / hard-kill respawn / idle-TTL) green on Linux/WSL. Windows: fork tests skip (Linux-gated), spawn tests pass.
• Live westus2 hosted Foundry agent: spawn validated (one-shot hard-kill+delete, not recovered; multi-turn drain); fork validated (deployed with AGENTSERVER_TASK_WORKER_FORK=1; parent container survived fork-child hard-kills and kept serving).

Follow-ups

• Fix rebase drift: adapt isolation + test handlers to main's new metadata model (blocks the isolation tests on this branch).
• Optional: PR_SET_PDEATHSIG orphan-child hard guarantee (Linux).
• Optional/deferred: clearer error when a handler calls the Tasks API (enforced boundary; see design §8/§11).

---

🤖 Draft opened via GitHub Copilot CLI.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
9 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.