For general server architecture, see
README.md. This module implements the watchdog process.
Watchdog state #
Most LSP clients only send us file diffs, so to facilitate sending entire file contents to freshly restarted workers, the watchdog needs to maintain the current state of each file. It can also use this state to detect changes to the header and thus restart the corresponding worker, freeing its imports.
TODO(WN): We may eventually want to keep track of approximately (since this isn't knowable exactly) where in the file a worker crashed. Then on restart, we tell said worker to only parse up to that point and query the user about how to proceed (continue OR allow the user to fix the bug and then continue OR ..). Without this, if the crash is deterministic, users may be confused about why the server seemingly stopped working for a single file.
Watchdog <-> worker communication #
The watchdog process and its file worker processes communicate via LSP. If the necessity arises, we might add non-standard commands similarly based on JSON-RPC. Most requests and notifications are forwarded to the corresponding file worker process, with the exception of these notifications:
- textDocument/didOpen: Launch the file worker, create the associated watchdog state and launch a task to asynchronously receive LSP packets from the worker (e.g. request responses).
- textDocument/didChange: Update the local file state so that it can be resent to restarted workers.
Then forward the
- textDocument/didClose: Signal a shutdown to the file worker and remove the associated watchdog state.
Moreover, we don't implement the full protocol at this level:
- Upon starting, the
initializerequest is forwarded to the worker, but it must not respond with its server capabilities. Consequently, the watchdog will not send an
initializednotification to the worker.
initialize, the watchdog sends the corresponding
didOpennotification with the full current state of the file. No additional
didOpennotifications will be forwarded to the worker process.
$/cancelRequestnotifications are forwarded to all file workers.
- File workers are always terminated with an
exitnotification, without previously receiving a
shutdownrequest. Similarly, they never receive a
Watchdog <-> client communication #
The watchdog itself should implement the LSP standard as closely as possible. However we reserve the right to add non-standard extensions in case they're needed, for example to communicate tactic state.
- terminated: Lean.Server.Watchdog.WorkerEvent
- importsChanged: Lean.Server.Watchdog.WorkerEvent
- crashed: IO.Error → Lean.Server.Watchdog.WorkerEvent
- ioError: IO.Error → Lean.Server.Watchdog.WorkerEvent
Events that worker-specific tasks signal to the main thread.
- crashed: Array Lean.JsonRpc.Message → Lean.Server.Watchdog.WorkerState
The watchdog can detect a crashed file worker in two places: When trying to send a message to the file worker and when reading a request reply. In the latter case, the forwarding task terminates and delegates a
crashedevent to the main task. Then, in both cases, the file worker has its state set to
crashedand requests that are in-flight are errored. Upon receiving the next packet for that file worker, the file worker is restarted and the packet is forwarded to it. If the crash was detected while writing a packet, we queue that packet until the next packet for the file worker arrives.
- running: Lean.Server.Watchdog.WorkerState
- doc : Lean.Server.DocumentMeta
- state : Lean.Server.Watchdog.WorkerState
The pending requests map contains all requests that have been received from the LSP client, but were not answered yet. We need them for forwaring cancellation requests to the correct worker as well as cleanly aborting requests on worker crashes.
- hIn : IO.FS.Stream
- hOut : IO.FS.Stream
- hLog : IO.FS.Stream
Command line arguments.
- initParams : Lean.Lsp.InitializeParams
We store these to pass them to workers.
- workerPath : Lake.FilePath
- srcSearchPath : Lake.SearchPath
Tries to write a message, sets the state of the FileWorker to
crashed if it does not succeed
and restarts the file worker if the
crashed flag was already set. Just logs an error if there
is no FileWorker at this
Messages that couldn't be sent can be queued up via the queueFailedMessage flag and
will be discharged after the FileWorker is restarted.