Test suite: make the remote shell server handle aborted client commands

anonym
2025-07-09T09:07:49Z

The remote shell server is a simplistic single threaded server. It waits for a request, upon receipt it executes the command, sends the response, and loops. If a client aborts a command on its side (i.e. stops listening for the response) and sends a new request, the server won't process the new request until it has finished executing the first command because the server was never told to abort it.

This code example might help:

  begin
    Timeout.timeout(5) do
      $vm.execute_successfully('sleep 20')
    end
  rescue Timeout::Error
    # The above timeout will trigger, but we ignore it and continue.
  end
  # This timeout will trigger since `sleep 20` is still executing and
  # thus blocking the remote shell server from processing new requests.
  Timeout.timeout(5) do
    $vm.execute_successfully('true')
  end

For a command that runs indefinitely (example: #20263 (comment 255352)), the client sending a new request will result in RemoteShell::Timeout after (currently) 20 minutes.

Note that in RemoteShell#communicate we handle "out-of-order remote shell response", which is fine for commands that doesn't take too long, i.e. shorter than the 20 minute RemoteShell::Timeout and shorter than any timeout used for the next command. But to fully handle aborted client commands the server cannot block while executing. So we should make it so that the server runs the commands in a thread and continues processing requests all the time.

Given how the client and server communicates by simply write()ing/read()ing on a Virtio channel we have to be a bit careful since this new concurrency can screw things up. It might be simplest to let the main thread be the only one interacting with the channel, i.e. it read()s new requests, but once a thread is done it somehow stops doing that and write()s the response. Maybe it's even worth making the client tell the server when it aborts a command so it can kill the corresponding thread, both to avoid concurrency on the server side and because it makes it easier to reason about what's going on since the client side and server side agrees that a command is aborted.