After 2 days I figured out the webrtc protocol I'm going to use for streaming video from the robot's camera and controlling the robot from the browser. Of course it's not just a simple http server with requests and responses. By this point I'd already run into a ton of edge cases and odd behavior during reconnects and reboots. I started with a smart signaling server but ended up just broadcasting the message through websocket like an idiot.
Result on the screen. The terminal in the bottom left is a python script sending the offer, the browser on the left is the robot UI with a dualsense connected. After the connection is set up commands from the joystick go through webrtc into the python script through the data channel and for now they're just logged there.
