serial port VirtualBox on Linux host(tested on ubuntu )

While installing Solaris on VirtualBox on Linux host, I was slightly annoyed by the graphical interface. I had to go out of my screen session, use mouse, click the virtual machine window, let it steal the input and then press right ctrl key to go out, and use mouse again to go back to my screen session.

Solaris x86 supports installation over a serial port. Why not use it?

VirtualBox supports attaching a serial device to a guest machine. By default, serial ports are disabled. If you go to the details tab, and see ‘Serial Ports’, you can click it and get a dialog box. Enable serial port COM1 (IRQ 4 I/O port 0x3F8). There are two port modes. One of them is ‘Host Device’. It would be very nice to use it, but VirtualBox run as a normal user lacks permissions to create devices. The other option is ‘Host Pipe’. Normal user permssions are sufficient to create a pipe, but when you run VirtualBox and examine the file type, you’ll find out that what has been actually created is a unix domain socket rather than a pipe.

How do you access serial port exposed as a socket? Use socat!

I’ve worked out two ways of accessing the serial port: by converting to TCP and by converting to a pty (pseudo teletype device). First, TCP solution. Let’s say, that the path to the socket is “/tmp/foo-socket”. We’ll use socat to convert it to a TCP server.

socat UNIX-CONNECT:/tmp/foo-socket TCP-LISTEN:8040

This command shouldn’t return to prompt. If it does, your serial port device on the guest virtual machine isn’t ready yet. If it doesn’t return, go to a different terminal window and type:

telnet localhost 8040

Voila! Another solution is to create a pty.

socat UNIX-CONNECT:/tmp/foo-socket PTY,link=/tmp/foo-pty

…and on another terminal:

screen /tmp/foo-pty

If socat returns immediately, it means your guest system doesn’t use the serial port at the moment. If you’re installing Solaris 10, you’ll have to select a serial port (ttya) installation mode first, in the bootloader.