The Unix tradition of “everything is a file” and using lots of small, specialized tools in a pipeline works quite well after you wrap your head around the philosophy and practice. The technique of combining utilities together in novel ways has withstood the test of time surprisingly well.
But today is different than the mid-70s –or even the mid-80s. More than ever before, we live in a networked world. And most of the tools in the standard Linux/Unix/Mac OS X toolbox are not network-aware. Sure, at a low level, a network socket acts a lot like a file descriptor for /etc/passwd, but in practice it’s not that easy.
Indeed, it’s hard to build “network pipelines” much of the time. Sometimes, you can get away with ssh and a pipe, such as ssh somehost cat /etc/foo | grep bar, but the ssh command often feels like it’s just getting in the way.
Swiss Army Plumbing
If Perl is the Swiss Army Chainsaw of scripting, then netcat and socat are the Swiss Army Plumbing of the Unix world. Both are incredibly powerful network utilities that try to make it easy to connect processes on various machines.
The Freshmeat project description for netcat says:
netcat is a simple Unix utility that reads and writes data across network connections. It’s a reliable backend tool that can be used directly or easily driven by other programs. It’s also a feature-rich network debugging and exploration tool, and has several interesting built-in capabilities.
netcat makes it easy to move data between two machines. For example, let’s say you need to copy a directory tree rooted at /opt/stuff from machine A to machine B and need the processes on both ends to do their work as root. Since most ssh installations don’t permit root logins, you’d likely resort to taring up the data on machine A, copying it to machine B as a non-root user, and then unpacking it. But if you’re moving a lot of data, that’s not so easy. That takes longer, and you may not have disk space for two copies of the data. You could compress it, but that takes even longer still.
With netcat it’s easy. On machine A:
# tar -cf – . | nc -w 3 -l -p 5432
And on machine B:
# nc -w 5 a.example.com 5432 | tar -xvf -
The command on machine A sets up a tar process on machine A that waits for a TCP connection on port 5432. When a client (such as machine B) connects, tar simply sends all the data through that TCP port, thanks to netcat. No temporary files are required. On machine B, another tar process reads from the network via netcat, writing the data to disk as it streams over the network. Neat, huh?
And that’s just the beginning. There’s a ton of cool tricks you can play with netcat. Check its README file.
And Then Some!
The socat web site describes socat as netcat++, implying that it’s inspired by netcat but goes even further. They key difference is that netcat works only on network sockets (typically TCP connections), but socat can connect many different types of I/O abstractions. Again, let’s look at an example.
Imagine that you have a MySQL-based application running on machine A. Because machine A is too busy, you decide to move MySQL to machine B, but keep the application on machine A. Normally, that’d be easy. You reconfigure the application to connect to MySQL remotely (via TCP/IP) rather than locally (via the /tmp/mysql.sock Unix socket). But what if the application doesn’t have a way to do that? Or, what if you no longer have the source code?
socat allows to you build a bi-directional pipe between the local socket on machine A and TCP port 3306 on machine B.
On machine A:
# socat UNIX-LISTEN:/tmp/mysql.sock,fork
That tells socat to listen to /tmp/mysql.sock, and to connect it to TCP port 3306 on machine B. The “fork” option forks a new copy of socat to handle each connection, so you can handle many instances of the application transparently without any funky multiplexing.
Now think about other applications that use local sockets for communication: X11, ssh-agent, and so on. There’s all kinds of fun to be had with socat.
Do you have an idea for a project we should feature? Drop a note to firstname.lastname@example.org and let us know.