Remosi’s Rants

Linux (and Unix in general) has a long and (imho) interesting history. It’s always evolving, always continuing to change. However in the rush there are some interesting things that things that have been buried and all but lost. Some of these are abominations that should have never ever seen the light of day (eg, gets(3)), but many of them were a good idea that’s got lost somewhere along the way, and often naively reinvented and all the knowledge that was gained the first time around has been lost.

I’m not saying that all of these technologies should be revived wholesale, many have been buried for good reason, but there are lessons to be learnt here. I refer to Linux and glibc here since thats what I know, much of this applies equally to any Unix.

Lets start with the fact on a modern Linux system you’d be hard pressed to realise that the GUI is network transparent. There is no dialog box “Run [application [v]] on host [hostname [v]] as user [username [v]].” Yet it was very common 10 years ago to be running xterm’s on a variety of boxes, executing cpu intensive commands on machines with cycles to spare. Today people are surprised when you explain that you’re running your web browser on a machine thats seperate to the machine you’re logged into. X11 doesn’t work well over high latency links I’ll admit, but over a local network it’s nearly impossible to tell that even a heavy application like a word processor isn’t running locally. Theres none of the blocky loading like you get with VNC. Yet this is becoming less and less well known.

Thin client Xterminals would speak “XDMCP” to query for which machines were the least loaded for you to login to. You could select from a list, selecting ones that were lightly loaded, but contained the resources you needed to get your work done, and being thin clients it didn’t matter which mchine you used. The support is all still there, but mostly disabled by default because of network security issues, a story that repeats itself further through the article.

Because there were only a few of these big machines, everyone would be logged into them. Commands like “ps” and “who” would show you what other people were doing. Since there were probably multiple computers around you could use rwho(1) and rusers(1) to find out where someone was logged into. You could talk to any user that hadn’t disabled it, with write(1), or talk(1). People would write blog-like entries in their .plan which you could read with finger(1). Contrary to popular belief, Mirabilis didn’t invent “Instant Messenging” in 1996, Mirabilis just made a limited form what Unix users had been doing for years available to non-Unix users. (No possibility to be logged into multiple places, no .plan/blog, having to run a seperate program to announce your presence, no rwall(1)) Almost all of these protocols have been disabled as being insecure, and have never been reimplemented in more secure ways.

rcmd(3) is still a function in glibc which allows programmers to run a command on an another machine. While rsh(1)/rexec(1)/rlogin(1) have all been replaced with ssh(1), rcmd(3) has never been upgraded and remains a forgotten relic from when the Internet was a more trusting place.

Unix machines have always been multiuser boxes, although with the rapid increase in computing power in PC’s, this is becoming less common. Unix machines have files called utmp(5) and wtmp(5) that log who’s currently logged in and who has been logged in, in the past. These files are nearly forgotten, and rarely are even written to unless someone opens an xterm(1). The sense of community that went with using a large multiuser system has disappeared.

It’s not just the community orientated commands mentioned above that made Unix machines truely part of the network rather than boxes that just use the network. With yp/NIS/NIS+ you could trivially enable large groups of machines to share critical databases with ease. Not only were passwd(5), shadow(5) and group(5) shared, but other information like services(5), networks(5), hosts(5), ethers(5), rpc(5) and probably others were also copied. To enable this you’d edit one or two lines and everything was set, extensive customisation was possible via netgroups. Compare to a more “modern” LDAP setup which requires a lot more effort to setup and maintain. Once again yp/NIS/NIS+ were horribly insecure, and LDAP at least can deliver it’s data securely over SSL, but is LDAP really an improvement?

You could tell a well administered network, as every machine had an up to date, and useful /etc/networks and often a useful /etc/ethers too. However it’s been years since I’ve heard of anyone even mention these files. Now days hosts(5) is mostly used as a kind of weird outbound firewall to prevent software talking to hosts you don’t like.

Networked file systems are also another interesting part of the story. NFS is remarkably easy to setup under Unix, it integrates well. It supports the unix semantics well, and with newer versions (eg NFSv4) supports running over a single easily (un?) firewalled port and avoiding the reliance on UIDs being identical between multiple machines.

Then there are other really interesting things that have gone on in the space like Kerberos and AFS/Coda. It’s a pity that they’re not quite so nicely integrated as YP/NIS/NIS+/NFS etc.

Old command line applications such as ftp(1) and the various r* tools supported netrc(5), a file that could contain usernames and passwords as well as aliases for machines and even would support macros to run. Programs such as curl(1) and wget(1) still support this, but this feature is rarely if ever used, and I’d doubt would be implemented in many new applications.

In fact, most config files were executable, or scriptable in some way. When you logged into a machine various scripts would run which could determine based on all kinds of criteria how to configure your setup. You could decide not to use the high colour wallpaper on the old machine that only had 256 colours (oooh! flashies! remember them?), you could dim the colours your window manager used if you logged in after dark. Your $PATH could be setup based on hardware platform, OS version or even if directories existed or not. Window manager menus would be tweaked based on if this computer had a two or three button mouse, or may decide between several different programs depending on what was installed. A “Run Xterm on…” menu might be populated from /etc/hosts.equiv. Because the majority of users prefer not to program a lot of this kind of functionality has been removed. Now if you use the same config from multiple computers you’d better aim for the lowest common denominator.

You used to be able rely on every machine being able to store mail locally, all kinds of programs still look in $MAIL to check if you have new mail. Programs used to assume they can send you email if they had something non urgent to tell you. Mail clients would check the local mail spool for your messages. Anyone remember having bash(1) tell you that you had new mail?

The current trend with SOAP and XMLRPC is also amusing, given that glibc still contains all the sunrpc xdr encoding systems. These are solved problems. ASN.1 is another encoding scheme thats been around for ages. Neither appears to have any real problems (other than they don’t have standardised methods of passing them over HTTP, although that’s perhaps more a feature than a bug). Yet people still insist on reinventing them.

There are other weird features in libc that are there for one reason or another strfry(3) and memfrob(3) being amusing examples. Other examples include hsearch(3), lsearch(3), tsearch(3), insque(3), dbopen(3) etc… None of these as far as I’m aware are part of any standard.

X11 has had several other interesting ideas that have been often overlooked. Window managers were always extremely extensible, and supported all kinds of interesting features. Virtual Desktops and Workspaces are something that everyone has needed to support to be even considered (and I really wish the other OS’s would pick up. I really feel cramped without my virtual workspaces!). Lots of interesting things have been done with window managers that are being slowly forgotten. Focus follows mouse is about as close to “focus follow eyes” as you can get vs Click to focus’s “is it safe to click on that part of the window right now or will that cause something to execute now?” issues. Meta-LeftClick to raise, Meta-RightClick to bury, Meta-RightDrag to resize, and Meta-LeftClick to move all being fairly standard, but modern window managers are starting to ignore. Window Manages

X has a “resource database” which has a standard config for everything. Want to change the font size of some random little bit of text? Easy. Want to change the colour of something? Simple. Want to make all your xterms use visual bells? These are per X server (”Display”), and thus applications running remotely will pick up the local styling, even if your /home is not shared between the two. However modern toolkits ignore that,

On the flipside modern desktop “environments” are doing really well at looking really pretty (ooh! cubes!). however you can find lots of people using window managers that are eye burningly highlighter blue because they support the way they work.

There have been some great ideas in unix that have been discarded for one reason or another, and often even “reinvented” with all the lessons of the past doomed to be reinvented.

(Postscript: In editing this post to link all the stuff I wanted to link I realised I could have used a regex(5) to link all the manpages, but my browser doesn’t do regex searches let alone regex search and replace! Why the hell not?!)

So as you may remember, I've been thinking about (and quietly implementing) a distributed filesystem.  Tonight I got as far as it mostly working, and since everyone around here is sick of me telling them about it I thought I'd tell you all about it instead :)

It uses the 9p protocol to talk to the kernel (although I'm using the 9p python client for testing).  I use the chimera DHT for indexing, and libnpfs for dealing with the low level details of the 9p protocol.  I've managed to find bugs in all of these pieces of software so far :)

 Anyway, onto the interesting part of the show:

9p> ls
foo/
9p> cd foo
9p> ls
readme
9p> cat readme
Hello World
9p> mkdir sample
9p> ls -l
-rw-r--r-- perry perry 0        readme
drwxr-xr-x perry perry 0        sample
9p> cd sample
9p> ls -l
9p> put testfile
9p> ls -l
-rw-r--r-- perry perry 0       testfile
9p> cat testfile
This file is a test
9p> 

You may notice that the file sizes are all 0, even though most of them seem to contain data, this is coz I've not finished implementing stat(2) yet.  Permissions etc are kinda implemented (they're stored, but not read off disk yet, although libnpfs appears to verify them for me).

There's a bit of polishing off to do before the "local" part of the filesystem is mostly viable. Then I just have to hook all the distributedness together and I should have a fully functional prototype! (yeah right…)

Networking policy is currently defined in 3 seperate places:

• The Routing Table
• Firewalling Rules
• QoS Rules

Each of these systems has different mechanisms for matching packets. Under Linux (and under most systems) the firewalling rules have the most sophisticated mechanisms for matching packets, QoS matching rules are somewhat random, and the routing rules are overly simplistic (You can match on source network, destination network, and ToS bits).

This leads to bizarre hacks like FWMARK, where you use Linux's firewalling matching rules to match a packet so you can do QoS or Routing based on it. You also can do the same thing in multiple different ways. To prevent the routing of a packet you can use a reject route, or you can REJECT it using firewalling.

Thus my solution is to merge QoS, Firewalling and routing into one networking policy engine. A packet can be matched using a pluggable matching system, have various effects applied to it (such as NAT, ToS mangling, IPsec, logging etc), and then can be destined to one or more queues of the local machine, an interface, or a locally connected host.

Some example rules for a host with 3 interfaces (eth0 an expensive low latency internet connection used for ssh, eth1 where most of the internet traffic goes, and eth2 which has a nat'd network behind it) might look like:

# Firewall rules
 protocol tcp, destination-port 137, reject
 protocol udp, destination-port 137, reject

# Policy routes
 from 10.1.0.0/16, protocol tcp, destination-port 22,
   via 192.168.1.1, dev eth0, queue lowlatency

# NAT
 from 10.1.0.0/24, dest 192.168.0.0/16, snat 192.168.1.1

# Routes for local processing
 dest 192.0.2.254, dev host
 dest 10.1.1.254, dev host
 dest 192.168.1.254, dev host

# Routes for local networks
 dest 10.1.0.0/16, dev eth2, queue normal
 dest 192.0.2.0/24, dev eth1, queue normal
 dest 192.168.0.0/16, dev eth0, queue normal

# Default route(s)
 dest 0.0.0.0/0, via 192.0.2.254, dev eth1, queue normal

Queues need to be configured seperately, probably much as they are done today. I think such an integrated system would ease system administrators burden of having to specify policy in several difference places using several different algorithms.

I have written a tool for answering ARP’s out of DHCP, as well as logging suspicious ARP behaviour. The program works by connecting to an isc-dhcp server using the omapi protocol, by sniffing arp requests via libpcap (I skipped on the libtrace requirement just this once — theres no point in sending arp replies from a trace anyway), and generates arp replies with libnet.

Some issues with this system is that you need some way to filter ARP replies on your network, you need a switching fabric smart enough to firewall arp replies from, and arp requests to access ports. We are doing this by firewalling traffic through a softmac 802.11 network. This is complicated at best, and downright impossible almost all the other times.

Another issue is that the daemon currently forges the source MAC for the reply packet to come from the destination device which keeps things as similar as possible from the protocol layer, however this may confuse a switch that sees a packet originating from a different port. This could be fixed by having it not forge the source MAC, but that has potential issues with client machines not accepting the arp. This is, however, untested.

The third issue is that the entire program is untested, so use at your own risk. It’s available under the GPL, from svn. My libconfig is required to build. Comments solicited.

Security tools should be simple. They should do one thing and do it well. The SSH people have said that they are adding VPN support. This I think is a particularly bad idea. Companies are paranoid about security. Many companies already ban VPN programs, they punch holes in their carefully blocked up firewall. Would they, should they ban SSH as being a VPN tool? Even the current portforwarding features in ssh have caused problems with peoples security arrangements, with it people can start punching holes through your precious firewall, imagine how much worse it will be if they have full IP access to every port on every machine inside your network. Some companies have already banned ssh because of the port forwarding argument. What advantage does SSH having VPN capabilities have? If you were a network manager, would you require people to use vpn software? Would you allow people to use ssh?

So after several long heated discussions about my previous blog entry, I’ve go another kinda related point to make. Emulating Windows will not help us Win. If you can target linux and windows with the same executable, which OS are you going to test on? Thats right, windows. Ask any Java developer about write once, run anywhere. They’ll tell you to write once, test everywhere. In fact, they get bug reports on such obscure hardware that they cannot easily reproduce or test the bug. So, you’ll end up with software that (probably due to some bug in the software) works well under Windows, but fails to run under Linux. So, if you are going to have to select which OS to run these applications on, which one are you going to choose? Windows of course. So, we’ve gone to all this effort to have people develop software under Windows for users running, uh, Windows. However, it’s even worse than that, instead of developing a Linux version of their software people can say “It should just work under Linux”. So, now we don’t have a Linux version of the software, we have a Windows version that runs under Linux, where everyone complains that it just doesn’t look right. Maybe because it calls things a “Wizard” instead of a “Druid”, or maybe it refers to the “Start Menu” and “Internet explorer” in the help and/or documentation. Linux isn’t a first class citizen.

• So, why write a business app for users for Linux? Write it for C# on windows, then those linux hippies can use it too
• Why port random-important-app to linux? We can just get them to run it under wine.

So, I say that Wine and .NET are bad for Linux. They don’t help, they just give people excuses for not helping us.

The current big thing seems to be VMs. Java’s got one, .NET has one, hell, even Perl’s getting one. Now, Java had one for several reasons mostly write-once-run-anywheriness. I’m not sure why .NET has one. Microsoft can’t want people to write code that can run anywhere but Microsoft operating systems, unless they’re planning on moving away from x86. (Watch out Intel, they’re onto you!). Perl seems to be getting one because it’s the fashionable thing to do. “Everyone else was doing it and I wanted to be cool.”[1]

Now, for the closed source community a virtual machine is important. It mostly obscures your source code er, “Intellectual property”, and it gives you hardware independance so you don’t have to develop for lots of different platforms. But these requirement doesn’t exist for open source software. We don’t want to hide the source code, in fact, we work very hard to make sure that everyone has access to the source code. Many open source scripting languages don’t use a VM (such as PHP), and really, noone cares particularly much. Putting a PHP script under the GPL is like selling a car with a license that says that everywhere the car is, you take the chassis along too. I mean, a car that doesn’t have a chassis isn’t a car.

Now, the mono people have apparently stated that the reason they used Mono was because they wanted an easy way to provide bindings to multiple languages. They can now target .Net and then every .NET “enabled” language gets the bindings for free without a lot of work being repeated. Presumably they also used .NET because Sun will get rather angry at them if they try and use Java with bindings to Gnome. Also, programmers desperately need a decent language for writing applications in. C is a great language for writing lowlevel code in, however it’s a royal PIA for doing highlevel application development. C++ has the potential for being great if it actually had a decent standard library. Want to fetch a URL? Whoops, you’ve got to write code for HTTP, and resolving and network sockets on top of the C api’s which don’t “mesh” with C++ very well. (Look at getaddrinfo(3), and consider how much effort it takes to nicely wrap that into some C++ classes and repeating that for every program you write, compare to Java/.Net/Python/Perl libraries). Python is quite popular for writing small to medium sized gui applications, particularly with the Twisted library. But it’s not exactly zippy, and many people won’t touch it on religious grounds.[2]

So, my thought here is why are we following this lead. We have the best representation of the program at our disposal, the source code. The source, as written by the programmer has the highest level of expressibility you can achieve. Instead of trying to JIT some abstract byte code, you can compile the source directly into executable code. Instead of .pyc files littering around the place, you can have “.so”’s and executables kicking around. In fact some distro’s are starting to do this, for example Gentoo where you download the source, then compile it to meet the specifications of your computer.

So what pieces of the puzzle are we missing? Well, the major feature .NET has brought to the table has been a standard ABI. Basically the idea that programs written in one .NET language can call functions written in another .NET language, or even outside the .NET framework using P/Invoke. What we should be doing is trying to come up with a standard ABI that encapsulates all these things that compilers for different languages can all implement. This ABI could even be a subset of the C++ ABI, as C++ seems to be a superset of every language feature ever. You want strings (std::wstring)? templates? even things like garbage collection can be “easily” retrofitted into C++ (boost::shared_ptr<> or even more complicated systems). VMS did this, you could call pascal code from C, and both from fortran. The Unix people have quite happily standardised on the C ABI, almost everything supports it.

I tried doing some of these things before. Swig provides a way to allow arbitary C/C++ code to be made available to a large range of languages. I’ve tinkered with writing code that using nm(1) and c++filt(1) to automatically generate a Swig file, compile it, then have this all triggered from a python import hook so you can say “import foo” and it will go off, read the symbols out of /usr/lib/libfoo.so, process them with c++filt to get the function prototype, have it generate a swig file, compile said swig file into a .so which python can load as a module, and then proceeding to do so.

The code to do this is all fairly straight forward, and can be easily cached in ~/.autoswig/python/ or whatever. It’s dependant on reasonably standard tools, and will work for a large set of languages (basically whatever swig supports). It can be easily improved by just dlopen(3)ing the library, using libbfd(3) to parse the debugging information to get the prototypes of the functions, and some magic code to marshal data to the library, which is nearly what P/Invoke does. P/Invoke however requires you to write your own prototypes, my suggeston is to use the debugging (stabs) information that is already in the binary to build the prototype directly. My biggest issues with this was not wanting to write platform specific “magic code” (as mentioned above) since I don’t have access to a wide range of hardware. My other major issue that calling conventions are a nightmare particularly to do with whose responsibility is it to free allocated memory.

So I ask, why are the open source community wasting time on VMs when we can use the strength of our source to provide a superior solution?

[1]: Ok, I’m harsh on the perl people again :) Both perl and python have a VM that they are “compiled to” to basically cache the result of parsing the source.

[2]: Apparently many people were fortran programmers in another life and have a genetic fear of whitespace being used by the compiler, irrespective of the fact that people will moan if that whitespace isn’t there anyway.

I’m getting sick of people muttering about Usability. Now, don’t get me wrong. I’m not against Usability, but usability as people propose comes at a cost, and that cost is functionality.

For instance, the gnome people have changed their default window manager from sawfish to metacity. Metacity’s “easier to use” because it has no features. Theres nothing confusing for users to configure. However, you lose a lot of functionality. You can’t have multiple workspaces with multiple desktops. You can’t bind scripts to keys anymore. Theres a whole heap of things you can’t do. This is apparently a “feature”. It’s apparently not confusing to users. Not that I knew of any user who would want to configure their window manager without expecting pain and suffering.

My problem isn’t that the programs are now magically usable. My problem is that there is now no way to do the things I was doing before. It’s not that the features have been hidden behind an “Advanced…” button, or even hidden so that they can only be configured via an obscure about:config interface, or via direct editing of the configuration files. Those features just don’t exist. I’d be happy to have to drop to vi and edit a config file to edit my window manager, in fact, that was how I did it for many years. I’d almost be insulted if there was an easy gui way of setting things up.

The reason I moved to Linux from ol’ MSDOS was because I liked the power and flexibility of Linux. I could see everything, and contrary to popular belief, for a programmer at least, Linux has more documentation than you could ever wish for. If you can’t find the documentation for something you can always just go and read the source! But the reasons I came to love linux are being eroded. Choices are being reduced in the name of usability.

Usability exports regularly complain that open source software is hard to use, and has thousands of tiny, obscure options. But the reasons are that Open source software has been made by programmers for programmers, and programmers want, if not need, to be able to tweak things. Can the two ideals coexist? I think they can. Advanced users are willing to take the time to read the documentation to figure out how to customise weird options. No advanced user would turn away the opportunity for making features easier and quicker to use in the common case. But don’t take away features that “new users” don’t want or need! They don’t have to be there right “in your face”. They can be hidden behind an Advanced… button, or in a command line option, or an option in a config file. Advanced users will look for the feature and will be plesantly surprised to find it, and users who are afraid of the myriad of options won’t see them.

The reason I came to linux was because of the choice. Please! Stop limiting them!