Things you probably never wanted to know about MP3 files

Isomer — Thu, 20 Mar 2008 10:26:13 +0000

I’ve been playing around with the mp3’s I’ve got here, trying to automatically find the ones that are broken and need to be reripped from my original CD’s. Some have been truncated over the years, some are just encoded at horrible bitrates that make your ears bleed, and some are missing tracks from the album. Some have ID3v1 tags, some have ID3v2 tags, some have both but they disagree on the information. Some don’t have any tags, some are just named wrong.

So I thought I would write a script using a nice library to find out the length of the song, and pull out all the metadata from the ID3v1 and ID3v2 tags and look the data up in The musicbrainz database to check for accuracy. Then write out nice clean mp3 files with consistant ID3v1 and ID3v2 tags (annoyingly some older hardware mp3 players can only read ID3v1 tags, but ID3v1 is so crappy you really want to use ID3v2 where you can). I thought this should be a nice straight forward evenings work. Oh boy was I wrong.

None of the libraries I looked at would tell me the length of a track. Most of them would decode the ID3v2 and ID3v1 tags, but most would only give you the ID3v2 information if there was an ID3v2 tag and ignore the ID3v1 information (so I couldn’t easily make sure both tags were consistant). Most didn’t appear to give me the option to write out a ID3v2 and a ID3v1 tag into the same file. Now I expect there probably are libraries out there that do this, but it was disheartening enough that I decided to write my own mp3 parser library. I’m glad I did, I learnt a lot of interesting things along the way.

I started off reading the mp3 frame header specification. An mp3 file is basically a series of frames. Each frame begins with the “frame sync”. If a frame sync isn’t found, you skip until you find one. This allows recovering from a corrupt datastream. Then each header has information about how this frame was compressed. (In general) Each frame is independant and can be considered in isolation. An mp3 file is just a series of these frames one after another. So my program started out by parsing all these headers keeping note of how long each frame is so I could calculate the total duration of the mp3 file.

Now, if an mp3 decoder doesn’t find an mp3 frame where it expects one, it just skips over the data in the file until it does find a valid frame sync. This means you can put arbitary data into an mp3 file and mp3 decoders will ignore it. So what happens? Every man and his dog decides it’s a great idea to squirrel some data away in mp3 files.

The most obvious of these is the ID3v1 tag. The ID3v1 tag is the last 128 bytes of an mp3 file and starts with the characters “TAG”. It then has some fixed length fields for the artist, album name, title, comment etc. There is also a “Genre” field that has a possible 80 fields, latter extended by winamp to 125. A varient of this tag (ID3v1.1) also has a track number by borrowing bytes from the comment field.

Because the ID3v1 tag is annoying in oh so many ways (limited length fields that a large propotion of songs exceed, limited number of fields, the tag being at the end of the file making getting the information when you are streaming a file impossible until the files entirely copied etc), ID3v2 was invented to solve these issues. ID3v2 tags go at the beginning of the file, they have an arbitary number of tags that are of dynamic length. Yay! You’d think.

Except there are ID3v2.2 tags (the original ID3v2 format — I have no idea what happened to ID3v2.0 or ID3v2.1),ID3v2.3 which is what a lot of the software I’ve seen today means when it says “ID3v2″, and then theres ID3v2.4 which not many people seem to implement yet.

So now my library parses ID3v1, ID3v2.3 tags (I fortunately didn’t find any ID3v2.2 or ID3v2.3 tags in my collection), and can give me summary statistics on the mp3 stream (duration, histogram of bitrates used, average bitrate, total number of frames etc). Running this over my collection starts turning up all kinds of interesting things. A large proportion of the mp3 files appear to have random data that can’t be attributed to the MP3 stream, nor the ID3v1 or ID3v2 tags.

Further investigation discovers that people put Lyrics 3 v2 tags in mp3 files. Not because they want lyrics, but because it was a convenient places to put the full version of the artist/album/title if the ID3v1 tag was too short to support it. So I implemented a Lyrics3 v2 parser.

I also discovered that MP3 renormalisation software puts undo information into the file in a tag format called “APEv2″ which has a very poor specification. APEv2 isn’t recommended for MP3 files because it uses “APETAGEX” as it’s header/footer marker, which means it can coincidently end up with “TAG” in just the right place in the file for it to be mistaken as an ID3v1 tag. Chances of this is very slim, and its impossible if there is a valid ID3v1 tag there anyway. APEv2 is very similar to ID3v2.

But still theres data in the mp3 files I can’t attribute to any of the mp3 stream, ID3v1, ID3v2.2, Lyrics3v2, or APEv2 headers. So then I discovered the completely obscure music match tagging format. At this point I gave up. Theres obviously more tagging formats in use than I really care for. The other one that would be worth investigating would be the Vorbis/Flac comment structure, but I’ve not seen any evidence of them being used in any of the .mp3 files I’ve found to test on. Not that I expect this to have stopped someone from deciding it was a great idea and doing it anyway. Sigh.

Theres still quite a bit data that is left over even if I ignroe the musicmatch tags, which I looks like it is compressed data that for some reason doesn’t have a valid mp3 syncronisation header on it. I’ve not found an adequate explaination yet as to what this data is doing there.

So now I have a long list of mp3 files that are inconsistant with their tagging, have obviously silly things going on with their tagging (eg trailing whitespace), have data I can’t attribute to the bitstream or any known tag, or have bizarre or crazy frame headers (eg have some frames that aren’t mpeg version 1 layer 3, and albums that are broken (eg two files that claim to have the same track number, or no file claiming track “2″ on a 5 track album).

I decide to compare this to the musicbrainz database to see what things “should” be like. Musicbrainz database isn’t perfect, but it’s at least reasonably internally consistant and well maintained and free for download. Immediately I discover that there are regularly multiple versions of the same CD. Urgh. I swear the entire music industry is out to get me. So now I compare the number of tracks, and the track lengths that I have vs musicbrainz and pick the one that appears closest to the version I have, and compare them for accuracy. It appears that based on the cdripper used, you get slightly different lengths, presumably due to different ideas of when the cd leadin/leadout times begin.

So I start reripping CD’s that I think need to be sorted out, using cdparanoia. At the same time I decide it would be useful to write out the TOC so I can do a CDDB lookup later, and so I can include the TOC data in the ID3v2.3 tag which allows later offline processing to determine which exact version of a CD a track came from, and what the length of the track should be vs the length of the mp3. But apparently ripping a TOC is a complicated business. Noone actually just writes out the subcode “Q” data. So I use cdrdao to generate me a TOC file, which is near enough. This however means cdrdao rips the CD again to verify track leadin/leadout times. cdrdao can be used to rip the cd itself (using libcdparanoia), but when it does so it writes out one single file instead of one file per track. Arrrggghh. So while I’m writing this I’m waiting for CD’s to rip — twice for each CD.

Then I discover that CD’s can have a GRiD (aka Catalog) number which uniquely identifies the contents of the CD! Yay! Surely this would be more reliable than CDDB? Ahh, but as far I can see there is no publically available GRiD<->information database on the Internet. There are (sometimes) also ISRC numbers for each track which identifies the recording. This sounds like a very useful thing until you realise that it can change if the recording is retouched or slightly different, but might not be modified if the song is for instance remixed. But still a set of ISRC’s in a specific order would be a good identifier for a CD, but again there appears to be no publically useable database on the Internet of ISRC’s. I guess if a CD has GRiD or ISRC information then generally it contains CDTEXT on the CD itself with the name of the tracks. Still it would be an interesting database to trawl.

Other bizarre things that you might have never known. Number of unique versions of a CD according to musicbrainz:

Artist Album Number of versions

Roberto Carlos Roberto Carlos 26

James Earl Jones The Bible 17

Artist	Album	Number of versions
Roberto Carlos	Roberto Carlos	26
James Earl Jones	The Bible	17
The realmode x86 chips had a weird memory model called "segmentation". Each segment was 64k, and segments overlapped with each other offset by 16 bytes. So offset 0010 in segment 0000 was the same as offset 0000 in segment 0001. You'd write these addresses as segment:offset. Programmers would regularly directly access addresses in memory that they knew contained things. I can still remember a lot of interesting addresses off by heart. 0000:0000 was/is the interrupt vector table 0000:7c00 - the bootsector was loaded into memory A000:0000 - the VGA Framebuffer was located. B000:0000 - the monocrome textmode display was located B800:0000 - the colour textmode display C000:0000 - this was "Adaptor ROM", or where network cards (and other things) put their stuff F000:FA6E - this was the VGA ROM BIOS Font You talked directly to the hardware. If you wanted to hook an IRQ, go for it, overwrite the address it would use in the interrupt vector table, and remember to acknowledge the interrupt when it came time to return. I remember to change the VGA palette you write the pallete entry you want to change to io port 0×3c8, then write the red value to 0×3c9, green to 0×3c9, then blue to 0×3c9. They could be from 0 to 63. In fact, I remember you could poll an IO port (I must admit I've forgotten which one, 0×3CA?) and see if a horizontal or vertical retrace was in effect. Depending on the screen mode there were only 16 or 256 or whatever palette entries. Most of the time this meant that you could only have 256 colours on the screen at once. If you modified the palette while the video card was drawing on the screen you'd get "Snow". Snow was caused when the video card tried to read a value that the host CPU was in the process of modifying. It would normally read out 0xFF or some other weird value instead so you'd get "white" in random places across the screen causing what looked like "Snow". But you had enough time to modify about 4 entries during the horizontal blanking time (on my hardware at least) before it started painting the line and therefore avoiding the snow. This meant you could use 2564+242 colours on the screen at once (on a 256 scanline resolution). Often this was used to change just the background colour, usually in a nice smooth gradient. This was called coppering. By investigating what the other video registers were used for people discovered lots about how video cards worked. Video cards were only allocated 64k of address space, so to fit higher resolutions or higher bit depths in they used "planes" (or banks) that you switched into the 64k address space, wrote your updates to it, and then flipped in the next plane. Lots of interesting tricks could be pulled using these techniques. A lot of video modes were "chained". A write to video memory would write the same value to all planes simultaniously. These modes were fast and easy to program for, but they were stuck at very low resolutions. But by programming some of these low resolution modes and turning off the "chained" bit in the video card you could effectively create your own unsupported, undocumented video modes. These modes were collectively called "Mode X". Because the machines were so slow, you spent a lot of time working on optimising code. Often you'd end up rewriting tight loops in assembly code. Even then you'd look at how you could optimise your assembly code. x86 instructions are of dynamic length. There minimum length is 1 byte. Their maximum length was something like 6 bytes. I suspect on a modern x86_64 machine it's probably closer to 12 bytes by now. Your bus was 8 or 16bits wide, and you had to wait for bus transactions to complete, so you'd want to choose the shortest opcode you could. Also memory was at a premium, every byte saved was going to be useful somewhere else. xor ax, ax was better than mov ax,0×0000 because xor ax,ax is a 1 byte opcode, where as mov ax,0×0000 was 3 bytes. Unlike the RISC machines you get taught on at universities where everything is nice and simple and elegant, and instructions take a fixed number of cycles, on x86 they can take different amounts of time. xor ax,ax might take one cycle. a multiply or a divide might take a few hundred cycles. A sin or cos might take a small iceage. So sometimes you wanted to use more opcodes because they were faster but the achieved the same result. One popular VGA video mode was mode 0×13. ("mode thirteen"). It was 320×200x256. To code a putpixel you needed to do (A000:0000 + Y320 + X). But that had a nasty multiplication in it. Remember how you were taught to do multiplication at primary school? 32 320 ---- 0 + 64 +96 ----- 10240 Imagine doing this in binary: 0000 0000 0001 0000 0000 0001 0010 0000 ------------------- 0000 0000 0000 0000 0 0000 0000 0000 000 00 0000 0000 0000 00 000 0000 0000 0000 0 0000 0000 0000 0000 0 0000 0000 0010 000 00 0000 0000 0000 00 000 0000 0000 0000 0 0000 0000 0001 0000 ---------------------------------- 0000 0000 0001 0010 0000 The thing to note is where there is a zero in the second argument to '', there is a line of all 0's in the working, and where there is a 1, the first number is copied entirely shifted left by the position of the bit. So to multiply by 320 you could instead do res=(y<<5)+(y<<8). This was faster, significantly so. I remember benchmarking it Other quirks of the x86 architecture were things like you could do complex addressing modes like [bx+4ax+3]. this was handy when you needed to find the address of a member of a structure in a array. There is an opcode called LEA. Load effective Address. This instruction takes an address as it's argument and loads that address (not the value at that address) into some other register. The fun thing was that LEA was a single cycle opcode (and quite short 1 or 2 bytes IIRC). This meant that you could do a shift by a power of two, and an addition in one opcode instead of two. And not only that it wouldn't use the ALU to do the arithmetic so another opcode could be busy using the ALU. One of my greatest "hacks" was a program I wrote that used one of the "Mode X"'s I mentioned above to create a 256×256x256 video mode. Most monitors hated this mode as it was effectively square, and they were 4:3, but with some protesting they'd do it. I created a subdivision style "plasma". Then each vertical refresh I'd blit onto the screen a tunnel going off to infinity with this plasma textured onto it. Originally I wrote this in Pascal (Turbo Pascal v7 for DOS was a brilliant language, compiler, and IDE). Except it was slow. Glacially slow. So I started rewriting chunks of it in assembler. But I soon ran into the problem that intel machines don't have many registers. So I stored all the registers in the code segment (writable code segments yay!), and then used all the registers (including the stack pointer — which is why I couldn't store them on the stack) as general purpose registers. It still wasn't fast enough. One of the reasons for this was I only had a handful of segment registers. Now when you're addressing memory in x86 realmode, you give an offset into one of the segment registers (CS (code segment), DS (Data segment), ES (Extra Segment), SS (Stack Segment) and I needed to access 4 buffers (vga buffer, texture, and x/y data for my tunnel). So, I did some research and found the opcodes for the new 386 instructions, and tried using them. But turbo pascal's internal assembler didn't know about these opcodes. Now when it comes to x86 assembler there are prefixes for changing some characteristic about the next opcode. for instance a "ES:" prefix would modify the address in the instruction to be relative to the ES (Extra segment) instead of the DS (Data segment). If you used 32bit registeres in real mode the assembler would automatically output a prefix saying to use 32bit values/registers instead of 16 bit ones for the next opcode. This assembler didn't know how to do that. So I would use "db 0×66 ; mov ax, sp" to make the instruction "mov eax, esp". But then I discovered that the 386 had introduced another couple of segment registers (gs/fs). howeve there was no prefix for them, I had to hand assemble and insert the raw hex values for them into my program. So now I had a program that I had hand assembled to make run, and now it ran fast enough to be useful on my machine, and it looked sweet! But around the time of the release of ID's game "Doom" all of this disappeared. Compilers didn't produce completely moronic code anymore, and processesors were fast enough that you could afford to waste a few cycles here or there. The program I'd painstakingly hand optimised above I showed to a friend. He wrote the entire thing in C and it ran fine on his pentium computer. (his version however didn't run at all reasonably on my 386). Doom was written almost entirely in C, including the 3d raycaster. Only the dissolve between missions was written in assembler. Soon computers were coming with 3d accelerated video cards and everyone forgot how to write assembler, except for the fringe lunatics and people who write compilers. You might think that this is progress. No more do people spend a week painstakingly hand assembling code to get things to run as fast as humanly possible. But, the skills I learnt over 10 years ago when I taught myself to program are still useful today. Knowing the layout of a bootsector means I can correctly identify a lot more problems people have with disk images. Knowing assembly language in far more detail than anyone thinks is necessary helps with debugging obscure problems in programs. You tell people today about hand assembling programs to get them to run at speed and they just won't believe you. :) “Or” considered harmful. Isomer — Mon, 30 Oct 2006 03:16:33 +0000 On the weekend we decided that "or" (as used in the english language) is ambiguous and leads to confusion. We decided that instead we should use three terms: "andor" (to mean "are any of these true?") "xor" or "either-or" or "exclusive-or" (to mean "are only one of these true?") "ewok" (to mean "which of these are true?") Some examples: Is it wet outside ewok is it fine? ("it's wet outside"), Is it wet outside andor is it fine ("yes"), Is it wet outside xor is it fine ("yes"). Am I mad ewok have small furry animals invaded this conversation? ("You are both mad, and small furry animals have invaded this conversation"), Am I mad andor have small furry animals invaded this conversation? ("Yes"), Am I mad xor have small furry animals invaded this conversation? ("No" both are true). Would you like green eggs and ham on a plane ewok on a train? ("Mu" — I don't want green eggs on either). Would you like green eggs and ham on a plane andor on a train? ("No"), Would you like green eggs and ham on a plain xor on a train? ("No"). Should this be blue ewok green ewok red? ("Green"), Should this be blue andor green andor red ("yes"), Should this be blue xor green xor red ("Yes"). so, in closing, I Think we should stop using "or" in every day conversation and instead use "andor", "xor" and "ewok" instead to be more precise about what question we're asking. Home theatre and the IP Revolution? Isomer — Mon, 09 Oct 2006 12:09:32 +0000 So recently I was setting up an home theatre system. TV, DVD Player, Sky, PS/2, etc. and an AMP/Reciver. After dealing with S/Video, Component, Composite, RF, Scart, HDMI, DVI, and VGA as well as power and speaker cables, I came to the conclusion that convergance couldn’t happen fast enough. Imagine having all your components plug in via ethernet. They could be anywhere in the house, audio video could be streamed over IP. Devices could be discovered via mDNS. No need for limited numbers of inputs, want 6 DVD players? Sure, no problem. Want 4 SKY decoders? No issue. Your reciever would have one ethernet uplink, and terminals for your speakers, maybe preamp outputs. It would decode whatever codecs are the current fads (much as it does now). Assuming enough bandwidth it may even use 802.11 wireless reducing the need for even more cables. Other components (DVD, TV, Sky etc) would all also take power and ethernet as their ownly source. You could converge your home theatre with your phone (video conferencing? why not!), when your phone rings your DVD (or DVR) pauses the current output, and brings up a nice overlay with the callerID. Your remote control could be 802.11, which while it might chew through batteries could easily detect all your components and provide a consistant interface to them anywhere in the house. If people really care about DRM (personally I think it hurts everyone more than it helps, but I could be wrong…), then all of this can be easily authenticated/encrypted by IPsec and X.500 certificate CA’s. HDMI is close. It provides an integrated point-to-point audio/video connector which sends uncompressed video and encoded audio. It has no theoretical maximum cable length although it has a pracatical limitation of about 15m. compared to 100m for Ethernet. You can get HDMI repeaters, you can get ethernet switches. Ethernet lets you address multiple devices on the same subnet allowing you to have multiple sources (eg DVD/CD/sky/etc) and sinks (TV’s/recievers/etc) on the same network. HDMI has i²C to do autodetection of devices (eg for EDID), and supports AV Link for remote control features for multirooming etc. IP has a wide variety of protocols that could be used for this. Ethernet is well established and things such as ethernet switches/cables/etc are all mature products. So, I wish everything spoke Ethernet, it would rock. I guess I’ll just have to wait until more than just DVD players have HDMI to get close. python, ctypes, and debugging symbols. Isomer — Fri, 01 Sep 2006 14:16:36 +0000 there is a python module called `ctypes`. This module lets you make calls to various functions inside a dynamically linked library without having to explicitly write wrapper code for it. But you must specify the complete type of the function you're calling before you call it, so that the ctypes library knows arguments to push onto the stack and what order etc. This always seemed to be a silly restriction, in an strong, but dynamically typed language such as python having to forward declare things seems to defeat the entire purpose. And besides, when you compile something, you can compile it with full debugging symbols, which has the full type information about everything in your program anyway. So, you could write a program to parse the debugging symbols out of an .so, and generate a python module with all the various type definitions and functions exported from the library, and you could do all of this on the fly. And thus, I wrote it. foo.c struct function2_return_t { int a; int b; }; struct function2_return_t function2(int a) { struct function2_return_t ret; ret.a=a; ret.b=a+1; return ret; } And the example usage: >>> import ct >>> ct.load("foo.so") >>> import foo >>> ret=foo.function2(1) >>> print ret >>> print ret.a 1 >>> print ret.b 2 It's all still very prototypeish, it doesn't deal with pointers or strings yet. As you can see it deals with structs (as if they were classes), and could fairly easily deal with arrays and pointers to structs (although once you get into pointers, you get into "who owns this memory", perhaps requiring a .free() method on classes). Strings should also be fairly straight forward if you can figure out who actually owns the memory (should it be freed?). At the moment I run and parse the output of readelf to get the debugging information, and I don't properly hook the "import" mechanism to make it all completely transparent, but as a prototype I thought it was pretty cool. Woot! My Filesystem workish! Isomer — Thu, 03 Aug 2006 10:49:56 +0000 So as you may remember, I've been thinking about (and quietly implementing) a distributed filesystem. Tonight I got as far as it mostly working, and since everyone around here is sick of me telling them about it I thought I'd tell you all about it instead :) It uses the 9p protocol to talk to the kernel (although I'm using the 9p python client for testing). I use the chimera DHT for indexing, and libnpfs for dealing with the low level details of the 9p protocol. I've managed to find bugs in all of these pieces of software so far :) Anyway, onto the interesting part of the show: 9p> ls foo/ 9p> cd foo 9p> ls readme 9p> cat readme Hello World 9p> mkdir sample 9p> ls -l -rw-r--r-- perry perry 0 readme drwxr-xr-x perry perry 0 sample 9p> cd sample 9p> ls -l 9p> put testfile 9p> ls -l -rw-r--r-- perry perry 0 testfile 9p> cat testfile This file is a test 9p> You may notice that the file sizes are all 0, even though most of them seem to contain data, this is coz I've not finished implementing stat(2) yet. Permissions etc are kinda implemented (they're stored, but not read off disk yet, although libnpfs appears to verify them for me). There's a bit of polishing off to do before the "local" part of the filesystem is mostly viable. Then I just have to hook all the distributedness together and I should have a fully functional prototype! (yeah right…) Fragmenting IP Isomer — Mon, 17 Jul 2006 02:31:59 +0000 When fragmenting IP packets, you usually split the packets up into "`n-1`" pMTU sized packets, and `1` packet of the remainder. Now with wireless networks the larger the packet is the more likely it is to be lost/corrupted. I suspect it would make more sense to fragment packets into equal sized pieces, so `totlen/n` bytes each. Most routing overhead is per packet, and the total number of packets hasn't been changed. Also this would tend to avoid so many pMTU issues as your packets are probably going to less than pMTU sized.

The realmode x86 chips had a weird memory model called "segmentation". Each segment was 64k, and segments overlapped with each other offset by 16 bytes. So offset 0010 in segment 0000 was the same as offset 0000 in segment 0001. You'd write these addresses as segment:offset. Programmers would regularly directly access addresses in memory that they knew contained things. I can still remember a lot of interesting addresses off by heart.

0000:0000 was/is the interrupt vector table
0000:7c00 - the bootsector was loaded into memory
A000:0000 - the VGA Framebuffer was located.
B000:0000 - the monocrome textmode display was located
B800:0000 - the colour textmode display
C000:0000 - this was "Adaptor ROM", or where network cards (and other things) put their stuff
F000:FA6E - this was the VGA ROM BIOS Font

You talked directly to the hardware. If you wanted to hook an IRQ, go for it, overwrite the address it would use in the interrupt vector table, and remember to acknowledge the interrupt when it came time to return.

I remember to change the VGA palette you write the pallete entry you want to change to io port 0×3c8, then write the red value to 0×3c9, green to 0×3c9, then blue to 0×3c9. They could be from 0 to 63.

In fact, I remember you could poll an IO port (I must admit I've forgotten which one, 0×3CA?) and see if a horizontal or vertical retrace was in effect. Depending on the screen mode there were only 16 or 256 or whatever palette entries. Most of the time this meant that you could only have 256 colours on the screen at once. If you modified the palette while the video card was drawing on the screen you'd get "Snow".

Snow was caused when the video card tried to read a value that the host CPU was in the process of modifying. It would normally read out 0xFF or some other weird value instead so you'd get "white" in random places across the screen causing what looked like "Snow".

But you had enough time to modify about 4 entries during the horizontal blanking time (on my hardware at least) before it started painting the line and therefore avoiding the snow. This meant you could use 256*4+242 colours on the screen at once (on a 256 scanline resolution). Often this was used to change just the background colour, usually in a nice smooth gradient. This was called coppering.

By investigating what the other video registers were used for people discovered lots about how video cards worked. Video cards were only allocated 64k of address space, so to fit higher resolutions or higher bit depths in they used "planes" (or banks) that you switched into the 64k address space, wrote your updates to it, and then flipped in the next plane. Lots of interesting tricks could be pulled using these techniques. A lot of video modes were "chained". A write to video memory would write the same value to all planes simultaniously. These modes were fast and easy to program for, but they were stuck at very low resolutions. But by programming some of these low resolution modes and turning off the "chained" bit in the video card you could effectively create your own unsupported, undocumented video modes. These modes were collectively called "Mode X".

Because the machines were so slow, you spent a lot of time working on optimising code. Often you'd end up rewriting tight loops in assembly code. Even then you'd look at how you could optimise your assembly code.

x86 instructions are of dynamic length. There minimum length is 1 byte. Their maximum length was something like 6 bytes. I suspect on a modern x86_64 machine it's probably closer to 12 bytes by now.

Your bus was 8 or 16bits wide, and you had to wait for bus transactions to complete, so you'd want to choose the shortest opcode you could. Also memory was at a premium, every byte saved was going to be useful somewhere else. xor ax, ax was better than mov ax,0×0000 because xor ax,ax is a 1 byte opcode, where as mov ax,0×0000 was 3 bytes.

Unlike the RISC machines you get taught on at universities where everything is nice and simple and elegant, and instructions take a fixed number of cycles, on x86 they can take different amounts of time. xor ax,ax might take one cycle. a multiply or a divide might take a few hundred cycles. A sin or cos might take a small iceage. So sometimes you wanted to use more opcodes because they were faster but the achieved the same result.
One popular VGA video mode was mode 0×13. ("mode thirteen"). It was 320×200x256. To code a putpixel you needed to do (A000:0000 + Y*320 + X). But that had a nasty multiplication in it. Remember how you were taught to do multiplication at primary school?

32
 *320
 ----
    0
+ 64
+96
-----
10240

Imagine doing this in binary:

0000 0000 0001 0000
          *0000 0001 0010 0000
           -------------------
           0000 0000 0000 0000
         0 0000 0000 0000 000
        00 0000 0000 0000 00
       000 0000 0000 0000 0
      0000 0000 0000 0000
    0 0000 0000 0010 000
   00 0000 0000 0000 00
  000 0000 0000 0000 0
 0000 0000 0001 0000

 ----------------------------------

0000 0000 0001 0010 0000

The thing to note is where there is a zero in the second argument to '*', there is a line of all 0's in the working, and where there is a 1, the first number is copied entirely shifted left by the position of the bit. So to multiply by 320 you could instead do res=(y<<5)+(y<<8). This was faster, significantly so. I remember benchmarking it

Other quirks of the x86 architecture were things like you could do complex addressing modes like [bx+4*ax+3]. this was handy when you needed to find the address of a member of a structure in a array. There is an opcode called LEA. Load effective Address. This instruction takes an address as it's argument and loads that address (not the value at that address) into some other register. The fun thing was that LEA was a single cycle opcode (and quite short 1 or 2 bytes IIRC). This meant that you could do a shift by a power of two, and an addition in one opcode instead of two. And not only that it wouldn't use the ALU to do the arithmetic so another opcode could be busy using the ALU.

One of my greatest "hacks" was a program I wrote that used one of the "Mode X"'s I mentioned above to create a 256×256x256 video mode. Most monitors hated this mode as it was effectively square, and they were 4:3, but with some protesting they'd do it. I created a subdivision style "plasma". Then each vertical refresh I'd blit onto the screen a tunnel going off to infinity with this plasma textured onto it.

Originally I wrote this in Pascal (Turbo Pascal v7 for DOS was a brilliant language, compiler, and IDE). Except it was slow. Glacially slow. So I started rewriting chunks of it in assembler. But I soon ran into the problem that intel machines don't have many registers. So I stored all the registers in the code segment (writable code segments yay!), and then used all the registers (including the stack pointer — which is why I couldn't store them on the stack) as general purpose registers.

It still wasn't fast enough. One of the reasons for this was I only had a handful of segment registers. Now when you're addressing memory in x86 realmode, you give an offset into one of the segment registers (CS (code segment), DS (Data segment), ES (Extra Segment), SS (Stack Segment) and I needed to access 4 buffers (vga buffer, texture, and x/y data for my tunnel).

So, I did some research and found the opcodes for the new 386 instructions, and tried using them. But turbo pascal's internal assembler didn't know about these opcodes. Now when it comes to x86 assembler there are prefixes for changing some characteristic about the next opcode. for instance a "ES:" prefix would modify the address in the instruction to be relative to the ES (Extra segment) instead of the DS (Data segment). If you used 32bit registeres in real mode the assembler would automatically output a prefix saying to use 32bit values/registers instead of 16 bit ones for the next opcode. This assembler didn't know how to do that. So I would use "db 0×66 ; mov ax, sp" to make the instruction "mov eax, esp". But then I discovered that the 386 had introduced another couple of segment registers (gs/fs). howeve there was no prefix for them, I had to hand assemble and insert the raw hex values for them into my program.

So now I had a program that I had hand assembled to make run, and now it ran fast enough to be useful on my machine, and it looked sweet!

But around the time of the release of ID's game "Doom" all of this disappeared. Compilers didn't produce completely moronic code anymore, and processesors were fast enough that you could afford to waste a few cycles here or there. The program I'd painstakingly hand optimised above I showed to a friend. He wrote the entire thing in C and it ran fine on his pentium computer. (his version however didn't run at all reasonably on my 386).

Doom was written almost entirely in C, including the 3d raycaster. Only the dissolve between missions was written in assembler. Soon computers were coming with 3d accelerated video cards and everyone forgot how to write assembler, except for the fringe lunatics and people who write compilers.

You might think that this is progress. No more do people spend a week painstakingly hand assembling code to get things to run as fast as humanly possible. But, the skills I learnt over 10 years ago when I taught myself to program are still useful today. Knowing the layout of a bootsector means I can correctly identify a lot more problems people have with disk images. Knowing assembly language in far more detail than anyone thinks is necessary helps with debugging obscure problems in programs.

You tell people today about hand assembling programs to get them to run at speed and they just won't believe you. :)

“Or” considered harmful.

Isomer — Mon, 30 Oct 2006 03:16:33 +0000

On the weekend we decided that "or" (as used in the english language) is ambiguous and leads to confusion. We decided that instead we should use three terms:

"andor" (to mean "are any of these true?")
"xor" or "either-or" or "exclusive-or" (to mean "are only one of these true?")
"ewok" (to mean "which of these are true?")

Some examples:

Is it wet outside ewok is it fine? ("it's wet outside"), Is it wet outside andor is it fine ("yes"), Is it wet outside xor is it fine ("yes").
Am I mad ewok have small furry animals invaded this conversation? ("You are both mad, and small furry animals have invaded this conversation"), Am I mad andor have small furry animals invaded this conversation? ("Yes"), Am I mad xor have small furry animals invaded this conversation? ("No" both are true).
Would you like green eggs and ham on a plane ewok on a train? ("Mu" — I don't want green eggs on either). Would you like green eggs and ham on a plane andor on a train? ("No"), Would you like green eggs and ham on a plain xor on a train? ("No").
Should this be blue ewok green ewok red? ("Green"), Should this be blue andor green andor red ("yes"), Should this be blue xor green xor red ("Yes").

so, in closing, I Think we should stop using "or" in every day conversation and instead use "andor", "xor" and "ewok" instead to be more precise about what question we're asking.

Home theatre and the IP Revolution?

Isomer — Mon, 09 Oct 2006 12:09:32 +0000

So recently I was setting up an home theatre system. TV, DVD Player, Sky, PS/2, etc. and an AMP/Reciver. After dealing with S/Video, Component, Composite, RF, Scart, HDMI, DVI, and VGA as well as power and speaker cables, I came to the conclusion that convergance couldn’t happen fast enough.

Imagine having all your components plug in via ethernet. They could be anywhere in the house, audio video could be streamed over IP. Devices could be discovered via mDNS. No need for limited numbers of inputs, want 6 DVD players? Sure, no problem. Want 4 SKY decoders? No issue.

Your reciever would have one ethernet uplink, and terminals for your speakers, maybe preamp outputs. It would decode whatever codecs are the current fads (much as it does now). Assuming enough bandwidth it may even use 802.11 wireless reducing the need for even more cables. Other components (DVD, TV, Sky etc) would all also take power and ethernet as their ownly source.

You could converge your home theatre with your phone (video conferencing? why not!), when your phone rings your DVD (or DVR) pauses the current output, and brings up a nice overlay with the callerID.
Your remote control could be 802.11, which while it might chew through batteries could easily detect all your components and provide a consistant interface to them anywhere in the house.

If people really care about DRM (personally I think it hurts everyone more than it helps, but I could be wrong…), then all of this can be easily authenticated/encrypted by IPsec and X.500 certificate CA’s.

HDMI is close. It provides an integrated point-to-point audio/video connector which sends uncompressed video and encoded audio. It has no theoretical maximum cable length although it has a pracatical limitation of about 15m. compared to 100m for Ethernet. You can get HDMI repeaters, you can get ethernet switches. Ethernet lets you address multiple devices on the same subnet allowing you to have multiple sources (eg DVD/CD/sky/etc) and sinks (TV’s/recievers/etc) on the same network. HDMI has i²C to do autodetection of devices (eg for EDID), and supports AV Link for remote control features for multirooming etc. IP has a wide variety of protocols that could be used for this. Ethernet is well established and things such as ethernet switches/cables/etc are all mature products.

So, I wish everything spoke Ethernet, it would rock. I guess I’ll just have to wait until more than just DVD players have HDMI to get close.

python, ctypes, and debugging symbols.

Isomer — Fri, 01 Sep 2006 14:16:36 +0000

there is a python module called ctypes. This module lets you make calls to various functions inside a dynamically linked library without having to explicitly write wrapper code for it. But you must specify the complete type of the function you're calling before you call it, so that the ctypes library knows arguments to push onto the stack and what order etc.

This always seemed to be a silly restriction, in an strong, but dynamically typed language such as python having to forward declare things seems to defeat the entire purpose. And besides, when you compile something, you can compile it with full debugging symbols, which has the full type information about everything in your program anyway.

So, you could write a program to parse the debugging symbols out of an .so, and generate a python module with all the various type definitions and functions exported from the library, and you could do all of this on the fly.

And thus, I wrote it.

foo.c

struct function2_return_t {
        int a;
        int b;
};

struct function2_return_t function2(int a)
{
        struct function2_return_t ret;
        ret.a=a;
        ret.b=a+1;

return ret;
}

And the example usage:

>>> import ct
>>> ct.load("foo.so")
>>> import foo
>>> ret=foo.function2(1)
>>> print ret

>>> print ret.a
1
>>> print ret.b
2

It's all still very prototypeish, it doesn't deal with pointers or strings yet. As you can see it deals with structs (as if they were classes), and could fairly easily deal with arrays and pointers to structs (although once you get into pointers, you get into "who owns this memory", perhaps requiring a .free() method on classes). Strings should also be fairly straight forward if you can figure out who actually owns the memory (should it be freed?).

At the moment I run and parse the output of readelf to get the debugging information, and I don't properly hook the "import" mechanism to make it all completely transparent, but as a prototype I thought it was pretty cool.

Woot! My Filesystem workish!

Isomer — Thu, 03 Aug 2006 10:49:56 +0000

So as you may remember, I've been thinking about (and quietly implementing) a distributed filesystem. Tonight I got as far as it mostly working, and since everyone around here is sick of me telling them about it I thought I'd tell you all about it instead :)

It uses the 9p protocol to talk to the kernel (although I'm using the 9p python client for testing). I use the chimera DHT for indexing, and libnpfs for dealing with the low level details of the 9p protocol. I've managed to find bugs in all of these pieces of software so far :)

Anyway, onto the interesting part of the show:

9p> ls
foo/
9p> cd foo
9p> ls
readme
9p> cat readme
Hello World
9p> mkdir sample
9p> ls -l
-rw-r--r-- perry perry 0        readme
drwxr-xr-x perry perry 0        sample
9p> cd sample
9p> ls -l
9p> put testfile
9p> ls -l
-rw-r--r-- perry perry 0       testfile
9p> cat testfile
This file is a test
9p>

You may notice that the file sizes are all 0, even though most of them seem to contain data, this is coz I've not finished implementing stat(2) yet. Permissions etc are kinda implemented (they're stored, but not read off disk yet, although libnpfs appears to verify them for me).

There's a bit of polishing off to do before the "local" part of the filesystem is mostly viable. Then I just have to hook all the distributedness together and I should have a fully functional prototype! (yeah right…)

Fragmenting IP

Isomer — Mon, 17 Jul 2006 02:31:59 +0000

When fragmenting IP packets, you usually split the packets up into "n-1" pMTU sized packets, and 1 packet of the remainder. Now with wireless networks the larger the packet is the more likely it is to be lost/corrupted. I suspect it would make more sense to fragment packets into equal sized pieces, so totlen/n bytes each. Most routing overhead is per packet, and the total number of packets hasn't been changed. Also this would tend to avoid so many pMTU issues as your packets are probably going to less than pMTU sized.

Remosi's Rants

Things you probably never wanted to know about MP3 files

“Or” considered harmful.

Home theatre and the IP Revolution?

python, ctypes, and debugging symbols.

Woot! My Filesystem workish!

Fragmenting IP