Yes, the setup for BrickOS is a bit involved. In addition to what was noted in the USB driver thread, another reason for my interest in a Linux distribution was providing a means to greatly reduce the complexity of the setup. Most of BrickOS is written in C, so it is well-suited to the Linux environment.
As to 4800 baud and changing the protocol, BrickOS supports that. Technically (i.e. based on Lego's documentation), the USB Tower supports baud rates even higher than that. Carrier frequency plays a role, too. In their standard configuration, newer RCX units have a carrier frequency double that of the original units. For more details, there is some discussion on lugnet about that. Also, some people have hacked the IR serial communication capabilities to put Bluetooth support on the RCX. The synopsis of what is involved with that mod is incorporating a serial to Bluetooth adapter on the RCX, and then the computer establishes the serial connection via Bluetooth.
For the protocol, there are a couple protocol layers which generally are referred to collectively as LNP. Using one of the LNP layers, you can access data transmissions on a byte-by-byte basis. I would strongly recommend trying to leverage and/or build off the capabilities of LNP rather than attempting to start from scratch. I believe leJOS also incorporated LNP support.
Combine the baud rate and protocol modifications, and you have firmware downloads that occur in ¼ of the time of the standard Lego firmware download. Take either the baud rate modification by itself or the protocol modification by itself, and the transmission speed is still ½ that of what the standard Lego software does.
On of the capabilities I like about the RCX is its ability to generate DCC waveforms, such as are used to control model railroads and accessories. A pbForth and a patched BrickOS both provide these capabilities (leJOS and the standard firmware can't provide the necessary time granularity). As I'm sure you already know (but for the benefit of others), with DCC, you can actually control more than three motors (provided the motor is equipped with a DCC decoder) without resorting to multiplexing (though an additional power source [booster in DCC parlance] would likely be necessary). I see from
another post that you are into model railroading, too.
Also, perhaps it's just my perception, but the RCX--possibly because of its novelty--seemed to inspire a broader development community and more diversity in the alternative firmware, languages, and tools that were created, not to mention all the reverse engineering that went into it. One downside to that was sustainability and fragmentation, but I still find all the diversity intriguing. I've found record of at least 17 firmware projects, plus a few firmwares had support for more than one language. Obviously, quite a few alternative programming language implementations were created around Lego's standard firmware, and I've found record of at least 6 different programming languages for BrickOS.