SqueakNOS: A simple guide to writing HardwareDevices

2006-06-10 06:39:34 - gera

SqueakNOS currently has support for PC Keyboards, PS/2 Mouse, Serial ports and the Real Time functionality of the CMOS chip on PCs. In this article I will use the latter as an example to explain how new HardwareDevice could be implemented.

I will also include a list of the most wanted devices, and some open questions I'd like to discuss with interested people. The current status of SqueakNOS calls for collaboration on many aspects, and supporting more hardware is one of the most important.

We try to keep as much as possible done in Squeak, interpreted by the Virtual Machine, in opcodes, not native code. This includes, of course, a lot of low level things, including I/O ports access, and IRQ handling. For I/O the trick is easily solved with a couple of new primitives (we can add a few new primitives after all the others we removed :-). For IRQ handling it's a little bit more complicated, but lets just start with the simplest things.

Getting the right information

The first and most important thing we'll need when coding a HardwareDevice is information. and although it should be available somewhere, it's not always easy to find. I always like to start with simple things and build more functionality while I'm getting a stronger grip on the problem, so I tend to avoid reading the code of existing device drivers (Linux' for example), because they tend to be too complicated to extract the right information from them, however we know that they are working, and we can always use them as last resource :-)

So far, for the simple things we have implemented, a good source of information was old documents from tutorials going back to DOS days, where things used to be simple. But the truth is that the further we got the more we needed the real information (chip's datasheets for example).

As an example we are going to use the CMOS/RTC chip of the PC computers. This is where the BIOS configuration is stored and mantained across boots, and also where the time is maintained, even when the PC is turned off and unplugged.

I remembered from the old days that Ralph Brown's interrupt list had a good Appendix on I/O ports, and I went straight for that, and got it right away!. As you can see it has a fairly good description, and we can start trying it right away. So I opened my SqueakNOS and created a new class:

PortHardwareDevice subclass: #CMOS
    instanceVariableNames: ''
    classVariableNames: ''
    poolDictionaries: ''
    category: 'SqueakNOS-Devices-Base'

I immediately ran to a Workspace and inspected (we'll have MorphicWrappers soon enough, don't worry):

CMOS onPort: 16r70.

This gave me a CMOS device already, but what to do with it? Ralph Brown explains it really clear. Quoting from the sited webpage:

Accessing the CMOS

The CMOS memory exists outside of the normal address space and cannot contain directly executable code. It is reachable through IN and OUT commands at port number 70h (112d) and 71h (113d). To read a CMOS byte, an OUT to port 70h is executed with the address of the byte to be read and an IN from port 71h will then retrieve the requested information.

So I jumped to the Browser and added:

CMOS>>at: index
    self byteAt: 0 put: index.
    ^ self byteAt: 1

CMOS>>at: index put: value self byteAt: 0 put: index. ^ self byteAt: 1 put: value

Accessing I/O Ports from aPortBasedHardwareDevice

All PortBasedHardwareDevices have a base port, and to perform input and output operations you use byteAt: offset and byteAt: offset put: aByte respectively [1]. The quoted description says that to access a given value in the CMOS we first need to output the index to port 16r70 and then read/write from/to port 16r71, this two ports are respectively ports at offset 0 and 1 for our just instantiated CMOS onPort: 16r70. It may sound a little bit complicated, but when you have several devices of the same class on different ports (think serial com ports) it makes a lot of sense to access them as a base port and an offset.

With this ready, I wanted to test it, so I couldn't help watching my fingers type in the inspector:

self at: 0

This should access CMOS' register #0, and from a little bit simplified page at BiosCentral we can see that register #0 contains Real Time Clock's seconds (in BCD), so I continuously printed it like crazy, and to my surprise (or not) it did change every second! Wow! it's kind of working, wasn't it?!

The next and obvious thing was to implement easy accessors for all the interesting values:

    ^ (self valueAt: 0)
        bitAnd: 127

CMOS>>second: aNumber ^ self valueAt: 0 put: aNumber

CMOS>>minute ^ self valueAt: 2

CMOS>>hour ^ self valueAt: 4

CMOS>>time ^ Time hour: self hour minute: self minute second: self second

CMOS>>statusB ^ self at: 11

CMOS>>isBCD ^ (self statusB anyMask: 4) not


Where #valueAt: and #valueAt:put: will decide whether to use BCD encoding or not based on #isBCD.

So... what now? heh! The CMOS' RTC has an alarm that, AFAIK, nobody cared to ever use, but we are going to abuse.

SqueakNOS IRQ Handling

As explained in SqueakNOS' swiki:

The Squeak VM lets you signal Squeak Semaphores from the native world [C/assembly code] by calling signalSemaphoreWithIndex(). We want to serve IRQs from Squeak, using interpreted code, not native code. We seriously think that with hardware close to 1000 times faster than 20 years ago we should be able to do it without any problems. Of course we cannot set the native IRQs handlers to jump to Squeak directly, so our idea is to have a different Semaphore for every IRQ and have a Squeak Process with highIOPriority blocking on each Semaphore.

The code inside Squeak blocking on the Semaphore looks like:

InterruptRequestDispatcher>>installOn: aComputer
    self registerSemaphore.
    process := [
            semaphore wait.
            self handleOn: aComputer.
            aComputer interruptController signalEndOfInterrupt: interruptNumber.
        ] repeat
    ] fork priority: Processor highIOPriority.
    aComputer interruptController enableIRQ: interruptNumber.

And the native code to signal the Semaphore, taken from ints.h, is similar to:

void irq_1_handler();

asmlinkage void ISR_1() { if (0!=IRQSemaphores[number]) signalSemaphoreWithIndex(IRQSemaphores[number]); }

asm( " .text " " .align 16 " " irq_1_handler: " " pusha " " call ISR_1 " " popa " " iret " )

There is an interesting detail in all this: the software interrupt ending (IRET) and the hardware IRQ ending (outb(0x20,0x20)) are detached in SqueakNOS, were they are almost always done at the same time in every other code we saw. This gives a really desirable result: The IRET lets the software continue, going back to the interpreter and letting the Process waiting on the Semaphore be rescheduled, however, the hardware part (Interrupt Controller) still thinks the IRQ has not been served yet, and will wait until the Squeak side of the handler (shown above) signals the end of the interrupt (aComputer interruptController signalEndOfInterrupt: interruptNumber.

We believe that with computers close to 1000 times faster than 20 years ago this should be ok, however we are not sure yet if it may bring any problems latter, and we are open, although reluctant, to the possibility of coding some glue code, or even complete "device drivers" natively (or hopefully using Exupery). Probably for sound and video I/O, we'll see.

We were successfully using the very same scheme in the old SqueakNOS, so we just ported it to the new code base.

With all this, the next thing to do is to set the alarm, so we added an instance variable and a few more methods:

PortHardwareDevice subclass: #CMOS
    instanceVariableNames: 'alarmBlock'
    classVariableNames: ''
    poolDictionaries: ''
    category: 'SqueakNOS-Devices-Base'

CMOS>>alarmSecond: aNumber ^ self valueAt: 1 put: aNumber

CMOS>>alarmMinute: aNumber ^ self valueAt: 3 put: aNumber

CMOS>>alarmHour: aNumber ^ self valueAt: 5 put: aNumber

CMOS>>alarm: aTime self alarmHour: aTime hour; alarmMinute: aTime minute; alarmSecond: aTime second

CMOS>>onAlarmDo: aBlock self alarmBlock: aBlock.

CMOS>>enableAlarm self statusB: (self statusB bitOr: 16r20)

CMOS>>disableAlarm self statusB: (self statusB bitAnd: 16rDF)

And a few more, for the IRQ setup and handling:

CMOS>>handleIRQfor: anIRQDispatcher
    "this method will be called when the IRQ is signaled"
    self alarmBlock value.

CMOS>>installAlarm Computer current interruptController addHandler: self forIRQ: 8. self enableAlarm.

CMOS>>uninstallAlarm Computer current interruptController removeHandler: self forIRQ: 8. self disableAlarm.

Ahahahahhaaa!!! can't wait to try it! can you? In the inspector do:

self onAlarmDo: [Display flash: Display boundingBox].
self alarm: (self time addSeconds: 30).
self installAlarm.

Count! 30, 29, 28, 27, 26, ("Arg! next time I'll put 10"), 3, 2, 1, wow! flash! ah-ah! uhm... hey! ouch... reboot...

What did you expect? we are playing with low level stuff! we should be forced to reboot at least once!

What happened is that the alarm was triggered at the right time, and the block was actually evaluated (Display flashed), but not only once. It was done forever, like if the IRQ was triggered again and again and again. And since IRQ 8 (CMOS') has higher priority than IRQ 14 (mouse's), the mouse died (and killed the keyboard with it). But if you try carefully, you'll see that while the screen is flashing like mad, the keyboard is still working (until you move the mouse). This means two things: we should stop this IRQs! and more important, IRQ priorities are honored, and even when one IRQ runs like crazy, another more important can be served... nice!

Solving this small problem took me a while, until I found the right information: The chip's datasheet. There they "clearly" say the IRQs will repeat until you read the state on register C:

Any time the IRQF bit is a 1, the IRQ pin is driven low. All flag bits are cleared after Register C is read by the program or when the RESET pin is low.

So we had to tweak a little bit our #handleIRQfor: to include this:

CMOS>>handleIRQfor: anIRQDispatcher
    | flags |
    flags := self statusC.
    (flags anyMask: 16r20) ifTrue: [
        "registerC must be read to clear the interrupt bits,
         otherwise, the interrupt is generated again"
        self alarmBlock value.

With this we are pretty much done with the CMOS/RTC hardware. Of course there are more things to it (retrieving and storing BIOS configuration and password, periodic timer IRQs and update IRQs, etc), and of course, it could be made a lot more robust and cleaner, by checking the status flags and verifying error conditions and assigning names to constants, for example. But we just wanted to add something more to SqueakNOS base, and actually, we implemented CMOS thinking that it was going to be a good example for this article, simple but still rich.

Implementing a new PortBasedHardwareDevice

Implementing simple devices should be almost easy following the steps described here, for more complicated things it may be a little bit hairy. In any case, questions are welcome (and expected!), and we hope to build an interesting discusion group to talk about all this [2].

We know there is a big limitation right now, but we hope to find a nice solution for the time we really need it: Some hardware devices work accessing memory (RAM), for this we'll either need to add some external (native) memory management functionality, or find a way to lock ByteArrays (or any other Object) in memory so the GarbageCollector doesn't move them around while they are being accessed by the hardware. And then, even if we can solve this in some decent way, there are some old hardware devices that require low memory addresses (below 1M or even below 640K). For this cases we will definitely need to implement some external memory manager, or hack a big ByteArray in a fixed location and administer it from Squeak. The good news is that we currently don't have anything located in this low memory area (yes, we may be wasting 1M of RAM), so it may not be so complicated after all (if you are curious see how we implemented calloc() for SqueakNOS' VM.

Some interesting devices to work on

  • Networking. The first thing we want is connectivity, we currently have serial ports support, but it's not enough. It doesn't matter if we don't have a file system, or if we can't save the image in the hard disk. If we have connectivity, we can solve all this. And of course, we also want networking so we can use Scamper and Celeste! We have a partially working implementation of TCP/IP (made by Luciano a while ago), but we can't use it on real hardware! We are trying to integrate it into SqueakNOS over slip over the serial port, but no need to say that real ethernet support would be great!

  • ACPI. It may be not too complicated (if you can find the info), and it will be really interesting to have. Not only because knowing how much battery you have is important, and because if we don't have storage, suspending the computer may be an interesting way of surviving. But also, because we think that the main loop of the interpreter is really power consuming, and that we may have do something to improve that, but it's going to be complicated to test unless we can measure the differences. (the swiki has a little more about this)

  • USB. No need to explain why... Although we think that SqueakNOS should not have a real file system (but something a.la. GemStone instead), storage is important for sharing things. USB would be a great way to do storage (think USB keys). Also we'll want to connect cameras, microfones, mouses and all the other USB devices out there.

  • PCI. PCI support is not something we really want to have, it's more something we can't live without. Anything else will quite likely depend on PCI (Networking, USB, Video, Audio, etc.)

  • Video. We are currently relying on GRUB to enter the video mode, and we cannot change it after SqueakNOS has started. Having support for changing the video mode and playing with the video card would be nice (and probably not too complicated with the right info), however, we'll be happy with the current support for a while (mmm... ok, we'll probably improve video mode selection in GRUB).

  • HardDrives. probably EATA/IDE as they are the most common. Persistency is an issue, however implementing the HardDrive and some FileSystems is not the solution. We would like to forget about FileSystems, and be able to implement an Object Storage system a. la. GemStone as we said. Or probably starting with something simpler like ImageSegments or even spoon. To do this we may need to have some underlying FileSystem support, but we seriously don't want to see it from SqueakNOS :-)

    Open questions to discuss

  • Accessing The Computer. Currently Computer is a singleton, and you usually do Computer current to access its devices (for example, Computer current reset or Computer current cmos time). HardwareDevices get installed with a call to installOn: aComputer, so they get the Computer where they are being installed as an argument. They can either save this in an instance variable, or just use Computer current every time they need. Of course, if they are only going to use the interruptController they could just save it, or even just the InterruptRequestDispatcher they are using, instaed of the hole Computer. This design issue is not solved yet, and you'll see some things saving the computer, some accessing the singleton, etc... What do you think's the best option?... are we ever going to have more than one Computer?

  • #handleIRQ: interface. Related to the previous, when an IRQ "consumer" is going to be "signaled", it's currently sent the message #handleIRQfor:, and receives the InterruptRequestDispatcher. We've never been sure what information would be good to pass to this method... We also thought on implementing a hole family of methods like #handleIRQ:, #handleIRQ:onComputer:, #handleIRQ:dispatcher, etc. Now I think the best will probably be to implement handleIRQfor: anIRQDispatcher, and handleIRQfor: anIRQDispatcher on: aComputer, make a default implementation of the latter calling the former, and letting the user decide which want to implement. I just did it, What do you think is the best choice?

  • Unregistering IRQ dispatchers. If you take a look at the code for InterruptRequestDispatcher>>uninstall you'll see this weird line of code:
        [process terminate] forkAt: Processor lowIOPriority.
    We needed this because otherwise, if you wanted to uninstall the IRQ handler from within the #handleOn: you would end up killing the Process waiting on the Semaphore, which is the Process signaling the end of the IRQ to the interruptController. If you do so, the IRQ end would not be signaled, and the interruptController would end up in a weird (and non working) state. By forking at a lower priority than highIOPriority we are sure that this new Process will not be scheduled until the IRQ handling process is idle, and has already signaled the end of IRQ... If you got this confusing explanation, do you think the forking is a nice solution? should we use a queue for "uninstall requests" instead?

  • Native IRQ pseudo-handling. Currently, if for some strange reason, an IRQ is triggered by the hardware, but no Semaphore is registered for it, the end of the IRQ will never be flagged, and the interruptController will stay in this stalled state we talked about earlier. A possibility would be to add a very very simple IRQ handling code (just outb(0x20,0x20)) to every native IRQ handler, to be done only when no Semaphore is set... This should never be used, however, if something unpredictable happens, it may save SqueakNOS from hanging... should we put it? probably yes. Ok, I'm doing it right away, and waiting your comments. (I tried it, and it does make a desirable difference on cases where a hardware interrupt is enabled, but no InterruptRequestDispatcher is setm I think the code is staying).


    Oh well... This is it. We hope you at least enjoyed reading it, and of course, we'd like to see more HardwareDevices being born! Let's talk about this, and discuss more. We are opening a new mailing [2], and trying to maintain the swiki and diary entries updated, with information about new releases and small advances.

    gera for SqueakNOS

    [1] byteAt: and byteAt:put are implemented around the primitives primitiveInPortByte and primitiveOutPortByte. There are 4 more primitives (primitiveInPortWord, primitiveOutPortWord, primitiveInPortDword and primitiveOutPortDword) which can be used to implement more I/O methods.

    [2] We will create a mailing list for SqueakNOS discussion, but right now lets use Squeak-dev (at http://lists.squeakfoundation.org) to start with. This list has lots of traffic, so if you prefer just sent me private email. We will be prepending email subjects with SqNOS to make it easy to spot them.


    Good Article!
    2006-06-14 20:04:34 - KenCausey