peteg's blog - hacking - nixie clock

Skipping right over justifications, I've been trying to get the ts7250-based nixie clock project in the can, at least hardware-wise. To recap, I was hoping to use Linux RT to drive the multiplexed nixies from user space, and that worked. The problem was that if you want to decode mp3s on the board at the same time then the CPU load is insane; one or the other is fine, with mp3 decoding taking maybe 70% CPU tops, and the display down in the noise. David G at NICTA suggested I write a kernel driver as it was probably the cache flushage (or other context switch overhead) that was killing it. Instead I took the less hairy chested and hopefully more reusable path of driving the display with an ATmega328P, interfaced to the ts7250 by TWI/I²C. That was the easiest hack ever: everything worked first-go (to the limit of the code on the AVR).

As I'm stuck with a 3G dongle for internet presently, I was hoping to use the ts7250 as an access point. To that end I bought a TP-Link TL-WN722N from MSY and set about getting it to work. The box claims it goes to 150Mbs, but the bloke at MSY told me I'm only going to get 54Mbs out of it from the MacBook Pro. I have zero idea about post-802.11g wifi; all I know is I can get several megabytes a second at UNSW on the UWN.

I've had luck with compat-wireless before, but really struggled with the 2.6.34 kernel, the most recent one that Matthieu's patches apply to cleanly. (Something really funky happened in 2.6.35 and onwards to do with SPARSEMEM — suffice it to say that if I can get one of these more recent kernels to boot then I only get 32Mb of memory, not 64Mb. Everything else seems fine though.) After much fruitless hackery I asked on IRC, and got the first useful advice I've ever received from that medium.

In brief, some long term stable Linux kernels are more popular than others, and as many distros picked up 2.6.32 it is the one to go for. (2.6.34 seems to be the least popular release around that time — apparently 2.6.35 got used by some embedded systems.) Sure enough, an hour of compiling later and I had the latest wifi drivers built and working on the ts7250. The moral is not to believe claims of compatibility with unpopular kernels.

Getting hostapd going is another story. I have a basic configuration that lets the MacBook Pro connect, but nothing exciting happens as I haven't configured the TCP/IP machinery on the ts7250 yet. The fun bit is figuring out what it wants to know about WPA2.

I also bought an ATEN UC232A USB-RS232 adaptor. MSY was promising them for $18 but had no stock, so I had to pay $25 in Little Tokyo. It's top notch, it even works through a hub.

The controller for the nixie clock is ridiculously complex, given how many buttons the remote control has and the combinatory explosion of modes it can be in. Ergo the attraction of Esterel, the venerable imperative synchronous language that is supposed to nail exactly these problems.

The only publicly-available tools for Esterel are from the crusty old v5.92 distribution of circa 2000, which presumably occurred before Gérard Berry et al tried to commercialise it. [*] Fortunately for me, Tim has tried them out recently and demonstrated that they're not too crusty for my kind of purpose.

Up to now I've spent more time than I should doing the boilerplate, and am only just getting started on the controller proper. It is difficult to work out the architecture ahead of time, given how much of neophyte I am with the language. The system interfaces are pretty good, and it seems possible that one day I could run some of this code on an AVR.

I meant to say that Rob gave me a remote that works fine with the ts7250. I think he got it to control MythTV. It's quite nice, and the HID driver in Linux has been very cooperative.

[*] Well, there is also the Columbia Esterel Compiler, but it hasn't seen any development for many years now.

My Hồ Chí Minh clock has a temperature reading on it, alongside the lunar calendar, so I figured the nixie clock should have one too.

In the past one-wire temperature sensors have been cheap and plentiful, and were traditionally hooked up to classic PC serial or parallel ports. Nowadays the sensors seem to start at $US5, though I was fortunate to get some DS18B20s from this bloke on eBay for $AU3 each plus postage. The ts7250's GPIO pins stand in for the less flexible interfaces of yore.

These DS18B20 devices are pretty fancy, being able to source power parasitically from the data bus and yielding 12 bits of temperature data. They're only accurate to 0.5° on average, though, or perhaps a bit better at room temperatures, so I don't understand why one would need more than the 9 bits of the other models.

Anyway, wiring it up took about 5 minutes, and finding a suitable cable another half-hour or so. Fortunately the first four GPIO pins of the DIO header on the ts7250 are supposed to have the requisite pull-up resistors, so the sensor can indeed be powered parasitically.

Getting Linux to talk to it was a bit challenging, as the one-wire drivers don't seem to be set up for actual use. Apparently you have to use the non-mainline w1-gpio-custom module to indicate the GPIO pin to the one-wire driver. I don't know how to do it any other way; perhaps it is supposed to be hardwired somewhere. Moreover to get timing right for the parasitic powering, one needs a few more patches, which this bloke has hacked together.

It works:

# cat /sys/bus/w1/devices/28-0000027d5e12/w1_slave
09 01 4b 46 7f ff 07 10 bf : crc=bf YES
09 01 4b 46 7f ff 07 10 bf t=16562

but often the w1-therm driver complains:

w1_slave_driver 28-0000027d5e12: 18S20 doesn't respond to CONVERT_TEMP.

and the CRC check fails with occasionally crap data. I'm suspicious about the device ID as the code apparently does know about the DS18B20. Unloading the w1 drivers doesn't work either. Still, this is all much better than I had any right to expect, I guess.

Finally I have managed to get a real-time Linux kernel running on the ts7250, resulting in a flicker-free display. Score one to patience.

Briefly, the Linux RT blokes skipped 2.6.32.x as the kernel's concurrency foundations got a reworking. They released an RT patch against 2.6.33.2, and as ynezz has forward-ported the ts72xx patches it was a cinch to move to the bleeding edge. YAFFS (a flash filesystem) broke as it uses some old-school mutexery. Grossly updating it all to the struct mutex way of doing things is straightforward and seems to work.

Hacking the clock driver code is trickier now though, as the process runs at the maximum real-time priority. If it loops without making syscalls, the system dies.

Slow and steady progress compiling software this week. It is tedious as hell, and I am mystified as to why mature projects still have such baroque configuration management systems. For example, GLib (a part of GTK) does not support cross-compilation out of the box. A few hacks later and it does compile relatively painlessly, so why haven't the hacks been folded back into the project itself? I think the lasting effect of Debian's packaging of the known universe is that these nasty problems get patched but not pushed (or accepted or whatever) upstream.

Anyway, I was shocked, surprised and relieved that my long-in-the-making cross-compiled MPD ran first-go on the ts7250. It took me an age to configure — ALSA calls the mixer "Speaker" instead of the conventional "Master", and ALSA is so overengineered that even something this simple requires forensic deobfuscation. Everyone's had problems with ALSA, so Google is full of unanswered questions from noobs with poor grammar, or pages from 2005 describing now-obsolete obscurities.

Well, yeah. Using the pleasant MPD client Theremin, I can now blast tunes from the Nixie clock and control it from the laptop. It sounds fine, and uses less than 60% of the CPU with the clock driver doing its thing. I feel like I have finally joined the class of 1998.

The last of the desiderata is a remote control, so I can park the clock on the mantelpiece and do less sophisticated things without the MacBook. The receiver half of this cheap-arse infra red thing I bought does not get along with the ts7250 too well, though it might be OK on stock hardware. Having pretty much given up on it, I will flog its carcase in all the fora known to Cirrus EP93xx sufferers as a public service. I would so dearly like to be cool and trendy and BlueToothy, but I have no cool and trendy mobile phone to pair a receiver with.

I spent the evening polishing a root filesystem for the ts7250. The flash is quite fat (128Mb) so I gave up on some of the buggy busybox applets (udhcpc in particular) in favour of their real counterparts from the Debian distro. This approach put dropbear back onto its branch, and as the board can reliably connect to the WiFi router I can now SSH into it almost always after boot. It displays the time now, and synchronises with ntp, and the built-in real-time clock works too, albeit with some hefty drift.

Unfortunately the system still does not schedule my program very satisfactorily: any perturbation in CPU load results in flicker, and it struggles to play an mp3 without skipping. This is with a low-latency Linux kernel (2.6.32.3). I get the impression that later kernels in that series are easy to get going on the ts7250, so I might try one, but apparently there will not be a full RT patch for 2.6.32. Bleeding edge it may have to be.

Part of the reason is that my program naively uses the kernel scheduler for all delays, not just the larger ones. Thus when there is contention for the CPU the system overhead spikes, taking roughly as much time as user code. The sirq (presumably clock interrupt) load is circa 10%. I can feel some busy waiting coming on.

Replacing the anode driver transistor was entirely routine. Its friend survived. As I was having one of those rare days when the crappiness of earlier decisions on this and other topics was not only manifest but within reach of correction, I decided to assemble the hardware. Here's some photos, taken with the Olympus μTough 6010 on Daz's shonky tripod.

It's running on one of Andrew T's old power supplies, as my old-school LM7805 arrangement couldn't handle the heat. These devices have good failure modes for the most part, but overheating manifests as a loop: the ts7250 draws less current when the CPU is idle, so when the regulator comes back from a shutdown due to heat the processor gets to run for a few seconds before the regulator overheats once more. Nasty.

I discovered the ts7250 puts out enough current on the USB port to power the WiFi and audio adaptors.

It seems that Google's Android platform is attracting a lot of people to ARM platforms.

Software-wise things are still on the slow. Debian's armel port features a working dropbear SSH server, so I reckon there's something fishy with my cross-compiler setup, maybe the C libraries. Conversely nothing ALSA-ish wanted to run. Generally things are looking pretty likely.

The options for getting software onto the ts7250 are unappetising; either hand-compiling everything or running the risk of someone else miscompiling something. I'm sick of the former so I thought I'd try the latter, in the form of Debian's armel port. Martin Guy's recipe makes this straightfoward. Andrew T gave me a little script that copies binaries and just the libraries they depend on, but what I really want is an easy way to recompile programs and their dependencies. I've used apt-get source blah in the past and been happy.

Bogosity: dpkg does not like running on NFS exported through nfs-user-server. It seems that lockd has been on their TODO list for approximately the last twelve years. Sanity and serenity is provided by nfs-kernel-server.

I sorted out the remaining issues on the nixie board, viz making the anode resistors uniformly 11kΩ. The display is bright but some PWM will cure that. So it was time to fit the whole show together, and just as I gave up on one of Andrew T's power supplies I managed to release some magic smoke from the nixie board by forgetting how parlous the power arrangement was when reaching over the board. I'd switched everything else off but not my power supply, which the cockroaches will be getting when the time comes. The ts7250 survived unscathed, and a ginger replugging of it all revealed that I'd only managed to toast at most two of the anode switching transistors, and their failure mode is to go short-circuit. Phew. Relief. The Russian K155ИД1 is still going like a champ, and John Taylor's power supply didn't notice a thing.

More RoHS-non-compliant repair tomorrow.

Well! It's been a long time since I wrote about this project. A lot has happened, even some good things. Hardware-wise I put the ts7260 beyond use by somehow trashing the onboard flash. All I did was ask it to write 6kb to the root filesystem! Instead it took out enough of Redboot (or perhaps one of the even more obeisant Technologic Systems boot loaders) that recovery became a matter of finding something with real serial ports, and trying my luck with the serial blaster utility that someone wrote for precisely this contingency. Suffice it to say I got far enough to know the board is not toast, but not as far as getting it working again.

I met up with Andrew T on Monday past and he gifted me with a pair of ts7250 boards, quite similar to the ts7260 but lacking the power supply magic; I must feed them 5v and nothing more. They both fired up fine, but with Linux systems too far out of date for my purposes. Fortunately their real-time clocks appear to work, and the world has regained its rosy tinge.

So I spent this last week, more off than on, building kernels and wireless drivers and whatnots for one of these boards, saving the other against calamity. It mostly works, albeit with some dodginess in connecting to the WiFi: the dhcp client in busybox takes a few goes to get a lease. I need it to reliably connect before I can cut the rats' nest of umbilical cords the ts7250 presently lives off.

Today I bought a Creative Sound Blaster Play! USB audio dongle. MSY is selling them for just $25, a steal for such an anachronistic device. (Creative itself wants $28 + $15 delivery.) Quality is fine to these non-discerning ears. It will take me a while to compile up all the ALSA libraries and things; I'm hoping to use MAD with an infra-red remote control.

Lesson of the day: say configure --prefix=$PREFIX yadda where $PREFIX is where the artefact will appear relative to the root of the destination filesystem, and say make DESTDIR=$DESTDIR yadda where $DESTDIR is the root of the destination filesystem on the host system. ALSA embeds absolute paths into the libraries. This approach screws up the paths in the .la files that libtool generates; it assumes that you'll be compiling relative to $PREFIX.

Software wise I hacked up a crossfader for the digits. It looks OK, but as Bernie observes it will certainly need tweaking to take care of the relative digit brightnesses and perhaps those amongst the tubes too.

I spent the final week of January in Orange. I helped Dad build a wooden case for the whole thing. It's not going to set any size or innovation records, but it looks tidy enough. I'll take a photo when the software is sorted out.

Finished wiring up the clock, and it seems to go fine. The video is pretty boring, I know, but I had to have some evidence of how I spent the past few weeks. :-) The tubes do have uneven brightness: the third one seems to have a blackened grill, and is dimmer even though it has a smaller anode resistor. The fourth also has a smaller anode resistor, making it a bit brighter than the first two.

Apart from some fiddly soldering with a fat wedge tip, this came together fairly easily. The next thing to do is sort out the software, rig up a case and see if Andrew T will part with another ts7260. I've yet to try out John Taylor's power supply, and am hoping I can run the whole show off the 5v the ts7260 is supposed to pump on the LCD port.

As for numeral systems, the answer is a big no — there are heaps of natural-language numeral systems in common use, even now.

I got bitten by the classic Linux-ism: the standard resolution of usleep is only 10ms on these ARM boards, which is way too slow to do the kind of multiplexing the clock needs: I need delays of about 2ms and could use 100µs. I'm allergic to busy-waiting in user-space too.

The short story is that I tried 2.6.31.x and couldn't get the board to boot, but 2.6.29.6 works fine, after some mild futzing in applying the patch in the ts7260 forum's file area at Yahoo. It does not provide high-resolution timers for the ARM ep93xx which the ts7260 is blessed with, though. One can readily apply the RT patches to 2.6.29.6, but these also do not include the hrtimers stuff; one needs a further, not so clean patch to get these working.

So the clock now seems to scan OK, with no flicker visible to my eyes (on the single tube I have installed so far). Well, that's true provided that no other process is running. I've selected the right scheduler, cranked up the priorities, locked the program's virtual memory, and generally futzed with no improvement in robustness under load.

As has been true for a long time, there is some ts7260 stuff in the latest Linux kernel (2.6.32.3), and apparently the RT patches have been merged to the mainline. (I cannot really tell, but the CONFIG_PREEMPT_RT and friends options are there.) I'll see if I can get that to boot and perhaps figure out the hrtimers story there.

An aside: here's a good post on the proper treatment of SIGINT.

I wired up most of the first nixie tube to the board today. It's a bit hairy for a few reasons: a misconnection could fry me or the ARM board, and small mistakes could lead to extensive resoldering. Thus far things have gone OK; it turns out my 33kΩ-series/33kΩ-pull-down setup is strong enough while the ARM board is unpowered, but once the ARM board is fired up the 100kΩ pull-up on the ts7260 is enough to fire the anode switcher. I cured this empirically, by reducing the series to 11kΩ and the pull-down to 4.7kΩ. It works, and the voltage levels seem plausible, but they still imply the current is very weak. I wonder about noise.

Now I can switch amongst the cathodes (those I've wired up) using the K155ИД1, and that's about that. I need to replace all my resistor pairs and solder up a further three tubes. Joy.

In other news John Taylor's high-voltage power supply turned up. Took about four days to get here from California. I haven't fired it up yet, but will after I get the display board completed.

I constructed a further three anode switchers and they work fine. Now I'm up to wiring in the nixie tubes, but as I only want to do this once, I'm getting a bit ginger. My concern is that the multiplexing setup I'm using, which is essentially what everyone seems to use, allows more than one tube to be on at a time. So in the worst case the poor K155ИД1 chip would have to pass approx 12mA (four tubes at 3mA each), which exceeds its rated 7mA.

From the software I've found, no-one seems to do anything clever, so I expect their microcontrollers get into the main scanning loop quickly enough that nothing blows up. Unfortunately the ts7260 takes several seconds to boot Linux, with the boot loader delaying three seconds for recovery purposes, so I don't think I can ignore this.

According to the ep9301 ARM chip specs, the ts7260 is supposed to configure all these GPIO pins as inputs on bootup, which leaves them floating. This is probably safe from the chip's point of view, but not the nixie board's.

A solution is to pepper the anode drivers with pull-down resistors on the bases of the buffer transistors (the MPSA42s). Ha! That's what this guy was doing. Hmm, perhaps everyone does it. :-) I'm using 33kΩ which seems to do the trick. The voltage drop across the series 33kΩ (between the ARM and the buffer transistor) increased from 1.7v to 2.3v, so the current for a logic-high has gone up to about 70μA, if I've done my sums correctly. That seems barely plausible.

This still leaves the circuit at the mercy of the software. A complementary approach is to gate the high tension, which I'll investigate doing when John Taylor's power supply turns up. I want to switch the nixies off under software control anyway.

I soldered up a stock PNP-NPN transistor tree for the anode control, and surprise, it worked first go. This was reassuring as it is the first time the ts7260 has been hooked up to the high voltage.

The next step is to replicate that a further three times, and then wire up the nixies. Board layout is the toughest part of this project, I am geometrically weak.

I managed to hook the ts7260 ARM board up to the K155ИД1 TTL BCD-decoder/nixie driver chip, the Russian clone of the American 74141. The pinouts are identical. The short story I gleaned from Don Lancaster's venerable TTL Cookbook is that the CMOS levels of the ARM (3.3v) can directly drive one or maybe two TTL gates without anything blowing up. I am thankful that electronic technology has a forward-looking longevity that software should envy.

The ts7260 refuses to generate the promised 5v on its LCD port, so I think the little switch-mode power supply that is supposed to do this has been fried (and I'm not getting into any finger pointing). The same unit apparently services the USB port, which may lend weight to the idea that that port's brains are intact but have become unstuck from its brawn. Blowing up this kind of hardware is not much fun, it simply silently stops working.

Why do the world's languages use the same numerals? Do any deviate?

I caved and bought one of John Taylor's power supplies. You cannot beat $AU12 or so, you just can't. I will persevere with the one I built for now, and see if I can quell the EMI. In any case I have enough tubes for two or three clocks (depending on whether I go for four or six digits), so it won't go to waste.

Rob gifted me with some wire, so I hope I can improve the grounding situation too. You never know, it just might come good...

Well! After a few hours of burning some digits in, the sacrificial tube is looking pretty good. The '5' does not light the lower-left arc, and I'm not sure what I can do about that.

As for the power supply, I revise my earlier comment to say that now I would simply buy a pre-built unit from John Taylor. They're small, and for $US10, cheaper than acquiring the requisite bits.

My perf-board effort is getting tidier and more finished, albeit at the cost of much solder and desolder braid. It seems stable, with tight regulation at 182v from no load up to about 3mA after replacing my 1.5Ω current sense resistor with 0.33Ω. Apparently I should run the tubes at something like 1.5 times the rated current, allowing for multiplexing, so the supply really needs to go to 8mA. I'll leave that to another day.

I found that the multimeter leads (hooked across the anode resistor) radiated enough to interfere with the radio. However even when removed, the TV still got a pincushion effect. (I'm glad I still have an analog TV.) Placing the whole thing on top of my venerable regulated power supply eliminated any visible or audible effects, which is reassuring but not yet sufficient for confidence. I'm wondering how I can improve the grounding and shielding without huge amounts of solder or metalwork.

Late last year in a fit of procrastination I decided to try to build a nixie clock, following in the footsteps of multitudes. To that end I ordered some tubes and driver ICs from a bloke in Moscow who was amenable to a bit of bargaining. I have to wonder how the Russians get their hands on these new-old-stock devices from the cold war era.

I received the tubes — IN-8-2's, with long solderable wires — yesterday after about a fortnight, well packed. They are a bit larger than I expected.

For a power supply I synthesised something using an MC36043 switch-mode controller from a few designs around the net. If I was going to start from scratch now I would simply follow this guy's design and advice. I think he's quite active on the nixie mailing list.

Anyway, what I've got does fire up the tube, as you can see. It generates an open-circuit 180v or so that sags to 160-170v with the tube connected. I think this is due to my overly conservative current limiting resistor in the power supply - with a 11kΩ anode resistor the tube itself seems happy, though I won't be using such a small one long-term. From what I've measured the efficiency is at best 70%.

Many of the elements of this sacrificial nixie tube don't fire up evenly, which might be due to cathode poisoning. I have found that burning each digit in for a few hours greatly improves the coverage, reducing this partial illumination, but it remains to be seen if all digits come completely good.

The supply also generates copious amounts of RF interference, which I'm hoping some more capacitors will fix. Haven't shocked myself yet, but I am sure I will. The next step is to try to hook the driver ICs up to Andrew T's ts7260 ARM board, and see if I can bring the nixie under software control. Yes, this board is massively overkill just for a clock, but as all engineers know, you can never have too much overkill.