Firmware Updates

27 Mar 2022

You may or may not know just how much I hate printers. They are the worst consumer product experience you could ever dream of. Drivers and DRM and ink that costs more than champagne – it's the type of subject matter that's easier to rant about than shooting fish in a barrel.

I solved my printer problems long ago, by moving away from the consumer tier and getting an industrial thermal printer we lovingly call a Zebra. It's a 6-by-4 label printer, so it doesn't really fill the same shoes as a run-of-the-mill inkjet, but in principle it fixes all of the terrible attributes in one swoop. There's no ink to replace, and it's built to a quality that will last. It's designed to spit out labels all day, every day, for decades at a time. Nothing to go wrong, nothing to replace except the labels we're printing on.

The death of my Zebra printer hit hard.

Aside from the misery induced by having to finish the day's parcels the old-fashioned way – writing out addresses by hand, queuing up in a post office behind the other filthy patrons, enduring their cringeworthy interactions with the incompetent staff and paying a premium for the privilege – the real pain is that of betrayal. Zebra printers are workhorses, they last for decades. They are not supposed to break.

So we're faced with a decision: to repair, or to replace. Consumer products get replaced. The Zebra deserves a repair. But the further we delved into the process, the further our printer, and the company that produced it, fell from that platform we'd elevated them to.

The symptom

Apart from the fact that it doesn't print, the only symptom we've got is that the status light is flashing red and green.

The official manual, and the internet at large, have conflicting interpretations of this status pattern. The only thing that can be agreed on is that it "needs service". Green and red could be a flash memory error, but red and green would be a printhead or motor failure!

I realize it's difficult to convey much meaning with a single status LED, but this level of ambiguity, where I can't even determine if the failure is mechanical or electrical, this is cutting out a new niche of poor user interface.

Based on the situation where the failure occurred (on power-up, not during a print) and also based on some prior knowledge of printers and CNC controllers of this era, I'm reasonably confident the problem is with the flash memory.

A little exposition

You're a manufacturer. You sell some hardware. That hardware has electronics in it, which these days means a microcontroller running some software. "Firmware" if you're feeling fancy.

That software needs to live somewhere. Back in the day, it'd be a UV-erasable EPROM chip, these days it's on flash memory.

One of those niggling little problems, the type that gets forgotten about until the very end of the development, is that after you've sold your hardware, it may be that you need to change that firmware. But it's only occasionally this needs to be done, right? So you hastily chuck together a firmware-update mode and duct tape it onto the product.

With the advent of IoT, and the myriad security flaws unleashed upon the consumer world, firmware updates have become routine and commonplace. About the only good thing to come out of the era of the WiFi coffee machine is that, by necessity, the firmware update process has been streamlined. It's still painful, but it's not on the same level as an un-networked printer firmware update. I have experienced far too many of them in my life.

Expect:

• a windows only .exe file which installs custom drivers and a special program for firmware updates
• a command line interface if you're lucky, or a broken/unintelligible GUI otherwise
• complete inconsistency of terminology, where "download" can mean either direction
• obfuscation or "encryption" of the firmware images
• being forced to register with the manufacturer to get hold of any of it

This isn't limited to printers, even today products are being released with the same mindset. I recently performed a firmware update on a newish MIDI keyboard, partly just to see if the process was any less abhorrent. After installing their software, which is also used to configure the keyboard, and downloading their obfuscated firmware image, the update took place over SysEx over USB. Par for the course. This culture is so pervasive that even the synthesizer I designed for Aaron was eventually released with a proprietary program that manages its firmware updates. It's what people expect.

Corruption

Flash memory doesn't last forever. It has a limited number of write cycles, often as little as 10,000. SD cards and USB memory sticks get around this by spreading the writes, wear-levelling so if you overwrite a file, it's actually getting written to somewhere else, rather than hammering the same physical part of memory.

I don't know if this information was secret ten years ago, but a lot of products seem to have completely ignored the concept. Many printers and CNC controllers, in addition to keeping their firmware there, write a copy of everything they print onto flash memory too. I don't know if there's a supposed advantage to this or if flash memory was simply cheaper than keeping the files in RAM. Either way, the more you print, the more you're spending your write cycles and heading for impending doom.

Even devices that don't smash their memory like this can suffer from the same problems, usually consumer-grade hardware with planned obsolescence anyway, but in some cases merely performing a firmware update is enough to recover the device. A full erase/rewrite may fix partial corruption, or maybe the system is clever enough, with some coaxing, to ignore bad pages.

So with great hardship I embarked on the process to update the firmware on my Zebra printer, but alas, it was not enough to bring it back to life.

Pain, or: how not to implement firmware updates

We've set the scene for the kind of experience we signed up for, but even I, with my jaded firmware update experiences, felt a new level of vexation.

To download the firmware image, we have to register. To register, we have to provide a huge amount of personal data, including our postal address and phone number. Like any sane individual, I use a password manager and random generated passwords. Arbitrary password restrictions are one thing ("you must include at least x special characters" etc) but you know you're in for a fun ride when the designers of the web form specifically went out of their way to break compatibility with password managers.

They force you to type this out in full twice to register, and then again to sign in, so of course I went with something substantially less secure than I would otherwise have used.

Zebra's firmware update utility is called ZDownloader, but after installing it, it can only be found in the Start Menu as the completely non-descriptive "FirmwareDownloader". How unlikely it could be, that someone keeps around a windows machine for precisely this purpose, and has half a dozen other firmware flashing utilities installed!

Click on it, and it's ZDownloader, but in the start menu Zebra or Z returns nothing.

The program itself has all the quirks we've come to expect as standard, with unexplained acronyms and terminology, but by far the most irritating aspect is the lack of a verification step. It's common practice to refuse reading out the flash memory, to prevent stealing the firmware image I suppose (even though we already have the firmware file). But after writing the image, and sitting through the progress bar, there's no confirmation of whether the process worked. All that happens is the selected "kit" (firmware) in the table returns to blank. If we could verify the memory, that would at least tell us if we're chasing the right problem here. (It still might be a critical motor or print head failure!)

Surgery

After dismantling the printer, the memory chip was easy to spot as it was hidden under a sticker with firmware version.

Although Spansion, the company that produced this chip, has long gone out of business, happily I found someone selling new old-stock replacements pre-flashed with the Zebra bootloader code. I suppose that says a lot. Enough people have faced this same problem that someone has made a business out of it.

A touch of the hot-air gun got us started.

The pads were cleaned up and the new chip soldered into place. 'Course we lost our RoHS compliance in the process.

Reassembly, and then we've got to do the firmware-update process yet again, to christen the new memory chip. With that, the printer came back to life.

Proprietary protocol, proprietary driver, proprietary software and a terrible UI. These are the decisions that lead to a world where replacing a 48 pin, 0.5mm pitch surface mount memory chip is the easiest part of the repair process.

How to do it right

I wouldn't be as bold in calling out this utterly atrocious state of affairs if there wasn't a simple, obvious, right way to do it.

At the time of writing, I've not yet released my Precision Clock Mk. IV. The design was finalized in 2020 but due to the chip shortage, I've not been able to mass produce it. That won't stop me boasting about its firmware update process though. I plan to release the design as open source, but these techniques are just as applicable to any hardware product.

The clock has a QSPI flash memory chip (SOIC-8, so much easier to solder than the Spansion chip above) which contains the timezone database, world maps and firmware images. An additional complication is that there are multiple processors on board running separate firmware images. I handle that smoothly with a chain-loading bootloader, but my approach would have been the same even for a single processor solution.

Here's how it works: You plug in the USB cable and it appears as a mass storage device. Any operating system, Windows, Linux, Mac, even a phone will be able to see the device without needing a driver. You copy and paste the firmware image onto the drive. Done.

In my case, the mass storage device maps directly to the QSPI flash memory chip, but it doesn't have to, the whole thing could be virtual. It's only on the next power-up that the image is verified and copied from the QSPI device to the internal memory, by a custom bootloader. The process is robust enough that even if you cut the power part-way through, it will gracefully recover.

Sure, it took a little more effort to design than the laziest possible bootloader. Sure, I could have rolled with the system bootloader or even asked my users to invest in a J-Link programmer. But I'm just an individual doing this in my spare time. An individual who wanted to do it right.

The solution is obvious, and any companies that release products in this day and age that inflict on their users the monstrosity that is the firmware update .exe should be ashamed.

Spare Time and Hard Work

17 Sep 2018

Half rant, half motivational speech. Mostly rant. Twice this week I was accused of having "a lot of spare time". I find this rather offensive.

Spare time is, presumably, time spent not working. It might also mean time spent not doing anything. Or maybe it means time having fun. By any definition, I don't have much of it!

We all, obviously, have the same amount of time. We sell some of it to our jobs, and we waste some of it sleeping and eating. Some of us sell more, some of us waste more. What's left after that? Enjoyment time? The micro-moments that exist between the things we have to do? We stick to schedules, perhaps, and try to make our lives more efficient. I think that everyone is inefficient. I think it's impossible to make good use of time. The only thing you can hope for is to do slightly better, and be slightly less inefficient, than other people.

Let me talk about a couple of years ago. I was working for a startup company and because I didn't know any better, I was working 60 to 70 hours a week for almost nothing. It was a nightmare. I had no social life, no enjoyment, no relaxation. There was conflict, and there were tears. In the precious moments when I wasn't working, all I wanted to do was to curl up in bed and watch a shitty anime.

But it was during this time, this horribly busy and miserable time, that I produced the LED Earrings. It took about three months, squeezing in maybe one or two hours per week. I am not fooling around when I say that the hardest part of that and many other projects is the commitment, the mental effort of finding the time to work on it, forcing oneself to working on it, instead of running back to the comforts of anime and chocolate and vodka and cigarettes.

I don't understand the reasoning when someone looks at those earrings and says "That guy had a lot of spare time on his hands". That guy succeeded because he put in a huge amount of hard work – hard work to find the time to dedicate to the project, hard work to overcome the emotional burdens, hard work to suppress the symptoms of his misery. You think this site exists as some temple to the author's satisfaction and contentedness in the universe? Jesus, the whole point of a creative outlet is so that you can get away, get from one day to the next, to spend an hour or even a few minutes without worrying about the realities of life. If I have made good use of the micro-moments, it is simply because I am terrified of idleness.

Whatever you think, and despite what I may have said in the past, projects like those earrings did not materialize because I was twiddling my thumbs with nothing better to do.

You're lying to yourself if you think that you haven't got enough "spare time" to do projects or learn new skills. You may not have the willpower, or the ability, but few people do when they start out. The point is that spare time is rarely the problem. Willpower and ability are things anyone can work on, can practice and improve upon. If it was easy we wouldn't be ranting about it.

For every project I've put here, there are probably an equal number of failures I haven't mentioned. And I don't mean a failed project where we learn from where we went wrong and try harder next time, I mean a failure of willpower. Failures where I give up because it's too hard, failures where I just don't have the dedication to see it through and the entire time I've invested into it has gone down the drain. I don't talk about these ones because it's depressing, and depression is contagious, and if we didn't put on a brave face then we might as well all just kill ourselves.

"Spare time" is a mindset, a broken mindset that puts every last problem away in a neat little box. But life is chaos. We make things so that we can pass time with the minimum amount of regret. I'm still just the miserable, disenfranchised misanthrope that I always was, but at least now I have something to add weight to my rebuttal. mitxela.com/rants is the one forum I have where I can tell the "spare time" accusers to fuck off.

There's something about midi.org...

10 Apr 2016

Update 2018: I was contacted by MIDI.org about this post, and I should point out that they have probably updated their website since this was written and it's unlikely to still be relevant. I'll leave it up anyway, because I don't like changing history, and also because I still find the archive link useful.

Cookies, the Legislation and the Grand Misunderstanding

29 Mar 2016

It's been seven years since I last posted a rant here. Rants tend to have a time limit on them; usually, eventually, the problem gets sorted out. I deleted the oldest rants a while ago, they were no longer valid or interesting. Maybe this one will become invalid too. I hope so.

Before this rant can even begin, I have to tell you about the moronic law that was passed by the UK government a few years ago regarding cookies. And before I can talk about that, I want to just cover the basics to make sure you're on board about what a cookie actually is.

Imagine you're in a queue at your local council. There's a man shouting at one of the desks. A woman with too many children is struggling to silence them. Little wooden beads on red metal wires have long lost the interest of any infant who would have played with them. You're tired, your knees ache as much as your head. You want to sit down, and you can, because the queue uses paper tickets.

These paper tickets are a lot like cookies, and are subject to the same rules. It's your responsibility to hold on to it. Nothing stops you tearing it up, throwing it away or giving it to someone else. You could tell the establishment that on principle, you do not accept paper tickets. Fine, but they would probably take no notice, and you would never advance in the queue.

When they announce who's next in the queue, they do so over an insecure channel (a loudspeaker, or a display board that everyone can see). Nothing stops you scribbling out your number and forging that of someone else's, and getting to the clerk's desk first. They might notice your handwriting, and this is one way that cookies differ from paper tickets, there is no handwriting.

At the end of your visit, it is expected that you throw away your ticket. It was intended to last for this session only. It has expired, but there's nothing stopping you keeping it.

Queues and raffles use paper tickets to identify you, and there are two reasons we don't complain and argue about privacy:

• you're in control, and can throw away the ticket whenever you like
• you want to be identified (you're at the font of the queue / you won a prize!)

Similarly, most of the time with cookies we want to be identified. Here's another analogy.

I go to my bank to make a withdrawal. I show my bank card or passbook to the cashier, or I sign some paper, and convince her that I am the owner of the account. Then she does the transaction and hands me my money. Why should she trust that the person she handed the money to was the same person who proved they owned that account? Because I was standing there the entire time.

But when we browse the web, each page view is a separate request to the web server, and initially there's no indication that it's the same person making each request. This is a little like me walking in to the bank, saying I'd like to make a withdrawal, and then leaving. A moment later I walk back in and show my bank details, then leave. I enter for a third time and expect the money to be handed to me. Well, maybe they still would if I was easy enough to identify. My hat and glasses might be unusual enough to single me out, but anyone could fake those.

My facial features are probably harder to fake, but I can't change them. Not without a certain amount of pain anyway. So if a criminal found my doppelgänger, or gave someone plastic surgery to look like me, they could swindle me repeatedly. Every time I left the bank after a transaction, they could walk in and say, "Me again, I'd like to empty the account." And they could do this every time, until I changed my face.

On the internet, these identifying clues are things like what browser you're using (hat and glasses?) and your IP address. IP addresses are difficult to fake, but not unique - yours can change, or you may even share it with the rest of the office.

So we come to the concept of logging in. I prove to a website who I am, once, by giving my password. In return they hand me a cookie. This is identical to the paper ticket, except instead of assigning numbers sequentially, they choose a very large random number that (hopefully) is impossible to guess. This is called the session ID. I show this session ID with every request, and the website believes who I am.

Of course if the channel is unencrypted, anyone listening could steal that session ID and pretend to be me. But encryption is outside the scope of this rant.

There are other uses of paper tickets, and there are other uses of cookies. This website, for instance, uses cookies. The title images are re-drawn in a different font for each time you visit. The only way to do this is through a cookie – the images themselves check and set this cookie, and all it contains is the name of the font. This makes all of the images on the home page the same font, but when you close your browser and visit later, they will all be in a different font. This cookie does not identify you in any way.

Another cookie this site uses is to optionally disable the fade effect between pages. This task could be better achieved through HTML5's localStorage, since there is no need to send this information to the server, but I use a cookie for maximum compatibility (and I wrote the site before localStorage existed).

In both of these cases, if you refuse the cookie I don't care, it's only your user experience which will be affected.

If you're thinking that all this about cookies is super-obvious, I'd agree with you. But apparently the folks in charge have no idea about what cookies are or what they do.

Tracking, and the Right Thing To Do

If websites can identify you, of course they can track you. Your browser is only supposed to show the cookies to the website that set them, but companies get around this by embedding frames to other domains. But if you don't want to be tracked, don't play the game! Remember that you are in control of what your cookies are and what they do.

The need for cookies and their legitimate uses means that disabling them entirely is not an option. In addition, most websites that want to track you will punish you for not accepting them – you'll always be stuck at the back of the queue. But there's a way of satisfying all these sites without letting them track you:

• Allow all cookies for the session only
• White-list sites you want to stay logged in to

As simple as this seems, it solves all problems. Tracking cookies are normally set with expiry dates several years in the future. By accepting them for the session the website functions perfectly, but when you close your browser it completely forgets who you are. The next visit it thinks you're a new customer. For certain forums and sites that I really do want to stay identified (logged in) I white-list them and they behave perfectly too.

This is how I've had my browser configured for more than ten years. This is what everyone should do. But it seems that almost no one does.

The Law

Instead of educating everyone on the ways of the online world, the government brought in a baffling law. It's baffling because I cannot tell if it was devious in its motives, or just good-willed and moronically stupid.

It seems that the law stated: all websites that 'use cookies' must tell you that they do so.

(I actually tried to look up what the law said exactly. It is was brought in as "The Privacy and Electronic Communications (EC Directive) (Amendment) Regulations 2011" and was again amended in 2015. However the two minutes I spent trying to read these documents was not sufficient to get past the legalese. I'm basing my rant on what companies and websites actually do.)

I'd wager that 99.9% of websites use cookies in one form or another, so what, every website in existence has to warn you? Apart from anything else, anybody ignorant enough to learn something from these warnings is unlikely to care and will just close it without thinking. And since every website will have this warning, it becomes meaningless. To close the box you have to 'agree' to their terms – convincing the user that it is the website, not them, that is in control of their cookies.

I like the BBC, and I think this message and their system is well-intended. But those 'settings' are all cookies, and by 'accepting' them you are handing control over to the website as to whether or not you are tracked. Would you rather meekly ask a website not to track you, or simply prevent tracking from being possible in the first place?

At least Google doesn't bother with the pretence of giving you a choice.

The Joke

None of this post so far has been rant. All of that above was just preamble. I thought that the people of Britain would rise up and overthrow this idiocy. You want to know what grinds my gears? You want to know the great irony to it all?

In order to close that warning box, the website has to remember that you've closed it. How does it do that? Via a cookie of course.

Yes. You have to permanently accept a cookie, or else always be hounded with the same message about cookies every time you visit the site. Anyone who has their cookies set up the correct way, as I described above, will be punished for it by being shown a message about cookies every time on every site they ever visit.

I have put up with these messages for years. No one cares because everyone else is willing to play the game.

The result of the law is that ordinary people continue to be tracked, and people who do the right thing are punished for it. Since tracking data is so valuable to marketing companies, and it would be a disaster if everyone configured their cookies the correct way, I cannot help but suspect an evil motive behind the law. Either that, or the people in charge are just really, really stupid.

Next time, on mitxela.com/rants: Google's crusade on unsecured websites – not everyone has something to hide.

Tune in in another seven years.

Out with a Haiku

25 Sep 2009

This was once a blog.
It might have been amusing
but now it is dead.

I got rather bored
of writing crap about me.
Hence no more updates.