Field notes

• The RockBLOCK 9704 on Zephyr, and the pin wired upside down
• I built a LoRa analyser because I was sick of guessing
• Zephyr is the DOS-to-Linux moment for embedded
• We open-sourced our Antarctic wave buoys

The RockBLOCK 9704 on Zephyr, and the pin wired upside down

A microcontroller on my back step sent 16 bytes through an Iridium satellite. The message was “Stamp-C3 says hi”, and it was in Cloudloop about ten seconds after the shell command that sent it. The Zephyr module that got it there is now public: rockblock9704-zephyr, MIT.

Why this modem

The modem is Ground Control’s RockBLOCK 9704, the Iridium Certus one. I have put about 70 of its older cousins, the 9602 and 9603, into Antarctic wave buoys since 2012. Those carry 340 bytes a message, so a buoy has to reduce a whole wave record to statistics on board before anything leaves. That has worked, and the science has always run on the statistics.

The 9704 carries messages of about 100 kB. With that much room the 2 Hz motion record itself can come home when a scientist on shore asks for it. The next buoy and the OGLAS satellite hub both use it.

There is no AT mode on the 9704. It speaks JSPR, JSON lines over a 230400 baud UART, with messages going out in segments, and Ground Control published the C library that speaks it under MIT.

Ground Control’s library

The module does not reimplement the protocol. west pins Ground Control’s library at a commit and the build compiles it as is, no patches. The module is the part Zephyr needs around it.

• A serial port on whatever UART the devicetree names, with the modem as a node on it and its three control lines as optional GPIOs.
• One worker thread that owns every call into the library. The library has no locking and keeps its state in static buffers, so I put all of it on one thread. The thread sleeps until a request or a byte from the modem arrives, so an idle modem costs nothing.
• An API where every request carries a timeout. The library’s own send has none, and its async path only reacts to lines from the modem, so a modem that goes quiet mid-send leaves a message in flight for good. The module adds a cancel for that.
• A shell command, so rb9704 info, rb9704 signal and rb9704 send work from any Zephyr console, and a spy that prints every JSPR line both ways.
• A fake modem in Python, on a pty or a serial port. The same tests run against Ground Control’s own Python binding, against Zephyr on native_sim, and against Zephyr on a board through a two-board rig.

The sample runs on an STM32U575, an ESP32-S3 and an ESP32-C3, which is Arm, Xtensa and RISC-V, with nothing in the module tied to a CPU. That is the Zephyr argument from August paid back on a smaller scale: the modem is a few lines in a board overlay, and the same code builds for all three.

Reading the library before trusting it

Before writing the fake modem I went through the library’s source to find out what it sends and what it expects back, and kept a list as I went. It runs to 21 items. A few that a caller has to know:

• The library asks for provisioning once and caches the answer, even when the answer was “not provisioned”. A modem provisioned after its first send refuses every topic until a reboot. The module clears the cache before a send whose topic it has not seen, and after a refusal.
• The blocking calls stop reporting success once a callback is registered. The module uses the async path only.
• A query during a send swallows the modem’s segment requests, because the reader discards lines it is not waiting for. Info and signal are refused with -EBUSY while a message is moving.
• A line with no status code reads past the library’s buffer.

The module works around the ones it can. The rest are written up in the repository’s docs/JSPR.md for upstream, with line numbers.

What the hardware taught me

None of these three are in the source, and I found them in the wrong order.

USB-C shares the UART

The 9704’s USB-C goes through an FTDI chip to the modem’s serial lines, on the same pins as the header. With the cable plugged in, the FTDI drives the modem’s RX, so a board on the header can hear the modem but the modem never hears the board. Every begin failed with -EIO. It did not occur to me for a while that the cable was the problem; once it came out and the modem was powered from the header, begin worked.

Boot output before begin

On power-up the modem sends its boot lines whether or not anything has opened the library. On the Stamp-C3 those overflowed a 128-byte UART FIFO, the next begin heard fragments of old lines, and the one after that crashed the board inside the library’s line parser, which is the last item in the list above. The module now reads the UART from boot and discards what it was not open for, and when it opens part-way through a line it discards the rest of that line too.

P_EN

The third was a mistake of my own. Outside, the modem answered for two to five minutes after each begin, then went silent: every request -EIO, the signal reports stopped, nothing until the control lines were cycled. I did this four times before I went back to the header pinout.

P_EN is the supercapacitor charge enable, and Ground Control’s hardware page says so, in capitals: active low. My overlay said GPIO_ACTIVE_HIGH, so I had either not read that line or not believed it. Every begin drove the pin high, switched the charger off, and the modem ran on whatever the supercapacitor held until it was flat.

Pulling the wire off the header proved it: the same firmware stayed up and sent its first message. I fixed the overlay, put the wire back, and the second message went through. The modem then ran through the night, 11 hours 54 minutes without a drop and satellites visible throughout.

The module now also releases P_EN while the modem is off, so the board’s own pull-down keeps the charger on, and it follows Ground Control’s shutdown sequence: I_EN low, then wait for the booted pin to fall before touching anything else, which their page says matters.

Not done yet

I have not yet had a message come down to the board through a real satellite. The receive path is there and runs against the fake modem on the host and on the boards, single messages and two-segment ones. The next step is to do that under open sky.

Where this fits

This is the kind of work I do: Zephyr firmware for hardware that has to keep running where nobody can visit, satellite and LoRa links included. If you have a 9704 on the desk and a board to put it on, the module is MIT, and I am happy to talk.

I built a LoRa analyser because I was sick of guessing

Debugging a LoRa link used to go like this. A node stops reporting, so I swap the antenna, move the gateway and drop the spreading factor. Something improves, or it doesn’t, and I have no idea why, because I cannot see the band.

So I built the thing I kept wishing I had. LoRaScope is a small radio that listens to the band and shows what is on the air: LoRa, MeshCore and LoRaWAN, with live switching between them. There is no gateway or network server in the path. The radio receives the packet, the box decodes it, and it shows up on a screen.

The product site is lora.dd.com.au. This is the field note version of what it does and why.

What you see

It shows who is transmitting, how often, at what RSSI and SNR, how busy the channel is and whether the link is healthy. Most of the guesswork went away once I could look.

For LoRaWAN and MeshCore it reads the headers without any keys: DevAddr, frame counter, port and airtime. You cannot see the payload without the session keys, but you do not need it to find the node that is hogging the channel, or to notice that a device is rejoining every ten minutes. When you do have the keys, it decrypts the payload as well.

Decoding on the box, or in the browser

Plain binary payloads and encrypted ones decode on the device itself and go straight to its screen. That covers the sensors I build and most of what I run into in the field: a few bytes of packed fields, sometimes encrypted with a symmetric key.

For anything more involved there is a JavaScript plugin runtime in the browser. You write a small decoder against the live feed, reload, and your packets show up as fields instead of hex. I have used it for site-specific formats and for vendor encodings that were never documented. It is the same feed the rest of the dashboard uses, so a plugin’s output goes into the plots and the logs like everything else.

Walking the site

The dashboard is served from the box, so a laptop or a phone on the same WiFi is all you need. A lot of LoRa problems depend on where you are standing, so there is a screen on the device as well.

It also has GPS and a ping test. Walk away from the other end, press ping, and distance, RSSI and SNR come up on the screen and get written to the SD card with the position. A range survey becomes one trip with the box in your hand, and nobody has to sit at the far end reading numbers over the phone.

Everything else gets logged to SD too, raw frames and decoded, and you can browse and plot the captures in the browser afterwards.

Other things I use

Airtime per station, with bandwidth and duty cycle. That is usually the number you need, and it is the one people are most often surprised by.

Channel scanning shows what is where before you pick a frequency. There is a LoRaWAN mode that watches for join requests and uplinks across the sub-band.

The box can also emulate a hub or a repeater, so you can test a single node in isolation without touching the field hardware.

Handing it to an agent

Everything the browser can see and do is also exposed over telnet and REST. That was a deliberate choice, and it paid off sooner than I expected.

I had a run of packet corruption on the bench. Instead of reading hex myself, I gave an AI agent the telnet and REST addresses and let it watch the feed, form a hypothesis, inject a test packet and look at what came back. It ran that loop on its own and found the fault. The full session is written up on the LoRaScope site: How I let an AI debug my LoRa network.

Being able to query and drive the analyser is what makes automated testing possible.

What building it covered

I am building LoRaScope to sell. I am also building it because it is a compact demonstration of the work I do for clients.

The firmware is C++ on an ESP32. Getting three protocols onto one radio meant pulling apart the binary framing for each and writing decoders that run on a microcontroller. The decrypt path is AES, with the key handling that goes with it. There is a capture store on SD with an API over the top, a web dashboard served from flash, and the telnet and REST surface that the agent used above.

The same skills show up in client work: ESP-IDF and Zephyr firmware, LoRa and BLE, binary protocol design and reverse engineering, encryption on constrained devices, device provisioning, and the database and API end that the data lands in.

If you have a mesh that drops packets or a binary protocol with no documentation, I take on that kind of work. The LoRa and LoRaWAN work I do is described in more detail there, or get in touch.

Zephyr is the DOS-to-Linux moment for embedded

I’ve written embedded firmware for a long time, on AVR, ESP32, nRF52 and RP2040, and mostly the default was bare metal or a small superloop. It worked. It was also the DOS era of microcontrollers: you own the whole machine, you write every driver yourself, and scaling up means more #ifdef blocks. FreeRTOS if the project was bigger, but the same shape.

Zephyr is the move to Linux. It brings what Linux got right down to a 64KB MCU.

I was late to this

In May I wrote myself a note about build systems. PlatformIO was under “Current” and Zephyr was under “Alternatives watched”, with the line “worth it if I move heavier on nRF.” The same note listed the trigger that would change my mind: “a specific project’s complexity outgrows PlatformIO’s project model.”

Then exactly that happened. I started the second-generation OGLAS repo on 23 July. By 16 August, 184 commits later, the whole fleet (sensor, repeater and hub tiers, plus a handheld and a remote control) was running on Zephyr.

Hardware goes in devicetree

This was the first thing that sold me. Hardware goes in devicetree overlays, features go in Kconfig fragments, and each product tier gets its own app directory. The three vary independently, so adding a board or a feature does not multiply the others.

In practice I collapsed a pile of per-role firmware variants into one config-driven image with a runtime node config store. There is one binary, configured per node, where before there was a build matrix that grew every time a customer wanted something slightly different.

The same app on ARM and RISC-V

The same OGLAS sensor code runs on nRF52840 (RAK4631, XIAO BLE, T-Echo Lite), ESP32-S3 (T3-S3, Heltec Wireless Tracker), ESP32-C6 and ESP32-C3. The last two are RISC-V.

The application did not change between instruction sets because the HAL is Zephyr’s and not the vendor’s. I did not expect that to work without effort. Changing silicon mid-project has normally meant a rewrite, and here it was a board file.

CI for firmware

ztest and twister are the parts most people miss. I have twelve test suites that build and run across a board matrix on every push, which is the regression discipline web code has had for years.

It still bit me. The platform list in CI has to name every board a sample.yaml mentions, or those tests skip silently rather than failing. That is how two of my apps went unbuilt for a week without anyone noticing.

The rest is less visible but adds up. Networking is part of the tree, so BLE, Thread, Wi-Fi and MQTT are maintained subsystems and not vendor code copied into the project. Power management has policy hooks: I gated a GNSS UART between acquisitions and stopped the GPS drawing current between fixes. The build reports enough that I noticed a board at 87.6% DRAM, moved the big arrays into PSRAM, and got it back to 73.2%.

No flag day

The interop contract for OGLAS is the wire protocol. A Zephyr node has to be byte-identical over the air to a legacy Arduino one, and the binary framing and radio parameters are frozen, so old and new firmware coexist in the field. There was no migration weekend and no fleet-wide reflash, and at no point did the whole network have to be on the new firmware at once.

Being able to deploy the rewrite one node at a time is the reason it got finished.

The potholes

Bringing up a Heltec Wireless Tracker, nothing I wrote to the SX1262 appeared to take. The sync word read back 0x00 0x00 instead of 0x14 0x24, and every transmit timed out.

I worked through the obvious suspects (software chip-select, reclaiming the CS pin from its pinmux, a push-pull reset, dropping SPI from 4 MHz to 1 MHz) and got nowhere. So I bit-banged a GetStatus on the same pins with plain GPIO, bypassing the driver entirely. The status byte came back 0x52, chip mode RX. The radio had been in RX exactly as commanded, so my writes had been landing all along and only the read path was dead.

The cause was that the MISO pad’s input buffer was never enabled (FUN_IE = 0 in IO_MUX), so the GPIO matrix was reading a constant zero. Boards where MISO happens to land on a strapping pin work by accident, because the boot ROM enables the input buffer for them, which is presumably why this had gone unnoticed. Between that and a board-file defect, that in-tree board’s LoRa radio had never worked.

That took two days. I could read the driver, find the fault, fix it and write it up for upstream. On a vendor SDK I would have filed a ticket and waited.

Where the line falls

Kconfig, devicetree and west are a lot to take in on day one. For a 2KB ATtiny or a one-off blinky I still reach for PlatformIO and a superloop, and I’d tell a client the same.

Once a project has more than one bus, more than one transport, or more than one person on it, Zephyr pays back quickly.

If you’re weighing the same move, the Zephyr work I take on covers firmware architecture, driver work, LoRa and mesh, low-power design, and getting a team onto Zephyr past the first month. I’m happy to talk through where the line falls for your hardware; get in touch.

We open-sourced our Antarctic wave buoys

For over a decade I built and maintained the firmware for the Waves In Ice buoys, instruments that are placed on Antarctic sea ice, freeze in, and report wave motion back over Iridium satellite for months at a time. The data from the earlier generations contributed to peer-reviewed science, including a 2014 paper in Nature on storm-induced sea-ice breakup.

That work is now open source as WII5 at wii5.sh3d.com.au: AVR firmware, schematics, PCB designs, enclosure notes, the on-board wave-statistics mathematics, and deployment logs going back to 2012.

Why open it up?

The reason is the bus factor. These buoys have outlived research grants, hardware generations and more than one institution’s involvement. The knowledge of how they work lived in too few heads: why the power budget is shaped the way it is, how the IMU maths turns accelerations into wave spectra on an 8-bit microcontroller, and what fails in sea ice.

Publishing everything means the next research group doesn’t start from zero. They can read the firmware source, see a decade of deployment history, and build on hardware that has already survived the worst environment we could find for it.

What’s in it for an engineer?

Even if you never go near sea ice, the interesting bits are the constraints: ultra-low-power AVR design that runs for months on a fixed battery budget, on-board signal processing because satellite bandwidth is too expensive to ship raw data, and mechanical design that survives being frozen into moving pack ice.

Remote monitoring, satellite telemetry and power-constrained firmware are the areas I consult in.