My Lowest-Friction Embedded Project Was Also the One That Shouldn't Have Worked

We have the best embedded development tools in history. Free, world-class compilers. Inexpensive debuggers and oscilloscopes and logic analysers on every developer's desk. Source code management with git, either self-hosted or as a service. Containers so your build works on any machine. Test frameworks that catch regressions before they ship. So why does shipping firmware still feel, for so many teams, like pushing a boulder up a hill and hoping it doesn't roll back down?

I've been an embedded developer for about forty years, and I've thought about that question a lot. I was thinking about it especially hard in the second year of COVID, when the firmware team I led at the LEGO Group was handed a burning platform.

The Project That Shouldn't Have Worked

The 8-bit micro we'd been using across the Powered Up product line was going end-of-life. We had a little over a year to rework a family of seven motors and four sensors onto a new 32-bit family. On paper, the project was looking risky. Consultants unfamiliar with the internal operating model, extreme pressure to get it right the first time, zero room to move any of the external constraints.

And yet this turned out to be one of the lowest-friction development environments I have ever worked in.

I can't tell you exactly how it happened or give you a recipe to reproduce it. But I can describe the pieces I noticed, because some of them are cheaper to try than you'd expect.

It Starts Before Anyone Writes a Line of Code

We kicked off in person, for two days, with the whole team in one room. I cannot oversell how much of the project's later resilience came from that meeting. Not from the technical agenda, which was secondary, but from establishing real trust between the humans who were going to have to rely on each other under pressure for a year.

We ran two exercises from the Liberating Structures catalogue. What I Need From You surfaces concrete asks between functions, in public, with no room for polite fog. TRIZ asks, with a completely straight face: "What could we do as a team to guarantee this project fails spectacularly?". The list you produce is a perfect mirror of every bad habit your organization already has. Once it's on a wall, nobody can pretend those aren't the things that will sink you.

At the end of the two days, we sat down and looked each other in the eyes. No managers, no director. Just the people who had to do the work. We asked each other whether we believed this could be done. The answer was yes. Not reluctantly, not with the caveats that everything has to go just right. A plain, and well-considered yes. Walking out of a meeting with that answer is a feeling I've only had a handful of times in my career.

Then You Have to Actually Ship Something

The execution was almost aggressively boring. Default Jira project with to-do, doing, and done, no custom fields. Plain sprint planning, one sprint of lookahead, goal set by the team. A single Confluence page as the overall source of truth, with each device, its schematic revision, its firmware image, and its current status. Containers for development so it genuinely did not matter whether a given consultant was on macOS or Windows. Sphinx with the Read the Docs template for all documentation, built by CI/CD on every push, with API sections pulled directly from source so they could never drift out of sync with the code. Behavior-driven development from day one on dev boards while we waited for the first prototypes to arrive. Trunk-based development, small focused commits. Merge conflicts became so rare they stopped being a topic.

None of those tools are clever. None of them are where the friction was hiding. The friction was hiding in the handoffs we didn't do, the long-lived branches we didn't create, the status meetings we didn't need because the Confluence matrix was already up to date, and the design reviews we didn't stage as formal gates because the collaboration was continuous.

The Part Nobody Tells You

If you've ever sat through a teamwork training workshop, you've seen the Formula 1 pit stop video. The 1950s pit stop takes a minute of guys with hammers. The modern one takes two seconds and looks effortless. Your team just needs to communicate and plan better, and a growth mindset wouldn't hurt either - right?

Sure. But the modern pit crew also has redundant staff, redundant tools, a single unambiguous success criterion, continuous training, and a mostly predictable problem space. Your firmware team almost certainly has none of those. You're understaffed. Priorities change. The success criterion shifts depending on who's asking.

Adding more process to an already-struggling team usually doesn't help. In my experience, process changes only move the needle if they build one or more of these 5 things:

• Trust
• Purpose
• Collaboration (not coordination)
• Support
• Confidence

Those are soft qualities. You can't really measure them, you can't buy them, and you can't train them in a workshop. Your existing process already either builds them or erodes them, and you can tell which by looking at what your team does when there's an urgent release and people start negotiating the checklist to save a few days.

My talk at the Embedded Online Conference, dives into bulding the 5 key pillars with sections on:

• Trunk-based development habits that made our code reviews a formality instead of a battle.
• The four types of documentation that every project actually needs (and why two of the diagrams you probably think are architecture diagrams aren't)
• Building a learning organization to ensre that your entire team is growing capabilities

If your team is shipping, but every delivery feels like a push to get it over the finish line, and that's been normalized, the talk might help you name a few of the things you'd otherwise have to discover by accident.

We are in the golden age of tools for embedded systems development. The question worth asking is whether your organization is using it to help your engineers shine, or to give them more rocks to push.

Sign in