Hidden Gems from the Embedded Online Conference Archives - Part 3
In the years since its inception, the Embedded Online Conference has hosted hundreds of excellent talks. The topics cover an impressive range, from high level introductions, to tool and framework comparisons, to case studies of really annoying bugs! This is the third post in a series looking at a different talk from the archives each week, appraising the content and providing some extra context. Here are the previous installments: Part 1, Part 2.
Last week we looked at a very successful project to send a rover to Mars. This week's talk, Learning from Disaster by Jack Ganssle, looks at when things go wrong! Sometimes very wrong!
Jack Ganssle is a highly esteemed engineer in the world of embedded systems. Over 27 years, he ran 500 issues of the Embedded Muse newsletter, finally finishing in 2024. An EE by training, Jack has worked on various aspects of embedded development and software. From 2003 to 2008 he was the embedded systems representative on NASA's Super Problem Resolution Team, part of the NASA Engineering and Safety Center. On top of all that, he gives engaging and thought provoking talks, including several at the EOC.
The first half of this talk is a series of case studies collected by Jack over several decades. A wide range of projects are covered, from bridges to helicopters to spacecraft - all of which failed for different reasons. Even if you have heard about some of the specific incidents before, Jack offers his own insight and a comparison between these failures.
The main lesson from this part of the talk is that a lot of projects do not learn from previous failures. The same mistakes are repeated. Jack places a particular focus on artificially compressed timelines and pushing engineers too hard. This has often led to important tests being skipped or warnings ignored in order to meet expectations. Consider how much time anyone in embedded systems spends carefully reviewing failures of similar projects in any given year. Probably not a lot. Public databases such as the FDA Medical Device Recalls database will provide plenty of food for thought if you simplly search "software". Jack is suggesting that engineers look at these failures as part of the process of development.
Having set the scene, Jack then moves on to tools which have proven effective in his career. Because of his background investigating failure in safety critical systems, Jack can speak with authority on this topic. There are no fads and little hype in Jack's list. Indeed, I think most engineers would find little to disagree with. However, it is worth reminding ourselves of the tools we already know about which are necessary for smooth progress through a project. There are also some useful quotes to bring to non-technical people who may question this approach as being too slow or too careful. One of my favourites, regarding the importance of the design stage:
If you think good design is expensive, try bad design
The talk ends on an optimistic note, with Jack hoping that people take this knowledge forward into their work. As this was a keynote talk, the Q&A section is at the end of the video and has some good nuggest to look out for such as:
- how to handle asserts in production code
- Jack's real opinion of Linux
- why some "agile" development is really just "chaos"
It must be difficult to condense lessons from a long career into a 45 minute presentation. Thankfully, Jack has a talent for making his expertise accessible to all. I'm sure you will learn something from this trip to the Embedded Online Archives. Join me again next week for another "hidden gem"!
Further Reading:
- Design by Contract - Object Oriented Software Construction by Bertand Meyer
- Agile! - The Good, the Hype and the Ugly (Book) by Bertrand Meyer
- The Best Defense is Offensive Programming by Tyler Hoffman
- Comments
- Write a Comment Select to add a comment
Blessed to see Jack's Learning from Disaster presentation twice back-in-the-day when conferences were in-person. Jack is great! Thank you for the reminder!
That's so cool! I think EOC is a nice format because it is so accessible and people can drop in and out, but it has been really nice to meet some of the other speakers in person :) I'm sure people will be learning from Jack's talks and newsletter archive for a long time to come!
To post reply to a comment, click on the 'reply' button attached to each comment. To post a new comment (not a reply to a comment) check out the 'Write a Comment' tab at the top of the comments.
Please login (on the right) if you already have an account on this platform.
Otherwise, please use this form to register (free) an join one of the largest online community for Electrical/Embedded/DSP/FPGA/ML engineers:
