My device crashed in the field. How do I figure out why?

• RTOS & Embedded OS
• Firmware & Programming
• Dev Tools & Testing

Logging is often enough to diagnose ordinary software bugs. When an embedded system encounters an unrecoverable fault, however, log messages alone are frequently insufficient. By the time the system resets, the state that caused the crash may already be gone.

Traditional embedded debugging typically requires reproducing the failure with a debugger attached, something that is often impossible once a device has been deployed.

This is where coredumps become valuable. A coredump preserves information about the system immediately before the crash, allowing developers to reconstruct what happened after the device has rebooted.

In this article, I will demonstrate how to configure Spotflow to upload coredumps on the FRDM-MCXN947 running Zephyr RTOS and explore how they can be used to analyze real crashes remotely.

What exactly is a Coredump?

Imagine you are debugging a firmware bug and your device suddenly crashes.

The watchdog reboots the system, the UART output stops, and when the device comes back online, the information needed to diagnose the crash is usually gone. RAM contents may be overwritten during startup, CPU registers are lost, and reproducing the failure under a debugger may be impossible once the device has been deployed.

A coredump solves this problem by capturing a snapshot of the system at the instant the crash occurred. Rather than simply reporting that the firmware crashed, it preserves enough information to reconstruct what the processor was doing immediately before the fault.

Depending on the configuration, a Zephyr coredump can include information such as:

• CPU register values (Program Counter, Stack Pointer, Link Register, etc.)
• The thread that was executing when the crash occurred
• Stack contents
• Selected regions of RAM
• Global and static variables
• The exception or fault that caused the system to stop

This information is typically written to non-volatile storage before the device resets. After rebooting, the coredump can be uploaded and analyzed later, even if the device is no longer connected to a debugger.

In the previous blog post, we configured Spotflow to collect logs from a running FRDM-MCXN947 over Ethernet. While this provided excellent visibility into normal application behavior, intentionally crashing the firmware using k_panic() or k_oops() initially produced only a "Device disconnected" event inside Spotflow. There was no information explaining why the device had crashed.

To enable meaningful crash analysis, we first need to configure Zephyr to generate coredumps and provide a location where those coredumps can be stored. Once that is done, Spotflow can upload the crash information automatically after the board reboots, allowing us to inspect stack traces, register values, global variables, and even AI-assisted crash analysis from the dashboard.

Intentionally Crashing the Device

With an understanding of what coredumps are and why they are useful, the next step was to intentionally crash the application and observe how Spotflow handled the event.

To simulate an unrecoverable fault, I modified the application to invoke Zephyr's k_panic() API after the firmware had been running normally for a short period of time:

LOG_ERR("Triggering intentional crash...");
k_sleep(K_SECONDS(10));
k_panic();

Unlike ordinary runtime errors, k_panic() immediately halts normal execution and enters Zephyr's fatal error handler. This makes it an effective way to verify that the entire crash reporting pipeline is functioning correctly without needing to wait for an actual software bug to occur.

After flashing the modified firmware, the device behaved exactly as expected: it rebooted almost immediately after the panic was triggered. At first glance, everything appeared to be working correctly.

An Unexpected Result

However, when I opened the Spotflow dashboard, I noticed something unexpected. Instead of reporting that the device had crashed, Spotflow only displayed a "Device disconnected" system event.

Although the final log message ("Triggering intentional crash...") had successfully reached Spotflow, there was no crash report, stack trace, or indication of what caused the reset. From Spotflow's perspective, the device had simply disappeared.

This immediately raised an important question: If Zephyr had crashed, why wasn't Spotflow recognizing it as a crash?

Why Spotflow Couldn't Detect the Crash

The answer turned out to be fairly straightforward. Although Zephyr provides built-in coredump support, it is not enabled by default on the FRDM-MCXN947. Without coredumps configured, the operating system has no mechanism to preserve processor state before rebooting.

As a result, Spotflow receives no crash data to upload or analyze. The device simply disconnects, reboots, and reconnects, making the event appear almost identical to an unexpected power loss.

To enable crash reporting, two pieces needed to be configured:

1. Enable Zephyr's coredump subsystem through Kconfig.
2. Reserve non-volatile memory for storing the coredump before the board resets.

Adding the Spotflow SDK

Before configuring coredumps, the Spotflow SDK first needs to be added to the Zephyr workspace. This is accomplished by adding the Spotflow repository to the workspace manifest (west.yml).

manifest:
  projects:
    - name: spotflow
      revision: main
      path: modules/lib/spotflow
      url: https://github.com/spotflow-io/device-sdk

Afterward, running west update downloads the Spotflow SDK into the workspace. At this point, the project now has access to the Spotflow libraries, networking helpers, logging backend, and coredump functionality used throughout the remainder of this article.

Enabling Spotflow Coredumps

With the Spotflow SDK integrated into the project, the next step was enabling Zephyr's coredump support. Spotflow provides several Kconfig options that extend Zephyr's built-in coredump subsystem, allowing crash information to be collected and uploaded after the device reboots.

The following options were added to prj.conf:

CONFIG_SPOTFLOW_COREDUMPS=y
CONFIG_DEBUG_COREDUMP_THREADS_METADATA=y
CONFIG_DEBUG_COREDUMP_MEMORY_DUMP_THREADS=y

The first option enables Spotflow's coredump integration, while the remaining options instruct Zephyr to include additional information about running threads and their memory in the generated coredump. This additional context greatly improves the usefulness of post-mortem debugging by preserving more of the system's state at the time of the crash.

At this point, everything appeared to be configured correctly, and I expected Spotflow to begin reporting crash information automatically. Unfortunately, that was not what happened.

The First Attempt

To verify the configuration, I modified the application to intentionally trigger a kernel panic after several seconds of normal execution.

LOG_INF("Triggering intentional crash...");
k_panic();

When the firmware was flashed onto the FRDM-MCXN947, however, the build process immediately failed with the following error:

#error "Need a fixed partition named 'coredump-partition'!"

At first glance, this error was somewhat confusing. None of the Kconfig options referenced a flash partition, and I had not modified the board's memory layout in any way. So why was Zephyr suddenly asking for a partition named coredump-partition?

Why the Build Failed

The compiler output also told us exactly where the error originated. Following C:/spotflow-ws/zephyr/subsys/debug/coredump/coredump_backend_flash_partition.c, I was able to find the section relevant to the error I was receiving:

The source immediately explains the problem: This backend stores coredump data inside a flash partition, and it expects a Devicetree partition labeled coredump-partition to exist. Before compiling, the preprocessor checks whether that partition has been defined:

#define FLASH_PARTITION coredump_partition

#if !PARTITION_EXISTS(FLASH_PARTITION)
#error "Need a fixed partition named 'coredump-partition'!"
#endif

This was the missing piece. My board's default Devicetree did not define a partition named coredump-partition, so the preprocessor intentionally stopped the build before any code was compiled. Since the required partition did not exist, the next step was to extend the board's Devicetree using an overlay file.

Adding a DeviceTree Overlay

Now that I understood why the build failed, the solution became much clearer. The coredump backend was simply asking for a location in non-volatile memory where crash data would be stored.

However, writing into flash raises an important question: Where should the coredump be stored? By default, the FRDM-MCXN947's Devicetree does not reserve any flash space for coredumps. Zephyr therefore has no safe location to write crash information, which is exactly why the build failed.

The solution is to extend the board's Devicetree using an overlay. Unlike modifying the board's original Devicetree files, an overlay allows an application to add or override hardware configuration without changing Zephyr itself. During the build process, Zephyr automatically merges the overlay with the board's existing Devicetree to create the final hardware description used by the application.

I added the following overlay:

&w25q64jvssiq {
    partitions {
        storage_partition: partition@0 {
            label = "storage";
            reg = <0x0 0x700000>;
        };

        coredump_partition: partition@700000 {
            label = "coredump-partition";
            reg = <0x700000 0x100000>;
        };
    };
};

This overlay modifies the board's external 8 MB Winbond flash device by defining two fixed partitions.

The first 7 MB is reserved for general application storage, while the final 1 MB is dedicated exclusively to storing coredumps.

The partition label itself is also significant. Earlier, we saw that Zephyr's build system explicitly checked for a partition named coredump-partition. Once the overlay defines that partition, the compile-time check succeeds, allowing the coredump backend to determine exactly where crash data should be written.

Building the Application Again

With the Devicetree overlay in place, I rebuilt the project, expecting the problem to be resolved. This time, the application compiled successfully without any errors. The previous build failure disappeared because Zephyr was now able to locate the required coredump-partition during compilation.

After flashing the updated firmware and intentionally triggering another panic, Spotflow immediately detected that the device had crashed. Unlike before, the dashboard no longer reported only a "Device disconnected" event. Instead, it generated a crash report containing information extracted directly from the captured coredump.

Even before uploading the firmware's symbol (.elf) file, the crash report already contains valuable low-level information. For example, Spotflow displays the processor's register values exactly as they were captured at the instant the exception occurred.

These registers represent the CPU's execution state at the moment of the crash and often provide the first clues about what the processor was doing immediately before execution stopped.

In addition to displaying the raw hexadecimal values, Spotflow allows each register to be viewed in several formats, making it easier to interpret addresses, signed and unsigned integers, and other numerical representations during debugging.

While this information is already useful, the register values alone are still just numbers. To determine which functions, variables, and source code correspond to those addresses, I needed to provide Spotflow with the firmware's symbol file.

Uploading the Firmware Symbols

Although Spotflow was now successfully capturing coredumps, the crash report still lacked one important piece of information: symbol names.

Without symbol information, the captured stack consists primarily of raw memory addresses. While register values and memory contents are still available, determining which functions were executing requires additional debugging information.

Zephyr generates this information as part of every build inside:

build/
└── zephyr/
    └── zephyr.elf

Unlike the firmware image (zephyr.bin) that is flashed onto the board, zephyr.elf contains debugging symbols, including function names, global variables, and source-code mappings.

After creating a firmware version within Spotflow, I uploaded the generated zephyr.elf file. Spotflow uses it purely as a lookup table that translates memory addresses into function names, source files, and variables.

Once the upload completed successfully, Spotflow was able to associate future coredumps with this firmware version. To do this, when another crash report appears, you can upload the .elf file directly from Spotflow instead of having to go to your project folder and locate the files manually.

Analyzing the Crash

With the firmware symbols uploaded, I rebuilt the application one final time and intentionally triggered another kernel panic.

This time, the resulting crash report looked very different. Rather than simply reporting that the device had disconnected, Spotflow reconstructed enough of the processor's execution state from the captured coredump to perform a true post-mortem analysis. Instead of relying solely on UART logs and educated guesses, I now had enough information to investigate exactly what happened before the system crashed.

Understanding the Exception

The first thing I looked at was the exception summary.

Spotflow identified that the firmware had terminated because Zephyr entered a kernel panic. In this example, we intentionally called k_panic() to verify that the coredump pipeline was functioning correctly, so that result was expected.

In a real application, however, this summary often provides the first indication of what actually caused the crash. Depending on the fault, it might instead report a stack overflow, an invalid memory access, an assertion failure, or another unrecoverable exception.

Knowing why the operating system halted is an important first step, but it still does not explain where execution was when the crash occurred.

Following the Call Stack

Whenever I am investigating a firmware crash, the stack trace is usually the next place I look.

A stack trace reconstructs the sequence of function calls that were active immediately before execution stopped. Without debugging symbols, this information would consist almost entirely of raw memory addresses. Because the firmware's symbol file had been uploaded, Spotflow was able to resolve those addresses back into meaningful function names and source code locations.

In this example, the stack trace shows that the main thread was executing inside k_sleep(), which had been called from main(). Although this particular example contains only two stack frames, real embedded applications often produce much deeper call stacks spanning multiple application layers.

Being able to reconstruct the chain of function calls is invaluable when tracing an unexpected crash back to the code that ultimately caused it.

The stack trace tells us how execution reached the fault, but I also wanted to understand what the processor itself looked like when execution stopped.

Examining the Processor State

To answer that question, I turned to the processor registers.

Spotflow preserves the CPU registers exactly as they were captured before the device rebooted. Registers such as the Program Counter (PC), Link Register (LR), and Stack Pointer (SP) provide a snapshot of what the processor was executing, where it expected to return next, and the state of the current stack.

For developers accustomed to working with a debugger, this information should feel familiar. The key difference is that the board no longer needs to be connected to a development machine. Instead, the processor state is captured automatically during the crash, stored in flash, uploaded after reboot, and made available remotely through Spotflow.

The processor registers explain the CPU's state at the instant of the crash, but crashes are rarely caused by registers alone. To understand why the software reached that state, it is often necessary to inspect the application's own data.

Inspecting Variables and Memory

Because the uploaded symbol file also contains debugging information, Spotflow can associate portions of the captured memory with variables defined in the application.

Depending on how Zephyr's coredump subsystem is configured, this can include global variables, static variables, thread information, stack contents, and selected regions of RAM. Rather than viewing only raw memory, Spotflow presents this information in a form that is much easier to relate back to the original application.

Being able to inspect the application's state after deployment is one of the biggest advantages of coredumps over traditional logging. Logs can tell you what happened leading up to a crash, but a coredump often reveals why it happened.

At this point, I had reconstructed most of the system's state. The remaining challenge was interpreting all of that information together.

AI-Assisted Analysis

One feature I found particularly interesting was Spotflow's AI-assisted crash analysis.

Rather than manually piecing together the exception information, stack trace, register values, and captured variables, Spotflow generates a summary that describes the likely cause of the crash and highlights the most relevant parts of the captured coredump.

I do not consider this a replacement for understanding the underlying firmware or performing a proper investigation. Instead, I see it as another tool that helps point the investigation in the right direction—especially when working with unfamiliar codebases or triaging crash reports from deployed devices.

Putting it All Together

After working through this example, I found myself following roughly the same investigation process each time I analyzed a crash:

1. Read the exception summary to determine why Zephyr halted execution.
2. Examine the stack trace to identify which thread and functions were active.
3. Inspect the processor registers to understand the CPU's execution state.
4. Review captured variables and memory to determine what the application was doing.
5. Use the AI-generated summary as an additional aid when interpreting the collected information.

Individually, each of these pieces of information is useful. Together, they reconstruct enough of the system's state to make meaningful post-reboot debugging possible, even after the device has crashed.

Conclusion

Before configuring coredumps, intentionally crashing the firmware produced little more than a "Device disconnected" event in Spotflow. Although the device rebooted successfully, there was almost no information available to explain why the crash occurred.

By enabling Zephyr's coredump subsystem, reserving dedicated storage through a Devicetree overlay, and uploading the firmware's symbol file, that same crash became a detailed report containing stack traces, processor registers, variables, and source-level information.

The most important takeaway is not just Spotflow displaying crash reports, but rather that coredumps fundamentally change how deployed embedded systems can be debugged. Rather than hoping to reproduce a failure with a debugger attached, developers can retrieve the information they need directly from devices running in the field and begin investigating immediately.

Sign in