Bug 6585

Daniel, can you please use the dediprog to read off the bad image (8M) and attach to this bug?

Max, please attempt to replicate the failed scenario following Daniel's instructions.

Could we have some kind of memory leak around the UEFI NVRAM variable space, that every boot to shell is eating up a little bit of variable space until, there is none left, and the shell can't boot without the new record?  (this is just a theory, but it explains how the firmware can stop booting when nothing should be writing to it on boot.  NVram space can be updated within the boot, and with a leak it could eventually run out).

Maybe something associated with USB (USB 3.0 support) or some other aspects of USB boot/enumeration?

If this is the case: Getting a bad flash image dump and comparing it to the original image loaded into the flash, we should not see corruption as much as unused spaces (FF's) becoming used data spaces (non-FF values), to the point that the number of FF's in that contiguous space of the flash are all but gone. 

Since John Hawley has provided such a diff file:
 
There is a transition of FF to other data (particularly 00's) in large continuous blocks at 0050923C to 0054DFFA.  That is  a lot of once clear spaces now utilized (282046 or over 1/4 megabyte).  Will have to check with the Firmware developers, to make sure that this is the NVRAM data space range in the image, but it looks like the theory may be proving out.  Someone is filling a lot of "blank Space" in the firmware with a lot of data. 

Now the need is to find the culprit responsible for all this NVRam writing.  

Tim He may be best suited for this, as he has all of the firmware sources, several MAX A1 boards and an ITP.

Created attachment 2062 [details]
firmware diff

This is a quick dump of the diff between the two firmware images G = Good B = Bad

When people say "a bootable USB drive," do they mean
1. any USB drive at all?
2. any USB drive with an OS on it?
3. specifically a USB drive that the firmware autodetects as bootable?

And if 2., then is it okay if the firmware skips the EFI shell and I end up at a GRUB screen, or must I change the boot order in the firmware so I actually see an EFI shell prompt in order to replicate the bug?

-Max

> At power on the bar will move across the bottom of the screen, but then all you
> get is a blank screen.

Do you mean a completely blank screen, or might there be some periods on there too? Because if that's the case I may have replicated it. I'm not sure. I'm going to re-flash my firmware and start over just to make sure.

Today's insight is brought to you by hexcompare[1], which is an awesome program that makes it visually obvious what's going on.

If I flash a new firmware and then follow Hart/Jackson's procedure, you can actually see that the nvram changes in a pattern that "creeps forward" on every reboot cycle. I have ripped firmware images from the minnowboard after 20 reboot cycles, 21 reboot cycles, and 22 reboot cycles. Since I think the firmware itself is still not to be made public, I'm going to attach a series of screenshots of hexcompare, which I think will illustrate what's going on. Anyone within Intel who wants the .bins of the ripped firmware images can email me. 

It's definitely clear that SOMETHING nasty is going on and things *are* getting overwritten.

[1] http://sourceforge.net/projects/hexcompare

Quick Firmware lesson:  

In Firmware, there is a need for non-volatile storage of data between boots of a platform.  many firmware solutions use a part of the Flash device as that data store.  This is a highly limited resources, but as long as the firmware drivers behave and utilize the storage correctly, there should be enough flash space for the needs of the system.  

There are mechanism to take back space that has been superseded by newer data, and  to even compact the data to defragment the flash space usage.  

However, IF the flash data space gets full, then problems can occur.  There should be a recovery mechanism to attempt to defrag the data, and failing that: clear settings, push back to defaults, log and error, reboot, and print a message.  Thus the system may act 'funny' but not lock up.  But an ill formed driver (or drivers) could lock the system up before any recovery is possible.  

The point here, is that while the writing is occurring, it may NOT be a "Nasty overwriting of program space", but normal non-volatile data storage in the space allocated for that function.  The nasty part is not the writes, but that too many writes have occurred without 'cleanup' and/or the data space is full, and the system has responded to that condition by locking up.

Created attachment 2063 [details]
Screenshots of hexcompare between different firmware images

This has comparisons between
* firwmare after 20 cycles vs. firmware after 21 cycles
* firmware after 21 cycles vs. firmware after 22 cycles
* original (uncorrupted firmware) vs. firmware after 20 cycles
* original (uncorrupted firmware) vs. firmware after 21 cycles
* original (uncorrupted firmware) vs. firmware after 22 cycles
You can actually see the damage "creep forward" with successive reboots. 

Eventaully it's bound to hit something important. I will continue rebooting until it does.

more information. i accidentally left one of the boards with the corrupted firmware powered on. after about 90 seconds the board booted to the uefi shell, but i did not get any hdmi/dvi output. i have been able to duplicate this with 3 other boards that had corrupted firmware.

@Michael Krau,

Fair enough. In fact, looking closer, all the bytes that have been overwritten so far were 0xFF in the original. I don't start seeing any actual data in the original firmware until around offset 0x5ffffc. So theoretically nothing should break until those areas get overwritten. If the firmware is working correctly, then those areas won't get overwritten at all. If there's a bug, then they will.

Is that about right?

-Max

(In reply to comment #9)
> more information. i accidentally left one of the boards with the corrupted
> firmware powered on. after about 90 seconds the board booted to the uefi
> shell, but i did not get any hdmi/dvi output. i have been able to duplicate
> this with 3 other boards that had corrupted firmware.

Getting an EFI shell on the serial console but not on the monitor sounds *very* familiar to me, because it happens to me on about 50% of boots. I always blamed display detection issues.

Also, I've had big hangs prior to getting an EFI shell, but I always blamed netboot taking forever to time out.

-Max

@ Max Eliaser (per comment #10)

Actually, the danger is not over overflowing or over-righting the area designated for storage.  Just filling that area, demanding more, then locking up the system when more is NOT available.  

The Firmware infrastructure (UEFI) will keep the data store from going out of bounds (writing values on top of code or such), but when the area fills up, it will respond with an error condition to the driver requesting the space.  If that driver reacts badly (in example: goes infinite loop on this error), one can see where the problem becomes a systemic crash).

The firmware is still good, none of the executable code is corrupted, only something is not playing by the rules when dealing with a limited resource. (think an offshoot of the 'deadly embrace' scenario.  A process holding the thread while waiting for a resource that will never become free, because the thread is necessary to free it).

I'm 100 reboot cycles in, and so far it's pretty well-behaved. Every 10 reboots or so, I get a longish pause before the EFI shell (not anything like 90 seconds, more like 5-15 seconds.) I guess this is the NVRAM defragmenter/cleaner kicking in. 

Next time it happens, I'll try yanking the power before it's finished, to see if this yields an unbootable system.

@max

i've been trying to replicate it myself. i'll get with my junior enginer in the morning to see exactly how he was able to replicate the problem. it seems some small aspect is missing...

Question:  Are the systems you are using going through video only, video and serial, or are they serial only?  the reports in the field are video output only, which may have some impact on the conditions.

we are testing with both video and uart console

(In reply to comment #15)
> Question:  Are the systems you are using going through video only, video and
> serial, or are they serial only?  the reports in the field are video output
> only, which may have some impact on the conditions.

I'm using both. As I said, sometimes that's the only way I can see if the UEFI shell has come up, although that's an existing display detection issue with this Lilliput monitor I'm using.

-Max

@michael

another piece of info, on the boards that we have been able to replicate this on, it appears that a couple of blocks in the nvram section have been marked write-protected. is this normal?

Some notes from my testing today:

- A MinnowBoard MAX A1 with the 7/18/2014 firmware
- A USB2 stick, formatted Fat32, non-bootable in the USB2 port
- Power cycled the board 70 times (full removal of power)

I have not been able to reproduce the issue with this setup.  I have pulled the firmware off of this build, I'm not sure it's worth adding to the bug.

(In reply to comment #17)
> (In reply to comment #15)
> > Question:  Are the systems you are using going through video only, video and
> > serial, or are they serial only?  the reports in the field are video output
> > only, which may have some impact on the conditions.
> 
> I'm using both. As I said, sometimes that's the only way I can see if the
> UEFI shell has come up, although that's an existing display detection issue
> with this Lilliput monitor I'm using.
> 
> -Max

Could it be that the person reporting this is just having the same display issues I'm having? Have you tried the "corrupted" minnowmaxs with different displays? 

Also, do you know if you can rip the NVRAM on a corrupted minnowmax, put it on a working minnowmax, and get another corrupted minnowmax? Otherwise all this firmware ripping I'm doing is a bit of a waste of time. :)

@max

both failed boards are on the way back to circuitco, but it will a few days. as a side note i have experience the corrupt firmware myself a few times in the last 4 months. the question here, is this the same issue or another?

I am hoping Tim He will be in soon,  and he can take all of the data everyone has provided and follow the clues to the root cause and possible solution.

(In reply to comment #22)
> I am hoping Tim He will be in soon,  and he can take all of the data
> everyone has provided and follow the clues to the root cause and possible
> solution.

If he wants the firmware images I've extracted, have him email me. I have 15 of them.

Created attachment 2065 [details]
diff of firmware after 70 reboots

Diff of the firmware after 70 reboots using the debug 7-18 firmware

Adding Tim to the bug so he's seeing what we are putting up here

Created attachment 2066 [details]
MinnowBoard MAX 7-18 DEBUG firmware w/ updater program

This is a *DEBUG* build of the MinnowBoard MAX firmware, issues 7-18-2014.  This is mostly being posted as a "I was not able to reproduce the bug on this" and as a suggested "please upgrade to" in case you do hit the bug mentioned here.

Downsides:
- There's *LOTS* of debug messages that come out on the serial port during boot
- It boots slightly slower than the normal firmware

--------------------------------
Simplest way to upgrade:
--------------------------------

1) take a fat32 usb stick, unzip and copy the two files in the zip to
that.

2) boot up the MinnowBoard MAX without any other storage installed.
This should get you to the UEFI shell.  If that doesn't work hit F2
while booting and use the boot manager to select the efi shell

3) at the shell type:
fs0:
FirmwareUpdate.efi MNW2_IFWI_X64_D_2014_07_18_1654_SecEnabled.bin

Tab completion works, as well as you want to check what the you are on
the right drive.

This will run for a while, and will shut the board down.  Just power it
back on (pressing SW1 should power it back up) and you are done, the
board is usable normally.

slightly different pathology, and this may be a new bug.  I've got a board that I started an install of Fedora 20 on, had some issues getting the display to look correct on a Lilliput (Lilliput's problem) killed it, changed monitors and had some issues with the new monitor.  In the course of rebooting things (pulling power) it's possible I interrupted the clean-up phase of the nvram, and now the board won't boot.  Nothing comes out on the serial line.

I'm going to pull the firmware, and pass it along.  This might be a slightly different issue than what has been root caused.

As the image than John pulled on 08/02, I did some test with it to check the nvram variable. it should be that Firmware Volume is not corrupted or the signature of FvHeader is broken. 

John, Do you remember what are exact steps to reproduce?, I'd like to reporduce it then try to root cause it on my side.

Created attachment 2087 [details]
Non-Debug Beta build Firmware R75695

This is a slightly newer, non-debug version, of the firmware with explicit fixes relating to this bug

Created attachment 2088 [details]
GPG signature for the firmware zip file

GPG signature for the zip file using my key

This was fixed in firmware release 0.71+, available at https://uefidk.com/content/minnowboard-max

It is highly recommended that users update their firmware to this, or later, versions.