Gigabyte Z170X-* (skylake) Motherboards - duplicate boot entries UEFI BIOS bug, TAKE 2

Hello. This is a bug report for certain Gigabyte Skylake motherboard customer(s). It tries to explain simply about this duplicate boot entries problem.

Symptom

Every time a new boot (or reboot), the boot menu list changes, and extra 'UEFI: ' boot entry is added
The new entrie(s) are duplicated of other entries. And more than the 'normal'. Eg. 3x same entry (or higher).
After only 5-10 reboot, the boot list becomes very long

Example: in Gigabyte BIOS pages

Example: in windows EasyUEFI tool

Who?

This bug affect Gigabyte GA-Z170X-* series Motherboards, and their customers. For example:

Gigabyte GA-Z170X Gaming 3
Gigabyte GA-Z170X Gaming 5
Any similar product which share same common "new" UEFI BIOS for skylake. ~Z170 Gaming 7, UDH5, H170, and more.

The problem also seen on many OLDER gigabyte motherboards, and some OLDER ASUS mothboards. But the difference is those ones can be solved with a special editing of the boot menu. Or other workaround fix. That categorically is not the case anymore for newer Skylake - gigabyte motherboards. For example the Gigabyte GA-Z170X Gaming 5 which is very popular now. Perhaps the Gigabyte firmware has changed, and is now based upon a NEWER AMI bios product. And makes other changes (improved UEFI and other feature) for the BIOS for skylake era.

AFAIK this is a long time bug. Never properly fixed / not yet. The rest of this bug report assumes a "Gigabyte GA-Z170X Gaming 3", "Gigabyte GA-Z170X Gaming 5" or similar Gigabyte motherboard (of sklake era).

Workaround

It is possible to clear the duplicate entries manually, yes?

These 2 ways:

-1) manually by using a special UEFI tool:

'EasyUEFI' freeware program on windows (choco install -y easyuefi)
Clover console, in clover bootloader's boot menu

Problem:

not all customers run windows, and the other tools are not simple to operate.
customer may not be technically knowledgable to install or use special UEFI tools.
the entries will continue to accumulate again. Even after clearing the boot table 100%
this may lead to forgetting / and a NOT clearing of duplicate boot entries. Which is potentially dangerous (see below, next section)

-2) manually by:

Removing the "UEFI: " disk
then rebooting the BIOS. Will clear the boot entries.
then power-down, and re-connect the disk

Problem:

not all customers can easily access the disk drive inside the computer case.
the entries will continue to accumulate again. Even after clearing the boot table 100%
this may lead to forgetting / and a NOT clearing of duplicate boot entries. Which is potentially dangerous (see below, next section)

What is the concern? - Why problem? - Why fix it?

This bug looks harmless or just inconvenient at first. However there is a potential danger because of possible brick.

NEW QUESTION: How brick is possible?

If the boot entries accumulate too much:
It can have potential to overflow the table of boot entries on to next addresses. IF:
- Boot entries table is of a pre-allocated table size
- And there is other essential data written in an areas coming after the boot entries table
Then too many boot entries may result in a BIOS brick, if not properly safeguarded.
- And perhaps not even recoverable (flashable) afterwards.

Question: OK so why is potential brick a suspect in relation to this issue?

Because at least 1 customer (this time ASUS) has already reported an unstable behaviour (BIOS crashes). And they believe it was due to this 'too many boot entries' issue. Here: http://www.tonymacx86.com/general-help/175274-guide-remove-extra-clover-bios-boot-entries-prevent-further-problems-2.html#post1155192
From memory, also remember reading about a bricked (unrecoverable) BIOS too. From other user. Sorry, just cant find link again / remember where it was.
Because since some of the BIOS code is coming from external AMI BIOS supplier. It is not certain to be fully aware of how the AMI BIOS is arranged / configured in respect to boot entries table (- if Gigabyte dont know all intricacies of AMI code).
As not-yet-bricked Gigabyte customers, we are not going to push our luck to try and disprove it cannot brick. Since testing so far to the failure point might not be recoverable, and lead to RMA of boards. (which would be reckless and very inconvenient for customer).

Therefore it would be more useful to all, if not customer, but Gigabyte BIOS team can take it further now, and be the ones to test or reproduce the issue in-house. At minimum to check if:

If it is indeed a potential brick possible.
To re-assure customer if no brick
To raise awareness to AMI BIOS supplier (if it is not a Gigabyte issue).

But also in the hopes, maybe to fix this longtimes issue, once and for all -/W-

Why is AMI code / involvement also suspect?

Because some much fewer ASUS customer also reported seeing a similar bug. For exmaple here:

http://www.tonymacx86.com/general-help/181720-boot-entries-duplicate-clover-boot.html#post1171717

There is also some few others scattered historical report in forums / elsewhere.

However:

ASUS customer can stop the boot entries from coming back again. This is not true for Gigabyte. That 'work-around' does not work except for ASUS boards.
Other NEWER ASUS customer (for z170 / syklake), now also report the opposite.
That ASUS has no such bug or problem anymore.
Which suggest that perhaps ASUS have already fixed their issue.

By contrast:

Many Gigbyte Z170X Gaming customer do see duplicate entries BIOS bug.
Very few Gigabyte customer don't report issue (who have these non-standard bootloader).
Trying the 'ASUS workaround' on Gigabyte boards is not effective / does not work.
Which suggest that Gigabyte problem is not solved.

It also suggests that:

The 2 may be related together, or just a co-incidence - We do not know 100% sure if there is a link.
The AMI involvement is not-conclusive (from customer perspective). Just that is is SUSPECT.
Because UEFI is a very complex area (many components / pieces, some pieces shared, some not).
It could instead be a similar BIOS configuration... because why no ASROCK, or MSI customer report this issue too?
So we really need Gigabyte BIOS team and experts to do for us, make a more technical investigation.

Pre-condition

The bug seems to occur when:

A not-recognized UEFI bootloader is attached to the system

'not-recognized', meaning: less common ones, which is not

GRUB UEFI bootloader (for ubuntu)
Windows UEFI bootloader ('BCD')

Example:

Clover UEFI bootloader
PassMark Memtest x86 UEFI bootloader
Other minority bootloader (e.g. what about FreeBS DUEFI boot?)

Found with

Gigabyte GA-Z170X Gaming 3 Rev 1.0 EU
Also seen by many other Gigabyte customers

How reproduce bug?

This is new recommended, simplified proceedure. Using PASSMARK Memtest x86 boot disk.

Test steps:

Download PASSMARK Memtest x86 image. For UEFI disk.

http://www.memtest86.com/download.htm

Select "Image for creating bootable USB Drive". For example this one:

http://www.memtest86.com/downloads/memtest86-usb.tar.gz

Burn USB disk image --> a USB disk. Or to a SATA hard drive.

On windows: Use 'win32diskimager' program

On linux: Use 'dd' program

Attach disk to GIGABYTE motherboard Z170X Gaming 3 (or a similar product)

Boot into "UEFI: ..." entry --> select "Exit" reboot,

Repeat this for at least 3x times, or more.

Test result

Duplicated boot entries are seen.
For Memtest disk, +2 new 'UEFI: ' entries are added per boot

No bug?

Here is fool-proof disk image, manually uploaded:

PASSMARK Free edition - re-uploaded memtest dd image:

Still no luck? Try burning to SATA hard disk instead of USB flash key (requires linux). Although it should not be necessary. Those were the exact test conditions.

FAQ

But did you re-test with newest Gigabyte BIOS?

Yes. The newest BIOS this bug is already reproduced on, is version 'F6' (Gigabyte GA-Z170X Gaming 3). Which is the latest version currently (at time of writing).

But this same problem also does affect ASUS customer too, yes?

Well 'yes' and 'no'. While it is indeed true that ASUS customer may see this same symptom / problem. The difference is that ASUS customer can successfully modify the boot menu list in a particular way. Such that the boot entries will stop accumulating. However this same 'workaround fix' is not possible with the Gigabyte product. The boot entries always still accumulating, its no solution. That may be because the kinds of entries added are different on the affected ASUS system. We do not see similar type of boot entries under closer scrutiny with UEFI tools.

Is this bug because of dual-boot. Is that true?

No. A dual boot system is not needed in order to see this but. It can happen with just only 1 disk attached to the system. However it is true that many customers are confused and / or report this. Or it may be reported for non-gigabyte customer (like ASUS customer etc).

Is this bug only affect SATA disk?

Actually it depends the circumstance. But strictly 'no' is the answer.

This duplicate entries bug has been seen with USB 2.0 HDD caddy device --> USB boot of Passmark Memtest x86 UEFI image. And nothing else. No SATA HDDs attached.

However if you are having difficulty reproducing the bug, then by all means please try SATA HDD instead of USB flash key. In testing USB Key was not reproducible for clover bootloader (on USB). Clover had to be coped to SATA hdd to see the bug. This is another reason why not use clover for testing purposes, and instead test with the MEMTEST x86 UEFI image, as recommended.

Why is it taking so long to fix?

We do not know. But there are several reason:

It is easily to think or assume this bug is not serious. However we should not assume that without a further testing.
UEFI systems are comlicated. With many different parts working together. That can mean it is difficult to know which one(s) is responsiable / difficult debug.
These new reproduce instruction for PASSMARK MEMTEST was not known until very recently. Wheras clover bootloader is very difficult to install without also an extra Macintosh computer.
The important thing is to test, and re-test thoroughly, and try to avoid or limit the potential for BRICK situation.
Is AMI supplier informed / aware yet? Can AMI espertise help to narrow / solve the problem?

Where is the previous instruction to reproduce this issue?

https://gist.github.com/anonymous/f5b09529f279348ac6e6

Other observations

The 1st 2 boots, there is some mixing up (confusion) of the boot table. Like the BIOS has not settled yet fully. So different kinds of boot entries briefly appear, then are replaced by other ones. Until eventually a new rhythm is settled, to making the excess entries.
It is suspect that Windows and Linux bootloader are specially recognized by the BIOS.
Since for those popular OS, there is no 'UEFI: ...' boot entry. Instead it says 'windows boot', or 'ubuntu' / name of the operating system.
Whenever 'UEFI: ' entries appear, then perhaps the BIOS does not explicitly recognize it as a 'known' OS like windows or linux.
This is when the duplicate boot entry bug can happen
Sometimes the 'UEFI: ' entry is also seen on a USB installer disk, for example Ubuntu USB installer. Then there is no bug. Rather the bug CAN happen under the 'UEFI: ' presense, for example with PASSMARK Memtest x86.
It is presumed the contents of the ESP '/EFI/' boot folder is different for all these different disks. Which may affect success / fail. But the customer is not technical expert on UEFI. And does not know how the BIOS is scanning those EFI/ folder, and constructing the boot menu.
In any case its presumed to be a bug in the BIOS software. Since there should be some fail-safe check against accumulated boot entries. Since they are exact same thing, just duplicate of some previous ones.

/gb-mbe-bug-TAKE2.md