Add an option to truncate the ME file about me_cleaner HOT 59 CLOSED

I've pushed everything to dev: now me_cleaner tries to relocate the data as close as possible to FPT. I've also added a -O option to specify an output file.

Untested

from me_cleaner.

Works! Good job @corna

from me_cleaner.

Great! We tried to reduce the ME image in a very early version of me_cleaner but it didn't work:

• Shrinking the "reduced" ME partition in IFD to the minimum size allowed (FTPR
  offset + FTPR size + 4 kB alignment) bricks the device too (but we're not
  sure about this, maybe we made a mistake during the image creation).

I was able to reduce it to cover 0x3000 to 1 MB. There appears to be some
data being written into part of the SPI flash by the ME on each boot,
logging someting about the startup?, so the IFD must include that portion.
I did change the IFD to allow the CPU to read the ME region so that I can
include it in my TPM measurements.

This discussion was on a ME image with the FTPR partition intact, I suppose the code that writes stuff in the ROM is in one of the removed Huffman modules (therefore I suppose this won't work on Skylake images, where the modules are intact).
I've tested it on my X220 and everything works perfectly (668 kB).

Thanks for sharing your discoveries!
Just a couple of notes:

1. You put the same layout.txt both before and after the modification
2. The Intel Flash Descriptor minimum block size is 4096 bytes (0x1000), all the starting addresses must end with 000 and the ending must end with fff, therefore the layout.txt should be corrected to
   00000000:00000fff fd
   000DF000:007fffff bios
   00003000:000DEFFF me
   00001000:00002fff gbe

from me_cleaner.

Cheers for the catch.
The layout is verbatim pulled from my working ROM, so I'll keep it as is.

I would have written a patch by now but I've got no clue about python. I might work on it tomorrow as an exercise to learn it.

from me_cleaner.

Btw, I'd like not to truncate the image: if the user provides the extracted ME image it can also truncate it with an external tool (like dd), while if it provides a full image it can modify the layout with ifdtool. Either way, there are external tools that already do the job, and I think me_cleaner should just print the minimum allowed size, without modifying the size of the image.

from me_cleaner.

Oh yeah, at no point was I asking for it to be a default, merely asking that it provides a nice way to see the minimum length.

from me_cleaner.

Here it is
https://github.com/corna/me_cleaner/tree/experiments_truncate_image

Can you check that the result of your manual inspection of the binary and the output of me_cleaner match?

from me_cleaner.

nroach44@normandy:~/media/code/coreboot/me_cleaner-git$ python me_cleaner.py ~/local/downloads/coreboot_x220_seabios_20170124.rom 
Full image detected
The ME/TXE region goes from 0x3000 to 0x4fffff
Found FPT header at 0x3010
Found 1 partition(s)
Found FTPR header: FTPR partition spans from 0xcf000 to 0x145000
Removing extra partitions...
Removing extra partition entries in FPT...
Removing EFFS presence flag...
Correcting checksum (0xed)...
ME/TXE firmware version 7.1.20.1119
Reading FTPR modules list...
 UPDATE           (LZMA   , 0x1131c5 - 0x113257): removed
 BUP              (Huffman, fragmented data    ): NOT removed, essential
 KERNEL           (Huffman, fragmented data    ): removed
 POLICY           (Huffman, fragmented data    ): removed
 HOSTCOMM         (LZMA   , 0x113257 - 0x11880c): removed
 RSA              (LZMA   , 0x11880c - 0x11d2bd): removed
 CLS              (LZMA   , 0x11d2bd - 0x121ccf): removed
 TDT              (LZMA   , 0x121ccf - 0x127e7a): removed
 FTCS             (Huffman, fragmented data    ): removed
The ME minimum size is 901120 bytes (0xdc000 bytes)
The ME region can be reduced up to:
 00003000:000defff me
Checking FTPR RSA signature... VALID
Done! Good luck!

Aside from the fact it doesn't match mine (because I'm not doing it properly) it all looks good - I'll try it on my test mule tomorrow (GMT+8)

from me_cleaner.

nroach44@normandy:~/media/code/coreboot/me_cleaner-git$ ifdtool -f layout ~/local/tmp/meshrink/coreboot_x220_seabios_20170124.rom.new
File /home/nroach44/local/tmp/meshrink/coreboot_x220_seabios_20170124.rom.new is 8388608 bytes
Wrote layout to layout
nroach44@normandy:~/media/code/coreboot/me_cleaner-git$ cat layout
00000000:00000fff fd
000de000:007fffff bios
00003000:000defff me
00001000:00002fff gbe


nroach44@normandy:~/media/code/coreboot/me_cleaner-git$ python me_cleaner.py ~/local/tmp/meshrink/coreboot_x220_seabios_20170124.rom.new 
Full image detected
The ME/TXE region is corrupted or missing

I assume this is because it's not aligned properly? (This is straight from my working directory from manually doing it, I've not tried it in a clean folder for this)

Edit: No, It looks the same...

Edit2: I'm a twit - It looks like at some point I either did the math differently or something silently fixed it - the IFD on the test mule was indeed 0x1000 aligned.

Edit3: I'm going to bed - the BIOS is overlapping the ME region, and I assume that is why your tool is baulking on the new rom. I will do this all again tomorrow to double check things.

from me_cleaner.

OK, I did it completely from scratch, and from what I can tell (I haven't given it 30 minutes yet) everything seems fine (and yes, my IFD regions are fixed now).

My working ROM is here: https://nroach44.id.au/upl/coreboot_x220_seabios_20170125.rom

Interestingly, if I extract the truncated ME region, me_cleaner will end up making the file bigger. Presumably if I run it on the whole BIOS image it will end up writing into the BIOS region?

from me_cleaner.

Thanks for your findings!

On my T420, I used the 1.5MB ME image found on the coreboot wiki:
https://www.dropbox.com/s/gesgk9ot2ho24zm/me.bin

me_cleaner reported a minimum size of 294912 bytes (0x48000 bytes).

So I was able to reduce the image to 294kB. Here is my layout:

00000000:00000fff fd
0004b000:007fffff bios
00003000:0004afff me
00001000:00002fff gbe

Before flashing, unhuffme gave me a [MATCH] for the BUP region. Just passed the 30 minute mark.

Now I have a whopping CONFIG_CBFS_SIZE=0x7b5000 for payloads. Might be interesting for squeezing a linux kernel in there.

On a side note, I noticed that me_cleaner left over some bytes in the middle. I shaved them off without danger.

$ hexdump 1.5MB_me_cleaned.bin
[...]
0047de0 d0f8 f80c 0cd0 fff8 ffff ffff ffff ffff
0047df0 ffff ffff ffff ffff ffff ffff ffff ffff
*
0077bf0 ffff ffff ffff ffff ffff ffff df36 7f7f
0077c00 7f7f 7f7f 7f7f 7f7f 7f7f 7f7f 7f7f 7f7f
*
0077c40 607f ffff ffff ffff ffff ffff ffff ffff
0077c50 ffff ffff ffff ffff ffff ffff ffff ffff
*
017d000

from me_cleaner.

Awesome! I've added that as an addendum, and when this functionality gets mainstreamed I'll rewrite the blog post. I've just started running my own version of this and it looks like it's all good.

from me_cleaner.

So far, everything works just fine. You might want to post your blog on the coreboot mailing list - coreboot was originally all about executing linux from flash, people might be happy to hear they can reclaim large portions from the ME for this.

from me_cleaner.

@alegru me_cleaner has just been included in coreboot, but the integration on the menuconfig has received a negative review. Once we understand if the relocation of the code is possible (see the end) I'll push the updates to coreboot.

from me_cleaner.

I've finally understood how to relocate the partitions, now the FTPR can be pushed to the beginning of the ME region
I've merged everything in the dev branch, can you guys test it? The syntax is me_cleaner.py -r <image>
On my X220 the ME region is now ~80 kB and the CBFS size is 0x7e0000

from me_cleaner.

Very nice @corna. What was the problem at the previous ftpr move script? I'll test it at a me8 system later. The parameters are also a great addition. Something that should be there, in my opinion, is a file output. Meaning, not write over the original but expect an input for reading and an output for writing (example: me_cleaner me.bin me_cut.bin)

from me_cleaner.

@corna Great work! No problems on my T420, just finished waiting 30 minutes. I could reduce the 1.5 ME file to ~74 kB:

00000000:00000fff fd
00015000:007fffff bios
00003000:00014fff me
00001000:00002fff gbe

This makes my CBFS size 0x7eb000 large. Impressive. I don't think there is much more to gain.
I agree on @platomav that it would be helpful to generate a truncated file as output, instead of using dd and converting manually.

from me_cleaner.

What was the problem at the previous ftpr move script?

The previous script didn't correct the bytes 0x0c:0x10 of the LLUT. These bytes aren't documented in me.bios.info, but they are referred in me_unpack.py as "SpiBase: offset of the LUT in the ME region". This field has an absolute offset close to the beginning of the LLUT, but not exactly the same. Anyways, summing it the difference between the new and the old FTPR offset fixed everything.

Meaning, not write over the original but expect an input for reading and an output for writing

it would be helpful to generate a truncated file as output, instead of using dd and converting manually

Doable

I've just noted that the minimum size returned by me_cleaner is not reliable, as it returned 4 kB less than the "real" minimum and bricked my X220. Check manually that you're not removing valid data.

OT: @platomav do you have write access on me.bios.io? I'd like to fix a couple of things there

from me_cleaner.

Tested the 5MB image that I extracted from my factory bios. Interestingly, the truncated image (~70 kB) is (marginally) smaller compared to the 1.5MB one after relocation.

00000000:00000fff fd
00014000:007fffff bios
00003000:00013fff me
00001000:00002fff gbe

CBFS size is 0x7ec000

I've just noted that the minimum size returned by me_cleaner is not reliable

In both cases I had no problems with this; for testing I deliberately shaved off too much.
Tried the -c flag but I guess me_cleaner just checks the signature. unhuffme gave me errors, at least.

from me_cleaner.

@corna

Valid data are cut because the FTPR is set to move at 0x1000 (4KB) and the size reported by me_cleaner at the end relates to FTPR and not the entire ME binary (from $FPT). Apart from that issue, here are some notes:

1. We can save an extra 3.92KB by moving FTPR exactly after the last $FPT entry. Take the $FPT header length at 0x1A and add 0x20 for the solo FTPR entry. For ME6-10 & TXE1-2 that means 0x50 starting offset and for ME11 & TXE3 that would be 0x40 since $FPT is 0x20 sized at new firmware. By the way, that's the proper way to know how many bytes to check for the checksum.

2. Intel always 1K aligns Engine partitions at $FPT entries. The actual partition size is found at $MCP and its alignment goes into $FPT. I suggest you first align FTPR by itself and then align the entire Engine region ($FPT + FTPR). That way no data will be lost, everything will be aligned and the other SPI regions (BIOS) won't start at weird non-1K offsets.

3. After the two alignments, check that the rest of the cut file is filled with padding. A simple "remaining_bytes" == "remaining_bytes" * b'0xFF' check will do just fine.

Note: I'm not mentioning SPS stuff yet. However, in case you missed it, this person tested the script at an SPS 4 (x86 Engine) system.

The previous script didn't correct the bytes 0x0c:0x10 of the LLUT.

Igor to the rescue! We would be nowhere without that guy! :)

do you have write access on me.bios.io? I'd like to fix a couple of things there

No, I'm not related to that. I have noticed as well that it has various mistakes or omissions here and there. I've seen some of them corrected at me_unpack and I've added some more at MEA structs.

from me_cleaner.

Valid data are cut because the FTPR is set to move at 0x1000 (4KB) and the size reported by me_cleaner at the end relates to FTPR and not the entire ME binary (from $FPT)

Both the Huffman and the LZMA/uncompressed end addresses should be relative to $FPT.
I've noticed that the data cut away is not mapped by any module, the hashes returned by unhuffme are correct. They are just a bunch of "F8 D0 0C", I don't know why they are still there

We can save an extra 3.92KB by moving FTPR exactly after the last $FPT entry.

I didn't want to move the FTPR too high because the mysterious value "spibase" (bytes 0x0c:0x10 of the LLUT) points to an address before the start of the FTPR in my image (the relocated FTPR starts at 0x1000 while this address is 0xB40). For now I'm keeping the new FTPR address a bit higher than the real minimum.

I suggest you first align FTPR by itself and then align the entire Engine region ($FPT + FTPR). That way no data will be lost, everything will be aligned and the other SPI regions (BIOS) won't start at weird non-1K offsets.

The IFD addresses store only the bits 24:12, so they are intrinsically 4 kB aligned.

After the two alignments, check that the rest of the cut file is filled with padding. A simple "remaining_bytes" == "remaining_bytes" * b'0xFF' check will do just fine.

I'm not sure about this. Probably the extra "unmapped" bytes can be truncated, I'll investigate

I'm not mentioning SPS stuff yet. However, in case you missed it, this person tested the script at an SPS 4 (x86 Engine) system.

Yep, I've missed it! ;)
Thank you for pointing it out.

Igor to the rescue! We would be nowhere without that guy! :)

Indeed :)

from me_cleaner.

Tried the -c flag but I guess me_cleaner just checks the signature.

Yes, the -c flag just checks the structure and the signature
I've added it to integrate in coreboot, when you just want to check if the supplied image is valid or just garbage and you don't want to modify it

from me_cleaner.

I've just rebuilt and flashed to my test machine - I shrunk the image down to 12000B (me_cleaner advised 11000B, but there was some non-0xFF data at the end there) and the machine doesn't get to the stage where it turns on the back light. I'm not sure if that's a coreboot/SeaBIOS issue but I suspect it may be the ME.

from me_cleaner.

@nroach44 have you also relocated the FTPR partition?

from me_cleaner.

I didn't want to move the FTPR too high because the mysterious value "spibase"

Here is what I noticed. That "spibase" always starts before LUT, in weird places. Sometimes at the previous partition and some others within FTPR itself (for example right in between an $MME module). What is always common (GLUT, LLUT, LUT) is that the size between that offset and the start of LUT data (after Chipset/Revision, LUT + 0x40) is always shown inside bytes 2-3 of AddrBase (LUT + 0x9 : LUT + 0x9A). For example, if I take a GLUT partition:

Partition = 00 F0 0D 00 (0xDF000)
AddrBase = 00 C0 0D 10
SpiBase = 80 E2 0D 00 (0xDE280)
LUT_Data = LUT + 0x40 (0xDF040)

If you subtract SpiBase from LUT_Data, you get 0xDF040 - 0xDE280 = 0xDC0. Check 2nd and 3rd bytes of AddrBase which are C0 0D and thus 0xDC0.

from me_cleaner.

Edit: @corna I went and re-read the arguments, yes, I just used me_cleaner -r.

I can upload the resulting image somewhere if you want.

from me_cleaner.

@platomav
You're right, good job! So AddrBase[1:3] + SpiBase = Start of the chunks (LLUT + 0x40). I don't see the reason of all of this though

@nroach44
Can you try with me_cleaner -r without truncating it (i.e. leaving the partition layout untouched)?

from me_cleaner.

@corna Just noting it while I come across it again - it seems that when I run me_cleaner over my me region it makes the file larger:

-rw-r--r-- 1 nroach44 nroach44  901120 Jan 30 21:51 flashregion_2_intel_me.bin

nroach44@normandy:~/media/code/coreboot/me_cleaner-git$ ./me_cleaner.py -r flashregion_2_intel_me.bin

-rw-r--r-- 1 nroach44 nroach44 1318912 Jan 30 21:51 flashregion_2_intel_me.bin

This had me stumped for a while when trying to re-insert the image.

from me_cleaner.

Can you run a binary diff between the two?

Are you applying me_cleaner.py -r on an already truncated image?

from me_cleaner.

I don't see the reason of all of this though

Neither do I and it especially does not make any sense since that SpiBase offset can sometimes fall into random FTPR data such as an $MME module as I noticed at ME7. Maybe that field is useful for an extra LUT offset verification and nothing else. They could have added the actual LUT offset there instead of a previous offset + a size to calculate. Yes, I cannot understand the reasoning no matter how I look at it. Regardless, based on that finding, there are some things that should be tested:

1. Does it work to move FTPR to $FPT + AddrBase[1:3] (it should)
2. Does it work to move FTPR to $FPT and ignore the out-of-bounds SpiBase offset, meaning: does the ME even care if that offset is valid (probably does, how will it calculate it with AddrBase[1:3] then?)
3. Does it work to move FTPR to $FPT and also adjust AddrBase and SpiBase to the correct size/offset (it should - best case scenario to save even more space)

from me_cleaner.

1. Does it work to move FTPR to $FPT + AddrBase[1:3]

Probably, I'll test it

2. Does it work to move FTPR to $FPT and ignore the out-of-bounds SpiBase offset, meaning: does the ME even care if that offset is valid

Probably not, as the relocation didn't work before the adjustment of this field

3. Does it work to move FTPR to $FPT and also adjust AddrBase and SpiBase to the correct offset

Probably not, as AddrBase is used during the Huffman decompression and the loading addresses of the modules are signed

from me_cleaner.

@corna ran me_cleaner -r over a dumped working image, did not adjust layout, same result as before.

I will investigate the enlarging issue a bit more and report on that separately.

from me_cleaner.

Ah, alright. So, since AddrBase is both checked and used for something else non-adjustable, the only test is moving FTPR to $FPT + AddrBase[1:3]. There are two cases here:

a) If AddrBase[1:3] > $FPT (example: 0x50 for ME6-10), move FTPR to 0x0 + AddrBase[1:3]
b) Else, move FTPR to $FPT (0x50) + AddrBase[1:3]

I think "a" should work since that SpiBase offset can already fall into irrelevant (to LUT) data such as the $MME module previously mentioned, so it won't make any difference if that irrelevant data is $FPT. That way we can cut an extra $FPT_size (0x50) data from the ME region. I know it is kind of ridiculous to try and squize those extra 0x50 bytes from the ME region but if we can do that, and since so much progress has been made already, why not? 😛

As for "b", a check is nice but there cannot be any firmware with such small AddrBase[1:3].

from me_cleaner.

I think "a" should work since that SpiBase offset can already fall into irrelevant (to LUT) data such at the $MME module previously mentioned, so it won't make any difference if that irrelevant data is $FPT.

I agree

I know it is kind of ridiculous to try and squize those extra 0x50 bytes from the ME region but if we can do that, and since so much progress has been made already, why not? 😛

You really want those bytes, don't you? ;)

from me_cleaner.

You really want those bytes, don't you? ;)

I cannot live without them gone... :)

Regarding these extra data you mention, maybe you are referring to the ME8+ "uncharted" (not at $FPT) WCOD/LOCL partitions found at 5MB/Corporate SKUs exactly after last $FPT partition. These should be removed as well from such images. Another possible issue is the fact that ME6 does not have a size at the last $FPT entry. But I suppose, in both cases, me_cleaner can just nuke everything after FTPR and call it a day.

from me_cleaner.

Nope, I'm referring at the extra data remaining in the FTPR Huffman stream
Currently me_cleaner removes each chunk of every removable module but there is some data remaining, not part of any module (3 or 4 blocks of ~200 bytes each, random data like F8 D0 0C F8 D0 0C...)

from me_cleaner.

Okay, finally got around to re-doing my testing (X220):
Normal cleaned ME image: All good
Relocated ME image: All good, some funkyness that required a battery pull but meh
Relocated and truncated ME image: no coreboot (using the suggested me_region layout)

The truncated image was done simply by moving the region, not actually regenerating the image - the coreboot region hadn't changed. Unless there's some requirement that all the regions be contiguous, the ME didn't like the suggested size. I'll reflash back to a working image and see how far I can cull it down.

from me_cleaner.

For reference, me_cleaner says "The ME minimum size is 65536 bytes (0x10000 bytes)" which seem roughly correct.

The only data cut off is:

0010680 ffff ffff ffff ffff 36ff 7fdf 7f7f 7f7f
0010690 7f7f 7f7f 7f7f 7f7f 7f7f 7f7f 7f7f 7f7f
*
00106c0 7f7f 7f7f 7f7f 7f7f 7f7f 7f7f 7f7f ff60
00106d0 ffff ffff ffff ffff ffff ffff ffff ffff
*
0010b70 ffff ffff ffff ffff 7fff fe8c 5e5e d058
0010b80 f80c 0cd0 d0f8 f80c 0cd0 d0f8 f80c 0cd0
0010b90 d0f8 f80c 0cd0 d0f8 f80c 0cd0 d0f8 f80c
0010ba0 0cd0 d0f8 f80c 0cd0 d0f8 f80c 0cd0 d0f8
0010bb0 f80c 0cd0 d0f8 f80c 0cd0 d0f8 f80c 0cd0
0010bc0 d0f8 f80c 0cd0 d0f8 f80c 0cd0 d0f8 f80c
0010bd0 0cd0 d0f8 f80c 0cd0 d0f8 f80c 0cd0 d0f8
0010be0 f80c 0cd0 d0f8 f80c 0cd0 d0f8 f80c 0cd0
0010bf0 d0f8 f80c 0cd0 d0f8 f80c 0cd0 d0f8 f80c
0010c00 0cd0 d0f8 f80c 0cd0 d0f8 f80c 0cd0 d0f8
0010c10 f80c 0cd0 d0f8 f80c 0cd0 d0f8 f80c 0cd0
0010c20 d0f8 f80c 0cd0 fff8 ffff ffff ffff ffff
0010c30 ffff ffff ffff ffff ffff ffff ffff ffff
*
04fd000

from me_cleaner.

All good, some funkyness that required a battery pull but meh

What do you mean?

I'm starting to think that the boot fails if ME can't compute the hash of the subsequent module, which might happen if the image is truncated "too close" to the ending address. Can you try again using the suggested size + 0x1000?

from me_cleaner.

I usually just shut the machine down and power it on again, but when I flashed the relocated image it wouldn't even finish the EC initialisation - no think light flash. I assume that's because the ME was half booted and didn't like the region being shifted out from underneath it.

Edit: that particular issue only applied to the relocation that did end up booting after I pulled the battery.

I had flashed an image that I thought would be 0x20000 (double the size), but I missed a zero and bricked it. I'm going to be back home soon and will try it properly.

from me_cleaner.

All good, some funkyness that required a battery pull but meh

ME with 5MB firmware is powered separately from the Host system (M3 power state supported). A global (ME + Host) reset is needed in such case. A simple shutdown only resets the Host, not ME. So, to do a manual global reset, power off the system (not standby), disconnect all power (AC, main battery - RTC battery doesn't matter) and wait for 30 seconds.

ME with 1.5MB firmware is not powered separately from the Host system (M3 power state not supported). A system power off is enough in such case to perform a global (ME + Host) reset.

from me_cleaner.

Right, that explains it.

Also this might explain why shrinking the ME region to 0x20000 still doesn't work:

nroach44@normandy:~/local/tmp/meshrink6/test$ ifdtool -x ../01_reloc.rom.new 
File ../01_reloc.rom.new is 8388608 bytes
  Flash Region 0 (Flash Descriptor): 00000000 - 00000fff 
  Flash Region 1 (BIOS): 00500000 - 007fffff 
  Flash Region 2 (Intel ME): 00003000 - 00022fff 
  Flash Region 3 (GbE): 00001000 - 00002fff 
  Flash Region 4 (Platform Data): 00fff000 - 00000fff (unused)
nroach44@normandy:~/local/tmp/meshrink6/test$ hexdump flashregion_2_intel_me.bin | tail
0000000 ffff ffff ffff ffff ffff ffff ffff ffff
*
0020000

So this whole time ifdtool seems to be wiping the region when I resize it...? When I finish unbricking my X220 I'll flash an image that doesn't have a zeroed ME region...

from me_cleaner.

Okay, I shrunk the region to

00000000:00000fff fd
00500000:007fffff bios
00003000:00022fff me
00001000:00002fff gbe

And then truncated the ME file, and then re-inserted it. This still results in a brick.

If you wouldn't mind having a squiz at the file here: https://nroach44.id.au/upl/coreboot_x220_seabios_20170204_broken.rom.new
And double checking that I'm not doing something stupid?

Also to double check, the regions can have gaps in them yeah? They don't need to be end-to-end?

from me_cleaner.

Ah, there is a bug. The SpiBase of LUT is calculated from FD instead of $FPT.

In my test, it worked (by accident) because I cleaned a bare Engine region and not a full SPI image, so 0x0 coincided with $FPT offset.

from me_cleaner.

Hmm, I'm not knowledgeable enough to know if that's behind my issues, but just for completeness' sake, I rebuilt coreboot around the "cleaned, relocated and truncated to 0x20000 length" image. This also meant that the BIOS region was resized to start where the ME region ended.

Still results in a brick. I'm going to bed and I'll have a squiz at possibly truncating it to the minimum size before relocation to see if that changes things.

from me_cleaner.

It is a bug in me_cleaner, nothing to do with your actions. To avoid it, use me_cleaner on your Engine region only (not the whole SPI) and then insert into the SPI to flash back. Or you can wait until it gets fixed. 👍

from me_cleaner.

No, I meant that because it works when I don't truncate I don't believe that this is causing my issue. Although I don't remember if the image that I'm working from was cleaned as a rom or just a region.

Cheers though!

Edit: That said, I'm quite tired (so I might be mis-reading your message) and will check in tomorrow.

from me_cleaner.

me_cleaner on your Engine region only

Tried this with latest commits, but still bricked my machine (0x10000 for me)

from me_cleaner.

It is a bug in me_cleaner, nothing to do with your actions. To avoid it, use me_cleaner on your Engine region only (not the whole SPI) and then insert into the SPI to flash back. Or you can wait until it gets fixed. 👍

Yes, thanks for pointing it out. Fixed in ab4e583

from me_cleaner.

Ok, so I've gone through and tested stuff again but with a battery pull after each flash.
On the current commit
Cleaning works fine.
Relocating allows the machine to boot but:

ME: FWS2: 0xa0000
ME:  Bist in progress: 0x0
ME:  ICC Status      : 0x0
ME:  Invoke MEBx     : 0x0
ME:  CPU replaced    : 0x0
ME:  MBP ready       : 0x0
ME:  MFS failure     : 0x0
ME:  Warm reset req  : 0x0
ME:  CPU repl valid  : 0x0
ME:  (Reserved)      : 0x0
ME:  FW update req   : 0x0
ME:  (Reserved)      : 0x0
ME:  Current state   : 0xa
ME:  Current PM event: 0x0
ME:  Progress code   : 0x0
Waited long enough, or CPU was not replaced, continue...
PASSED! Tell ME that DRAM is ready
ME: FWS2: 0xffffffff
ME:  Bist in progress: 0x1
ME:  ICC Status      : 0x3
ME:  Invoke MEBx     : 0x1
ME:  CPU replaced    : 0x1
ME:  MBP ready       : 0x1
ME:  MFS failure     : 0x1
ME:  Warm reset req  : 0x1
ME:  CPU repl valid  : 0x1
ME:  (Reserved)      : 0x3
ME:  FW update req   : 0x1
ME:  (Reserved)      : 0xf
ME:  Current state   : 0xff
ME:  Current PM event: 0xf
ME:  Progress code   : 0xf
ME: Requested BIOS Action: Continue to boot

It causes the machine to reboot after 30 minutes. I never noticed because I would unbrick the machine, boot it and then flash another image.

Running -r on the region as opposed to the whole image doesn't change anything.

from me_cleaner.

What is the output of unhuffme and intelmetool -s? Have you also truncated the image or just relocated?

from me_cleaner.

Just relocated, no truncation yet.

nroach44@normandy:~/local/tmp/meshrink6_5$ unhuffme 04_reloc.rom
number of flash regions: 4
	region: 0 in range: 00000fff-00000000
	region: 1 in range: 007fffff-00500000
	region: 2 in range: 004fffff-00003000
	region: 3 in range: 00002fff-00001000

Flash partition table (1 entries):
	partition:  FTPR(type:0) at 00000060, size:76000

Code partition: FTPR(9 modules, v7.1.20.1119)
		 0           UPDATE     lzma at:   441c5 va:2003d000+2000
		 1              BUP  huffman at:     780 va:20040000+11000
		 2           KERNEL  huffman at:     780 va:20055000+29000
		 3           POLICY  huffman at:     780 va:20089000+1b000
		 4         HOSTCOMM     lzma at:   44257 va:20506000+b000
		 5              RSA     lzma at:   4980c va:2051b000+c000
		 6              CLS     lzma at:   4e2bd va:2052a000+a000
		 7              TDT     lzma at:   52ccf va:2053b000+e000
		 8             FTCS  huffman at:     780 va:200a6000+7000

        UPDATE	13f1e1e6479e383099dcf7bb2db126b55d1d64c2daa082cbe138d433940ec97b lzma
           BUP	ad58e144164a9112c97f195f5423139ff56f6a241436207d16f6cde597101bbe [MATCH]
        KERNEL	d79e66a78379a17d0dfd650e754a3a904cd057cee06700fc1d14a84037992925 incomplete
        POLICY	961d8327469459bf8bb40e683bc5089b337bd7931c79fad174e936b4547cf3db incomplete
      HOSTCOMM	53a003e4451b2ead67236bab45bdc1f8279f418be873df33969c62d05eff8ac0 lzma
           RSA	15f4be4ed1d95d3b148d7907e4a85c4d3d5e76ffbbcf9c6eae09dae17a5eea6b lzma
           CLS	0c0970d720cb109046e68c7c048d5d1cc5e0cb4011acb4deebb7e081d6b524df lzma
           TDT	cbb0a4676d192cc8da7352562c2b9414fecab0d88ce67e3292566a74d12fde94 lzma
          FTCS	2a286569910e5d375f1de0ad628fe5ac87723d0c984c0d87fe00328d49e50f32 incomplete

intelmetool -s output is simply:

Bad news, you have a `QM67 Express Chipset Family LPC Controller` so you have ME hardware on board and you can't control or disable it, continuing...

MEI device not found

from me_cleaner.

Are you using the very latest version of intelmetool from coreboot? It was recently updated

from me_cleaner.

Ah, no.
Cleaned only ROM:

Bad news, you have a `QM67 Express Chipset Family LPC Controller` so you have ME hardware on board and you can't control or disable it, continuing...

MEI was hidden on PCI, now unlocked
MEI found: [8086:1c3a] 6 Series/C200 Series Chipset Family MEI Controller #1

ME Status   : 0x1e003052
ME Status 2 : 0x10320000

ME: FW Partition Table      : OK
ME: Bringup Loader Failure  : NO
ME: Firmware Init Complete  : NO
ME: Manufacturing Mode      : YES
ME: Boot Options Present    : NO
ME: Update In Progress      : NO
ME: Current Working State   : Recovery
ME: Current Operation State : M0 with UMA
ME: Current Operation Mode  : Normal
ME: Error Code              : Image Failure
ME: Progress Phase          : BUP Phase
ME: Power Management Event  : Clean Moff->Mx wake
ME: Progress Phase State    : M0 kernel load

ME: Extend SHA-256: 0311dee9ea06e2ed7a972729ddbefd07e4406ed70f136e3bcca940fa0b705e13

ME: has a broken implementation on your board with this BIOS
ME: failed to become ready
ME: failed to become ready
ME: GET FW VERSION message failed
ME: failed to become ready
ME: failed to become ready
ME: GET FWCAPS message failed
Re-hiding MEI device...done

Cleaned and relocated, no truncation:

Bad news, you have a `QM67 Express Chipset Family LPC Controller` so you have ME hardware on board and you can't control or disable it, continuing...

MEI was hidden on PCI, now unlocked
Re-hiding MEI device...done
MEI device not found

from me_cleaner.

I've just tried on my X220 with a ME7 5MB image (I usually use a ME8 1.5MB dumped from a Chromebook) and it bricked my laptop, I'll work on it

from me_cleaner.

Yeah, when I have some more time I'll sit down and try and figure out some of the inconsistencies - I'm 99% sure the new "don't work"s from me are with the factory 5MB, but the "works fine" were with the 1.5 from the coreboot wiki.

Sorry for the time wasting!

from me_cleaner.

I've noticed that with my 5 MB image the FTPR is moved up to the minimum possible (0x60) and after changing it to 0x400 (1 kB) it worked. Can you change line 26 to min_ftpr_offset = 0x400 and try again?

from me_cleaner.

Boots fine, unlike every other bad image I didn't need to pull the battery twice (bad images would mean the power light would flash the first time you tried powering it on)

ME: FWS2: 0x164e0002
ME:  Bist in progress: 0x0
ME:  ICC Status      : 0x1
ME:  Invoke MEBx     : 0x0
ME:  CPU replaced    : 0x0
ME:  MBP ready       : 0x0
ME:  MFS failure     : 0x0
ME:  Warm reset req  : 0x0
ME:  CPU repl valid  : 0x0
ME:  (Reserved)      : 0x0
ME:  FW update req   : 0x0
ME:  (Reserved)      : 0x0
ME:  Current state   : 0x4e
ME:  Current PM event: 0x6
ME:  Progress code   : 0x1
Waited long enough, or CPU was not replaced, continue...
PASSED! Tell ME that DRAM is ready
ME: FWS2: 0x162c0002
ME:  Bist in progress: 0x0
ME:  ICC Status      : 0x1
ME:  Invoke MEBx     : 0x0
ME:  CPU replaced    : 0x0
ME:  MBP ready       : 0x0
ME:  MFS failure     : 0x0
ME:  Warm reset req  : 0x0
ME:  CPU repl valid  : 0x0
ME:  (Reserved)      : 0x0
ME:  FW update req   : 0x0
ME:  (Reserved)      : 0x0
ME:  Current state   : 0x2c
ME:  Current PM event: 0x6
ME:  Progress code   : 0x1
ME: Requested BIOS Action: Continue to boot

Bad news, you have a `QM67 Express Chipset Family LPC Controller` so you have ME hardware on board and you can't control or disable it, continuing...

MEI was hidden on PCI, now unlocked
MEI found: [8086:1c3a] 6 Series/C200 Series Chipset Family MEI Controller #1

ME Status   : 0x1e003052
ME Status 2 : 0x16320002

ME: FW Partition Table      : OK
ME: Bringup Loader Failure  : NO
ME: Firmware Init Complete  : NO
ME: Manufacturing Mode      : YES
ME: Boot Options Present    : NO
ME: Update In Progress      : NO
ME: Current Working State   : Recovery
ME: Current Operation State : M0 with UMA
ME: Current Operation Mode  : Normal
ME: Error Code              : Image Failure
ME: Progress Phase          : BUP Phase
ME: Power Management Event  : Pseudo-global reset
ME: Progress Phase State    : M0 kernel load

ME: Extend SHA-256: 0311dee9ea06e2ed7a972729ddbefd07e4406ed70f136e3bcca940fa0b705e13

ME: has a broken implementation on your board with this BIOS
ME: failed to become ready
ME: failed to become ready
ME: GET FW VERSION message failed
ME: failed to become ready
ME: failed to become ready
ME: GET FWCAPS message failed
Re-hiding MEI device...done

Not been up 30 minutes yet but from the logs it looks like it should be fine.

Edit: Passed the 30 minutes mark fine.

from me_cleaner.

Original request done in 0ac4b4b.

from me_cleaner.