Bug 15075 - filemap.py does not handle 64KB block size
Summary: filemap.py does not handle 64KB block size
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: Scripts and Tools (show other bugs)
Version: 5.99
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 4.3 M1
Assignee: Andrew
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2023-03-21 01:04 UTC by Andrew
Modified: 2023-06-21 00:17 UTC (History)
2 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Andrew 2023-03-21 01:04:12 UTC
I'm attempting to get yocto building on a ppc64le POWER9 based server. I'm building master from the OpenBMC project (which builds find on x86 machines).

I hit the following errror:

```
| Traceback (most recent call last):
|   File "/home/fspcibld/284637575/openbmc/scripts/wic", line 547, in <module>
|     sys.exit(main(sys.argv[1:]))
|              ^^^^^^^^^^^^^^^^^^
|   File "/home/fspcibld/284637575/openbmc/scripts/wic", line 542, in main
|     return hlp.invoke_subcommand(args, parser, hlp.wic_help_usage, subcommands)
|            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|   File "/home/fspcibld/284637575/openbmc/poky/scripts/lib/wic/help.py", line 83, in invoke_subcommand
|     subcmd[0](args, usage)
|   File "/home/fspcibld/284637575/openbmc/scripts/wic", line 222, in wic_create_subcommand
|     engine.wic_create(wks_file, rootfs_dir, bootimg_dir, kernel_dir,
|   File "/home/fspcibld/284637575/openbmc/poky/scripts/lib/wic/engine.py", line 190, in wic_create
|     plugin.do_create()
|   File "/home/fspcibld/284637575/openbmc/poky/scripts/lib/wic/plugins/imager/direct.py", line 99, in do_create
|     self.assemble()
|   File "/home/fspcibld/284637575/openbmc/poky/scripts/lib/wic/plugins/imager/direct.py", line 194, in assemble
|     self._image.assemble()
|   File "/home/fspcibld/284637575/openbmc/poky/scripts/lib/wic/plugins/imager/direct.py", line 624, in assemble
|     sparse_copy(source, self.path, seek=part.start * self.sector_size)
|   File "/home/fspcibld/284637575/openbmc/poky/scripts/lib/wic/filemap.py", line 540, in sparse_copy
|     for first, last in fmap.get_mapped_ranges(0, fmap.blocks_cnt):
|   File "/home/fspcibld/284637575/openbmc/poky/scripts/lib/wic/filemap.py", line 458, in get_mapped_ranges
|     first_prev, last_prev = next(iterator)
|                             ^^^^^^^^^^^^^^
|   File "/home/fspcibld/284637575/openbmc/poky/scripts/lib/wic/filemap.py", line 440, in _do_get_mapped_ranges
|     assert extent_len % self.block_size == 0
|            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| AssertionError
| WARNING: exit code 1 from a shell command.
ERROR: Task (/home/fspcibld/284637575/openbmc/meta-phosphor/recipes-phosphor/images/obmc-phosphor-image.bb:do_image_wic) failed with exit code '1'
NOTE: Tasks Summary: Attempted 5214 tasks of which 5213 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
  /home/fspcibld/284637575/openbmc/meta-phosphor/recipes-phosphor/images/obmc-phosphor-image.bb:do_image_wic
```

I added some debug and found the following on my system:

```
| DEBUG: Python function extend_recipe_sysroot finished
| DEBUG: Executing python function set_image_size
| DEBUG: 197444.000000 = 151880 * 1.300000
| DEBUG: 197444.000000 = max(197444.000000, 65536)[197444.000000] + 0
| DEBUG: 197444.000000 = int(197444.000000)
| DEBUG: 197444 = aligned(197444)
| DEBUG: returning 197444
| DEBUG: Python function set_image_size finished
| Var <EXTRA_USERS_PARAMS:pn-obmc-phosphor-image>:1: DeprecationWarning: invalid escape sequence '\$'
| Var <EXTRA_USERS_PARAMS>:1: DeprecationWarning: invalid escape sequence '\$'
| DEBUG: Executing shell function do_image_wic
| INFO: Creating image(s)...
| 
| extend_start and block_size(0, 65536)
| extend_len and block_size(65536, 65536)

| extend_start and block_size(0, 65536)
| extend_len and block_size(65536, 65536)

| extend_start and block_size(0, 65536)
| extend_len and block_size(274432, 65536)
<ASSERT>
```

The issue appears to be that this is a 65536 block_size system and the script does not handle that well(?). The 64KB does not evenly divide into the 274432. A quick hack was to just have get_block_size() return 4KB and all worked fine. I could send a patch up that adds a little logic to return a max of 4KB?
Comment 1 Randy MacLeod 2023-03-23 14:38:45 UTC
Where is the 64 KB block size coming in?
Propsing a patch on the email list would be great and a good start to gettting to the root cause of the issue.
Comment 2 Andrew 2023-03-24 20:16:52 UTC
> Proposing a patch on the email list would be great and a good start to getting to the root cause of the issue.

ok, I sent https://lists.openembedded.org/g/openembedded-core/message/179079 upstream.

> Where is the 64 KB block size coming in?

The 64KB is coming from the logic in get_block_size() from within the script. It appears to do an ioctl into the kernel.
Comment 3 Andrew 2023-06-20 12:26:35 UTC
Patch to mailing list was accepted and merged.

https://lists.openembedded.org/g/openembedded-core/message/179079
Comment 4 Randy MacLeod 2023-06-21 00:17:40 UTC
Thanks Andrew.
If you could it's appreciated if you can include a link to the commit.
Here, that's: https://git.openembedded.org/openembedded-core/commit/?id=1e23b803af6991fc20e4a4e88a0ef0541399e722