Bug 6957 - [toolchain/uclibc]very long paths
Summary: [toolchain/uclibc]very long paths
Status: RESOLVED INVALID
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: devtools / tool chain (show other bugs)
Version: unspecified
Hardware: Galileo Multiple
: Medium normal
Target Milestone: 2.2 M4
Assignee: Robert Yang
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2014-11-12 16:10 UTC by Krzysztof Sywula
Modified: 2016-08-01 08:13 UTC (History)
9 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Krzysztof Sywula 2014-11-12 16:10:19 UTC
Yocto becomes used more widely outside of Linux and ext3/4 filesystem, especially in learning and hacker space (e.g. Arudino). Very long paths in toolchain archive cause troubles so people try to hack/work around it with mixed results.
Windows limits full path length, meaning path + file name, to something below 260 characters, OSX (HFS+) limits to 255 characters. 

relative path to one of a toolchain files:
./sysroots/i586-poky-linux-uclibc/usr/src/debug/uclibc/0.9.33+gitAUTOINC+946799cd0ce0c6c803c9cb173a84f4d607bde350-r8.4/git/libpthread/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_timedwrlock.S
takes 201 characters already

Something like
+gitAUTOINC+946799cd0ce0c6c803c9cb173a84f4d607bde350
takes 52 characters

I would consider shortening uclibc path by changing PV to something like "0.9.33-r8.4".
Finally, why does uclibc in Yocto do 'fetch git latest & greatest'? That's not reliable! What about sticking to the tag like Yocto does for many other packages taken from git? We already know uclibc is not well supported, however, due to growing IoT market it is being used more often.

PS. I based my examples on toolchain from 1.4 Yocto, but the issue still exists on master branch.
Comment 1 Saul Wold 2014-11-13 01:15:36 UTC
First to address your concern about fetching glibc latest and greatest, it should not be doing that, you might notice that there is a SRCREV set in the uclibc recipe, we fix the revision on a per-release based on a stable point of uclibc.

Regarding the first issue, we have reduced the git has from the 52 charaters to around 30, depending on the Version and Revision by reducing the hash to 10 characters, I am not sure if that is enough of a reduction for your purposes or not.
Comment 2 Brendan Le Foll 2014-11-13 10:09:01 UTC
(In reply to comment #0)
> Something like
> +gitAUTOINC+946799cd0ce0c6c803c9cb173a84f4d607bde350
> takes 52 characters

As Saul mentioned, at least on 1.6 they seem to be 32chars. Basing it on the tag is not a good idea since tags can be easily moved but hashes are much more difficult to 'fake'. I guess it might be possible to shorten it a little more. Is the issue you're seeing mostly on uclibc?

> I would consider shortening uclibc path by changing PV to something like
> "0.9.33-r8.4".
> Finally, why does uclibc in Yocto do 'fetch git latest & greatest'? That's
> not reliable! What about sticking to the tag like Yocto does for many other
> packages taken from git? We already know uclibc is not well supported,
> however, due to growing IoT market it is being used more often.

I'm a little confused, at least the package form yocto definately has a SRCREV set and from what I can tell this is definatly respected. See meta/recipes-core/uclibc/uclibc-git.inc:1
Comment 3 Krzysztof Sywula 2014-11-13 11:56:53 UTC
Thanks for explaining this
"we fix the revision on a per-release based on a stable point of uclibc"
since from a distance it just looks like SRCREV is being updated every some time to 'latest & greatest' from uclibc.

Every (or nearly every) other component has a proper versioning schema, sometimes based on git-tags. Even Yocto uses git-tags for versioning, but let's not argue about them, that's just an implementation detail.
On the other hand uclibc is still 0.9.33. That's werid, isn't it? If it changes, why isn't the version bumped?
On top of that I don't really see the point of including git SHA to package name. I spotted it's lenght has been reduced on master, but it's still taking some space that could be saved - especially knowing it becomes a problem for others.

What about using only PR, as it increments for new recipe revisions and is present always anyway?

'opkg info <pkg-name>' shows description, maybe we can preserve full git SHA in there? I'm pretty sure yum/apt-get will easily show git SHA in some description field as well. We can ship additional file to the target/toolchain containing git SHA if necessary.

I had many reports for uclibc toolchain long path problem, it might affect eglibc as well, not sure about that.

I'm willing to deliver some patches once we agree to some solution.
Comment 4 Brendan Le Foll 2014-11-14 16:03:44 UTC
(In reply to comment #3)
> Thanks for explaining this
> "we fix the revision on a per-release based on a stable point of uclibc"
> since from a distance it just looks like SRCREV is being updated every some
> time to 'latest & greatest' from uclibc.

Yes, you're correct this is suboptimal. I'm trying to work out exactly what the rational behind those updates are and see if we can maybe do better there.

> Every (or nearly every) other component has a proper versioning schema,
> sometimes based on git-tags. Even Yocto uses git-tags for versioning, but
> let's not argue about them, that's just an implementation detail.
> On the other hand uclibc is still 0.9.33. That's werid, isn't it? If it
> changes, why isn't the version bumped?
> On top of that I don't really see the point of including git SHA to package
> name. I spotted it's lenght has been reduced on master, but it's still
> taking some space that could be saved - especially knowing it becomes a
> problem for others.
> 
> What about using only PR, as it increments for new recipe revisions and is
> present always anyway?
> 
> 'opkg info <pkg-name>' shows description, maybe we can preserve full git SHA
> in there? I'm pretty sure yum/apt-get will easily show git SHA in some
> description field as well. We can ship additional file to the
> target/toolchain containing git SHA if necessary.
> 
> I had many reports for uclibc toolchain long path problem, it might affect
> eglibc as well, not sure about that.
> 
> I'm willing to deliver some patches once we agree to some solution.

For your particular issue in the short term I think there are two easy fixes, don't build the debug packages (they're not used by the 'duino toolchain are they?) or bbapend and replace PV with something that's smaller.

Are there any other packages that are a particular worry to you in reguards to the 255 char limit with HFS+?
Comment 5 Krzysztof Sywula 2014-11-17 15:29:01 UTC
Brendan,
Not building debug packages effectively strips off a toolchain & image functionality in case someone wanted to use gdb. Adding bbappend with a PV change is the winner since it is harmless, and I am going to do it locally.  However I think there is a value in upstreaming it. That is why I started the discussion.

I am not aware of any other problematic packages.
Comment 6 Stephen K Jolley 2014-12-01 18:54:28 UTC
Brendan, it appears the information requested has been provided.  I have moved this to an ACCEPTED state.
Comment 7 Randy MacLeod 2015-05-21 15:04:29 UTC
Shorten the sha1 to make the path shorter.
Comment 8 Mark Hatle 2015-05-21 15:54:47 UTC
Most of the tooling within the system uses the Windows mechanisms to start with a 'long' path and shrink it (internally) to an 8.3 style path.  This allows for a significantly longer path that what would initially appear possible.

I agree that super long git SRC revs should be shortened just to make them more useful, but this should not be a problem on most Windows machines due to this conversion.  (Note if installed from a .zip archive, WinZip is clueless and can't figure out how to do this properly.  Using 7zip, and the archive will extract properly with the correct name.)

I'm not familiar with using the SDK on OS X to comment there.
Comment 9 Richard Purdie 2016-01-14 16:10:36 UTC
Whilst we may remove uclibc, this issue would apply to any SCM based recipe. We really need to use a truncated version in the debug paths.
Comment 10 Robert Yang 2016-08-01 08:13:08 UTC
uclibc is gone