Bug 11310 - host packages wrong for Ubuntu 16?
Summary: host packages wrong for Ubuntu 16?
Status: RESOLVED FIXED
Alias: None
Product: Quick Start
Classification: Documentation
Component: quick-start (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 2.3 M4
Assignee: Scott Rifenbark
QA Contact:
URL:
Whiteboard: 26 April 2017: RESOLVED
Depends on:
Blocks:
 
Reported: 2017-04-07 17:15 UTC by Robert Berger
Modified: 2017-04-26 18:30 UTC (History)
7 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Done (doc changes complete)


Attachments
script to test install for different distros (2.66 KB, application/octet-stream)
2017-04-20 03:03 UTC, brian avery
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Robert Berger 2017-04-07 17:15:17 UTC
see:

http://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#packages

sudo apt-get update

sudo apt-get install gawk wget git-core diffstat unzip texinfo gcc-multilib  [OK]

sudo apt-get install build-essential chrpath socat cpio python python3 python3-pip python3-expect

E: Unable to locate package python3-expect

I guess it should be 
sudo apt install python3-pexpect

sudo apt-get install xz which libsdl1.2-dev xterm
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package xz
E: Unable to locate package which

sudo apt install xz-utils

No idea what's the which package.

http://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#ubuntu-packages

sudo apt-get install gawk wget git-core diffstat unzip texinfo gcc-multilib

sudo apt-get install build-essential chrpath socat cpio python python3 python3-pip python3-expect
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package python3-expect

same as above

sudo apt-get install xz which
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package xz
E: Unable to locate package which

same as above

sudo apt-get install libsdl1.2-dev xterm [OK]

sudo apt-get install make xsltproc docbook-utils fop dblatex xmlto [OK]

sudo apt-get install python-git [OK]
Comment 1 Robert Berger 2017-04-07 17:16:54 UTC
cat /etc/issue
Ubuntu 16.04 LTS \n \l
Comment 2 Scott Rifenbark 2017-04-10 18:04:24 UTC
Hi, 

I recently worked Bug 10854 and worked with Brian Avery on this.  I am going to copy him on the bug.  This package requirement stuff for the hosts seems to be in constant flux... depending on various distro versions etc.  

I would like for Brian to look at this bug and your comments before I change stuff.

Thanks, 
Scott
Comment 3 brian avery 2017-04-11 00:28:28 UTC
yipes. either I messed up (possible) or things moved.
The broken pkgs should be:

 python3-pexpect
 debianutils
 xz-utils

The broken versions were:
python3-expect
which
xz

------
tested on official ubuntu docker contianer ubuntu:16.04 
# cat /etc/issue
Ubuntu Xenial Xerus (development branch) \n \l
Comment 4 Scott Rifenbark 2017-04-11 01:19:23 UTC
Brian, 

I don't know which distros this correction applies to (Ubuntu....all?)  So, I am cutting and pasting the four "Essentials" packages that are listed right now in the YP Reference Manual.  It would really help me clear this up if you could edit this comment with some substitutions for each of these distros.

Please tell me exactly how each of these should be.

Thanks,
Scott

================================================


This is for Ubuntu (see http://www.yoctoproject.org/docs/2.3/ref-manual/ref-manual.html#ubuntu-packages)

Essentials: Packages needed to build an image on a headless system:

         $ sudo apt-get install gawk wget git-core diffstat unzip texinfo \
         gcc-multilib build-essential chrpath socat cpio python python3 \
         python3-pip python3-expect xz which


================================================


This is for Fedora (see http://www.yoctoproject.org/docs/2.3/ref-manual/ref-manual.html#fedora-packages)

Essentials: Packages needed to build an image for a headless system:

     $ sudo dnf install gawk make wget tar bzip2 gzip python3 unzip perl \
       patch diffutils diffstat git cpp gcc gcc-c++ glibc-devel texinfo \
       chrpath ccache perl-Data-Dumper perl-Text-ParseWords \
       perl-Thread-Queue perl-bignum socat findutils which file cpio \
       python python3-pip python3-expect xz which


================================================



This is for OpenSUSE (see http://www.yoctoproject.org/docs/2.3/ref-manual/ref-manual.html#opensuse-packages)

Essentials: Packages needed to build an image for a headless system:

     $ sudo zypper install python gcc gcc-c++ git chrpath make wget \
       python-xml diffstat makeinfo python-curses patch socat python3 \
       python3-curses tar python3-pip python3-expect xz which


===============================================



This is for CentOS (see http://www.yoctoproject.org/docs/2.3/ref-manual/ref-manual.html#centos-packages)

Essentials: Packages needed to build an image for a headless system:

     $ sudo yum install -y epel-release
     $ sudo yum makecache
     $ sudo yum install gawk make wget tar bzip2 gzip python unzip perl \
     patch diffutils diffstat git cpp gcc gcc-c++ glibc-devel texinfo \
     chrpath socat perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue \
     python34-pip xz which SDL-devel xterm
Comment 5 Robert Berger 2017-04-11 09:15:44 UTC
I can confirm that for Ubuntu 16 those install:

$ sudo apt-get install gawk wget git-core diffstat unzip texinfo \
         gcc-multilib build-essential chrpath socat cpio python python3 \
         python3-pip python3-pexpect xz-utils debianutils

instead of 

$ sudo apt-get install gawk wget git-core diffstat unzip texinfo \
         gcc-multilib build-essential chrpath socat cpio python python3 \
         python3-pip python3-expect xz which
Comment 6 Robert Berger 2017-04-11 09:20:57 UTC
BTW in order to get bitbake core-image-minimal -g -u depexp to work with Morty on Ubuntu 16 I had to add:

sudo apt-get install -y libgtk-3-0 libgtk-3-bin libgtk-3-common libgtk-3-dev python3-gi
Comment 7 brian avery 2017-04-11 15:03:33 UTC
yep, the "essentials" are just to make images.
Comment 8 Scott Rifenbark 2017-04-11 19:04:49 UTC
Brian, 

What about Comment 4?  Can you give me some feedback on those lists to be sure I am changing the right packages?

Also, for Robert's Comment 5... those packages do not appear anywhere.  What distros and groupings would those apply to?

thanks,
Scott
Comment 9 brian avery 2017-04-11 20:25:09 UTC
Rebecca, could you ask someone on the team to go through each of these and verify?

Easiest approach:
run the std distro container for each of the os's scott has below

do the apt-get/dnf/zypper/yum install that scott lists. - make sure they all install.

$useradd -m tester

$su - tester
$ git clone git://git.yoctoproject.org/poky
$ ./poky/oe-init-buildenv
$ bitbake core-image-minimal

- check that it works.
testimage still has issues inside a container, so don't worry about that right now

ty!!
Comment 10 brian avery 2017-04-11 20:41:59 UTC
Scott, my Comment 3 correction was from a quick test on the ubuntu-16.04 container. I asked Rebecca's team for help to verify the matrix.
Comment 11 Rebecca Chang 2017-04-12 00:43:15 UTC
PG team will take over the verification. I will post the progress over here after the checking.
Comment 12 Rebecca Chang 2017-04-12 04:58:24 UTC
We have ran tests for all 4 distros, python3-expect was not found in all the package management repos.

Without installing python3-expect, bitbake image build still can be executed flawlessly. So python3-expect should not be inside the essential list since it is not discoverable and not needed.
Comment 13 Paul Eggleton 2017-04-12 09:56:24 UTC
I believe pexpect was added because the code to support testimage on real hardware uses it in order to interact with the bootloader. That isn't functionality that everyone will use, but it does still exist.
Comment 14 Scott Rifenbark 2017-04-12 15:04:05 UTC
Ok - I am going to remove "python3-expect" from the list of the essentials for Ubuntu, Fedora, and OpenSUSE.  Leaving all else alone unless I hear further. 

Scott
Comment 15 Scott Rifenbark 2017-04-13 20:45:13 UTC
Hi, 

See http://www.yoctoproject.org/docs/2.3/ref-manual/ref-manual.html#required-packages-for-the-host-development-system in the ref-manual and http://www.yoctoproject.org/docs/2.3/yocto-project-qs/yocto-project-qs.html#packages in the QS for the latest rendition of this stuff. 

Thanks, 
Scott
Comment 16 Rebecca Chang 2017-04-14 04:05:13 UTC
Reviewed doc changes.
Comment 17 Scott Rifenbark 2017-04-14 14:51:10 UTC
Hi, 

Since the docs have been reviewed with no comments, I am assuming the changes and the state of the packages are good.  I am marking the bug as RESOLVED and setting the doc flag to "done".

Scott
Comment 18 brian avery 2017-04-17 16:57:08 UTC
Sadly nope.
Rebecca, 
apt-get install python3-pexpect works on ubuntu 16.04, for example. And dnf install python3-pexpect works on fedora:24. The names change periodically, which is what is causing this to be soo frustrating.  

As Paul mentioned, python expect is being used to talk to the real hardware in testing.

At the top of the Poky tree, if you do 
$ grep -rH expect . | egrep import 
you will see:
./meta-yocto-bsp/lib/oeqa/controllers/grubtarget.py:import pexpect
./meta-yocto-bsp/lib/oeqa/controllers/edgeroutertarget.py:import pexpect
./meta-yocto-bsp/lib/oeqa/controllers/beaglebonetarget.py:import pexpect
Comment 19 Scott Rifenbark 2017-04-17 17:12:00 UTC
Ok - thanks for reopening the bug.  I will need some instruction on what we want for packages and for what distributions they apply. 

Scott
Comment 20 Scott Rifenbark 2017-04-19 21:27:53 UTC
Brian, 

I am changing the owner to you because I NEEDINFO for this bug.  Need someone to give me the definitive packages for the four distros.  

Thanks, 
Scott
Comment 21 brian avery 2017-04-19 22:36:34 UTC
adding saul, 
Rebecca did you all have a chance to follow up on this?
Comment 22 Rebecca Chang 2017-04-20 00:45:00 UTC
(In reply to comment #21)
> adding saul, 
> Rebecca did you all have a chance to follow up on this?

I will follow up on this. I have just synced up with Saul, PG team will take the next step on checking "python3-pexpect" and find out its replacement on all 4 distros and report out.
Comment 23 Rebecca Chang 2017-04-20 02:52:31 UTC
For Ubuntu 16.04, OpenSUSE, Fedora:
python3-pexpect is working. The broken pkg was python3-expect.

CentOS:
We are able to install pexpect package with yum. python3-pexpect was not found for CentOS.
Comment 24 brian avery 2017-04-20 03:03:25 UTC
Created attachment 3741 [details]
script to test install for different distros
Comment 25 brian avery 2017-04-20 03:06:08 UTC
I ended up writing a script to test installs (attached)
Here;s the results:
DEBIAN_HOST_PACKAGES_ESSENTIAL="gawk wget git-core diffstat unzip texinfo gcc-multilib \                                                                                          
     build-essential chrpath socat cpio python python3 python3-pip python3-pexpect\                                                                                               
     xz-utils debianutils"                                            |
UBUNTU_HOST_PACKAGES_ESSENTIAL="gawk wget git-core diffstat unzip texinfo gcc-multilib \                                                                                          
     build-essential chrpath socat cpio python python3 python3-pip python3-pexpect\                                                                                               
     xz-utils debianutils"                                            |
FEDORA_HOST_PACKAGES_ESSENTIAL="gawk make wget tar bzip2 gzip python3 unzip perl patch \                                                                                          
     diffutils diffstat git cpp gcc gcc-c++ glibc-devel texinfo chrpath \                                                                                                         
     ccache perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue perl-bignum socat python3-pexpect \                                                                           
     findutils which file cpio python python3-pip xz which"           |
OPENSUSE_HOST_PACKAGES_ESSENTIAL="python gcc gcc-c++ git chrpath make wget python-xml \                                                                                           
     diffstat makeinfo python-curses patch socat python3 python3-curses tar python3-pip python3-pexpect \                                                                         
     xz which"                                                        |
CENTOS_HOST_PACKAGES_ESSENTIAL="gawk make wget tar bzip2 gzip python unzip perl patch \                                                                                           
     diffutils diffstat git cpp gcc gcc-c++ glibc-devel texinfo chrpath socat \                                                                                                   
     perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue python34-pip xz \                                                                                                    
     which"                                                           |
-----
I also see python3 expect not being available in centos-7 . didn't test building any images as that is more complex to set up...
--
Reassigning back to scott. is ok?
Comment 26 Scott Rifenbark 2017-04-20 14:59:49 UTC
Hi,

Thanks for the effort testing these out. See http://www.yoctoproject.org/docs/2.3/ref-manual/ref-manual.html#required-packages-for-the-host-development-system for the link into the ref-manual.  Essentially, they stayed the same except for adding "python3-pexpect" into Ubuntu/Debian, Fedora, and OpenSUSE.

See http://www.yoctoproject.org/docs/2.3/yocto-project-qs/yocto-project-qs.html#packages for the same in the QS Manual. 

I am setting the bug to IN PROGRESS REVIEW again.

Thanks, 
Scott
Comment 27 Robert Berger 2017-04-25 15:02:59 UTC
Just started testing on the pyro branch and the first error I get is:

ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed:
  ping

----

Package ping is a virtual package provided by:
  inetutils-ping 2:1.9.4-1build1
  iputils-ping 3:20121221-5ubuntu2
You should explicitly select one to install.

-----

sudo apt insatll iputils-ping 

fixes the problem on Ubuntu 16.x
Comment 28 Robert Berger 2017-04-25 15:03:59 UTC
sudo apt install iputils-ping
Comment 29 Scott Rifenbark 2017-04-25 15:22:25 UTC
Hi Robert, 

See http://www.yoctoproject.org/docs/2.3/ref-manual/ref-manual.html#ubuntu-packages for the change for Ubuntu/Debian in the ref-manual.  See http://www.yoctoproject.org/docs/2.3/yocto-project-qs/yocto-project-qs.html#packages for the same change in the YP Quick Start. 

Let me know of any other failing stuff regarding packages for Pyro.

Thanks, 
Scott
Comment 30 Robert Berger 2017-04-25 16:39:55 UTC
Hi Scott,

I am happy with what's in your latest links.

Thanks!

Regards,

Robert
Comment 31 Scott Rifenbark 2017-04-26 18:30:58 UTC
Thanks Robert.

Setting to RESOLVED and putting the doc flag to "done."

Scott