Bug 15135 - Can bitbake provide a setup.py or pyproject.toml to package it like gentoo portage
Summary: Can bitbake provide a setup.py or pyproject.toml to package it like gentoo po...
Status: RESOLVED NOTABUG
Alias: None
Product: BitBake
Classification: Build System, Metadata & Runtime
Component: bitbake (show other bugs)
Version: unspecified
Hardware: All Multiple
: Undecided normal
Target Milestone: 0.0.0
Assignee: Richard Purdie
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2023-06-06 15:28 UTC by wuzhenyu@ustc.edu
Modified: 2023-06-08 14:56 UTC (History)
3 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Yes (doc changes required)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description wuzhenyu@ustc.edu 2023-06-06 15:28:47 UTC
For example,
<https://github.com/gentoo/portage/blob/master/setup.py> make portage can be packaged and uploaded to [PYPI](https://pypi.org/project/portage/#files), and it make it more easy to be pacakged by other linux distribution like [Arch Linux](https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=portage). So Why not bitbake do it? The uploading to PYPI can be done easily by CI/CD.
Comment 1 wuzhenyu@ustc.edu 2023-06-06 16:39:25 UTC
```diff
From d8edeb24a63d467e8084a3b464bccd3271e1f9a5 Mon Sep 17 00:00:00 2001
From: Wu Zhenyu <wuzhenyu@ustc.edu>
Date: Wed, 7 Jun 2023 00:21:01 +0800
Subject: [PATCH] support package

use `python -m build` to build package
set PYPI_API_TOKEN to upload it to PYPI
---
 .github/workflows/python.yml |  44 ++++++++++++
 .gitignore                   | 128 ++++++++++++++++++++++++++++++++++-
 lib/bb/server/process.py     |   5 +-
 pyproject.toml               | 108 +++++++++++++++++++++++++++++
 4 files changed, 282 insertions(+), 3 deletions(-)
 create mode 100644 .github/workflows/python.yml
 create mode 100644 pyproject.toml

diff --git a/.github/workflows/python.yml b/.github/workflows/python.yml
new file mode 100644
index 0000000..ab3f7ba
--- /dev/null
+++ b/.github/workflows/python.yml
@@ -0,0 +1,44 @@
+---
+"on":
+  push:
+  pull_request:
+  workflow_dispatch:
+
+# https://github.com/softprops/action-gh-release/issues/236
+permissions:
+  contents: write
+
+env:
+  PYTHONUTF8: "1"
+  python-version: 3.x
+  cache: pip
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - uses: actions/setup-python@v4
+        with:
+          python-version: ${{env.python-version}}
+          cache: ${{env.cache}}
+      - name: Install dependencies
+        run: |
+          pip install build
+      - name: Build
+        run: |
+          python -m build
+      - uses: pypa/gh-action-pypi-publish@release/v1
+        if: startsWith(github.ref, 'refs/tags/')
+        with:
+          password: ${{secrets.PYPI_API_TOKEN}}
+      - uses: actions/upload-artifact@v3
+        if: "! startsWith(github.ref, 'refs/tags/')"
+        with:
+          path: |
+            dist/*
+      - uses: softprops/action-gh-release@v1
+        if: startsWith(github.ref, 'refs/tags/')
+        with:
+          files: |
+            dist/*
diff --git a/.gitignore b/.gitignore
index fbf9eb2..c646b0f 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,5 +1,3 @@
-*.pyc
-*.pyo
 *~
 pyshtables.py
 /doc/manual/html/
@@ -16,3 +14,129 @@ lib/toaster/contrib/tts/backlog.txt
 lib/toaster/contrib/tts/log/*
 lib/toaster/contrib/tts/.cache/*
 lib/bb/tests/runqueue-tests/bitbake-cookerdaemon.log
+
+# Python.gitignore:
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# celery beat schedule file
+celerybeat-schedule
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
diff --git a/lib/bb/server/process.py b/lib/bb/server/process.py
index 7616ef5..e5d1cbd 100644
--- a/lib/bb/server/process.py
+++ b/lib/bb/server/process.py
@@ -32,6 +32,7 @@ import stat
 import bb.server.xmlrpcserver
 from bb import daemonize
 from multiprocessing import queues
+from shutil import which
 
 logger = logging.getLogger('BitBake')
 
@@ -619,7 +620,9 @@ class BitBakeServer(object):
         os.close(self.readypipe)
         os.set_inheritable(self.bitbake_lock.fileno(), True)
         os.set_inheritable(self.readypipein, True)
-        serverscript = os.path.realpath(os.path.dirname(__file__) + "/../../../bin/bitbake-server")
+        serverscript = which("bitbake-server")
+        if serverscript is None:
+            serverscript = os.path.realpath(os.path.dirname(__file__) + "/../../../bin/bitbake-server")
         os.execl(sys.executable, "bitbake-server", serverscript, "decafbad", str(self.bitbake_lock.fileno()), str(self.readypipein), self.logfile, self.bitbake_lock.name, self.sockname,  str(self.server_timeout or 0), str(int(self.profile)), str(self.xmlrpcinterface[0]), str(self.xmlrpcinterface[1]))
 
 def execServer(lockfd, readypipeinfd, lockname, sockname, server_timeout, xmlrpcinterface, profile):
diff --git a/pyproject.toml b/pyproject.toml
new file mode 100644
index 0000000..f189284
--- /dev/null
+++ b/pyproject.toml
@@ -0,0 +1,108 @@
+[build-system]
+requires = ["setuptools_scm[toml] >= 6.2"]
+build-backend = "setuptools.build_meta"
+
+# https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html
+[project]
+name = "bitbake"
+description = ""
+readme = "README.md"
+requires-python = ">= 3.7"
+keywords = ["yocto"]
+version = "2.4.1"
+classifiers = [
+  "Development Status :: 3 - Alpha",
+  "Intended Audience :: Developers",
+  "Topic :: Software Development :: Build Tools",
+  "License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
+  "Operating System :: Microsoft :: Windows",
+  "Operating System :: POSIX",
+  "Operating System :: Unix",
+  "Operating System :: MacOS",
+  "Programming Language :: Python :: 3 :: Only",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.7",
+  "Programming Language :: Python :: 3.8",
+  "Programming Language :: Python :: 3.9",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: Implementation :: CPython",
+  "Programming Language :: Python :: Implementation :: PyPy",
+]
+dynamic = ["optional-dependencies"]
+
+[[project.authors]]
+name = "Tim Ansell"
+email = "mithro@mithis.net"
+
+[[project.authors]]
+name = "Phil Blundell"
+email = "pb@handhelds.org"
+
+[[project.authors]]
+name = "Seb Frankengul"
+email = "seb@frankengul.org"
+
+[[project.authors]]
+name = "Holger Freyther"
+email = "holger@moiji-mobile.com"
+
+[[project.authors]]
+name = "Marcin Juszkiewicz"
+email = "marcin@juszkiewicz.com.pl"
+
+[[project.authors]]
+name = "Chris Larson"
+email = "kergoth@handhelds.org"
+
+[[project.authors]]
+name = "Ulrich Luckas"
+email = "luckas@musoft.de"
+
+[[project.authors]]
+name = "Mickey Lauer"
+email = "mickey@Vanille.de"
+
+[[project.authors]]
+name = "Richard Purdie"
+email = "rpurdie@rpsys.net"
+
+[[project.authors]]
+name = "Holger Schurig"
+email = "holgerschurig@gmx.de"
+
+[project.license]
+text = "GPL v2"
+
+[project.urls]
+Homepage = "https://bitbake.readthedocs.io"
+Download = "https://github.com/Freed-Wu/bitbake/releases"
+"Bug Report" = "https://github.com/Freed-Wu/bitbake/issues"
+Source = "https://github.com/Freed-Wu/bitbake"
+
+[tool.setuptools]
+script-files = [
+  "bin/bitbake",
+  "bin/bitbake-diffsigs",
+  "bin/bitbake-dumpsig",
+  "bin/bitbake-getvar",
+  "bin/bitbake-hashclient",
+  "bin/bitbake-hashserv",
+  "bin/bitbake-layers",
+  "bin/bitbake-prserv",
+  "bin/bitbake-selftest",
+  "bin/bitbake-server",
+  "bin/bitbake-worker",
+  "bin/git-make-shallow",
+  "bin/toaster",
+  "bin/toaster-eventreplay",
+]
+
+[tool.setuptools.packages.find]
+where = ["lib"]
+
+[tool.setuptools.data-files]
+"share/man/man1" = ["doc/bitbake.1"]
+
+[tool.setuptools.dynamic.optional-dependencies.toaster]
+file = "toaster-requirements.txt"
-- 
2.40.1
```
Comment 2 wuzhenyu@ustc.edu 2023-06-06 16:43:31 UTC
Sorry, this is right:

```diff
From 83220c57dd7075e6836d890d6114c2a9b6584c0e Mon Sep 17 00:00:00 2001
From: Wu Zhenyu <wuzhenyu@ustc.edu>
Date: Wed, 7 Jun 2023 00:21:01 +0800
Subject: [PATCH] support package

use `python -m build` to build package
set PYPI_API_TOKEN to upload it to PYPI
---
 .github/workflows/python.yml |  44 ++++++++++++
 .gitignore                   | 128 ++++++++++++++++++++++++++++++++++-
 lib/bb/server/process.py     |   5 +-
 pyproject.toml               | 108 +++++++++++++++++++++++++++++
 4 files changed, 282 insertions(+), 3 deletions(-)
 create mode 100644 .github/workflows/python.yml
 create mode 100644 pyproject.toml

diff --git a/.github/workflows/python.yml b/.github/workflows/python.yml
new file mode 100644
index 0000000..ab3f7ba
--- /dev/null
+++ b/.github/workflows/python.yml
@@ -0,0 +1,44 @@
+---
+"on":
+  push:
+  pull_request:
+  workflow_dispatch:
+
+# https://github.com/softprops/action-gh-release/issues/236
+permissions:
+  contents: write
+
+env:
+  PYTHONUTF8: "1"
+  python-version: 3.x
+  cache: pip
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - uses: actions/setup-python@v4
+        with:
+          python-version: ${{env.python-version}}
+          cache: ${{env.cache}}
+      - name: Install dependencies
+        run: |
+          pip install build
+      - name: Build
+        run: |
+          python -m build
+      - uses: pypa/gh-action-pypi-publish@release/v1
+        if: startsWith(github.ref, 'refs/tags/')
+        with:
+          password: ${{secrets.PYPI_API_TOKEN}}
+      - uses: actions/upload-artifact@v3
+        if: "! startsWith(github.ref, 'refs/tags/')"
+        with:
+          path: |
+            dist/*
+      - uses: softprops/action-gh-release@v1
+        if: startsWith(github.ref, 'refs/tags/')
+        with:
+          files: |
+            dist/*
diff --git a/.gitignore b/.gitignore
index fbf9eb2..c646b0f 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,5 +1,3 @@
-*.pyc
-*.pyo
 *~
 pyshtables.py
 /doc/manual/html/
@@ -16,3 +14,129 @@ lib/toaster/contrib/tts/backlog.txt
 lib/toaster/contrib/tts/log/*
 lib/toaster/contrib/tts/.cache/*
 lib/bb/tests/runqueue-tests/bitbake-cookerdaemon.log
+
+# Python.gitignore:
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# celery beat schedule file
+celerybeat-schedule
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
diff --git a/lib/bb/server/process.py b/lib/bb/server/process.py
index 7616ef5..e5d1cbd 100644
--- a/lib/bb/server/process.py
+++ b/lib/bb/server/process.py
@@ -32,6 +32,7 @@ import stat
 import bb.server.xmlrpcserver
 from bb import daemonize
 from multiprocessing import queues
+from shutil import which
 
 logger = logging.getLogger('BitBake')
 
@@ -619,7 +620,9 @@ class BitBakeServer(object):
         os.close(self.readypipe)
         os.set_inheritable(self.bitbake_lock.fileno(), True)
         os.set_inheritable(self.readypipein, True)
-        serverscript = os.path.realpath(os.path.dirname(__file__) + "/../../../bin/bitbake-server")
+        serverscript = which("bitbake-server")
+        if serverscript is None:
+            serverscript = os.path.realpath(os.path.dirname(__file__) + "/../../../bin/bitbake-server")
         os.execl(sys.executable, "bitbake-server", serverscript, "decafbad", str(self.bitbake_lock.fileno()), str(self.readypipein), self.logfile, self.bitbake_lock.name, self.sockname,  str(self.server_timeout or 0), str(int(self.profile)), str(self.xmlrpcinterface[0]), str(self.xmlrpcinterface[1]))
 
 def execServer(lockfd, readypipeinfd, lockname, sockname, server_timeout, xmlrpcinterface, profile):
diff --git a/pyproject.toml b/pyproject.toml
new file mode 100644
index 0000000..3d22005
--- /dev/null
+++ b/pyproject.toml
@@ -0,0 +1,108 @@
+[build-system]
+requires = ["setuptools_scm[toml] >= 6.2"]
+build-backend = "setuptools.build_meta"
+
+# https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html
+[project]
+name = "bitbake"
+description = ""
+readme = "README.md"
+requires-python = ">= 3.7"
+keywords = ["yocto"]
+version = "2.4.1"
+classifiers = [
+  "Development Status :: 3 - Alpha",
+  "Intended Audience :: Developers",
+  "Topic :: Software Development :: Build Tools",
+  "License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
+  "Operating System :: Microsoft :: Windows",
+  "Operating System :: POSIX",
+  "Operating System :: Unix",
+  "Operating System :: MacOS",
+  "Programming Language :: Python :: 3 :: Only",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.7",
+  "Programming Language :: Python :: 3.8",
+  "Programming Language :: Python :: 3.9",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: Implementation :: CPython",
+  "Programming Language :: Python :: Implementation :: PyPy",
+]
+dynamic = ["optional-dependencies"]
+
+[[project.authors]]
+name = "Tim Ansell"
+email = "mithro@mithis.net"
+
+[[project.authors]]
+name = "Phil Blundell"
+email = "pb@handhelds.org"
+
+[[project.authors]]
+name = "Seb Frankengul"
+email = "seb@frankengul.org"
+
+[[project.authors]]
+name = "Holger Freyther"
+email = "holger@moiji-mobile.com"
+
+[[project.authors]]
+name = "Marcin Juszkiewicz"
+email = "marcin@juszkiewicz.com.pl"
+
+[[project.authors]]
+name = "Chris Larson"
+email = "kergoth@handhelds.org"
+
+[[project.authors]]
+name = "Ulrich Luckas"
+email = "luckas@musoft.de"
+
+[[project.authors]]
+name = "Mickey Lauer"
+email = "mickey@Vanille.de"
+
+[[project.authors]]
+name = "Richard Purdie"
+email = "rpurdie@rpsys.net"
+
+[[project.authors]]
+name = "Holger Schurig"
+email = "holgerschurig@gmx.de"
+
+[project.license]
+text = "GPL v2"
+
+[project.urls]
+Homepage = "https://docs.yoctoproject.org/bitbake"
+Download = "https://git.openembedded.org/bitbake"
+"Bug Report" = "https://bugzilla.yoctoproject.org"
+Source = "https://git.openembedded.org/bitbake"
+
+[tool.setuptools]
+script-files = [
+  "bin/bitbake",
+  "bin/bitbake-diffsigs",
+  "bin/bitbake-dumpsig",
+  "bin/bitbake-getvar",
+  "bin/bitbake-hashclient",
+  "bin/bitbake-hashserv",
+  "bin/bitbake-layers",
+  "bin/bitbake-prserv",
+  "bin/bitbake-selftest",
+  "bin/bitbake-server",
+  "bin/bitbake-worker",
+  "bin/git-make-shallow",
+  "bin/toaster",
+  "bin/toaster-eventreplay",
+]
+
+[tool.setuptools.packages.find]
+where = ["lib"]
+
+[tool.setuptools.data-files]
+"share/man/man1" = ["doc/bitbake.1"]
+
+[tool.setuptools.dynamic.optional-dependencies.toaster]
+file = "toaster-requirements.txt"
-- 
2.40.1
```
Comment 3 Ross Burton 2023-06-08 14:56:19 UTC
We encourage distributions to not package bitbake, but instead expect people using bitbake to bundle it with the layers they're using.  This is mainly because the bitbake API changes in each major release so there's a high chance that the distribution-provided bitbake isn't compatible with the layers that you're trying to use, at which point having a distribution-provided bitbake is pointless.

There's a related question over having pyproject.toml so that you can build a wheel of bitbake, and whether pypi should have all of the releases, but right now the maintainers stance is that you bundle bitbake with your layers.

Also considering bitbake's git repository is hosted at git.openembedded.org I'm not sure how useful a .github/workflows/python.yml would be.