Bug 14814

Summary: ncurses version of taskexp.py
Product: [Build System, Metadata & Runtime] BitBake Reporter: David Reyna <david.reyna>
Component: bitbakeAssignee: David Reyna <david.reyna>
Status: RESOLVED FIXED QA Contact:
Severity: enhancement    
Priority: Medium+ CC: poky.bs.watcher, poky.watcher, randy.macleod, richard.purdie
Version: 5.0   
Target Milestone: 5.0 M4   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Yes (doc changes required)

Description David Reyna 2022-05-25 01:55:23 UTC
1. Create an ncurses version of the GTK app "taskexp.py".
2. Add these features:
   (a) Sort tasks in recipes by their dependency order
   (b) Print individual and/or recipe-wide dependencies to a file
   (c) Add a wild card filter
   (d) Show the target recipes on BOLD
Comment 1 David Reyna 2022-05-25 01:56:41 UTC
Presented at the YP Summit 2022.5.
Comment 2 David Reyna 2023-06-22 09:40:51 UTC
I have a fix in test to the self-test blocker.
Comment 3 David Reyna 2024-02-18 10:50:47 UTC
I have prepared and submitted a re-partition of the previous submission. It adds a non-ncurses version of the unit test to support ptest and Autobuilder. 

That test will specifically request the bitbake dependecy data and do a quick sanity test of the results. This is the most vulnerable part of the integration over releases.

Normal execution example:
  $ bitbake -g -u taskexp_ncurses.py acl zlib 

GUI self-execution unit test example:
  $ TASK_EXP_UNIT_TEST=1 bitbake -g -u taskexp_ncurses quilt

Non-ncurses unit test example::
  $ TASK_EXP_UNIT_TEST_NOTERM=1 bitbake -g -u taskexp_ncurses quilt
  * UNIT TEST:START
  * UNIT TEST:BITBAKE FETCH
  Loading Cache: (0 of 0)
  Bitbake... : (1 of 2)
  Processing recipes: (0 of 914)
  ***************************************
  
  Generating dependency tree: (0 of 3)
  Generating dependency tree: (1 of 3)
  Generating dependency tree: (2 of 3)
  Generating dependency tree: (3 of 3)
  Pass:bitbake dependency count = 11456
  Pass:Quilt-native depends count = 11
  Pass:Quilt-native rdepends count = 351
  Pass:Quilt depends count = 138
  Pass:Quilt rdepends count = 41
  * UNIT TEST:STOP
  $ echo $?
  0
  $

NOTE: since the previous version I observed that for example the task "do_recipe_qa" now precedes "do_fetch", at least for "zlib". We will have to run the GUI self test each release to track the latest task tree organization.
Comment 4 David Reyna 2024-02-18 10:55:29 UTC
FYI per previous comment: 

Changes to the task order list does not break the tool. It may only affects the order they appear in the task window if there is order ambiguity in the inner task dependencies.

I only made the note to keep the GUI self test clean of false errors.
Comment 5 Randy MacLeod 2024-02-18 19:18:37 UTC
David, 

Maybe I need to read the manual or source but if I run your example on poky/master 

$ git log  -3
c754218614   2024-02-18   oeqa/selftest/rust: Exclude failing riscv tests
f62e4d81ec   2024-02-18   bitbake: taskexp_ncurses: ncurses version of taskexp.py
3c08950557   2024-02-17   testsdk: Avoid PATH contamination


yocto/b/taskexp 
❯ bitbake -g -u taskexp_ncurses.py acl zlib 
FATAL: Unable to import extension module "taskexp_ncurses.py" from bb.ui. Valid extension modules: knotty, ncurses, taskexp, taskexp_ncurses or teamcity
WARNING: <frozen importlib._bootstrap>:671: ImportWarning: DynamicImporter.exec_module() not found; falling back to load_module()

WARNING: <frozen importlib._bootstrap>:671: ImportWarning: DynamicImporter.exec_module() not found; falling back to load_module()

What do you suggest?
Comment 6 David Reyna 2024-02-18 20:14:34 UTC
Hi Randy and Richard,

Randy: Richard removed the ".py" from the script's name. You can see the new name "taskexp_ncurses" in the offered list in your error output.

Richard wrote:
>> I did make one small tweak to the patch when I queued it to drop the
>> .py and .py_2 from the above command examples since it caused them
>> not to work.
>> 
>> I've also queued a script to autobuilder-helper to allow us to run
>> these tests.
>
> That helper script didn't work. I do now have the tests working but it
>took hours to get it to work correctly due to the lack of a suitably
>capable terminal in the test environment on the autobuilder.
>
>I did want to see us resolve this though!

Richard: I am sorry for the extra work. The update purposefully skipped the entire ncurses setup and only runs pure Python commands. I tested that in an SSH shell that had "TERM=xterm", which was the minimal TERM I had available. I am now VERY curious what shell AutoBuilder has, because it confounds me. I did use "print(...,end='',flush=True) for a progress bar. Was that the issue?

David
Comment 7 David Reyna 2024-05-09 16:58:48 UTC
Deployed in YP 5.0!