Bug 11451

adding JL for interest, SW for assignee thoughts

just some status update on this bug, fixed up and patch several issues mainly, missing dependencies from juci, npm shrinkwrap extraneous error, iptables from openwrt patch failing, makefile build error.

currently Juci managed to compile into the core-image-sato, however, the Juci directory and libraries are missing. Still continuing the investigation as to why this happens. Tree diagram of Juci image shown below.

.
├── etc
│   └── hotplug.d
├── usr
│   ├── bin
│   │   └── juci-update
│   ├── lib
│   │   └── juci
│   │       ├── acl
│   │       ├── api
│   │       │   └── juci
│   │       └── services
│   └── share
│       ├── juci
│       │   └── juci.config.example
│       └── lua
└── www
    ├── css
    └── js

16 directories, 2 files

some new updates, found out that the Juci Makefile was trying to call some programs from nodejs e.g. uglifyjs, but it fails to find despite uglifyjs succeeded in compiling. This is because, npm install will install into a local node_modules and running this locally will not register the programs installed in nodejs into the PATH environment.

Simplest solution is to use npm install -g which would install all the dependencies and programs into the global directory/path so that Makefile would be able to find and view the programs but juci seems to have implemented a "offline" node_modules tarball as seen here https://github.com/mkschreder/juci/blob/master/node_modules.tar.gz

In order to prevent duplicating packages, rather than install npm install -g which would globally install the packages for the Makefile to view, it might be better to move those modules from node_modules into the global location recipe-sysroot-native and make use of the existing modules. 

Another thing is yuicompressor seems to rely on Java as yuicompressor is a .jar class file and it seems that we require to run apt-get install yui-compressor. So far, checking from layers.openembedded.org, there isn't a recipe for yui-compressor, might need to create a recipe using devtool.

yuicompressor's need on java on the host is unfortunate.

Can it be avoided somehow by specifying a different compressor:
   https://stackoverflow.com/questions/28932/best-javascript-compressor
perhaps UglifyJS would be suitable since it's a JavaScript compressor/minifier written in JavaScript.

Hmmm, it is very possible to change the type of compressor used to compress the files. In fact, Juci is already adopting Uglify-js for compressing js files but it seems that it is relying on yuicompressor for compressing css files.

But I do agree that if we can avoid Java, it would be for the best. There's a few options in the market for css compressors, mainly csso and minify, although both of them depends on nodejs and it's not a standalone binary, so there's tons of dependencies in the node_modules directory.

I'll try changing from yuicompressor to minify and see how it goes.

Managed to fix up the global install of the offline node_modules and Makefile is now able to compile :)

Here's more updates.

After discussing with Joshua and Brian, here's some clear deliverable for M1     * initial goal: understand the gaps and how to address them
* primary goal: patches for meta-openwrt ready for integration that makes Juci functional on a default OE distro
* future work: consider moving juci and uci to some other layer

Status on the build, I did some changes as I realized that the Juci inside the recipe of meta-openwrt is a very old version. 1+ year old, so I updated the revision and remove the older patches. Fixed up the "offline" node_modules tarball, fixed the package.json, fixed the npm install, fixed the Makefile, fixed the bootstrap, fixed the luci that was the dependency of juci, fixed lua recipe that was breaking and lots more broken stuff. Changed the yui-compressor which relied on Java to minify. And finally it builds on devshell at least.

Two issues still arises.
1. I can build using devshell manually, however, running over bitbake juci still fails stating that it cannot find "minify" despite minify existing inside the recipe-sysroot-native and "minify" can be used during devshell. So I'm still investigating why this is happening.

2. After building it manually using devshell, somehow the bin/ directory is still empty. This should show some css or js files inside their directories. Checking the Makefile seems to show some clues but it's really complicated inside. I'll probably have to debug that Makefile thoroughly to figure out why the website isn't installed.

root:.../juci/git-r0/git (master) # tree bin
bin
├── etc
│   └── hotplug.d
├── usr
│   ├── bin
│   │   └── juci-update
│   ├── lib
│   │   └── orange
│   │       ├── acl
│   │       ├── api
│   │       │   └── juci
│   │       └── services
│   └── share
│       ├── juci
│       │   └── juci.config.example
│       └── lua
└── www
    ├── css
    └── js

16 directories, 2 files

(In reply to comment #6)
> Two issues still arises.
> 1. I can build using devshell manually, however, running over bitbake juci
> still fails stating that it cannot find "minify" despite minify existing
> inside the recipe-sysroot-native and "minify" can be used during devshell.
> So I'm still investigating why this is happening.

I don't know anything about the Juci build machinery but it's possible/probably that when building it manually within the devshell it's using various utilities from your host.

If you haven't already, I recommend the devshell section of the YP manual:
http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#platdev-appdev-devshell

(In reply to comment #7)
> (In reply to comment #6)
> > Two issues still arises.
> > 1. I can build using devshell manually, however, running over bitbake juci
> > still fails stating that it cannot find "minify" despite minify existing
> > inside the recipe-sysroot-native and "minify" can be used during devshell.
> > So I'm still investigating why this is happening.
> 
> I don't know anything about the Juci build machinery but it's
> possible/probably that when building it manually within the devshell it's
> using various utilities from your host.
> 
> If you haven't already, I recommend the devshell section of the YP manual:
> http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#platdev-
> appdev-devshell

Thanks, it appears that minify had some issues due to permissions problem. After looking thru, I found out that inside node_modules there exist a .bin directory that is created by "npm install" each time it is called. The issue is that all of the softlinks inside the .bin directory had wrong permissions. I didn't spend too much time into investigating why this is happening yet, I'll probably re-visit and optimize this better once I've completed resolving other more critical issues inside Juci. Currently I'm prioritizing integrating Juci into Poky.

The investigation on why the Juci build dir is empty is still on-going and I have some pre-early reports which is not conclusive yet but I'll require some effort and time to attempt this and find more clues. 

Looking thru the documentation on Juci, it looks like there's more steps to go thru to integrate Juci that the recipe provided by meta-openwrt might not have covered. Then again, this is just some early reporting and this is merely a guess based on Juci docs and recipes from meta-openwrt. Looking thru the Juci documentation, it looks like they are tightly bound to OpenWRT's method of building, i.e. using feeds and defconfig to configure the builds of Juci. (see here https://github.com/mkschreder/juci-openwrt-feed)

Since I'm not sure if they are building it correctly, my proposal to understand how they build it would be to attempt building an OpenWRT with Juci on an actual router and try to compare between their environment and their builds. The purpose is to find any gaps in the current recipe provided inside meta-openwrt and check if it is building correctly. And if required, modification on the recipe maybe needed in order to make this work. So my next step is now to replicate OpenWRT with Juci on my own device (LinkIt SMART 7688) and see how it goes. Hopefully I'm not going about the long way :/ I'll probably increase the Person-Days to 17 since the amount of effort seems to have increased.

Some status update after a while, the great news is I have managed to build and compile Juci on Poky however, since Juci is just the front-end GUI, the back-end which is powered by UCI looks like is having some incompatibility issues. The changes done on the Juci isn't reflecting on Poky system e.g changing the VLAN or Firewall doesn't reflect on the Poky even though it is changed on the website. Still investigating on this, if this is an UCI issue, I might file another bug on UCI rather than putting this under Juci.

Another issue is, when I was about to commit this changes up, I did a new git pull on Poky and realized that gcc 7 is causing quite a number of errors on Juci compilation. I tried resolving them, but there's now a new issue on some missing files.

| In file included from /usr/include/limits.h:25:0,
|                  from /usr/lib/gcc/x86_64-linux-gnu/5/include-fixed/limits.h:168,
|                  from /usr/lib/gcc/x86_64-linux-gnu/5/include-fixed/syslimits.h:7,
|                  from /usr/lib/gcc/x86_64-linux-gnu/5/include-fixed/limits.h:34,
|                  from host/minilua.c:33:
| /usr/include/features.h:367:25: fatal error: sys/cdefs.h: No such file or directory
| compilation terminated.
| Makefile:664: recipe for target 'host/minilua.o' failed
| make[1]: *** [host/minilua.o] Error 1
| make[1]: *** Waiting for unfinished jobs....
| In file included from host/buildvm_fold.c:6:0:
| host/buildvm.h:9:23: fatal error: sys/types.h: No such file or directory
| In file included from host/buildvm_lib.c:6:0:
| host/buildvm.h:9:23: fatal error: sys/types.h: No such file or directory
| compilation terminated.
| compilation terminated.
| In file included from host/buildvm_peobj.c:9:0:
| host/buildvm.h:9:23: fatal error: sys/types.h: No such file or directory
| In file included from host/buildvm_asm.c:6:0:
| host/buildvm.h:9:23: fatal error: sys/types.h: No such file or directory
| compilation terminated.
| compilation terminated.
| Makefile:664: recipe for target 'host/buildvm_peobj.o' failed
| make[1]: *** [host/buildvm_peobj.o] Error 1
| Makefile:664: recipe for target 'host/buildvm_asm.o' failed
| make[1]: *** [host/buildvm_asm.o] Error 1
| Makefile:664: recipe for target 'host/buildvm_lib.o' failed
| make[1]: *** [host/buildvm_lib.o] Error 1
| Makefile:664: recipe for target 'host/buildvm_fold.o' failed
| make[1]: *** [host/buildvm_fold.o] Error 1

Does anyone here have any pointers to this issue?

(In reply to comment #9)
> Some status update after a while, the great news is I have managed to build
> and compile Juci on Poky however, since Juci is just the front-end GUI, the
> back-end which is powered by UCI looks like is having some incompatibility
> issues. The changes done on the Juci isn't reflecting on Poky system e.g
> changing the VLAN or Firewall doesn't reflect on the Poky even though it is
> changed on the website. Still investigating on this, if this is an UCI
> issue, I might file another bug on UCI rather than putting this under Juci.

What do you mean by "The changes done on the Juci isn't reflecting on Poky system"? Is it that the changes aren't being written to a file on disk? That the files are updated but the changed configuration isn't reflected in the running system? i.e that the files UCI is editing are not affecting our system configuration?

I would guess the latter and would expect that we (Yocto Project) will have to do some work to make UCI compatible with our system configuration.

Based on a skim-read  of the UCI docs[1] it looks like UCI writes its configuration to /etc/config and relies on either:
a) in-house OpenWRT programs referencing these configuration files directly (for openWRT specific programs)
b) 3rd party applications having been taught to reference those files, or having those files mapped to their canonical configuration files by an initialisation script

Furthermore the expectation seems to be that the services are restarted after any configuration file change to ensure that the new configuration values are picked up.

I'd look at:
* whether changes made in JUCI are being reflected to /etc/config
* whether changes are triggering a service restart
* whether the service restarts map to service handling in the poky reference distro (note: we'll likely want/need to do some work to make UCI support systemd at some point)

I would guess that the file changes are being written to /etc/config but that we don't have any hooks in place to map those changes to the system configuration locations in poky. The UCI page on the openwrt wiki indicates this is usually handled by the component's init script during the service restart.

1. https://wiki.openwrt.org/doc/uci

(In reply to comment #10)
> What do you mean by "The changes done on the Juci isn't reflecting on Poky
> system"? Is it that the changes aren't being written to a file on disk? That
> the files are updated but the changed configuration isn't reflected in the
> running system? i.e that the files UCI is editing are not affecting our
> system configuration?
> 
> I would guess the latter and would expect that we (Yocto Project) will have
> to do some work to make UCI compatible with our system configuration.
> 
> Based on a skim-read  of the UCI docs[1] it looks like UCI writes its
> configuration to /etc/config and relies on either:
> a) in-house OpenWRT programs referencing these configuration files directly
> (for openWRT specific programs)
> b) 3rd party applications having been taught to reference those files, or
> having those files mapped to their canonical configuration files by an
> initialisation script
> 
> Furthermore the expectation seems to be that the services are restarted
> after any configuration file change to ensure that the new configuration
> values are picked up.


Yes Joshua, you're right about that. UCI requires some work done to become compatible. Inside the /etc/config there's only two files:

1. juci
2. rpcd

> I'd look at:
> * whether changes made in JUCI are being reflected to /etc/config
> * whether changes are triggering a service restart
> * whether the service restarts map to service handling in the poky reference
> distro (note: we'll likely want/need to do some work to make UCI support
> systemd at some point)
> 
> I would guess that the file changes are being written to /etc/config but
> that we don't have any hooks in place to map those changes to the system
> configuration locations in poky. The UCI page on the openwrt wiki indicates
> this is usually handled by the component's init script during the service
> restart.
> 
> 1. https://wiki.openwrt.org/doc/uci

I tried running very simple UCI commands to test if UCI is working and Juci is working. It seems that it is working as I'm seeing changes being reflected into the /etc/config/juci.

After a few days of running thru things here and there, there's a few things that I noticed that isn't working.

1. /etc/config only provides two files as stated above, juci and rpcd. I tried building my own OpenWRT router and checking out what are the correct files that are 'supposed' to be inside an actual OpenWRT router. This is what I found inside an OpenWRT router:

ddns       dropbear   network    qos-opkg   system     uhttpd     wshaper
ddns-opkg  firewall   p910nd     rpcd       ubootenv   upnpd
dhcp       luci       qos        sqm        ucitrack   wireless

This OpenWRT build doesn't have Juci installed but it has luci, so the Juci is missing is expected. This should explain why the changes on the firewall and ip address and wifi isn't being reflected. It's because it's missing these files which is binded or made connected to the programs inside OpenWRT. The services in OpenWRT will read the changes inside the /etc/config and reflect it to the programs.

2. rpcd seems to fail to connect to ubus which is one of my biggest headache as I've been trying to get it up. Running rpcd will show up the error of "Failed to connect to ubus".

3. I'm not too sure about this part, I tried checking around https://github.com/kraj/meta-openwrt/ for orangerpcd (https://github.com/respeaker/orangerpcd) but I don't see traces of OrangeRPCD. I tried building Juci on my own router, and it seems that OrangeRPCD is needed for Juci but I'm not too sure why the meta-openwrt is missing this component. However, I'm not too confident if Juci actually do need this component to function but I thought following the same exact setup of how Juci builds should be easier to get it up.

Conclusion: Juci builds over Poky now and the website is runs but without RPC server or the ubus working, I can't access the website. Also, even though I tried to bypass the website and run the UCI commands, the UCI commands aren't going thru as the back-end compatibility is currently "not compatible" with Poky.

Any advice on how I should proceed on this bug? Should I now concentrate on making the back-end components of Juci e.g. UCI compatiblility to Poky or ubus+rpcd working? or should these go into another bug?

Submitted to oe-devel mailing list for meta-openwrt to review merging.
http://lists.openembedded.org/pipermail/openembedded-devel/2017-July/113681.html

Closed as this has already been merged together with meta-configuration as discussed.

https://github.com/avalluri/meta-configuration/tree/juci/devel