genodelabs / goa Goto Github PK

Now that Sculpt 21.10 is released, it is time to update Goa so that software can be built for the new Sculpt system.

Analog to the extended validation of elements in the <parent-provides>-node in runtime-files worked on in #26 , I would like to extend the validation to throw an error on sessions from the <requires>-node, which can not be provided.

For example the following vm session reference would not trigger a validation error in the runtime-file:

<requires>
    <vm/>
    <timer/>
</requires>

This would speed up debugging for users a lot and especially help eliminating first frustrations with the tooling.

Would you like to implement this or if not, are there any valuable hints you might give me concerning the implementation?

Kind regards,
Lars

If a Goa project only has a src directory but no artifacts file, goa prints the following error:

$ goa build
Error: ... does not look like a goa project

However, in an initial project phase - e.g., while crafting an import file and trying to get some 3rd-party code compiled -
there exists no artifacts file yet because one hasn't got hold of any binaries yet (hence, no clue what to write into the artifacts file). It would be nice of Goa to account for this situation by giving an instructive message whenever a src directory is present but no artifacts file, like:

"The project has a 'src' directory but lacks an 'artifacts' file. You may start with an empty file."

Currently, the files and directories specified by an 'api' file are searched in the build directory of the library. However, it is not very common for 3rd-party libraries to copy their header files to the build directory. In order to export an api archive with Goa, one thus needs to patch the CMakeLists.txt. I'm therefore thinking about adding support for searching the specified files and directories not only in the build directory but also in the src directory.

@trimpim Does this sound reasonable to you?

The packages are referenced in share/goa/lib/run/linux.tcl but missing in share/goa/goarc.

Is Virtualbox supported for testing with goa run?
I've mostly just tried to run the example from: https://github.com/genodelabs/genode/tree/master/repos/ports/recipes/pkg/vbox5

I currently use the following pkg/vbox_test/runtime:

<runtime ram="4300M" caps="2500" binary="init">

        <requires>
                <file_system label="vm"/>
                <file_system label="shared"/>
                <vm/>
                <timer/>
                <gui/>
                <nic/>
                <rom label="capslock"/>
                <report label="shape"/>
                <report label="clipboard"/>
                <rom    label="clipboard"/>
                <rm/>
                <rtc/>
                <rom label="usb_devices"/>
                <usb/>
        </requires>

        <config verbose="yes">

                <parent-provides>
                        <service name="ROM"/>
                        <service name="PD"/>
                        <service name="RM"/>
                        <service name="CPU"/>
                        <service name="LOG"/>
                        <service name="VM"/>
                        <service name="Gui"/>
                        <service name="Timer"/>
                        <service name="Rtc"/>
                        <service name="Report"/>
                        <service name="File_system"/>
                        <service name="Usb"/>
                        <service name="Nic"/>
                </parent-provides>

                <default-route> <any-service> <parent/> <any-child/> </any-service> </default-route>

                <default caps="100"/>

    <start name="vfs" caps="400">
        <resource name="RAM" quantum="10M"/>
        <provides><service name="File_system"/></provides>
        <config>
            <vfs>
                <ram/>
                <rom name="machine.vbox"/>
            </vfs>
            <default-policy root="/" writeable="yes"/>
        </config>
        <route>
            <any-service> <parent/> <any-child/> </any-service>
        </route>
    </start>

                <start name="vbox" caps="2000">
                        <binary name="virtualbox5" />
                        <resource name="RAM" quantum="4G"/>
                        <exit propagate="yes"/>
                        <config vbox_file="machine.vbox" xhci="yes" vm_name="linux" capslock="ROM">
                                <vfs>
                                        <dir name="dev">
                                                <log/>
                                                <rtc/>
                                         </dir>
                                        <dir name="shared"> <fs label="shared" writeable="yes"/> </dir>
                                        <fs writeable="yes"/>
                                </vfs>
                                <libc stdout="/dev/log" stderr="/dev/log" rtc="/dev/rtc"/>
                        </config>
                        <route>
                                <service name="Audio_out"> <parent/> </service>
                                <service name="File_system" label="shared"> <parent label="shared"/> </service>
                                <service name="File_system">                <child name="vfs"/> </service>
                                <service name="ROM" label="usb_devices"> <parent label="usb_devices"/> </service>
                                <service name="ROM" label="capslock"> <parent label="capslock"/> </service>
                                <service name="ROM" label="platform_info">
                                        <parent label="platform_info"/> </service>
                                <service name="Nic"> <parent/> </service>
                                <service name="Report" label="shape"> <parent label="shape"/> </service>
                                <service name="ROM"    label="clipboard"> <parent label="clipboard"/> </service>
                                <service name="Report" label="clipboard"> <parent label="clipboard"/> </service>
                                <service name="Gui"> <parent label=""/> </service>
                                <any-service> <parent/> </any-service>
                        </route>
                </start>
        </config>

        <content>
                <rom label="ld.lib.so"/>
                <rom label="init"/>
                <rom label="virtualbox5"/>
                <rom label="libc.lib.so"/>
                <rom label="libm.lib.so"/>
                <rom label="libiconv.lib.so"/>
                <rom label="libyuv.lib.so"/>
                <rom label="qemu-usb.lib.so"/>
                <rom label="stdcxx.lib.so"/>
                <rom label="vfs.lib.so"/>
                <rom label="jpeg.lib.so"/>
                <rom label="platform_info"/>
                <rom label="machine.vbox"/>
                <rom label="vfs"/>
        </content>

</runtime>

The following pkg/vbox_test/archives:

TLC62CsRraK4hdw6/src/vbox5
TLC62CsRraK4hdw6/src/libc
TLC62CsRraK4hdw6/src/init
TLC62CsRraK4hdw6/src/posix
TLC62CsRraK4hdw6/src/zlib
TLC62CsRraK4hdw6/src/libiconv
TLC62CsRraK4hdw6/src/libyuv
TLC62CsRraK4hdw6/src/stdcxx
TLC62CsRraK4hdw6/src/vfs
TLC62CsRraK4hdw6/src/jpeg

The following raw/platform_info (since I read it's not provided on Linux? taken from https://github.com/genodelabs/genode/blob/master/repos/os/run/vmm_x86.run#L75 :

                        <platform_info>
                                <kernel name="nova"/>
                                <acpi revision="2" rsdt="0x2ffe20c5"/>
                                <affinity-space width="2" height="2"/>
                                <boot/>
                                <hardware>
                                        <features svm="true" vmx="true"/>
                                        <tsc invariant="false" freq_khz="2555589"/>
                                        <cpus>
                                                <cpu id="0" package="0" core="0" thread="0" family="0x10" model="0x2" stepping="0x3" platform="0x0" patch="0x0"/>
                                        </cpus>
                                </hardware>
                        </platform_info>

And a machine.vbox from https://github.com/genodelabs/genode/blob/master/repos/gems/recipes/raw/download_coreplus/machine.vbox

The following is output by goa run:

Genode sculpt-22.04 <local changes>
17592186044415 MiB RAM and 18997 caps assigned to init
[init -> nic_drv] MAC address 02:00:00:00:00:01
[init -> nic_drv] using tap device "tap0"
[init -> nic_drv] Error: could not configure /dev/net/tun: no virtual network emulation
terminate called after throwing an instance of 'Genode::Exception'
[init -> vbox_test] parent provides
[init -> vbox_test]   service "ROM"
[init -> vbox_test]   service "PD"
[init -> vbox_test]   service "RM"
[init -> vbox_test]   service "CPU"
[init -> vbox_test]   service "LOG"
[init -> vbox_test]   service "VM"
[init -> vbox_test]   service "Gui"
[init -> vbox_test]   service "Timer"
[init -> vbox_test]   service "Rtc"
[init -> vbox_test]   service "Report"
[init -> vbox_test]   service "File_system"
[init -> vbox_test]   service "Usb"
[init -> vbox_test]   service "Nic"
[init -> vbox_test] child "vfs"
[init -> vbox_test]   RAM quota:  9992K
[init -> vbox_test]   cap quota:  366
[init -> vbox_test]   ELF binary: vfs
[init -> vbox_test]   priority:   0
[init -> vbox_test]   provides service File_system
[init -> vbox_test] child "vbox"
[init -> vbox_test]   RAM quota:  4194056K
[init -> vbox_test]   cap quota:  1966
[init -> vbox_test]   ELF binary: virtualbox5
[init -> vbox_test]   priority:   0
[init -> vbox_test] child "vfs" announces service "File_system"
error: XDG_RUNTIME_DIR is invalid or not set in the environment.
Authorization required, but no authorization protocol specified
[init -> drivers -> fb_sdl] Error: fb_sdl works on X11 only. Your SDL backend is offscreen.
terminate called after throwing an instance of 'Sdl_videodriver_not_supported'
[init -> vbox_test -> vbox] Log created: 2023-01-21T15:24:00.002000000Z
[init -> vbox_test -> vbox] main     Executable: /virtualbox
[init -> vbox_test -> vbox] Error: SystemPropertiesWrap : Cannot determine default Guest Additions ISO location. Most likely they are not available
[init -> vbox_test] child "vbox" requests resources: cap_quota=4

And then it's just stuck there.

Please either document that virtualization is not possible or provide an example in this repository.
Thanks for the great work!

At the moment, the versions of components come from share/goa/goarc. So at the moment, versions that my software is compiled with are changed when I update the tool. This is undesirable.

Ideally, Goa would write the current dependencies into a lock file when dependencies have been resolved. This lock file can then be checked into version control.

Good examples of this workflow are the Haskell Stack, or Cargo lock files, npm lock files, etc.

The examples can be found at https://github.com/ssumpf/goa-projects and are used to demonstrate Genode's Phone SDK.

Currently, goa run will download and prepare dependencies before starting Genode. In some situations it might be desirable to separate these steps. For example: while testing my Goa projects I'd like to interact with the spawned Genode process via expect and time out if a certain output is not produced. This does not work if I call goa run because downloading the dependencies might take an arbitrarily long time.

One idea would be to create a new prepare-run subcommand that does everything that run currently does but does not call run_genode. run would then depend on this and only call run_genode. Is there a specific reason why it does not currently work this way?

This allows access to ROM modules that are provided from outside of the project. The feature is activated as in the code snippet below:

<requires>
    <rom label="name_of_rom"/>
</requires>

To provide the ROM a lx_fs and a fs_rom component are started. A file with the name name_of_rom is expected in the sub directory roms.

Currently, there is no reference in Goa where to find documentation about the structure of 'import' files. Since Goa uses the port tools from the Genode repo, the guide at genode.org can be used as documentation. However, this is not immediately clear for newcomers.

Instead, I'd rather integrate documentation in terms of a goa help import sub-command into Goa, similar to goa help artifacts. This way Goa becomes more self-contained in terms of documentation.

Analog to the extended validation of elements in the <parent-provides>-node in runtime-files worked on in #26 , I would like to extend the validation to throw an error on files from the <content>-node, which can not be provided.

For example the following inexistent file reference would not trigger a validation error in the runtime-file:

<content>
    <rom label="existent">
    <rom label="inexistent">
</content>

This would speed up debugging for users a lot and especially help eliminating first frustrations with the tooling.

Would you like to implement this or if not, are there any valuable hints you might give me concerning the implementation?

Kind regards,
Lars

I know this isn't a huge deal, but when goa is run via a relative symbolic link, e.g. "../goa/bin/goa", it doesn't properly find its install directory. Either it should resolve relative links properly, or there should be a notice in the readme that goa cannot be added to the shell path via a relative symbolic link.

This idea is spinning in my head for a while now, and I'm admittedly not quite sure about it.

My original intention behind the .goarc files was to allow for user-specific settings, following the usual pattern (like .bashrc).

However, by now, I have seen several instances where .goarc files are checked in into Git repositories (goa-project repos), which in my opinion, makes things a little bit too obscure because those files are, well, hidden. So should we better change the naming to Goa (with an upper-case first letter like in Makefile)? Or should both variants be allowed?

I'm particularly worried since .goarc files can contain arbitrary Tcl code. So when a user clones a Git repo and invokes goa commands, this code get executed. Hence, it would be fair to bring this code to the user's attention by not hiding it.

I tried to extend the unix shell a bit and noticed that some api archives are reported missing as soon as I run goa publish. I believe the api archives are automatically downloaded, however, if goa is unable to pull a certain archive once it seems to forget some dependencies.
I'm able to reproduce this issue with the following modifications on the intro/unix_3rd in goa-playground:

1. remove the var directory to start on a clean slate
2. add alex-ab/src/which to pkg/unix_3rd/archives
3. add set version(alex-ab/src/which) 2021-02-22 to .goarc
4. run goa add-depot-user alex-ab --depot-url https://depot.genode.org --pubkey-file /path/to/alex-ab/pubkey
5. run goa run - this will fail on downloading https://depot.genode.org/alex-ab/api/libc/2021-02-22.tar.xz
6. run goa run again - this will succeed
7. yet, running goa publish will report a bunch of missing api archives

Due to ABI changes in FreeBSD 12, the libc crate used by rustc currently tries to link against versioned symbols of some library functions to preserve compatability with FreeBSD 11.

To make the standard library distributed with the x86_64-unknown-freebsd target work, statically link versioned stat() and fstat() symbols from the the compat-libc api package.

Hello Norman,

Today I've installed your goa program, I've followed every step you described in your installation file, but every time I try to execute it, (goa build), it appears the following error:

/usr/local/genode/tool/current/bin/genode-x86-gcc: 1: /usr/local/genode/tool/current/bin/genode-x86-gcc: Syntax error: "(" unexpected
while executing
"exec $cross_dev_prefix\gcc -print-libgcc-file-name"
(file "/home/sgrjlhgwl/goa/share/goa/lib/flags.tcl" line 14)
invoked from within
"source [file join $tool_dir lib flags.tcl]"
invoked from within
"if {$perform(build-dir)} {

#
# Prepare depot content for the used APIs and generate ABI stubs
#
# This must happen before assembling the compile ..."
(file "/home/sgrjlhgwl/goa/bin/goa" line 495)

Honestly I don't know how I can fix it, any helps?

Thanks.

PS: I've installed everything needed: from Genode toolchain to expect tool (& the other your wrote)

:)

As of now, Goa is limited to the x86_64 architecture. Since Sculpt 21.10 has become routinely be available for AARCH64 in addition to x86_64, it would be nice being able to target both architectures, or even cross compile.

Is Virtualbox supported for testing with goa run?
I've mostly just tried to run the example from: https://github.com/genodelabs/genode/tree/master/repos/ports/recipes/pkg/vbox5

I currently use the following pkg/vbox_test/runtime:

<runtime ram="4300M" caps="2500" binary="init">

        <requires>
                <file_system label="vm"/>
                <file_system label="shared"/>
                <vm/>
                <timer/>
                <gui/>
                <nic/>
                <rom label="capslock"/>
                <report label="shape"/>
                <report label="clipboard"/>
                <rom    label="clipboard"/>
                <rm/>
                <rtc/>
                <rom label="usb_devices"/>
                <usb/>
        </requires>

        <config verbose="yes">

                <parent-provides>
                        <service name="ROM"/>
                        <service name="PD"/>
                        <service name="RM"/>
                        <service name="CPU"/>
                        <service name="LOG"/>
                        <service name="VM"/>
                        <service name="Gui"/>
                        <service name="Timer"/>
                        <service name="Rtc"/>
                        <service name="Report"/>
                        <service name="File_system"/>
                        <service name="Usb"/>
                        <service name="Nic"/>
                </parent-provides>

                <default-route> <any-service> <parent/> <any-child/> </any-service> </default-route>

                <default caps="100"/>

    <start name="vfs" caps="400">
        <resource name="RAM" quantum="10M"/>
        <provides><service name="File_system"/></provides>
        <config>
            <vfs>
                <ram/>
                <rom name="machine.vbox"/>
            </vfs>
            <default-policy root="/" writeable="yes"/>
        </config>
        <route>
            <any-service> <parent/> <any-child/> </any-service>
        </route>
    </start>

                <start name="vbox" caps="2000">
                        <binary name="virtualbox5" />
                        <resource name="RAM" quantum="4G"/>
                        <exit propagate="yes"/>
                        <config vbox_file="machine.vbox" xhci="yes" vm_name="linux" capslock="ROM">
                                <vfs>
                                        <dir name="dev">
                                                <log/>
                                                <rtc/>
                                         </dir>
                                        <dir name="shared"> <fs label="shared" writeable="yes"/> </dir>
                                        <fs writeable="yes"/>
                                </vfs>
                                <libc stdout="/dev/log" stderr="/dev/log" rtc="/dev/rtc"/>
                        </config>
                        <route>
                                <service name="Audio_out"> <parent/> </service>
                                <service name="File_system" label="shared"> <parent label="shared"/> </service>
                                <service name="File_system">                <child name="vfs"/> </service>
                                <service name="ROM" label="usb_devices"> <parent label="usb_devices"/> </service>
                                <service name="ROM" label="capslock"> <parent label="capslock"/> </service>
                                <service name="ROM" label="platform_info">
                                        <parent label="platform_info"/> </service>
                                <service name="Nic"> <parent/> </service>
                                <service name="Report" label="shape"> <parent label="shape"/> </service>
                                <service name="ROM"    label="clipboard"> <parent label="clipboard"/> </service>
                                <service name="Report" label="clipboard"> <parent label="clipboard"/> </service>
                                <service name="Gui"> <parent label=""/> </service>
                                <any-service> <parent/> </any-service>
                        </route>
                </start>
        </config>

        <content>
                <rom label="ld.lib.so"/>
                <rom label="init"/>
                <rom label="virtualbox5"/>
                <rom label="libc.lib.so"/>
                <rom label="libm.lib.so"/>
                <rom label="libiconv.lib.so"/>
                <rom label="libyuv.lib.so"/>
                <rom label="qemu-usb.lib.so"/>
                <rom label="stdcxx.lib.so"/>
                <rom label="vfs.lib.so"/>
                <rom label="jpeg.lib.so"/>
                <rom label="platform_info"/>
                <rom label="machine.vbox"/>
                <rom label="vfs"/>
        </content>

</runtime>

The following pkg/vbox_test/archives:

TLC62CsRraK4hdw6/src/vbox5
TLC62CsRraK4hdw6/src/libc
TLC62CsRraK4hdw6/src/init
TLC62CsRraK4hdw6/src/posix
TLC62CsRraK4hdw6/src/zlib
TLC62CsRraK4hdw6/src/libiconv
TLC62CsRraK4hdw6/src/libyuv
TLC62CsRraK4hdw6/src/stdcxx
TLC62CsRraK4hdw6/src/vfs
TLC62CsRraK4hdw6/src/jpeg

The following raw/platform_info (since I read it's not provided on Linux? taken from https://github.com/genodelabs/genode/blob/master/repos/os/run/vmm_x86.run#L75 :

                        <platform_info>
                                <kernel name="nova"/>
                                <acpi revision="2" rsdt="0x2ffe20c5"/>
                                <affinity-space width="2" height="2"/>
                                <boot/>
                                <hardware>
                                        <features svm="true" vmx="true"/>
                                        <tsc invariant="false" freq_khz="2555589"/>
                                        <cpus>
                                                <cpu id="0" package="0" core="0" thread="0" family="0x10" model="0x2" stepping="0x3" platform="0x0" patch="0x0"/>
                                        </cpus>
                                </hardware>
                        </platform_info>

And a machine.vbox from https://github.com/genodelabs/genode/blob/master/repos/gems/recipes/raw/download_coreplus/machine.vbox

The following is output by goa run:

Genode sculpt-22.04 <local changes>
17592186044415 MiB RAM and 18997 caps assigned to init
[init -> nic_drv] MAC address 02:00:00:00:00:01
[init -> nic_drv] using tap device "tap0"
[init -> nic_drv] Error: could not configure /dev/net/tun: no virtual network emulation
terminate called after throwing an instance of 'Genode::Exception'
[init -> vbox_test] parent provides
[init -> vbox_test]   service "ROM"
[init -> vbox_test]   service "PD"
[init -> vbox_test]   service "RM"
[init -> vbox_test]   service "CPU"
[init -> vbox_test]   service "LOG"
[init -> vbox_test]   service "VM"
[init -> vbox_test]   service "Gui"
[init -> vbox_test]   service "Timer"
[init -> vbox_test]   service "Rtc"
[init -> vbox_test]   service "Report"
[init -> vbox_test]   service "File_system"
[init -> vbox_test]   service "Usb"
[init -> vbox_test]   service "Nic"
[init -> vbox_test] child "vfs"
[init -> vbox_test]   RAM quota:  9992K
[init -> vbox_test]   cap quota:  366
[init -> vbox_test]   ELF binary: vfs
[init -> vbox_test]   priority:   0
[init -> vbox_test]   provides service File_system
[init -> vbox_test] child "vbox"
[init -> vbox_test]   RAM quota:  4194056K
[init -> vbox_test]   cap quota:  1966
[init -> vbox_test]   ELF binary: virtualbox5
[init -> vbox_test]   priority:   0
[init -> vbox_test] child "vfs" announces service "File_system"
error: XDG_RUNTIME_DIR is invalid or not set in the environment.
Authorization required, but no authorization protocol specified
[init -> drivers -> fb_sdl] Error: fb_sdl works on X11 only. Your SDL backend is offscreen.
terminate called after throwing an instance of 'Sdl_videodriver_not_supported'
[init -> vbox_test -> vbox] Log created: 2023-01-21T15:24:00.002000000Z
[init -> vbox_test -> vbox] main     Executable: /virtualbox
[init -> vbox_test -> vbox] Error: SystemPropertiesWrap : Cannot determine default Guest Additions ISO location. Most likely they are not available
[init -> vbox_test] child "vbox" requests resources: cap_quota=4

And then it's just stuck there.

Please either document that virtualization is not possible or provide an example in this repository.

Thanks for the great work!

The cmake_step2 example relied on Github's Subversion support, which will be discontinued. In order to keep the example working, add a new repository type, sparse-git, and convert the example to use a sparse checkout.

By default, Goa creates a default depot at var/depot in the project directory. This is done "magically" when Goa tries downloading/extracting any depot archive and encounters a non-existing var/depot directory. However, when calling goa export, the var/depot folder gets created but will not be populated with the default depot users. The same effect occurs for goa add-depot-user.

By introducing a depot-dir action similar to the build-dir action, we could make the goa run, goa export and goa add-depot-user dependent on this. Moreover, the action could be used separately to initialise a certain directory as a depot.

@nfeske I noticed that Goa only populates the depot directory with the default depot if the depot_dir configuration has not been customised. I'm wondering whether we could change this. When working with several projects, the --depot-dir argument or depot_dir config variable can be used to use a shared depot directory. However, there is no way to conveniently set this up.

@jschlatow noticed in 60ef785#r117846730 that there is an artifact that exports the API to the wrong location.

The code in question looks like an artifact of integrating the possibility to build libraries. It is not needed to successfully build our test scenario, which uses libraries.

I have a test scenario, which uses the component vbox5.
Running it on top of the linux-runtime is not supported.
I would like to add a qemu-runtime so the scenario can be tested like in the genode-repository:

• https://github.com/genodelabs/genode/tree/master/repos/ports/recipes/pkg/vbox5

Would you like to implement this or if not, are there any valuable hints you might give me concerning the implementation?

Kind regards,
Lars

Since its release, Goa has been augmented with support for additional build systems (autoconf, qmake, vivado) that require some sort of user documentation.

Goa unexpectedly requires an API version when exporting a regular src project.

This can be observed when removing the goa/examples/hello_make/version file and issuing the following commands:

goa/examples/hello_make$ rm version
goa/examples/hello_make$ touch /tmp/license
goa/examples/hello_make$ goa export --depot-user x --depot-overwrite --license /tmp/license 
Error: version for archive x/api/hello_make undefined
 
 Create a 'version' file in your project directory, or 
 define 'set version(x/api/hello_make) <version>' in your .goarc file.

In this case, Goa should only require version for x/src/hello_make, not for an API archive.

If a Goa project has an empty used_apis file, the exported src recipe is missing the used_apis file. However, Genode's build system requires this file to be present when importing the recipe via import_from_depot.

The linux_nic_drv previously had a nic_server-mode that was used to implement the NIC requirements. Since removal of this mode from the driver, the support for NIC sessions is broken in Goa. As the NIC driver solely operates as an Uplink client, we need to additionally instantiate a NIC router that we use as a NIC server for the runtime.

I just noticed that the lib.so files generated by Goa are all ~17MB large. This also holds for the cmake_library example. I found that the -Wl,-Ttext=0x1000000 argument is passed as ldflags to cmake also for shared libraries, which inflates the resulting file by 16MB.

When changing the version of an api archive used by a Goa project, the implemented symbols might change as well. However, the abi stubs stored in the var/abi/ directory are not regenerated automatically. Currently, we therefore need to remove the var/ directories when switching between Goa releases or when using --versions-from-genode-dir.

Commit nfeske@60ef785 breaks qmake (ldlibs_exe missing) and Genode dynamic binaries (Component::construct gets garbage collected by ld, leading to broken binaries).

When trying to run goa build for the first time, it tries to copy the default depot into the source directory. If goa was installed in a read-only location, the resulting files will also be read-only. This issue came up during packaging of goa.

%  find .
./src
./src/Makefile
./src/hello.cc
./used_apis
% goa build 
download nfeske/api/base/2019-11-23.tar.xz
download nfeske/api/base/2019-11-23.tar.xz.sig
Error: could not verify 'nfeske/api/base/2019-11-23', signature does not match
       public key '/home/julian/src/own/genode-hello/var/depot/nfeske/pubkey'
Error: failed to download the following depot archives:
 nfeske/api/base/2019-11-23

The reason is that gpg cannot create the key in the read-only directories goa created:

gpg: can't create '/home/julian/src/own/genode-hello/var/depot/nfeske/pubkey.dearmored': Permission denied

% ls -lh var/depot
total 8.0K
dr-xr-xr-x 2 julian users 4.0K Jan  1  1970 genodelabs
dr-xr-xr-x 2 julian users 4.0K Jan  1  1970 nfeske

This can be hotfixed by running chmod -R +w var.

The real fix seems to be fixing this Tcl code:

proc prepare_depot_with_archives { archive_list } {
	global depot_dir public_dir tool_dir jobs

	if {![depot_exists]} {

		if {[customized_variable depot_dir]} {
			exit_with_error "cannot install APIs because" \
			                "there is no depot at $depot_dir" }

		# create default depot local to the project at var/depot/
		file mkdir var
		file copy [file join $tool_dir default_depot] $depot_dir # <- This needs to be fixed
	}

Unfortunately, my Tcl skills are zero. :(

According to the documentation, only a single pkg shall be exported if --pkg <pkgname> is provided to goa export. Yet, this is only implemented for goa publish.

Updating a user's depot index with packages published with Goa for a new Sculpt release is still a manual task. With an increasing number of Goa packages to be managed and published, this becomes a significant effort. Adding support for publishing an index to a user's depot, will automate this task and eliminate the need to switch to the depot tools within the Genode repo.

Here is a collection of ideas regarding this:

• Similar to how packages are exported/published, an index is exported/published by the to-be-added commands goa export-index and goa publish-index.
• Both commands take a version string (e.g. 23.04) as a positional argument. Moreover, the commands evaluate the depot_user and depot_overwrite variables.
• The index is built from a index file in the working directory that is similar to the exported/published file but lacks username and version information.
• The archive versions are taken from the goarc. Optionally, it might be possible to look up versions from the subdirectories that contain the Goa projects.
• It may be worth thinking about whether goa publish-index should call goa publish for not-yet-published archives.

Related to this, a command goa bump-versions <version> could be helpful for setting the versions of all Goa projects present in the subdirectories.

These are fixes for some problems I encountered while building OpenSC with GOA. The autoconf support had to be improved at some points and support for using the PCSC-lite API was required.

m-stein/goa@b9432d2 autoconf: improve appliance of verbosity settings
m-stein/goa@f231a3b autoconf: define CPP and CXXCPP for configure
m-stein/goa@9ea55a6 quirks: support using pcsc-lite api
m-stein/goa@7024fa9 autoconf: autoreconf if there's only configure.ac

To simplify testing of projects that use multiple NIC interfaces we should start a nic_router and anic_drv for each required <nic label=""/>.

This comes from the discussion on the mailing list see: https://lists.genode.org/pipermail/users/2023-June/008754.html

When trying to build the lvgl_support library without strict warnings disabled, I'm getting the following build error:

/home/johannes/repos/genode/depot/genodelabs/api/gui_session/2023-03-12/include/gui_session/gui_session.h: In instantiation of ‘Gui::Session::Command::Command(ARGS) [with ARGS = Gui::Session::Command::To_front]’:
/home/johannes/repos/genode/depot/genodelabs/api/gui_session/2023-03-12/include/gui_session/client.h:103:12:   required from ‘void Gui::Session_client::enqueue(ARGS ...) [with CMD = Gui::Session::Command::To_front; ARGS = {Genode::Handle<Gui::View>, Genode::Handle<Gui::View>}]’
/home/johannes/repos/goa-pkgs/lvgl/lvgl_support/src/lvgl_support.cc:172:69:   required from here
/home/johannes/repos/genode/depot/genodelabs/api/gui_session/2023-03-12/include/gui_session/gui_session.h:139:29: error: dereferencing type-punned pointer will break strict-aliasing rules [-Werror=strict-aliasing]
  139 |    reinterpret_cast<ARGS &>(nop) = args;
      |                             ^~~
/home/johannes/repos/genode/depot/genodelabs/api/gui_session/2023-03-12/include/gui_session/gui_session.h:139:34: error: dereferencing type-punned pointer will break strict-aliasing rules [-Werror=strict-aliasing]
  139 |    reinterpret_cast<ARGS &>(nop) = args;
      |    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~
cc1plus: all warnings being treated as errors

Apparently, this is due to a missing -fno-strict-aliasing which is present in the Genode build system but not in Goa.

Since the genodelabs depot archives needed for goa run are only published for the official Sculpt releases, testing/developing for other Genode versions is quite cumbersome. Having a way to configure that the required depot archives shall be taken from another user than genodelabs will greatly simplify this.

Note this configuration is different from the depot_user variable, which defines the depot user for exporting/publishing. The new variable should only affect the archives loaded by goa run. The archives referenced by any Goa package will remain unaffected by this.

Regarding the naming, I am open to suggestions. Maybe something along the line of run_as or run_as_depot_user.

@nfeske, @ssumpf Any suggestions?

While building some library and binary archives on different computers we noticed, that the published archives contain symbolic links to absolute file paths from the machine creating the archive.

I see two ways to fix this.

1. during extract_api_artifacts do not create a symbolic link but copy the files.
   • disadvantage: more disk usage
2. when creating the tar de-reference the files

If there are no objections, I would create and test a patch that implements the second solution.

First of, I'm very excited to see the recent love and well deserved attention being given to Goa. It will help nurture a rich ecosystem in the Genode world.

I was happy to see the article on Rust support being improved in the Genode 23.08 release, such that the official Rust toolchains could be used unmodified with the Goa tool. This is an incredible improvement in user experience. Thanks for working on this @ssumpf, @atopia and others of the Genode team!

I followed the instructions of the https://genodians.org/atopia/2023-08-24-enabling-the-upstream-rust-toolchain article using the 23.08 release of Genode.

The example code worked perfectly using the official Rust toolchain and the Genode 21.05 toolchain:

$ rustup toolchain install stable-x86_64-unknown-linux-gnu
 $ rustup target add x86_64-unknown-freebsd

However, I noticed that the Goa tool is yet to be updated to use the 23.05 Genode toolchain (as it still relies on the Genode 21.05 toolchain):

So, to test it out, I switched from 21.05 to 23.05.

 if {![info exists cross_dev_prefix]} {
        switch $arch {
-       arm_v8a { set cross_dev_prefix "/usr/local/genode/tool/21.05/bin/genode-aarch64-" }
-       x86_64  { set cross_dev_prefix "/usr/local/genode/tool/21.05/bin/genode-x86-"  }
+       arm_v8a { set cross_dev_prefix "/usr/local/genode/tool/23.05/bin/genode-aarch64-" }
+       x86_64  { set cross_dev_prefix "/usr/local/genode/tool/23.05/bin/genode-x86-"  }
        default { exit_with_error "tool-chain prefix is not defined" }
        }
 }

The hello world Rust example gave a link time error when compiled with the 23.05 toolchain, whereas the example compiled and ran perfectly when using the Genode 21.05 toolchain.

$ time goa run -C examples/hello_rust --versions-from-genode-dir genode --depot-dir genode/depot

Error output:

error: linking with `/usr/local/genode/tool/23.05/bin/genode-x86-gcc` failed: exit status: 1
  |
  = note: LC_ALL="C" VSLANG="1033" "/usr/local/genode/tool/23.05/bin/genode-x86-gcc" "-m64" "/tmp/rustclRzYhg/symbols.o" ... "-L.../goa/examples/hello_rust/var/abi/x86_64" "-l:libc.lib.so" "-l:libm.lib.so" "-l:posix.lib.so" "-Wl,--dynamic-linker=ld.lib.so" "-Wl,--dynamic-list=.../goa/share/goa/ld/genode_dyn.dl" "-T" ".../goa/share/goa/ld/genode_dyn.ld" "-l:ld.lib.so"
  = note: .../rustlib/x86_64-unknown-freebsd/lib/libstd-b98b1995c63467c0.rlib(std-b98b1995c63467c0.std.c179cdd7ef8a0dd5-cgu.0.rcgu.o): in function `std::sys::unix::fs::File::file_attr':
          /rustc/1b198b3a196442e14fb06978166ab46a4618d131/library/std/src/sys/unix/fs.rs:1086: undefined reference to `fstat@FBSD_1.0'
          .../rustlib/x86_64-unknown-freebsd/lib/libstd-b98b1995c63467c0.rlib(std-b98b1995c63467c0.std.c179cdd7ef8a0dd5-cgu.0.rcgu.o): in function `std::sys::unix::fs::stat::{{closure}}':
          /rustc/1b198b3a196442e14fb06978166ab46a4618d131/library/std/src/sys/unix/fs.rs:1619: undefined reference to `stat@FBSD_1.0'
          /usr/local/genode/tool/23.05/bin/../lib/gcc/x86_64-pc-elf/12.3.0/../../../../x86_64-pc-elf/bin/ld: /rustc/1b198b3a196442e14fb06978166ab46a4618d131/library/std/src/sys/unix/fs.rs:1619: undefined reference to `stat@FBSD_1.0'
          /usr/local/genode/tool/23.05/bin/../lib/gcc/x86_64-pc-elf/12.3.0/../../../../x86_64-pc-elf/bin/ld: /rustc/1b198b3a196442e14fb06978166ab46a4618d131/library/std/src/sys/unix/fs.rs:1619: undefined reference to `stat@FBSD_1.0'
          .../rustlib/x86_64-unknown-freebsd/lib/libstd-b98b1995c63467c0.rlib(std-b98b1995c63467c0.std.c179cdd7ef8a0dd5-cgu.0.rcgu.o): in function `std::sys::unix::fs::File::file_attr':
          /rustc/1b198b3a196442e14fb06978166ab46a4618d131/library/std/src/sys/unix/fs.rs:1086: undefined reference to `fstat@FBSD_1.0'
          collect2: error: ld returned 1 exit status

  = note: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
  = note: use the `-l` flag to specify native libraries to link
  = note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#cargorustc-link-libkindname)

error: could not compile `hello-rust` (bin "hello-rust") due to previous error
[hello_rust] Error: build via cargo failed: child process exited abnormally

Let me know if any additional information is required to troubleshoot this issue.

$ uname -a
Linux x1 6.4.11-arch2-1 #1 SMP PREEMPT_DYNAMIC Sat, 19 Aug 2023 15:38:34 +0000 x86_64 GNU/Linux

Cheerful summer wishes from Sweden,
Robin

When Sculpt deploys a pkg, it looks up the declaration of ROM content of the new subsystem in all pkg dependencies (see genodelabs/genode#3981). Goa should do the same to avoid the duplication of <content> nodes when customizing an existing pkg.

E.g., when crafting a new runtime based on https://github.com/nfeske/goa-playground/tree/master/unix, e.g., to supplement only make.tar, one can create a new goa project as follows:

pkg/unix_with_make/archives:

nfeske/pkg/unix
nfeske/src/gnumake

pkg/unix_with_make/runtime:

<runtime ram="170M" caps="5000" binary="init" config="unix.config">
	<requires>
		...
	</requires>
	<content>
		<rom label="make.tar"/>
	</content>
</runtime>

raw/vfs.config:

config ld_verbose="yes">
  <vfs>
    <tar name="make.tar" />
    ...
  </vfs>
  ...
</config>

Note that one does not need to supply a copy of unix.config because the ROM from the unix project can be used as is. Also the <content> does not need to list any content that is already declared in unix's runtime.

When trying this with the current version of Goa, Goa complains that the unix.config is missing. Since I have used Goa like that in the past (e.g., https://github.com/nfeske/goa-playground/tree/git/test/unix-git), I guess that the current behavior is a regression that was introduced with adding the sanity checks for the plausibility of the config.

While using Goa for porting a variant of binutils, I encountered the issue that AR and AS where not explicitly passed as environment variables by Goa. So the project's Makefile.in was falling back to a default.

I'd suggest extending the assignments at https://github.com/genodelabs/goa/blob/master/share/goa/lib/build/autoconf.tcl#L40 to cover AS and AR as well.

I recently noticed a bunch of errors when running a test project using qt5_textedit. These errors are caused by unrouteable services such as clipboard ROM/Report. These can be easily resolved emulated by a report_rom component. Furthermore, by adding a black-hole component, we could easily satisfy service requirements such as Audio_in, Audio_out and state reports.

The libSDL2 and friends (like Mesa) are readily available at genode-world. However, to use the library from a Goa project, using CMake in particular, a few pieces are missing in Goa. Fortunately, @ssumpf happens to have those puzzle pieces in a branch https://github.com/ssumpf/goa/commits/doom3.

@ssumpf I would very much appreciate you upstreaming the libSDL2 support! Maybe even along with a simple example of using it with CMake? I'm asking because the latest version of Hatari relies on SDL2 and uses CMake.

So far, goa has only added include paths to library headers to the CPPFLAGS variable, breaking any explicit C compilation rules only using CFLAGS because of unavailable (standard) library headers.

To support explicit use of (only) CFLAGS, add include directories to the CFLAGS variable as well.

Goa follows the scheme that (most) configuration variables can be overridden by command line arguments. However, for some configuration variables the corresponding command-line argument is missing. These are the variables with no equivalent command-line argument:

• ld_march
• cc_march
• bin_dir
• abi_dir
• contrib_dir
• build_dir
• run_dir
• import_dir

• add a heuristic to detect Rust projects via presence of Cargo.toml in the source directory
• implement a little helper to generate dummy static libraries that satisfy linking requirements of the Rust toolchain
• add support for cargo as a build system
• add a Rust hello world example

We want to add a simple testing facility to Goa in order to identify regressions quickly. The current suggestion is that there will be a script in <goa>/share/goa/lib that executes everything in the examples directory and compares the actual output to the expected output (using expect). Any objections or suggestions are welcome.

Currently, Goa uses base-linux for its run stage. In principle, the tool is able to support other targets like qemu or real hardware. Since run/linux.tcl serves as a blueprint for any future run-target implementation, it should undergo some restructuring. Moreover, we should add a bit of documentation about the limitations of the linux target and how custom targets can be added.

I understand that goa import supports populating the raw folder. However, this prevents keeping a manually managed raw/ folder in conjunction with the import feature.