microsoft / cbl-marinerdemo Goto Github PK

Hi.

After running the command sudo make iso CONFIG_FILE=../imageconfigs/demo_iso.json
I'm getting a 404 not found error for one of the pkgs.

Trying to get the pkg via browser confirms that it does not exist.

Full error from terminal:

ERROR: Failed to download toolchain package: gdbm-debuginfo-1.18-3.cm1.x86_64.rpm.
ERROR: Last 20 lines from log '/home/ohadm/CBL-MarinerDemo/build/logs/toolchain/downloads/gdbm-debuginfo-1.18-3.cm1.x86_64.rpm.log':

Downloading toolchain RPM: gdbm-debuginfo-1.18-3.cm1.x86_64.rpm
--2021-07-21 03:36:35--  https://packages.microsoft.com/cbl-mariner/1.0/prod/base/x86_64/rpms/gdbm-debuginfo-1.18-3.cm1.x86_64.rpm
Resolving packages.microsoft.com (packages.microsoft.com)... failed: Temporary failure in name resolution.
wget: unable to resolve host address ‘packages.microsoft.com’
--2021-07-21 03:36:45--  https://packages.microsoft.com/cbl-mariner/1.0/prod/update/x86_64/rpms/gdbm-debuginfo-1.18-3.cm1.x86_64.rpm
Resolving packages.microsoft.com (packages.microsoft.com)... 104.214.230.139
Connecting to packages.microsoft.com (packages.microsoft.com)|104.214.230.139|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2021-07-21 03:36:46 ERROR 404: Not Found.

Toolchain download failed. See above errors for more details.

Also pic included.

Regards,

https://github.com/microsoft/CBL-MarinerDemo#modify-the-demo-image-kernel

https://github.com/microsoft/CBL-MarinerTutorials/tree/main/build-in-container

I tried running ./build-in-container/mariner-docker-builder.sh -c then ./build-in-container/mariner-docker-builder.sh -b from the root of the repo. It wasn't clear to me where you're meant to run it from (somewhere outside the repo?) based on this:

# Setup the container for 1st use
./CBL-MarinerTutorials/build-in-container/mariner-docker-builder.sh -t

After that, I got an error in the middle of the build, but it was a bit hard to understand what was going on from the end of the output because there's a lot of Go output to scroll through:

------------ Building Mariner toolkit ------------
++ echo '------------ Building Mariner toolkit ------------'
+++ nproc
++ make -j8 -C CBL-Mariner/toolkit package-toolkit BUILD_DIR=/mariner/build CHROOT_DIR=/temp/DockerStage/ CONCURRENT_PACKAGE_BUILDS=12 CONFIG_FILE= LOG_LEVEL=info OUT_DIR=/mariner/out REBUILD_TOOLS=y
make: Entering directory '/mariner/CBL-Mariner/toolkit'
SUDO_USER is not set, running 'go get' as user ''
echo "mariner-official-base.repo mariner-microsoft.repo" | sed -E 's:([^ ]+[.]repo):/mariner/SPECS/mariner-repos/\1:g' | xargs cat > /mariner/CBL-Mariner/toolkit/scripts/sodiff/sodiff.repo
cat: /mariner/SPECS/mariner-repos/mariner-official-base.repo: No such file or directory
cat: /mariner/SPECS/mariner-repos/mariner-microsoft.repo: No such file or directory
make: *** [/mariner/CBL-Mariner/toolkit/scripts/analysis.mk:73: /mariner/CBL-Mariner/toolkit/scripts/sodiff/sodiff.repo] Error 123
...
go: downloading github.com/google/uuid v1.3.0
go: downloading github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8
go: downloading golang.org/x/net v0.18.0
go: downloading github.com/golang-jwt/jwt/v5 v5.0.0
make: Leaving directory '/mariner/CBL-Mariner/toolkit'

I randomly guessed at adding this line to build-in-container/scripts/setup.sh to get a bit further:

         echo "------------ Building Mariner toolkit ------------"
+        cp -r CBL-Mariner/SPECS/mariner-repos SPECS/ && \
         make -j$(nproc) \

Then I got through Adding RPM to worker chroot and eventually it seemed not to find any specs. I left the hello_world_demo in place and thought it would build:

INFO[0003][rpmssnapshot] Generating RPMs snapshot from specs inside (/mariner/SPECS). 
INFO[0003][rpmssnapshot] Found 0 compatible specs.                    
INFO[0003][rpmssnapshot] The specs build 0 packages in total.         

(I was getting this working to have a local dev loop for microsoft/azurelinux#7446, but for the time being the buddy build seems fine.)

I have installed the CBL-Mariner on my laptop and trying to SSH to the same from the laptop console. The host got the ECDSA key fingerprint but after that, it asks me to enter the password. It looks like the received public key is not matching with the stored version at the host. Any assistance you can provide would be greatly appreciated. Please find the log below.

ssh -v [email protected]

OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5
debug1: Connecting to 172.31.88.105 [172.31.88.105] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file C:\Users\v-praraju/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file C:\Users\v-praraju/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file C:\Users\v-praraju/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file C:\Users\v-praraju/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file C:\Users\v-praraju/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file C:\Users\v-praraju/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file C:\Users\v-praraju/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file C:\Users\v-praraju/.ssh/id_ed25519-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file C:\Users\v-praraju/.ssh/id_xmss type -1
debug1: key_load_public: No such file or directory
debug1: identity file C:\Users\v-praraju/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_for_Windows_7.7
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.5
debug1: match: OpenSSH_8.5 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 172.31.88.105:22 as 'root'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: [email protected] MAC: compression: none
debug1: kex: client->server cipher: [email protected] MAC: compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:bxV9a9vziCDEPZjonFKT3WByUxPc+HIql3hrvceQvA8
debug1: read_passphrase: can't open /dev/tty: No such file or directory
The authenticity of host '172.31.88.105 (172.31.88.105)' can't be established.
ECDSA key fingerprint is SHA256:bxV9a9vziCDEPZjonFKT3WByUxPc+HIql3hrvceQvA8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.31.88.105' (ECDSA) to the list of known hosts.
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: pubkey_prepare: ssh_get_authentication_socket: No such file or directory
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,[email protected],ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected]>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: publickey

debug1: Next authentication method: keyboard-interactive
debug1: read_passphrase: can't open /dev/tty: No such file or directory
Password:

I am doing a little testing - albeit outside the tested "x86_64 based machine using Ubuntu 18.04." I've made progress, but hit an issue with one failing test while building the toolkit.

Environment

I am testing in a 20.04 Gitpod environment - mimicking the 18.04 pre-requisites.

FROM gitpod/workspace-full:latest
RUN sudo apt update && \
    sudo apt upgrade --assume-yes && \
    sudo apt install rpm --assume-yes && \
    sudo apt install qemu-utils --assume-yes && \
    sudo apt install genisoimage --assume-yes && \
    sudo apt install pigz --assume-yes && \
    sudo ln -vsf /home/gitpod/go/bin/go /usr/bin/go

The above gives me the following as a starting point.

GNU Make 4.2.1
tar (GNU tar) 1.30
GNU Wget 1.20.3 built on linux-gnu.
curl 7.68.0 (x86_64-pc-linux-gnu) libcurl/7.68.0 OpenSSL/1.1.1f zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh/0.9.3/openssl/zlib nghttp2/1.40.0 librtmp/2.3
RPM version 4.14.2.1
bash: qemu-utils: command not found
go version go1.16.5 linux/amd64
genisoimage 1.1.11 (Linux)
Python 3.8.11
bison (GNU Bison) 3.5.1
GNU Awk 5.0.1, API: 2.0 (GNU MPFR 4.0.2, GNU MP 6.2.0)
pigz 2.4
Docker version 19.03.15, build 99e3ed8919

Build Context

I am building the toolkit, with:

git clone https://github.com/microsoft/CBL-Mariner.git
pushd CBL-Mariner/toolkit
git checkout 1.0-stable
sudo make package-toolkit REBUILD_TOOLS=y
popd

At line 345 of CBL-Mariner/toolkit/tools/internal/safechroot/safechroot.go

	343:	// Remove this chroot from the list of active ones since it has now been cleaned up.
	344:	// Create a new slice that is -1 capacity of the current activeChroots.
	345:	newActiveChroots := make([]*Chroot, emptyLen, len(activeChroots)-1)

there is an unexpected error:

ERRO[0000] Mount failed on (/workspace/yarely/CBL-Mariner/toolkit/tools/internal/safechroot/_tmp/TestInitializeShouldCreateRoot/dev). Error: operation not permitted 
WARN[0000] Error creating mountpoints for chroot        
--- FAIL: TestInitializeShouldCreateRoot (0.00s)
    safechroot_test.go:62: 
                Error Trace:    safechroot_test.go:62
                Error:          Received unexpected error:
                                operation not permitted
                Test:           TestInitializeShouldCreateRoot
    safechroot_test.go:71: 
                Error Trace:    safechroot_test.go:71
                Error:          Should be true
                Test:           TestInitializeShouldCreateRoot
panic: runtime error: makeslice: cap out of range [recovered]
        panic: runtime error: makeslice: cap out of range
goroutine 21 [running]:
testing.tRunner.func1.2(0x5ea340, 0x665190)
        /home/gitpod/go/src/testing/testing.go:1143 +0x332
testing.tRunner.func1(0xc00008b200)
        /home/gitpod/go/src/testing/testing.go:1146 +0x4b6
panic(0x5ea340, 0x665190)
        /home/gitpod/go/src/runtime/panic.go:965 +0x1b9
microsoft.com/pkggen/internal/safechroot.(*Chroot).Close(0xc0000a3410, 0xc00008b200, 0x0, 0x0)
        /workspace/yarely/CBL-Mariner/toolkit/tools/internal/safechroot/safechroot.go:345 +0x1c8
microsoft.com/pkggen/internal/safechroot.TestInitializeShouldCreateRoot(0xc00008b200)
        /workspace/yarely/CBL-Mariner/toolkit/tools/internal/safechroot/safechroot_test.go:75 +0x251
testing.tRunner(0xc00008b200, 0x6371f8)
        /home/gitpod/go/src/testing/testing.go:1193 +0xef
created by testing.(*T).Run
        /home/gitpod/go/src/testing/testing.go:1238 +0x2b3

If I am reading this correctly, the issue is with releasing the chrooted enviroment (vs creating it). I don't know if this is specific to my Gitpod container, 20.04 or other. Happy to test a further if you have pointers.

• Ubuntu 22.04.03 LTS
• x86_64
• Clean OS Install

user@machine:~/local/repos/CBL-Mariner/toolkit$ sudo make package-toolkit REBUILD_TOOLS=y
Makefile:17: CONFIG_FILE is undefined, defaulting to toolkit's core-efi.json.
make: setfacl: No such file or directory

/home/user/local/repos/CBL-Mariner/toolkit/scripts/tools.mk:27: *** Go version 'go1.17.13' is less than minimum required version 'go1.19'. Stop

https://github.com/microsoft/CBL-MarinerDemo/blob/001337e5fc2ad4b85e616b5121b4469f77794199/README.md?plain=1#L382

In the formatted markdown file the FTP link to the gnuchess tarball source shows up as a plain "here" text, no hyperlink, which is confusing to the reader. It looks like github's markdown formatter doesn't support FTP links. Perhaps the README could reference the HTTPS link (https://ftp.gnu.org/gnu/chess/)

after executed "sudo make package-toolkit REBUILD_TOOLS=y", I got Makefile:13: CONFIG_FILE is empty, defaulting to toolkit's core-efi.json unless CONFIG_FILE="" was set explicitly.
cd /home/my/CBL-Mariner/toolkit/tools/ &&
go test -covermode=atomic -coverprofile=/home/my/CBL-Mariner/build/tools/internal.test_coverage ./...
go: github.com/alecthomas/[email protected]: Get "https://proxy.golang.org/github.com/alecthomas/template/@v/v0.0.0-20190718012654-fb15b899a751.mod": dial tcp 216.58.200.49:443: i/o timeout
/home/my/CBL-Mariner/toolkit/scripts/tools.mk:89: recipe for target '/home/my/CBL-Mariner/build/tools/internal.test_coverage' failed

No matter what I setted proxy_all, it doesn't work. I tested curl www.google.com successfully. I don't know what to do

When downloading the toolchain via make build-packages

[azaugg@azaugg-ld3] [08:48:17] ~/CBL-Mariner/toolkit [2.0] $ sudo make build-packages CONFIG_FILE= Downloading toolchain RPM: filesystem-1.1-12.cm2.x86_64.rpm Downloading toolchain RPM: glibc-2.35-3.cm2.x86_64.rpm Downloading toolchain RPM: glibc-devel-2.35-3.cm2.x86_64.rpm Downloading toolchain RPM: glibc-i18n-2.35-3.cm2.x86_64.rpm Downloading toolchain RPM: glibc-iconv-2.35-3.cm2.x86_64.rpm Downloading toolchain RPM: glibc-lang-2.35-3.cm2.x86_64.rpm Downloading toolchain RPM: glibc-nscd-2.35-3.cm2.x86_64.rpm Downloading toolchain RPM: glibc-tools-2.35-3.cm2.x86_64.rpm Downloading toolchain RPM: zlib-1.2.12-2.cm2.x86_64.rpm Downloading toolchain RPM: zlib-devel-1.2.12-2.cm2.x86_64.rpm Downloading toolchain RPM: file-5.40-2.cm2.x86_64.rpm Downloading toolchain RPM: file-devel-5.40-2.cm2.x86_64.rpm Downloading toolchain RPM: file-libs-5.40-2.cm2.x86_64.rpm Downloading toolchain RPM: binutils-2.37-5.cm2.x86_64.rpm \nERROR: Failed to download toolchain package: binutils-2.37-5.cm2.x86_64.rpm. ERROR: Last 20 lines from log '/home/azaugg/CBL-Mariner/build/logs/toolchain/downloads/binutils-2.37-5.cm2.x86_64.rpm.log':\n Downloading toolchain RPM: binutils-2.37-5.cm2.x86_64.rpm https://packages.microsoft.com/cbl-mariner/2.0/prod/base/x86_64/binutils-2.37-5.cm2.x86_64.rpm: 2023-02-22 20:48:26 ERROR 404: Not Found. https://packages.microsoft.com/cbl-mariner/2.0/prod/base/debuginfo/x86_64/binutils-2.37-5.cm2.x86_64.rpm: 2023-02-22 20:48:27 ERROR 404: Not Found. https://packages.microsoft.com/cbl-mariner/2.0/prod/Microsoft/x86_64/binutils-2.37-5.cm2.x86_64.rpm: 2023-02-22 20:48:27 ERROR 404: Not Found. \nToolchain download failed. See above errors for more details. make: *** [/home/azaugg/CBL-Mariner/build/rpm_cache/cache/x86_64/binutils-2.37-5.cm2.x86_64.rpm] Error 1

18.04 does not come with git.
Need to add to apt-get install list.