Giter Club home page Giter Club logo

u-bmc's Introduction

u-root

Build Status codecov Go Report Card CodeQL GoDoc Slack License OpenSSF Best Practices

Description

u-root embodies four different projects.

  • Go versions of many standard Linux tools, such as ls, cp, or shutdown. See cmds/core for most of these.

  • A way to compile many Go programs into a single binary with busybox mode.

  • A way to create initramfs (an archive of files) to use with Linux kernels, embeddable into firmware.

  • Go bootloaders that use kexec to boot Linux or multiboot kernels such as ESXi, Xen, or tboot. They are meant to be used with LinuxBoot.

Usage

Make sure your Go version is >= 1.21.

Download and install u-root either via git:

git clone https://github.com/u-root/u-root
cd u-root
go install

Or install directly with go:

go install github.com/u-root/u-root@latest

Note

The u-root command will end up in $GOPATH/bin/u-root, so you may need to add $GOPATH/bin to your $PATH.

Examples

Here are some examples of using the u-root command to build an initramfs.

git clone https://github.com/u-root/u-root
cd u-root

# Build an initramfs of all the Go cmds in ./cmds/core/... (default)
u-root

# Generate an archive with bootloaders
#
# core and boot are templates that expand to sets of commands
u-root core boot

# Generate an archive with only these given commands
u-root ./cmds/core/{init,ls,ip,dhclient,wget,cat,gosh}

# Generate an archive with all of the core tools with some exceptions
u-root core -cmds/core/{ls,losetup}

Important

u-root works exactly when go build and go list work as well.

Note

The u-root tool is the same as the mkuimage tool with some defaults applied.

In the near future, uimage will replace u-root.

Tip

To just build Go busybox binaries, try out gobusybox's makebb tool.

Multi-module workspace builds

There are several ways to build multi-module command images using standard Go tooling.

$ mkdir workspace
$ cd workspace
$ git clone https://github.com/u-root/u-root
$ git clone https://github.com/u-root/cpu

$ go work init ./u-root
$ go work use ./cpu

$ u-root ./u-root/cmds/core/{init,gosh} ./cpu/cmds/cpud

$ cpio -ivt < /tmp/initramfs.linux_amd64.cpio
...
-rwxr-x---   0 root     root      6365184 Jan  1  1970 bbin/bb
lrwxrwxrwx   0 root     root            2 Jan  1  1970 bbin/cpud -> bb
lrwxrwxrwx   0 root     root            2 Jan  1  1970 bbin/gosh -> bb
lrwxrwxrwx   0 root     root            2 Jan  1  1970 bbin/init -> bb
...

# Works for offline vendored builds as well.
$ go work vendor # Go 1.22 feature.

$ u-root ./u-root/cmds/core/{init,gosh} ./cpu/cmds/cpud

When creating a new Go workspace is too much work, the goanywhere tool can create one on the fly. This works only with local file system paths:

$ go install github.com/u-root/gobusybox/src/cmd/goanywhere@latest

$ goanywhere ./u-root/cmds/core/{init,gosh} ./cpu/cmds/cpud -- u-root

goanywhere creates a workspace in a temporary directory with the given modules, and then execs u-root in the workspace passing along the command names.

Tip

While workspaces are good for local compilation, they are not meant to be checked in to version control systems.

For a non-workspace way of building multi-module initramfs images, read more in the mkuimage README. (The u-root tool is mkuimage with more defaults applied.)

Extra Files

You may also include additional files in the initramfs using the -files flag.

If you add binaries with -files are listed, their ldd dependencies will be included as well.

$ u-root -files /bin/bash

$ cpio -ivt < /tmp/initramfs.linux_amd64.cpio
...
-rwxr-xr-x   0 root     root      1277936 Jan  1  1970 bin/bash
...
drwxr-xr-x   0 root     root            0 Jan  1  1970 lib/x86_64-linux-gnu
-rwxr-xr-x   0 root     root       210792 Jan  1  1970 lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
-rwxr-xr-x   0 root     root      1926256 Jan  1  1970 lib/x86_64-linux-gnu/libc.so.6
lrwxrwxrwx   0 root     root           15 Jan  1  1970 lib/x86_64-linux-gnu/libtinfo.so.6 -> libtinfo.so.6.4
-rw-r--r--   0 root     root       216368 Jan  1  1970 lib/x86_64-linux-gnu/libtinfo.so.6.4
drwxr-xr-x   0 root     root            0 Jan  1  1970 lib64
lrwxrwxrwx   0 root     root           42 Jan  1  1970 lib64/ld-linux-x86-64.so.2 -> /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
...

You can determine placement with colons:

$ u-root -files "/bin/bash:sbin/sh"

$ cpio -ivt < /tmp/initramfs.linux_amd64.cpio
...
-rwxr-xr-x   0 root     root      1277936 Jan  1  1970 sbin/sh
...

For example on Debian, if you want to add two kernel modules for testing, executing your currently booted kernel:

$ u-root -files "$HOME/hello.ko:etc/hello.ko" -files "$HOME/hello2.ko:etc/hello2.ko"
$ qemu-system-x86_64 -kernel /boot/vmlinuz-$(uname -r) -initrd /tmp/initramfs.linux_amd64.cpio

Init and Uinit

u-root has a very simple (exchangable) init system controlled by the -initcmd and -uinitcmd command-line flags.

  • -initcmd determines what /init is symlinked to. -initcmd may be a u-root command name or a symlink target.

  • -uinitcmd is run by the default u-root init after some basic file system setup. There is no default, users should optionally supply their own. -uinitcmd may be a u-root command name with arguments or a symlink target with arguments.

  • After running a uinit (if there is one), init will start a shell determined by the -defaultsh argument.

We expect most users to keep their -initcmd as init, but to supply their own uinit for additional initialization or to immediately load another operating system.

All three command-line args accept both a u-root command name or a target symlink path. Only -uinitcmd accepts command-line arguments, however. For example,

u-root -uinitcmd="echo Go Gopher" ./cmds/core/{init,echo,gosh}

cpio -ivt < /tmp/initramfs.linux_amd64.cpio
# ...
# lrwxrwxrwx   0 root     root           12 Dec 31  1969 bin/uinit -> ../bbin/echo
# lrwxrwxrwx   0 root     root            9 Dec 31  1969 init -> bbin/init

qemu-system-x86_64 -kernel $KERNEL -initrd /tmp/initramfs.linux_amd64.cpio -nographic -append "console=ttyS0"
# ...
# [    0.848021] Freeing unused kernel memory: 896K
# 2020/05/01 04:04:39 Welcome to u-root!
#                              _
#   _   _      _ __ ___   ___ | |_
#  | | | |____| '__/ _ \ / _ \| __|
#  | |_| |____| | | (_) | (_) | |_
#   \__,_|    |_|  \___/ \___/ \__|
#
# Go Gopher
# ~/>

Passing command line arguments like above is equivalent to passing the arguments to uinit via a flags file in /etc/uinit.flags, see Extra Files.

Additionally, you can pass arguments to uinit via the uroot.uinitargs kernel parameters, for example:

u-root -uinitcmd="echo Gopher" ./cmds/core/{init,echo,gosh}

cpio -ivt < /tmp/initramfs.linux_amd64.cpio
# ...
# lrwxrwxrwx   0 root     root           12 Dec 31  1969 bin/uinit -> ../bbin/echo
# lrwxrwxrwx   0 root     root            9 Dec 31  1969 init -> bbin/init

qemu-system-x86_64 -kernel $KERNEL -initrd /tmp/initramfs.linux_amd64.cpio -nographic -append "console=ttyS0 uroot.uinitargs=Go"
# ...
# [    0.848021] Freeing unused kernel memory: 896K
# 2020/05/01 04:04:39 Welcome to u-root!
#                              _
#   _   _      _ __ ___   ___ | |_
#  | | | |____| '__/ _ \ / _ \| __|
#  | |_| |____| | | (_) | (_) | |_
#   \__,_|    |_|  \___/ \___/ \__|
#
# Go Gopher
# ~/>

Note the order of the passed arguments in the above example.

The command you name must be present in the command set. The following will not work:

u-root -uinitcmd="echo Go Gopher" ./cmds/core/{init,gosh}
# 21:05:57 could not create symlink from "bin/uinit" to "echo": command or path "echo" not included in u-root build: specify -uinitcmd="" to ignore this error and build without a uinit

You can also refer to non-u-root-commands; they will be added as symlinks. We don't presume to know whether your symlink target is correct or not.

This will build, but not work unless you add a /bin/foobar to the initramfs.

u-root -uinitcmd="/bin/foobar Go Gopher" ./cmds/core/{init,gosh}

This will boot the same as the above.

u-root -uinitcmd="/bin/foobar Go Gopher" -files /bin/echo:bin/foobar -files your-hosts-file:/etc/hosts ./cmds/core/{init,gosh}

The effect of the above command:

  • Sets up the uinit command to be /bin/foobar, with 2 arguments: Go Gopher
  • Adds /bin/echo as bin/foobar
  • Adds your-hosts-file as etc/hosts
  • builds in the cmds/core/init, and cmds/core/gosh commands.

This will bypass the regular u-root init and just launch a shell:

u-root -initcmd=gosh ./cmds/core/{gosh,ls}

cpio -ivt < /tmp/initramfs.linux_amd64.cpio
# ...
# lrwxrwxrwx   0 root     root            9 Dec 31  1969 init -> bbin/gosh

qemu-system-x86_64 -kernel $KERNEL -initrd /tmp/initramfs.linux_amd64.cpio -nographic -append "console=ttyS0"
# ...
# [    0.848021] Freeing unused kernel memory: 896K
# failed to put myself in foreground: ioctl: inappropriate ioctl for device
# ~/>

(It fails to do that because some initialization is missing when the shell is started without a proper init.)

Cross Compilation (targeting different architectures and OSes)

Cross-OS and -architecture compilation comes for free with Go. In fact, every PR to the u-root repo is built against the following architectures: amd64, x86 (i.e. 32bit), mipsle, armv7, arm64, and ppc64le.

Further, we run integration tests on linux/amd64, and linux/arm64, using several CI systems. If you need to add another CI system, processor or OS, please let us know.

To cross compile for an ARM, on Linux:

GOARCH=arm u-root

If you are on OSX, and wish to build for Linux on AMD64:

GOOS=linux GOARCH=amd64 u-root

Testing in QEMU

A good way to test the initramfs generated by u-root is with qemu:

qemu-system-x86_64 -nographic -kernel path/to/kernel -initrd /tmp/initramfs.linux_amd64.cpio

Note that you do not have to build a special kernel on your own, it is sufficient to use an existing one. Usually you can find one in /boot.

If you don't have a kernel handy, you can also get the one we use for VM testing:

go install github.com/hugelgupf/vmtest/tools/runvmtest@latest

runvmtest -- bash -c "cp \$VMTEST_KERNEL ./kernel"

It may not have all features you require, however.

Framebuffer

For framebuffer support, append a VESA mode via the vga kernel parameter:

qemu-system-x86_64 \
  -kernel path/to/kernel \
  -initrd /tmp/initramfs.linux_amd64.cpio \
  -append "vga=786"

For a list of modes, refer to the Linux kernel documentation.

Entropy / Random Number Generator

Some utilities, e.g., dhclient, require entropy to be present. For a speedy virtualized random number generator, the kernel should have the following:

CONFIG_VIRTIO_PCI=y
CONFIG_HW_RANDOM_VIRTIO=y
CONFIG_CRYPTO_DEV_VIRTIO=y

Then you can run your kernel in QEMU with a virtio-rng-pci device:

qemu-system-x86_64 \
    -device virtio-rng-pci \
    -kernel vmlinuz \
    -initrd /tmp/initramfs.linux_amd64.cpio

In addition, you can pass your host's RNG:

qemu-system-x86_64 \
    -object rng-random,filename=/dev/urandom,id=rng0 \
    -device virtio-rng-pci,rng=rng0 \
    -kernel vmlinuz \
    -initrd /tmp/initramfs.linux_amd64.cpio

SystemBoot

SystemBoot is a set of bootloaders written in Go. It is meant to be a distribution for LinuxBoot to create a system firmware + bootloader. All of these use kexec to boot. The commands are in cmds/boot. Parsers are available for GRUB, syslinux, and other config files to make the transition to LinuxBoot easier.

  • pxeboot: a network boot client that uses DHCP and HTTP or TFTP to get a boot configuration which can be parsed as PXELinux or iPXE configuration files to get a boot program.

  • boot: finds all bootable kernels on local disk, shows a menu, and boots them. Supports (basic) GRUB, (basic) syslinux, (non-EFI) BootLoaderSpec, and ESXi configurations.

More detailed information about the build process for a full LinuxBoot firmware image using u-root/systemboot and coreboot can be found in the LinuxBoot book chapter about LinuxBoot using coreboot, u-root and systemboot.

This project started as a loose collection of programs in u-root by various LinuxBoot contributors, as well as a personal experiment by Andrea Barberio that has since been merged in. It is now an effort of a broader community and graduated to a real project for system firmwares.

Compression

You can compress the initramfs. However, for xz compression, the kernel has some restrictions on the compression options and it is suggested to align the file to 512 byte boundaries:

xz --check=crc32 -9 --lzma2=dict=1MiB \
   --stdout /tmp/initramfs.linux_amd64.cpio \
   | dd conv=sync bs=512 \
   of=/tmp/initramfs.linux_amd64.cpio.xz

Getting Packages of TinyCore

Using the tcz command included in u-root, you can install tinycore linux packages for things you want.

You can use QEMU NAT to allow you to fetch packages. Let's suppose, for example, you want bash. Once u-root is running, you can do this:

% tcz bash

The tcz command computes and fetches all dependencies. If you can't get to tinycorelinux.net, or you want package fetching to be faster, you can run your own server for tinycore packages.

You can do this to get a local server using the u-root srvfiles command:

% srvfiles -p 80 -d path-to-local-tinycore-packages

Of course you have to fetch all those packages first somehow :-)

Build an Embeddable u-root

You can build the cpio image created by u-root into a Linux kernel via the CONFIG_INITRAMFS_SOURCE config variable or coreboot config variable, and further embed the kernel image into firmware as a coreboot payload.

In the kernel and coreboot case, you may need to configure ethernet. We have a dhclient command that works for both ipv4 and ipv6. Since v6 does not yet work that well for most people, a typical invocation looks like this:

% dhclient -ipv4 -ipv6=false

Or, on newer linux kernels (> 4.x) boot with ip=dhcp in the command line, assuming your kernel is configured to work that way.

Build Modes

u-root can create an initramfs in two different modes, specified by -build:

  • gbb mode: One busybox-like binary comprising all the Go tools you ask to include. See the gobusybox README for how it works. In this mode, u-root copies and rewrites the source of the tools you asked to include to be able to compile everything into one busybox-like binary.

  • binary mode: each specified binary is compiled separately and all binaries are added to the initramfs.

Updating Dependencies

go get -u
go mod tidy
go mod vendor

Building without network access

The u-root command supports building with workspace vendoring and module vendoring. In both of those cases, if all dependencies are found in the vendored directories, the build happens completely offline.

Read more in the mkuimage README.

u-root also still supports GO111MODULE=off builds.

Hardware

If you want to see u-root on real hardware, this board is a good start.

Contributions

For information about contributing, including how we sign off commits, please see CONTRIBUTING.md.

Improving existing commands (e.g., additional currently unsupported flags) is very welcome. In this case it is not even required to build an initramfs, just enter the cmds/ directory and start coding. A list of commands that are on the roadmap can be found here.

Website

The sources of u-root.org are inside the docs/ directory and are deployed to the gh-pages branch. The CNAME file is currently not part of the CI which deploys to the branch which shall be evaluated if this makes futures deployments easier.

u-bmc's People

Contributors

bluecmd avatar c0d3z3r0 avatar dependabot[bot] avatar hugelgupf avatar keno avatar mdlayher avatar mdr164 avatar orangecms avatar osresearch avatar rjoleary avatar shenki avatar walterchris avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

u-bmc's Issues

Implement fan controller

Being able to set the fans in the console and using RPC would make the hardware much nicer to be around.

For the initial work, just setting a value between 0-100% for duty cycle should be enough. Then a PID controller can be added at a later stage.

Restore MAC from NC-SI

On bootup u-bmc should read MAC address from NC-SI OEM commands and configure the interface accordingly.

Fix reset state on Leopard

After a power cycle the UART does not come back up.
Network works though, so debugging it should be straight forward.

Investigate what registers need to be twiddled to make that work.

Leopard: Ensure production grade

Right now the power LED is turned on straight away, but it would be nice if it can blink fast when verifying the firmware to indicate that the machine should not be booted right now, and blink slow when the machine is off but ready to be powered on.

We also want to capture and output the post code

Figure out automated regression/integration testing

I'm not sure how we want to do this.

The options I've been thinking about are:

  • Compile time redirection to use mocks.
    This is essentially unit tests, and we should do them anyway.

  • Using QEMU.
    QEMU is very limited when it comes to the ast2400 and creating a pseudo platform for it risks missing a lot of things.

  • Using gVisor.
    A lot of work and forces us to re-implement the ast2400 in gVisor.

  • Hardware
    Hardware is the best option, but it's slow to work with. Nonetheless it seems like the most promising candidate right now.

When the u-boot console issue has been solved and we can verify that NC-SI works in u-boot, then it should be possible to netboot from u-boot. That should be fast enough for on-demand testing in some fashion.

Persistent RTC across reboots

The ast2400 has an RTC built-in that is not battery backed up, but using it to carry time across restarts would be a nice reliability addition to allow less RPC downtime on reboots.

System comes up without GPIOs

This has happened twice. This particular one was after a power-cycle:

/# cat /tmp/u-bmc.log 
1970/01/01 00:00:05 Loading system configuration
1970/01/01 00:00:06 Starting debug SSH server
1970/01/01 00:00:06 Initialize system hardware
1970/01/01 00:00:06 Setting up Network Controller Sideband Interface (NC-SI) for eth0
1970/01/01 00:00:06 Starting GPIO drivers
1970/01/01 00:00:06 Starting fan system
1970/01/01 00:00:06 Configuring fan 0 for 20%
1970/01/01 00:00:06 Configuring fan 1 for 20%
1970/01/01 00:00:06 Configuring host UART console /dev/ttyS2 @ 57600 baud
1970/01/01 00:00:06 Starting gRPC interface
1970/01/01 00:00:06 Starting local shell

Reboot worked to restore functionality.

Optional USB subsystem

Some users want USB devices and network connectivity (mainly when using stock BIOS) and some do not. Both sides of the camp have good arguments.

I'd like to support a "u-bmc +usb" type of build where the USB support is compiled in, and one where the USB controller etc. is disabled. Interestingly this has implications not only in code, but also in the .dts file.

Split NCSI netlink functionality into own package

Hello again!

I had previously suggested that we split the NCSI netlink interaction code into its own package to enable further reuse in the Go ecosystem. I'd be happy to do the work and also write some tests (with genltest package) to lock in its behavior.

For now this could live under "u-bmc/pkg/ncsinl" or similar. I suggest the "nl" suffix because, if I understand, NCSI functionality can be manipulated with specially crafted Ethernet frames as well. I don't know how this all ties together.

Let me know if you'd like me to work on this! Thanks again.

DHCPv6

DHCPv6 will be nice for IPv6 only deployments.

While IPv6 can utilize RA for stateless configuration, we need to get a hostname for the TLS certificate.

gRPC reflection broken

There is a weird gRPC issue that I have not debugged much that is possibly related to vendoring. Essentially grpc_cli is no longer able to call any methods, claiming that there is no proto definition.

Sadly I did not save a long but it appears to be reproducible enough.

Will add more details as I get them

DHCPv4

dhclient used to be a bit broken, but we want to run it in process anyway.

Implement DHCPv4 and set hostname accordingly.

Support the NPCM750 EVB

Nuvoton has graciously provided an evaluation board to port u-bmc to. It will serve as a good second SoC to target to make source the abstractions hold (or most likely will cause some refactors).

As this will likely shake the API this is a pre 1.0 thing.

PressButton should block

Right now it returns as soon as the button has started to be pressed. It seems less surprising if it blocks for the duration of the press.

BMC firmware validation / Secure boot

Given that we're targeting specific platforms this is going to be optional and not useful for everyone.

However, it should be possible to create something that's reasonably secure by default.

The idea so far is to:

  • Ensure the BMC SPI flash has a protectable region that is volatile.
    Volatile is to prevent bricking, but the locking also needs to be persistent until power reset without any other option to unlock. For the Leopards I have access to that means replacing the SPI flashes with something like MT25QL256ABA (or even MT25QL512ABB for a nice upgrade).

  • Make u-boot verify Linux kernel, make the Linux kernel verify /init
    U-boot can already do this using signatures, and Linux can be made to do it using a initramfs.
    This would still allow for persistence for files around the /init - but not modifications of /init (bbin/bb) itself.

There are some interesting open issues that needs to be solved.

  1. Need to disable LPC super-duper access (iLPC2AHB) during this time and ASAP on power-up to reduce the timing window for attacks on that. Given that this is on power-on we should be good as the host system does not boot automatically.
  2. U-boot upgrades. How will they work?

Make fails on ssh_keys.go

~/go/src/github.com/u-root/u-bmc: make
make -C cmd/uinit ssh_keys.go
make[1]: Entering directory '/home/hudson/go/src/github.com/u-root/u-bmc/cmd/uinit'
make[1]: *** No rule to make target 'ssh_keys.go'.  Stop.
make[1]: Leaving directory '/home/hudson/go/src/github.com/u-root/u-bmc/cmd/uinit'
Makefile:78: recipe for target 'initramfs.cpio' failed
make: *** [initramfs.cpio] Error 2

There is no Makefile in cmd/uinit

Support different sized flashes

Right now u-bmc assumes 32MB flash. When putting in a 64MB flash u-boot correctly determined the size:

U-Boot 2016.07-00038-gc7a5629a1c (Sep 30 2018 - 22:40:21 +0200)

DRAM:  120 MiB
WARNING: Caches not enabled
Flash: 64 MiB
Using default environment

But the kernel was a bit more certain it was 32MB still:

[    1.567855] aspeed-smc 1e620000.flash-controller: Using 50 MHz SPI frequency
[    1.582237] aspeed-smc 1e620000.flash-controller: n25q512ax3 (65536 Kbytes)
[    1.596190] aspeed-smc 1e620000.flash-controller: CE0 window [ 0x20000000 - 0x24000000 ] 64MB
[    1.613363] aspeed-smc 1e620000.flash-controller: CE1 window [ 0x24000000 - 0x26000000 ] 32MB
[    1.630430] aspeed-smc 1e620000.flash-controller: read control register: 203b0641
[    2.192007] 2 fixed-partitions partitions found on MTD device bmc
[    2.204200] Creating 2 MTD partitions on "bmc":
[    2.213280] 0x000000000000-0x000000080000 : "u-boot"
[    2.228836] 0x000000080000-0x000002000000 : "ubi"

~# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 04000000 00010000 "bmc"
mtd1: 00080000 00010000 "u-boot"
mtd2: 01f80000 00010000 "ubi"

Watchdog heartbeat and reset

Sometimes we get weird lockups in u-boot. Those should of course be fixed, but it would be really nice with a watchdog that just resets the CPU in case where it happens as well as some sort of exported counter on when that has happened.

f20 works for one boot

First boot with the f20 build:

DRAM Init-DDR3                                                                  
CBR0-1234567012345670123456701234567                                            
CBR134Done                                                                      
                                                                                
                                                                                
U-Boot 2016.07-00036-g8b3fd4183a-dirty (Sep 10 2018 - 18:25:46 +0200)           
                                                                                
DRAM:  248 MiB                                                                  
WARNING: Caches not enabled                                                     
Flash: Unsupported SPI Flash!! 0x0019609d                                       
32 MiB                                                                          
Using default environment                                                       
                                                                                
In:    serial                                                                   
Out:   serial                                                                   
Err:   serial                                                                   
Net:   aspeednic#0                                                              
Error: aspeednic#0 address not set.                                             
                                                                                
Hit any key to stop autoboot:  0                                                
ubi0: attaching mtd1                                                            
ubi0: scanning is finished                                                      
                                                                                
. done                                                                          
\       

This hangs after a few thousand spinners and done messages. Second boot produces:

DRAM Init-DDR3                                                                  
CBR0-1234567012345670123456701234567                                            
CBR134Done                                                                      
                                                                                
                                                                                
U-Boot 2016.07-00036-g8b3fd4183a-dirty (Sep 10 2018 - 18:25:46 +0200)           
                                                                                
DRAM:  248 MiB                                                                  
WARNING: Caches not enabled                                                     
Flash: Unsupported SPI Flash!! 0x0019609d                                       
32 MiB                                                                          
Using default environment                                                       
                                                                                
In:    serial                                                                   
Out:   serial                                                                   
Err:   serial                                                                   
Net:   aspeednic#0                                                              
Error: aspeednic#0 address not set.                                             
                                                                                
Hit any key to stop autoboot:  0                                                
ubi0: attaching mtd1                                                            
ubi0: scanning is finished                                                      
                                                                                
. done                                                                          
\                                                                               
(a few hundred spinners)                                           
ubi0: attached mtd1 (name "mtd=1", size 31 MiB)                                 
ubi0: PEB size: 65536 bytes (64 KiB), LEB size: 65408 bytes                     
ubi0: min./max. I/O unit sizes: 1/1, sub-page size 1                            
ubi0: VID header offset: 64 (aligned 64), data offset: 128                      
ubi0: good PEBs: 504, bad PEBs: 0, corrupted PEBs: 0                            
ubi0: user volume: 2, internal volumes: 1, max. volumes count: 128              
ubi0: max/mean erase counter: 1/0, WL threshold: 4096, image sequence number: 97
1138040                                                                         
ubi0: available PEBs: 16, total reserved PEBs: 488, PEBs reserved for bad PEB ha
ndling: 0                                                                       
Loading file 'u-boot.boot.img' to addr 0x40000000...                            
Done                                                                            
## Executing script at 40000000                                                 
Loading file 'u-boot.env' to addr 0x40000000...                                 
Done                                                                            
## Info: input data size = 31706 = 0x7BDA                                       
Loading file 'f06c-leopard-ddr3.dtb' to addr 0x40000000...                      
Done                                                                            
Loading file 'zImage' to addr 0x40008000...                                     
Done                                                                            
Kernel image @ 0x40008000 [ 0x000000 - 0x24e408 ]                               
## Flattened Device Tree blob at 40000000                                       
   Booting using the fdt blob at 0x40000000                                     
   Loading Device Tree to 4f387000, end 4f38fa7b ... OK                         
                                                                                
Starting kernel ...                                                             
                                                                                
[    0.000000] Booting Linux on physical CPU 0x0                                
[    0.000000] Linux version 4.18.5-ubmc (bluecmd@bluecmd) (gcc version 6.3.1 20
170620 (15:6.3.1+svn253039-1)) #14 Fri Sep 14 11:38:36 CEST 2018

This makes it all the way to Linux and the interactive shell. Subsequent boots fail to produce any serial output at all.

i2cwatcher

Like gpiowatcher, i2cwatcher would snoop the BMC CPU to figure out what devices are being talked to, dump commands, replies, etc. to the extent where possible.

This would be useful in mapping undocumented motherboards.

BIOS flashing

Support flashing image to BIOS flash, simple as that.

Implementation is going to be a bit tricky maybe. I'd like to investigate doing BIOS flashing when the host is on, and only giving it control of the flash when it is in the BIOS, and as a soon as it is done reading it don't have it own the BIOS anymore. That will most likely break user-mode BIOS flashing/dumping however.

Still, the BMC is the one that should own the chip, and restricting the host from updating it can be a good thing as well.

OpenMetrics exporter

Link in the https://github.com/prometheus/client_golang library and export metrics.

I've allocated 9370 in https://github.com/prometheus/prometheus/wiki/Default-port-allocations.
It is also free in the IANA port assignment list it seems, if we want to get really serious about it.

Some metrics that I'm thinking that is needed:

  • BMC system load
  • BMC memory usage
  • BMC NC-SI state (low-prio)
  • BMC uptime
  • BMC network stats
  • Host power state
  • Host prochot/memhot/caterr/...
  • Debug board present
  • Mezzanine card present
  • BIOS selected
  • Fan speed
  • Fan percentage output
  • All the various sensors (pending sensor work)
  • SKU ID (Read from GPIO)
  • FRU IDs (I2C?)
  • BIOS version (maybe? Is going to be messy)
  • POST code

A lot of things in the list is doable straight away.

Console streaming

Right now the console streaming is very basic.

  1. Buffered history is needed.

  2. It would be nice if multiple connections are supported and that the RPC becomes indempotemt with some form of timestamp as the identifier.

  3. The same stream would be nice to have for the console output and maybe klog for the BMC itself, instead of having the /tmp/u-bmc.log file.

Flaky bootup on initial boot

I've seen this live but now also in the integration tests (which is excellent).
The full bootlog looks like this:

qemu-system-arm: Aspeed iBT has no chardev backend

DRAM Init-DDR3
CBR0-1357135713571357
CBR134Done


U-Boot 2016.07-00038-gc7a5629a1c (Oct 02 2018 - 21:27:29 +0000)

DRAM:  240 MiB
WARNING: Caches not enabled
Flash: 32 MiB
Using default environment

In:    serial
Out:   serial
Err:   serial
Net:   aspeednic#0
Error: aspeednic#0 address not set.

Hit any key to stop autoboot:  0 
ubi0: attaching mtd1
ubi0: scanning is finished

. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done
|
. done

Then nothing. The CPU hangs.

The easiest thing here is probably figuring out why u-boot wants to "fix" the filesystem in the first place, that would significantly speed up the initial boot after installation as well.

Link flaps on NC-SI when the host is booting

Right now, and in other BMCs as well, the link flaps when the power state changes for the host.

There might be nothing we can do about that, but we should have a look to be sure that's the case.

Upgrade U-boot

We're using OpenBMC's u-boot which is based on 2016. This means that there is no chance that we can upstream any u-boot modifications we would like to do.

u-boot mainline does not support AST24xx series so that would have to be ported, but the good thing is that it's not that much stuff we require from the CPU - so it could maybe be not that hard.

Still, collecting all the contributor material and making sure it's all licensed properly etc is going to be a time sink.

Pad with 0xFF instead of 0x00

Since the flash chips are 0xFF in their erased state it is faster if the empty parts of the ROM are filled with all ones rather than all zeros.

Hack-mode

Right now we compile everything and require a reflash every time something changes.

Given that u-root has a compilation mode for on the fly builds, investigate if it's possible to use that to be able to modify the source on the device to shorten the development iterations.

Submit Leopard device tree upstream (dup)

They ASPEED kernel maintainer eagerly awaits your patch. Ping me on irc if you need help making a kernel submission.

If you have other patches you're using then we should either get them upstream, or in the openbmc tree if they are not yet ready.

In-process sshd

To control lifetime and better adoption of ssh, I think moving the sshd to an in-process routine would be beneficial.

That could potentially help adding visibility for issues like u-root/u-root#957

Readability feedback

As discussed in person, here are a bunch of thoughts regarding readability improvements from a quick glance at the code:

  • A bunch of places in the code panic. They should either return an error, or — typically only for testing — follow the must* naming convention, e.g. mustOpenHostMemory. The same holds for log.Fatal, which also terminates the process, which is probably not desirable, especially as the code grows to fulfill more functions.
  • The syscall package in the standard library is frozen in favor of the golang.org/x/sys/unix. While you might not have a specific reason to use the x/sys/unix package yet (or maybe some constants you’re currently defining are to be found there?), it might be good style to start using it for consistency as the code grows.
  • Currently, flags are defined all over the place. Ideally, flags would only be defined in the main package, which should pass the flag values to the rest of the code. Otherwise, flags will be a hindrance to testing, might conflict, etc.
  • optionally, you might consider replacing some of the Makefiles with “go generate” instructions
  • For testing, it’s not idiomatic to pass *testing.T around. Instead, compare actual and expected state, return a diff, and call t.Errorf (or t.Fatalf) in your test if the diff is non-empty.

This list might not be exhaustive, but that’s what I could find right now :)

Hope that helps!

Something is amiss with vendoring

This is when it's building the busybox:

cmds/uinit/grpc.go:40:36: cannot use g (type *"google.golang.org/grpc".Server) as type *"github.com/u-root/u-bmc/vendor/google.golang.org/grpc".Server in argument to bmc.RegisterManagementServiceServer
, exit status 2

Looks like a vendoring issue.

HTTPS for gRPC

Currently it's all plaintext over port 80.

This should be switch to 443 and using HTTPS.

Generating a self-signed certificate + key and storing it in /etc/ is a good first step. Dealing with CA and certificate SANs and such is going to be annoying, so let's not do that right now.

Submit Leopard kernel patches upstream

They ASPEED kernel maintainer eagerly awaits your device tree patch. Ping me on irc if you need help making a kernel submission.

If you have other patches you're using then we should either get them upstream, or in the openbmc tree if they are not yet ready.

BIOS chip selection

The Leopard has 2 BIOS flashes. Being able to select which one to use is good.

  • Should it be persistent? Or should it be a one time thing?
  • Should it be controllable externally? Holding power while pressing reset or something like that?
  • If the BIOS fails for too long, should it switch over automatically? Probably not, better handled by an external rollout manager.

Fix host bootup

Right now when doing power on this is the sequence that u-bmc goes through:

1970/01/01 00:10:09 CPLD_CLK                       output driving high
1970/01/01 00:10:09 CPLD_CLK                       output driving low
1970/01/01 00:10:10 CPLD_CLK                       output driving high
1970/01/01 00:10:10 CPLD_CLK                       output driving low
1970/01/01 00:10:11 CPLD_CLK                       output driving high
1970/01/01 00:10:11 CPLD_CLK                       output driving low
1970/01/01 00:10:12 BMC_PWR_BTN_OUT_N              output driving low
1970/01/01 00:10:12 CPLD_CLK                       output driving high
1970/01/01 00:10:12 SLP_S3_N                       input became high
1970/01/01 00:10:12 GPIOG3/SGPSI1                  input became high
1970/01/01 00:10:12 GPION3/PWM3/VPIG3              input became high
1970/01/01 00:10:12 CPU0_PROCHOT_N                 input became high
1970/01/01 00:10:12 CPU1_PROCHOT_N                 input became high
1970/01/01 00:10:12 GPIOB1/SALT2                   input became high
1970/01/01 00:10:12 GPIOD6/SD2CD#                  input became high
1970/01/01 00:10:12 CPU_CATERR_N                   input became high
1970/01/01 00:10:12 CPLD_CLK                       output driving low
1970/01/01 00:10:12 GPIOB4/LPCRST#                 input became high
1970/01/01 00:10:12 PCH_PWR_OK                     input became high
1970/01/01 00:10:12 SYS_PWR_OK                     input became high
1970/01/01 00:10:12 GPIOG6/FLBUSY#                 input became high
1970/01/01 00:10:12 GPIOH0/ROMD8/NCTS6             became output
1970/01/01 00:10:12 GPIOH1/ROMD9/NDCD6             became output
1970/01/01 00:10:12 GPIOH2/ROMD10/NDSR6            became output
1970/01/01 00:10:12 GPIOH3/ROMD11/NRI6             became output
1970/01/01 00:10:12 GPIOH4/ROMD12/NDTR6            became output
1970/01/01 00:10:12 GPIOH5/ROMD13/NRTS6            became output
1970/01/01 00:10:12 GPIOH6/ROMD14/TXD6             became output
1970/01/01 00:10:12 GPIOH7/ROMD15/RXD6             became output
1970/01/01 00:10:12 GPIOH0/ROMD8/NCTS6             output driving high
1970/01/01 00:10:12 GPIOH1/ROMD9/NDCD6             output driving high
1970/01/01 00:10:12 GPIOH2/ROMD10/NDSR6            output driving high
1970/01/01 00:10:12 GPIOH3/ROMD11/NRI6             output driving high
1970/01/01 00:10:12 GPIOH4/ROMD12/NDTR6            output driving high
1970/01/01 00:10:12 GPIOH5/ROMD13/NRTS6            output driving high
1970/01/01 00:10:12 GPIOH6/ROMD14/TXD6             output driving high
1970/01/01 00:10:12 GPIOH7/ROMD15/RXD6             output driving high
1970/01/01 00:10:13 GPIOH1/ROMD9/NDCD6             output driving low
1970/01/01 00:10:13 GPIOH2/ROMD10/NDSR6            output driving low
1970/01/01 00:10:13 GPIOH5/ROMD13/NRTS6            output driving low
1970/01/01 00:10:13 GPIOH6/ROMD14/TXD6             output driving low
1970/01/01 00:10:13 GPIOH7/ROMD15/RXD6             output driving low
1970/01/01 00:10:13 BMC_PWR_BTN_OUT_N              output driving high
1970/01/01 00:10:13 CPLD_CLK                       output driving high
1970/01/01 00:10:13 CPLD_CLK                       output driving low
1970/01/01 00:10:14 CPLD_CLK                       output driving high
1970/01/01 00:10:14 CPLD_CLK                       output driving low
1970/01/01 00:10:15 CPLD_CLK                       output driving high
1970/01/01 00:10:15 CPLD_CLK                       output driving low
1970/01/01 00:10:16 CPLD_CLK                       output driving high
1970/01/01 00:10:16 CPLD_CLK                       output driving low

CPLD_CLK is a local modification that pulses GPIOB0 at 1Hz, not sure if that is needed yet.

As can be seen at:

1970/01/01 00:10:12 GPIOH0/ROMD8/NCTS6             became output

the kernel responds to something, maybe the LPCRST# change, and reconfigures stuff. That's most likely not what we want, and could be an issue.

SCU states:

SCU70: 4208c512 SCU80: 00000000 SCU84: 00fff0c0 SCU88: 01000000 SCU8C: 000000ff SCU90: 003fa000 SCU94: 00000000```

RTC

There is an RTC somewhere on the server.

  1. Find what it is, and on which bus it is on
  2. Enable it in the device tree

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.