How do we add 'blocked by Issue #8' (assuming that's appropriate)?

This command will build kmodules for xenomai-kernel against kernel in KERNELDIR:

make threads=xenomai-kernel KERNELDIR=/usr/src/kernels/3.5.7-5.el6.xenomai.x86_64

Even if already up to date, it will always rebuild all modules. Something messed up related to .separate-kbuild-artifacts-hack.stamp.

ceil() is needed by orient.ko

mah@ubuntu-10:/bighome/mah/emc2-rtos/rtlib/xenomai-kernel/3.5.7-xenomai-2.6.2.1$ nm xeno_math.ko |grep ceil
mah@ubuntu-10:/bighome/mah/emc2-rtos/rtlib/xenomai-kernel/3.5.7-xenomai-2.6.2.1$ nm orient.ko |grep ceil
         U ceil

(I dont understand the below - still ;)

make[2]: Entering directory `/usr/src/linux-headers-3.5.7-xenomai-2.6.2.1'
  Building modules, stage 2.
  MODPOST 143 modules
make[2]: Leaving directory `/usr/src/linux-headers-3.5.7-xenomai-2.6.2.1'
WARNING: "fabs" [/home/mah/emc2-ub/src/objects/xenomai-kernel/3.5.7-xenomai-2.6.2.1/steptest.ko] undefined!
WARNING: "floor" [/home/mah/emc2-ub/src/objects/xenomai-kernel/3.5.7-xenomai-2.6.2.1/sim_spindle.ko] undefined!
WARNING: "ceil" [/home/mah/emc2-ub/src/objects/xenomai-kernel/3.5.7-xenomai-2.6.2.1/sim_spindle.ko] undefined!
WARNING: "floor" [/home/mah/emc2-ub/src/objects/xenomai-kernel/3.5.7-xenomai-2.6.2.1/orient.ko] undefined!
WARNING: "ceil" [/home/mah/emc2-ub/src/objects/xenomai-kernel/3.5.7-xenomai-2.6.2.1/orient.ko] undefined!
WARNING: "pow" [/home/mah/emc2-ub/src/objects/xenomai-kernel/3.5.7-xenomai-2.6.2.1/joyhandle.ko] undefined!
WARNING: "pow" [/home/mah/emc2-ub/src/objects/xenomai-kernel/3.5.7-xenomai-2.6.2.1/genserkins.ko] undefined!
WARNING: "floor" [/home/mah/emc2-ub/src/objects/xenomai-kernel/3.5.7-xenomai-2.6.2.1/bldc.ko] undefined!

Six or seven runtests fail consistently on the buildbot. There's a definite pattern to them. Fix them.

these are now default - see changes to src/rtapi/rtapi_bitops.h

we currently do not have a runtest which exercises the userland threads PCI support logic, which is why the bug Andy noticed went under the runtests radar

downside: talking to PCI hardware during a runtest is potentially touchy

Something Jepler was saying that I don't want to forget:

http://linuxcnc.mah.priv.at/irc/%23linuxcnc-devel/2013-07-02.html#14:22:51

Michael?

root@mba:/tmp# comp --install hal_picnc.comp
/usr/local/share/linuxcnc/Makefile.modinc:68: /usr/local/src/Makefile.inc: No such file or directory
make: *** No rule to make target `/usr/local/src/Makefile.inc'. Stop.

Right now, the universal build (as of aa7c525) always build all detected flavors, regardless of ./configure --with- arguments.

Desired behavior:

• no flags: build all detected flavors
• --with- [...]: build only [...], even if other flavors could be built
• --without-: build all detected flavors except
• --with- --without-: just as if only --with- were supplied

Schooner/Mick reports all 128 runtests ok on his self-built rtai-3.5.7 kernel

his config is here: http://static.mah.priv.at/public/config-3.5.7-rtai

I get a ton of errors trying the same with Seb's 3.5.7 kernel from here: http://highlab.com/~seb/linuxcnc/rtai-for-3.5-prerelease/

obviously a difference in the kernel

this is fishy too:

Note that in Seb's rtai-modules-3.5.7-2-rtai 3.9+2013.04.14ubuntu1 module package rtai_fifos and rtai_sem are missing:
I had to disable them in etc/linuxcnc/rtapi.ini to get realtime to even start (just to fail later in a shm operation):

mah@amd64:/usr/realtime-3.5.7-2-rtai/modules$ ls -l rtai*ko
-rw-r--r-- 1 root root 137793 Apr 15 05:27 rtai_calibrate.ko
-rw-r--r-- 1 root root 321677 Apr 15 05:27 rtai_hal.ko
lrwxrwxrwx 1 root root 13 Apr 15 05:27 rtai_ksched.ko -> rtai_sched.ko
-rw-r--r-- 1 root root 2231120 Apr 15 05:27 rtai_lxrt.ko
-rw-r--r-- 1 root root 325793 Apr 15 05:27 rtai_math.ko
lrwxrwxrwx 1 root root 13 Apr 15 05:27 rtai_mup.ko -> rtai_sched.ko
-rw-r--r-- 1 root root 2231153 Apr 15 05:27 rtai_sched.ko
-rw-r--r-- 1 root root 88570 Apr 15 05:27 rtai_smi.ko
lrwxrwxrwx 1 root root 13 Apr 15 05:27 rtai_smp.ko -> rtai_sched.ko
lrwxrwxrwx 1 root root 13 Apr 15 05:27 rtai_up.ko -> rtai_sched.ko

this is the output on Mick's build:

-rw-r--r-- 1 root root 13132 2013-06-17 10:16 rtai_bits.ko
-rw-r--r-- 1 root root 6324 2013-06-17 10:16 rtai_calibrate.ko
-rw-r--r-- 1 root root 39759 2013-06-17 10:16 rtai_fifos.ko
-rw-r--r-- 1 root root 29875 2013-06-17 10:16 rtai_hal.ko
lrwxrwxrwx 1 root root 13 2013-06-17 10:16 rtai_ksched.ko -> rtai_sched.ko
-rw-r--r-- 1 root root 117215 2013-06-17 10:16 rtai_lxrt.ko
-rw-r--r-- 1 root root 75508 2013-06-17 10:16 rtai_math.ko
-rw-r--r-- 1 root root 16956 2013-06-17 10:16 rtai_mbx.ko
-rw-r--r-- 1 root root 17945 2013-06-17 10:16 rtai_mq.ko
-rw-r--r-- 1 root root 37300 2013-06-17 10:16 rtai_msg.ko
lrwxrwxrwx 1 root root 13 2013-06-17 10:16 rtai_mup.ko -> rtai_sched.ko
-rw-r--r-- 1 root root 33670 2013-06-17 10:16 rtai_netrpc.ko
-rw-r--r-- 1 root root 117216 2013-06-17 10:16 rtai_sched.ko
-rw-r--r-- 1 root root 37239 2013-06-17 10:16 rtai_sem.ko
-rw-r--r-- 1 root root 18035 2013-06-17 10:16 rtai_serial.ko
-rw-r--r-- 1 root root 17381 2013-06-17 10:16 rtai_shm.ko
-rw-r--r-- 1 root root 4119 2013-06-17 10:16 rtai_smi.ko
lrwxrwxrwx 1 root root 13 2013-06-17 10:16 rtai_smp.ko -> rtai_sched.ko
-rw-r--r-- 1 root root 18386 2013-06-17 10:16 rtai_tasklets.ko
-rw-r--r-- 1 root root 11520 2013-06-17 10:16 rtai_tbx.ko
lrwxrwxrwx 1 root root 13 2013-06-17 10:16 rtai_up.ko -> rtai_sched.ko

Note that in Seb's rtai-modules-3.5.7-2-rtai 3.9+2013.04.14ubuntu1 module package rtai_fifos and rtai_sem are missing:
I had to disable them in etc/linuxcnc/rtapi.ini to get realtime to even start:
#################################################

_cfg_flavor flavor section

[flavor_rtai-kernel]

These values do not normally need to be changed.

rtapi_app=/src/emc2-ub/libexec/rtapi_app_rtai-kernel
RTS=/usr/realtime-3.5.7-2-rtai/bin/rtai-config
RTDIR=/usr/realtime-3.5.7-2-rtai/modules
#MODULES=rtai_hal rtai_sched rtai_fifos rtai_sem rtai_math
MODULES=rtai_hal rtai_sched rtai_math

This was a weird one, since 'runtests' runs itself.

See commit e3237db for more details when this was broken and fixed recently.

RIP kernel modules are installed into ../rtlib/<flavor>/<kver>. The rtapi.conf and realtime scripts are already updated to load modules from this location for RIP.

The system installation by make install changes the picture:

• Kernel modules normally (RTAI is the usual exception) installed under /lib/modules/<kver>
• (At least Fedora) Distro packaging guidelines require kernel modules to be under that directory
• When depmod -a is run for any reason, symbols for all modules, incl. those under e.g. /lib/modules/<kver>/linuxcnc, will be read to compute modules.dep.
• This introduces the possibility of loading modules using 'modprobe' rather than 'insmod', which would simplify the realtime scripts greatly were the insmod complexity not required for RIP.
• This may influence the existing scheme of a module search path, currently <rtlibdir>/<flavor>/<kver>:<rtlibdir>/<flavor>:<rtlibdir>; when <rtlibdir> = /lib/modules/<kver>/linuxcnc, searching /lib/modules/<kver>/linuxcnc/<flavor>/<kver> doesn't make sense.

Assuming no input from others, I plan to put kernel modules into /lib/modules/<kver>/linuxcnc and modify the realtime scripts to load them, with the idea that a solution will come to mind in the process.

Agreed upon in 07/27 #linuxcnc-devel discussion with Seb, cradek, mhaberler, zultron

Fake author names like 'Joe Chipcutter' in git commits.

• find all the fake names
• identify real people
• rebase with real names

non-RIP runs error out:

Jun 14 14:55:28 foo rtapi:0: ${exec_prefix}/lib/linuxcnc/rtapi.so:
cannot locate module rtapi: No such file or directory

The problem is setting EMC_RTLIB_DIR to ${libdir}/linuxcnc. The shell
expands this to ${exec_prefix}/lib/linuxcnc, although
we expect (or at least hope) that it would expand to its final
/usr/lib/linuxcnc. In a Makefile, that happens, since there's one more
level of evaluation in the subshells make forks, but the unexpanded
value is included directly in config.h, causing problems for
module_path(), called from sim_rtapi_app.cc.

I propose (once again) that we do away with the module_path()
functions, use a single module directory located with an rpath, and use
common modules except for rtapi.so and ulapi.so, which would be given
new unique names based on flavor.

Michael and I have argued about this several times already, but this time
there's a new fact that has become clear as a result of the universal
build: with the exception of the rtapi.so and ulapi.so modules, other
modules are binary identical, which means they are indeed
interchangeable.

[Opened on behalf of mhaberler]

I have configured

./configure --with-xenomai --with-posix

those symbols.* runtests fail:

Runtest: 128 tests run, 126 successful, 2 failed + 0 expected
Failed: 
    /bighome/mah/emc2-rtos/tests/symbols.0
    /bighome/mah/emc2-rtos/tests/symbols.1

like so:

mah@ubuntu-10:/bighome/mah/emc2-rtos/tests/symbols.0$ cat stderr 
+ set -xe
+ comp --install test_define.comp
make: *** No rule to make target `modules'.  Stop.

I think it's because modules is a target only if BUILD_ALL_FLAVORS == yes?

ifeq ($(BUILD_ALL_FLAVORS),yes)
# Top-level modules target
#
# Re-run 'make modules', once for each configured userland threads
# flavor, and once for each detected kernel for each configured kernel
# threads flavor.  Set 'threads' and where applicable 'KERNELDIR'.
#
# Following tradition, this incarnation of the modules recipe is
# placed far away from all others.
FLAVORUP = $(shell echo $(1) | tr a-z- A-Z_)
FLAVORVAR = $($(call FLAVORUP,$(1))_THREADS_$(2))
modules:

• usage of shmdrv is per-machine, not per-config!! that is a tough one.
• realtime/Load: deal with shmdrv properly

Some module isn't cleaning up after itself?

In any case, this was breaking 'realtime stop', which unloaded shmdrv when no allocated segments existed.

I'm committing a workaround 339cd6db which unloads shmdrv when no open segments exist, and which prints a warning in case any allocated segments still exist.

Michael, can you shed some light on what's going on, and what should go on here?

in particular to copy the files:

/etc/security/limits.d/linuxcnc.conf
/etc/udev/rules.d/50-LINUXCNC-shmdrv.rules: rtapi/shmdrv/shmdrv.rules

reported by Chris Morley, reproduced on 10.04lts/ubuntu/atom

sequence:

DEBUG=5 realtime start

$ halcmd -f -k -V
HAL: initializing component 'halcmd24734'
halcmd: loadrt classiclader_rt
:1: /home/mah/emc2-dynload/libexec/rtapi_app_posix exited without becoming ready
:1: insmod failed, returned -1
See the log and output of 'dmesg' for more information.
log:

Jul 26 15:07:38 atom msgd: 0: hal_lib:24865:user /home/mah/emc2-dynload/libexec/rtapi_app_posix
Jul 26 15:07:38 atom msgd: 0: hal_lib:24865:user --instance=0
Jul 26 15:07:38 atom msgd: 0: hal_lib:24865:user load
Jul 26 15:07:38 atom msgd: 0: hal_lib:24865:user classiclader_rt
Jul 26 15:07:38 atom msgd: 0: hal_lib:24865:user
Jul 26 15:09:17 atom msgd: 0: rtapi_app:24841:user classiclader_rt: dlopen: classiclader_rt.so: cannot open shared object file: No such file or directory

the RUNPATH looks ok:

$ /usr/bin/objdump -p ../libexec/rtapi_app_posix | egrep 'RPATH|RUNPATH'
RPATH /home/mah/emc2-dynload/rtlib/posix:/home/mah/emc2-dynload/lib

$ ls -l /home/mah/emc2-dynload/rtlib/posix/classicladd*
-rwxr-xr-x 1 mah mah 104144 2013-07-26 14:10 /home/mah/emc2-dynload/rtlib/posix/classicladder_rt.so

very weird: the same sequence works fine on another 10.04LTS/ubuntu/i386 combo

probably need a bigger gun - like running rtapi_app under strace to see which paths it considers

currently rtapi_app fails if the linuxcnc user isnt in the xenomai group because various xenomai calls will fail

fail early - for now in scripts/realtime; eventually in rtapistat which checks preconditions for startup (not yet merged)

also, maybe add test to configure.in and/or Makefile 'sudo make setuid' step to check for xenomai gid membership

add reference to:

• current runtests status
• this issue tracker

xenomai-kernel threads; re-running succeeds:

failed test 9: pattern 'hm2/hm2_test.0: IDROM ClockLow is [[:digit:]]+, that's too low' not found

This is on my branch with realtime.in cleanups, based on -stab2, so feel free to throw this back at me.

halrun -f sometimes fails (at least with xenomai-kernel) like so, also during runtests
never observed on rtai-kernel

tests/and-or-not-mux.0$ halrun -f test.hal
mah@cnc:~/emc2-ub/tests/and-or-not-mux.0$ halrun -f test.hal
msgd:0 stopped
hal_lib not loaded
rtapi not loaded
xeno_math not loaded
shmdrv not loaded
insmod: error inserting '/home/mah/emc2-ub/rtlib/xenomai-kernel/3.5.7-xenomai-2.6.2.1/hal_lib.ko': -1 Invalid parameters

Inserting a delay after each iteration of the insmod loop in scripts/realtime makes this go away:
# load kernel modules
for MOD in $MODULES; do
# if loading rtapi.ko, tack on parameters
if test $MOD = rtapi; then
$linuxcnc_module_helper insert $MOD
rtapi_instance=$INSTANCE $NAME_CMD || return $?
else
$linuxcnc_module_helper insert $MOD || return $?
fi
sleep .1 # here
done

the error also does not show on runtests on a VM, hinting it depends on speed

dmesg shows:

[67881.130351] Xenomai math [xeno_math] loaded
[67881.130351] shmdrv: invalid size 0 pid 0
[67881.248008] threads: Unknown symbol hal_create_thread (err 0)
[67881.248008] threads: Unknown symbol hal_ready (err 0)
[67881.248008] threads: Unknown symbol hal_init_mode (err 0)
[67881.248008] threads: Unknown symbol hal_exit (err 0)

The second line is a hint hal_lib didnt initalize correctly and hence hal_lib failed to load properly

report by Gemi Orcullo:

PLATFORM: Raspberry Pi
KERNEL: 3.8.13
XENOMAI: v2.6.2.1
CONFIG: --with-platform=raspberry --with-posix --with-xenomai --disable-build-documentation --disable-usermode-pci

Runtest: 128 tests run, 125 successful, 3 failed + 0 expected
Failed:
/home/rpi/linuxcnc/tests/hm2-idrom
/home/rpi/linuxcnc/tests/toolchanger/toolno-pocket-differ/nonrandom
/home/rpi/linuxcnc/tests/toolchanger/toolno-pocket-differ/random

toolno-pocket-differ/nonrandom result: http://pastebin.com/DVLpVcJi

toolno-pocket-differ/random result: http://pastebin.com/cs1AS59V

After some investigation, it starts to look like the test is producing the expected output.

However, there's a lot of other junk that looks like error messages. Lines matching expected output do appear in /var/log/messages; this needs to be followed up on.

git log -S'<<<<<' master..unified-build-candidate-0

'make install' needs modification to put RTAPI modules in an appropriate location.

rtapi.conf and realtime scripts need modification to load and unload the RT environment

As of 32334bd, non-RIP installs will break unless autoconf variables like exec_prefix are explicitly defined in ./configure args. This is because ./configure seems to perform only one level of expansion, so a variable like ${libexecdir} expands to ${exec_prefix}/libexec; any further expansion needs to be performed outside of the configure script. Of course no further expansion happens in config.h, python or tcl files, so things break.

32334bd goes some way to solve this problem. Config variables are pulled out of e.g. config.h into rtapi.ini in completely expanded form. Consumers of these variables should use the facilities to read from the config file, but this is far from being finished:

• ${prefix} is defined in configure.in; this should not happen
• RIP and non-RIP builds define variables differently (this begs for breakage)
• still many path variables in config.h need to be pulled into rtapi.ini
• python and tcl need to learn to read rtapi.ini

http://buildbot.linuxcnc.org/buildbot/builders/precise-x86-xenomai-rip/builds/320/steps/compile/logs/stdio

Update:

Workaround in place, as below; however, Shabby and memleak may have this fixed in RTAI, so we should look at pulling in an update of the xeno_math code. Reopening.

module 'shmdrv.ko' not found in directory
/tmp/git-jman/linuxcnc-src/rtlib/xenomai/3.5.7-5.el6.xenomai.x86_64
/tmp/git-jman/linuxcnc-src/libexec/linuxcnc_module_helper: Unable to locate module 'shmdrv'

...
Compiling rtapi/posix/rtapi_app.cc
g++ -c -I. -Ilibnml/linklist -Ilibnml/cms -Ilibnml/rcs -Ilibnml/inifile -Ilibnml/os_intf -Ilibnml/nml -Ilibnml/buffer -Ilibnml/posemath -Irtapi -Irtapi_export -Ihal/drivers -Ihal -Iemc/nml_intf -Iemc/kinematics -Iemc/motion -Iemc/ini -Iemc/rs274ngc -Iemc/pythonplugin -I/usr/include/python2.6 -Os -g -Wall -DULAPI -UULAPI -I. -pthread -DTHREAD_FLAVOR_ID=0 -DRTAPI -D_GNU_SOURCE -D_FORTIFY_SOURCE=0
-MP -MD -MF "objects/rtapi/posix/rtapi_app.d" -MT "objects/rtapi/posix/rtapi_app.o"
rtapi/posix/rtapi_app.cc -o objects/rtapi/posix/rtapi_app.o

notice ' -DULAPI -UULAPI -DRTAPI'

technically rtapi_app is neither ULAPI or RTAPI, but it might be some usage of rtapi.h might require it

mah to investigate

root@mba:/tmp# comp --compile hal_picnc.comp
Compiling realtime hal_picnc.c
hal_picnc.c:64:5: error: unknown type name 'int64_t'
hal_picnc.comp:79:2: error: #error "This driver is for the Raspberry Pi platform only"
hal_picnc.comp: In function 'read_spi':
hal_picnc.comp:163:8: error: 'uint32_t' undeclared (first use in this function)
hal_picnc.comp:163:8: note: each undeclared identifier is reported only once for each function it appears in
hal_picnc.comp:163:17: error: expected ')' before 'rxBuf'
hal_picnc.comp: In function 'reset_board':
hal_picnc.comp:393:2: error: unknown type name 'uint32_t'
hal_picnc.comp: In function 'setup_gpio':
hal_picnc.comp:460:2: error: unknown type name 'uint32_t'
hal_picnc.comp: In function 'restore_gpio':
hal_picnc.comp:504:2: error: unknown type name 'uint32_t'
make: *** [hal_picnc.o] Error 1

Be sure the RTAI amd64 workaround for older systems is still in place for the universal build.

not many people will use it right away and the package deps might be troublesome:

sudo apt-get install libboost-serialization-dev libboost-thread-dev

Almost no documentation about RTOS and the Universal Build has been written.

the compiled-in HAL thread stack size (32k) is already a startup parameter; document it.

reenable the make flags for frame size checks just below that limit.

Kinda lost track of whether the deps are being built properly. Needs recheck.

several places use firmware path; at least the hm2 driver and hal_pru_generic.c

the path should be /lib/firmware/{hm2,pru}/pathname, but ideally overridable by an
environment variable FIRMWARE

NB: it should be avoided to link in get_rtapi_config() into RT components, since this
injects the c++ runtime environment into RT

root@mba:/tmp/HAL# comp --install hal_picnc.comp 
Compiling realtime hal_picnc.c
Linking hal_picnc.so
cp hal_picnc.so /usr/src/linuxcnc/rtlib/posix/

LCNC was configured with

./configure --with-posix --with-xenomai 

Please ignore as this was done under QEMU.

line 158

KMODDIR := ../rtlib/userland/$(KERNEL_VIRS)
--------------------------------------------------------^^^^^
not sure this is intended/works?

Reported by Andy:

Jul 30 21:14:13 precise msgd:0: startup instance=inst0
pid=22429 flavor=xenomai rtlevel=1 usrlevel=1 halsize=512000 shm=Posix
gcc=4.6.3 git=v2.6.0preunified-build-0~978da4a

loadrt hostmot2 -> {nothing}

loadrt hm2_pci->
:2:
/home/andypugh/linuxcnc.dev/libexec/rtapi_app_xenomai exited without
becoming ready
:2: insmod failed, returned -1
See the log and output of 'dmesg' for more information.
Jul 30 21:15:27 precise msgd:0: rtapi_app:22431:user hm2_pci:
dlopen: /home/andypugh/linuxcnc.dev/rtlib/xenomai/hm2_pci.so:
undefined symbol: pci_unregister_driver

check EXPORT_SYMBOL in userpci.c

The rtapi_msgd changes its name after invocation to add human-readable instance data to a ps -ef, with the CMD column displaying "msgd:0" (substitute 0 with the instance number). The -f option to ps displays the process's command line args, including ARGV[0]; ARGV[0] has changed, but not the

This breaks e.g. 'ps -C msgd' or 'ps -C msgd:0', and the pre-rename behavior of 'ps -C rtapi_msgd' remains, both in terms of the filter and the resulting displayed value of CMD.

Also, in 'ps -fwwC rtapi_msgd' output, 'msgd:0' is trailed by 100 characters, making wrapped output unnecessarily hard to read.