compiler-research / xeus-clang-repl Goto Github PK

This issue will expand upon issue #75 , and detail the problems that occur when building xeus-clang-repl natively within MacOS on Apple Silicon. To aid in reproducibility I will first outline some details of my system

• Processor: M1 Pro
• Ram: 16 GB
• OS: Sonoma 14.0
• Package Manger: Macports

One problem I came across while trying to build xeus-clang-repl was the use of the xeus/xserver_zmq.hpp header file. At least on MacOS when you install xeus this header file is not present. I had to install xeus-zmq, and change the header file to xeus-zmq/xserver_zmq.hpp .

I will make a proper start on detailing the problems with building this repo in MacOS this weekend, when I have a little more time. I will raise a separate issue in each one of the dependencies and link them back to this one.

Current version:

Proposed version:

Clad integrates with xeus-cling automagically. However, if we try the same approach it integrates but it fails to start as we pick up the conda build of clad that's compatible with clang5 (on which is still based xeus-cling).

We need to find out how to tell conda to install clad built with clang15 on which is currently based xeus-clang-repl.

Working with a vector type section shows the error:

In file included from <<< inputs >>>:1:
input_line_7:1:5: error: redefinition of 'new_python_var'
int new_python_var = 1327;
    ^
<<< inputs >>>:1:1: note: '' included multiple times, additional include site here
<<< inputs >>>:1:1: note: '' included multiple times, additional include site here
In file included from <<< inputs >>>:1:
input_line_10:1:5: error: redefinition of 'new_python_var'
int new_python_var = 1327;
    ^
<<< inputs >>>:1:1: note: '' included multiple times, additional include site here
<<< inputs >>>:1:1: note: '' included multiple times, additional include site here
In file included from <<< inputs >>>:1:
input_line_12:1:10: fatal error: 'iostream' file not found
#include <iostream>
         ^~~~~~~~~~

Was there any particular reason to license this under the Apache license?

Since Xeus is BSD-3-Clause and Xeus-cling are BSD-3-Clause, it would add some fluidity between these projects to license xeus-clang-repl BSD-3-Clause as well.

This repo currently has a release of 0.2.0 , but package_version is labelled as 0.1.0 in xeus-clang-replConfigVersion.cmake . This should be updated. Not sure what convention project has for labelling a version dev, but once informed a fix to this issue is easily done.

This works on my local built, but only with the relative path being passed on, not #include "xtl/xbase64.hpp"

In working around #74, I try to build the docker image (docker build . -t xeus-clang-repl) but fail to because of a linker error during building CppInterOp:

282.9 -- Build files have been written to: /home/jovyan/CppInterOp/build
282.9 ++ nproc --all
282.9 + cmake --build . --parallel 10
282.9 [  7%] Creating directories for 'googletest'
282.9 [  7%] Built target intrinsics_gen
...
314.8 [ 92%] Completed 'googletest'
314.9 [ 92%] Built target googletest
322.9 [ 92%] Built target obj.clangCppInterOp
322.9 [100%] Linking CXX shared library ../libclangCppInterOp.so
323.0 /usr/bin/ld.gold: error: /home/runner/work/xeus-clang-repl/xeus-clang-repl/clang-dev/inst/lib/libLLVMNVPTXCodeGen.a(NVPTXAsmPrinter.cpp.o): incompatible target
323.0 /usr/bin/ld.gold: error: /home/runner/work/xeus-clang-repl/xeus-clang-repl/clang-dev/inst/lib/libLLVMNVPTXCodeGen.a(NVPTXTargetMachine.cpp.o): incompatible target
323.0 /usr/bin/ld.gold: error: /home/runner/work/xeus-clang-repl/xeus-clang-repl/clang-dev/inst/lib/libLLVMNVPTXDesc.a(NVPTXMCTargetDesc.cpp.o): incompatible target

Not sure if it's because gold isn't smart enough or something else (i.e., I didn't try patching to use ld instead).

cc @vgvassilev

@alexander-penev This is the error which arises when trying to build the xeus-clang-repl Dockerfile with llvm-project version 17.x on an arm based architechture , mentioned in PR #77 . I don't currently know enough about the project to be able to fix this, but I'm happy to test the suggestions of others.

+ python -c 'import cppyy'
/home/jovyan/cppyy-backend/python/cppyy_backend/loader.py:154: UserWarning: No precompiled header available (/home/jovyan/cppyy-backend/python/cppyy_backend/etc not writable, set CLING_STANDARD_PCH); this may impact performance.
  warnings.warn('No precompiled header available (%s); this may impact performance.' % msg)
Failed to execute via ::process:Symbols not found: [ llvm_orc_registerEHFrameSectionWrapper ]
Failure value returned from cantFail wrapped call
Symbols not found: [ llvm_orc_registerEHFrameSectionWrapper ]
UNREACHABLE executed at /home/jovyan/llvm-project/llvm/include/llvm/Support/Error.h:815!
/bin/bash: line 1: 47760 Aborted                 (core dumped) python -c "import cppyy" 

Currently there is only a ci for Ubuntu on x86 . I suggest the creation of a workflow which compiles on macos, now the build issues that arise from building on an arm based apple machine have been resolved.

@vgvassilev @alexander-penev If either of you have any requirements on what you would like to see for such a workflow please put them in the comments below.

My first step towards doing this workflow would be to develop/improve the macos ci workflow for CppInterOp (issue compiler-research/CppInterOp#138 ). I am happy to be assigned to both these issues.

clang-repl seems to have some isses with resolution of templates:

Without a non-template overload:

With a template overload (included first)

Currently, if a cell with some definitions is executed twice, the second time will drop an error. I think, this behavior is undesired, because it doesn't allow modifying functions in cells.

Is there any way to overcome this?

It would be great if the Python magic was a plugin, so that we don't need CPython to build xeus-clang.

A byproduct of requiring CPython is that conda-forge needs to make variants of the build for all supported versions of Python.

Currently, the output is picked up automatically by the kernel as it is sent to std::cout (1), but it should be piped through the publish_execution_result(2)

(base) mlevental@maksims-MacBook-Pro xeus-clang-repl % docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
(base) mlevental@maksims-MacBook-Pro xeus-clang-repl % ./run-docker.sh
-:33 SC2046 warning: Quote this to prevent word splitting.
...
-:158 SC2086 info: Double quote to prevent globbing and word splitting.

Container engine initialization error: ('Check if docker is running on the host.', DockerException("Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))"))
Unable to find image 'xeus-clang-repl:latest' locally
docker: Error response from daemon: pull access denied for xeus-clang-repl, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
See 'docker run --help'.

Can't find it on dockerhub nor here as a package.

Note, maybe I need to build the image locally first (which is fine) but then the README should be updated...

cc @vgvassilev

Is there a means to build xeus-clang-repl without a patched llvm?

Maintaining patches flavors of clang and llvm on conda-forge has been a major hurdle for packaging Cling on conda-forge, and dropping this requirement would be a major upside of switching to xeus-clang-repl for us.

In the example above, the kernel simply restarts. Better recovery from exceptions / errors should be implemented so that the kernel doesn't automatically restart and lose its state.

Hi!

Recently I did many Profile-Guided Optimization (PGO) benchmarks on multiple projects (including static analysis tools and compilers like Rustc, Clang, Clangd, Clang Tidy, and many others) - the results are available here. For Clang PGO results are even described in the official documentation (20% improvements) - https://llvm.org/docs/HowToBuildWithPGO.html . I think it could be a good idea to enable PGO at least for Clang dependency in Xeus repl.

I can suggest the following things to do:

• Enable PGO at least for the Clang dependency.
• Document enabling PGO somewhere in the Documentation (possibly in the README file)

Here are some examples of how PGO is already integrated into other projects' build scripts:

• Rustc: a CI script for the multi-stage build
• GCC:
  • Official docs, section "Building with profile feedback" (even AutoFDO build is supported)
  • A part in a "wonderful" configure script
• Clang: Docs
• Python:
  • CPython: README
  • Pyston: README
• Go: Bash script
• V8: Bazel flag
• ChakraCore: Scripts
• Chromium: Script
• Firefox: Docs
  • Thunderbird has PGO support too
• PHP - Makefile command and old Centminmod scripts
• MySQL: CMake script
• YugabyteDB: GitHub commit
• FoundationDB: Script
• Zstd: Makefile
• Foot: Scripts
• Windows Terminal: GitHub PR
• Pydantic-core: GitHub PR

After PGO, I can suggest evaluating LLVM BOLT as an additional optimization step after PGO. Building with LLVM BOLT is already supported by Clang too in the upstream.