[SR-1109] Cannot run swift REPL in 2016-03-24-a on Ubuntu 15.10. Missing SwiftGlibc? about llvm-project HOT 26 CLOSED

Comment by Bill Abt (JIRA)

Note: Altho' the REPL is not working, everything else seems to be fine. Programs compile and run without problems. This problem just appears to be affecting the REPL.

from llvm-project.

Comment by Joe (JIRA)

@shahmishal, who owns the CI server that is building the Ubuntu 14.04 and 15.10 packages? I believe adding pexpect module to the server will expose that the packaging of Swift for both is breaking in importing Glibc into the REPL.

from llvm-project.

Comment by Ryan Lovelett (JIRA)

joeiachievedit (JIRA User) that is my concern as well. There are tests that could have caught this regression. They either are not running or are not running properly.

Both the bug and the fact that the CI didn't catch this should probably be addressed in this fix.

from llvm-project.

Comment by Joe (JIRA)

Reproduction of the issue separate from Ryan.

from llvm-project.

Comment by Ryan Lovelett (JIRA)

Using k8stone (JIRA User)'s suggestion that the issue arose sometime between swift-DEVELOPMENT-SNAPSHOT-2016-03-24-a and swift-DEVELOPMENT-SNAPSHOT-2016-03-16-a and git bisect. It's my conclusion that this regressed with this commit.

c6121d56b19305cf59148d46af54c06b771f3180 is the first bad commit
commit c6121d56b19305cf59148d46af54c06b771f3180
Author: Brian Gesiak <bgesiak@fb.com>
Date:   Wed Mar 16 13:29:42 2016 -0400

    [Un-revert][Glibc] Configure modulemap for target, not host
    
    This reverts commit f2154ee94d, which reverted 04e1cd5bda. The original
    commit needed to be reverted because of an issue in which install
    targets were added to OS X builds that did not target Linux. This
    addresses that issue by guarding all the Linux-only CMake logic with a
    check for the SDK being built.

:040000 040000 e92829c16aa22f20edfdf95f3bb18bb15a3fa226 90062ad44050a19fc0d5bc846409945e83619b01 M  lib
:040000 040000 bbe94bf4af832e154065bc0bcafffab2dacb839e 1a8be73f86884bda848cde22885bcd72a17660b2 M  stdlib
:040000 040000 abf55068f67ee44e2bd52169aa8b988fb8aead28 41db0f8ddec3281f51c6798dd47c86675b7118b3 M  tools

Full log if anyone is interested:

# bad: [7182c58cb2dbcceb5d4aaecb2c866b70270f8ba5] Merge pull request #​1821 from gregomni/typealias
# good: [d22638766e5907f77a5158699f533e5d962ec48d] Swap the order of arguments to expectEqual in Hashing test
git bisect start 'swift-DEVELOPMENT-SNAPSHOT-2016-03-24-a' 'swift-DEVELOPMENT-SNAPSHOT-2016-03-16-a'
# bad: [e3ec0703fd6dcb853b05da4dd5a315bea37740c6] Merge pull request #​1744 from trentxintong/FSO
git bisect bad e3ec0703fd6dcb853b05da4dd5a315bea37740c6
# bad: [09d2d635551f1fa444bab33eb21350c5c04225dc] [SIL] Teach type lowering to use _ObjectiveCBridgeable.
git bisect bad 09d2d635551f1fa444bab33eb21350c5c04225dc
# bad: [b793bab7608ae4f6a6560982e170a3d6c860512b] [docs][arc] Add a subsection on the relationships in between ARC and "copying" pointers.
git bisect bad b793bab7608ae4f6a6560982e170a3d6c860512b
# good: [c9ef32c4d034aa3400f1aed71c132ca252ba5e14] Move the objective-c bridging tests into the unit-tests folder
git bisect good c9ef32c4d034aa3400f1aed71c132ca252ba5e14
# skip: [ce9793fe15491d1011c90f3ee496a242902fab1e] [Test] Add module printing test for recently added module groups.
git bisect skip ce9793fe15491d1011c90f3ee496a242902fab1e
# bad: [10fefc532fa7ecd9833fb5b89390fde829028ab8] gardening: simplify an unnecessary unique pointer usage.
git bisect bad 10fefc532fa7ecd9833fb5b89390fde829028ab8
# bad: [aaa880943fc44eea9ecf16757e57fe63979dbe77] [docs][arc] Small change to admonishment in the header.
git bisect bad aaa880943fc44eea9ecf16757e57fe63979dbe77
# good: [a760b70827633619b09d4a8e4e2e82d10640adc5] Fixup recent formatting boo boos I've committed
git bisect good a760b70827633619b09d4a8e4e2e82d10640adc5
# bad: [38ceced77984188b25af9673ea61b28dc4c90b60] Merge pull request #​1704 from modocache/unrevert-target-specific-glibc
git bisect bad 38ceced77984188b25af9673ea61b28dc4c90b60
# bad: [c6121d56b19305cf59148d46af54c06b771f3180] [Un-revert][Glibc] Configure modulemap for target, not host
git bisect bad c6121d56b19305cf59148d46af54c06b771f3180
# first bad commit: [c6121d56b19305cf59148d46af54c06b771f3180] [Un-revert][Glibc] Configure modulemap for target, not host

from llvm-project.

CI now has pexpect module installed, and now the test is failing.

******************** TEST 'swift-package-tests :: repl/test-repl-glibc.py' FAILED ********************
Script:
--
rm -rf /tmp/swift-package-tests/repl/Output/test-repl-glibc.py.tmp.dir
mkdir /tmp/swift-package-tests/repl/Output/test-repl-glibc.py.tmp.dir
python /home/buildnode/jenkins/workspace/swift-PR-Linux/swift-integration-tests/repl/test-repl-glibc.py /home/buildnode/jenkins/workspace/swift-PR-Linux/buildbot_linux/none-swift_package_sandbox/usr/bin/swift > /tmp/swift-package-tests/repl/Output/test-repl-glibc.py.tmp.dir/output.txt
/home/buildnode/jenkins/workspace/swift-PR-Linux/buildbot_linux/llvm-linux-x86_64/bin/FileCheck --input-file /tmp/swift-package-tests/repl/Output/test-repl-glibc.py.tmp.dir/output.txt /home/buildnode/jenkins/workspace/swift-PR-Linux/swift-integration-tests/repl/test-repl-glibc.py
--
Exit Code: 1

Command Output (stdout):
--
Command 0: "rm" "-rf" "/tmp/swift-package-tests/repl/Output/test-repl-glibc.py.tmp.dir"
Command 0 Result: 0
Command 0 Output:


Command 0 Stderr:


Command 1: "mkdir" "/tmp/swift-package-tests/repl/Output/test-repl-glibc.py.tmp.dir"
Command 1 Result: 0
Command 1 Output:


Command 1 Stderr:


Command 2: "python" "/home/buildnode/jenkins/workspace/swift-PR-Linux/swift-integration-tests/repl/test-repl-glibc.py" "/home/buildnode/jenkins/workspace/swift-PR-Linux/buildbot_linux/none-swift_package_sandbox/usr/bin/swift"
Command 2 Result: 1
Command 2 Output:
None

Command 2 Stderr:
Traceback (most recent call last):
  File "/home/buildnode/jenkins/workspace/swift-PR-Linux/swift-integration-tests/repl/test-repl-glibc.py", line 49, in <module>
    repl.expect("init")
  File "/usr/lib/python2.7/dist-packages/pexpect/__init__.py", line 1418, in expect
    timeout, searchwindowsize)
  File "/usr/lib/python2.7/dist-packages/pexpect/__init__.py", line 1433, in expect_list
    timeout, searchwindowsize)
  File "/usr/lib/python2.7/dist-packages/pexpect/__init__.py", line 1535, in expect_loop
    raise TIMEOUT(str(err) + '\n' + str(self))
pexpect.TIMEOUT: Timeout exceeded.

from llvm-project.

I think this is going to end up going to somebody on the Swift side, but I'm going to track it down far enough to figure out who needs to address it.

from llvm-project.

I was able to repro this. Interestingly, I am finding that if something unrelated fails during the build (e.g. missing some packages), neither lldb nor llbuild seem to be able to install from a rebuild that is not a full build. That's a separate issue, though (see SR-1287 that I just filed).

In any event, I have it reproduced, so I can drill into it now.

from llvm-project.

I just attached the build script I'm using to mimic what the Ubuntu 15.10 packaging CI is doing. These are identical to that builder except that I've added an additional cmake flag to LLDB to export all of its internal symbols so I can debug it when I get to that point.

from llvm-project.

Comment by Ryan Lovelett (JIRA)

@trfiala I don't know if you've seen this or not.

I was able to track down the issue (and temporarily work around it) by sym linking /usr/lib/swift/linux/x86_64/glibc.modulemap -> /usr/lib/lldb/clang/linux/x86_64/glibc.modulemap.

I took the advice of [email protected] (JIRA User) and started debugging the import. I don't fully understand ClangImporter's role in the whole thing but my guess is that it, or some other Swift module, is improperly telling lldb/REPL where to find the Glibc.modulemap.

I know you said you thought this was going to be someone on the Swift side's problem. My investigation seems to indicate that as well too (but hey I'm a novice at this code base; so what do I know?).

from llvm-project.

HI Ryan!

Thanks for passing that along. Yes, Jim and I talked about it yesterday. It is likely the change listed above (c6121d5) is going to be the root cause. The ClangImporter needs to be able to reconstruct some info, and the module map (which is just a text file) needs to be found to do that right. The paths used to find that look like they may be built from the variables that were changed in c6121d5, but due to the high-level REPL tests not running due to the missing pexpect, that issue wasn't caught then.

Since I got it reproduced by the end of last night, I should be able to track that down and get it fixed up (or at least enlist the right people so they can do it). Unfortunately I think we're too far past c6121d5 to revert it.

from llvm-project.

One note on the build script I attached. It uses the buildbot preset build settings, which currently does a release build with no debug symbols. This is not ideal for debugging a problem. I've locally flipped the build presets to do a debug build so I have debug symbols. (Debugging what I initially built and reproduced was useless).

Separately, failing to import Glibc (or Foundation, which will use Glibc) should be getting caught by the LLDB test suite, which is a different test suite than the one that caught this. The one that caught this is a final packaging, high-level test suite. (And that has a separate issue, which is that it should have failed at the point where a key piece, pexpect, wasn't found). I filed SR-1295 to investigate.

from llvm-project.

If this cannot be resolved by today, we should XFAIL the test. Bots shouldn't stay red, it impacts @swift-ci usability and can lead to cascading failures.

from llvm-project.

Yes, I will figure out how to mark that test XFAIL if I can't resolve this today. I'm working on a debuggable build now.

from llvm-project.

I'm going to go ahead and make this XFAIL. Mishal is going to not publish any packages until we have this fixed. (I'm still working on it, but we can get build czars happy by making it green, and we'll continue to dig into the issue).

from llvm-project.

Updated the build script with one that doesn't do all the intermediate testing, and includes more debug info.

from llvm-project.

Temporarily marked the packaging test for glibc as XFAIL with the following commit in github.com/apple/swift-integration-tests:
e9e7054cbdb3d386fb3763406d4a81dfaccde2f4

I'm still working on this, and we don't plan on doing another Linux packaging until it gets resolved.

from llvm-project.

Making progress. We've got suspicious behavior under the debugger. Jordan and I are looking into it.

from llvm-project.

Okay, looks like we've got a fix for this. I'm just running the LLDB test suite against it and then I'll have the PR checker test it.

from llvm-project.

Testing this with:
swiftlang/swift#2279

from llvm-project.

I put up a Vagrant VM setup here that I'm using on a VMWare Fusion hypervisor:
https://github.com/tfiala/swift-vagrant

from llvm-project.

While the change in PR 2279 seems to address the issue, as is it triggers an assert in the LLDB test suite on Ubuntu. I need to resolve that before this can go further.

It appears as if the broken code path on Ubuntu that was incorrectly using the default clang resource path for Swift resource paths was masking that we have some SwiftASTContext instances that otherwise don't have a resource dir set. I'm going to figure out what those are. They may be the SwiftASTContext instances created for modules rather than the target. Looking at that next.

from llvm-project.

I have a change for the LLDB test failures that is working on Ubuntu 15.10. It ensures that a Swift resource path is set for all SwiftASTContext module instances if we had one previously set in a target.

I also think I figured out how to get the LLDB test suite running on Ubuntu in the package build layout. (This has an extra directory underneath the build dir, and this is throwing off some of our auto-find-the-swiftc-executable logic). We should be able to just specify the just-built swift compiler as the target compiler. I'll address that separately, though, since there might be a few wrinkles there.

from llvm-project.

This should now be addressed with the following 3 commits:

• swift: swiftlang/swift@42d97ed

• swift-lldb: apple/swift-lldb@ea2a5f2

• swift-integration-tests: swiftlang/swift-integration-tests@98f6c6d

from llvm-project.

Comment by Joe (JIRA)

Verified on Ubuntu 14.04:

➜  package-swift-3.0 git:(swift-3.0) ✗ ./install/usr/bin/swift
Welcome to Swift version 3.0-dev (LLVM 752e1430fc, Clang 1e6cba3ce3, Swift 42d97ed7dc). Type :help for assistance.
  1> import Glibc
  2>

Final packaging output:

  Expected Passes    : 12
  Unsupported Tests  : 4

Thanks @trfiala!

from llvm-project.

Comment by Joe (JIRA)

Verified on Ubuntu 15.10:

joe@ubuntu1510:~/package-swift$ ./install/usr/bin/swift
Welcome to Swift version 3.0-dev (LLVM 752e1430fc, Clang 1e6cba3ce3, Swift 42d97ed7dc). Type :help for assistance.
  1> import Glibc
  2>

Final packaging output:

  Expected Passes    : 11
  Unsupported Tests  : 5

from llvm-project.