Comments (33)
headupinclouds
commented on May 12, 2024
1
๐
All tests pass except the intentional heap-use-after-free asan test case.
3: Test command: /Users/dhirvonen/devel/drishti/_builds/osx-10-11-sanitize-address/src/lib/drishti/acf/ut/Release/test-drishti-acf "/Users/dhirvonen/devel/drishti/assets/images/lena512color.png" "/Users/dhirvonen/devel/drishti/assets/images/lena512gray.png" "/Users/dhirvonen/devel/drishti/assets/drishti_face_inner_48x48.mat"
3: Test timeout computed to be: 9.99988e+06
3: [==========] Running 1 test from 1 test case.
3: [----------] Global test environment set-up.
3: [----------] 1 test from ACFTest
3: [ RUN ] ACFTest.ACFDetection
3: -------------------------
3: File: /Users/dhirvonen/devel/drishti/assets/drishti_face_inner_48x48.mat
3: MATLAB 5.0 MAT-file, Platform: MACI64, Created on: Fri Oct 2 22:17:11 2015 Variables: detector vector<vector<MatlabIOContainer>>
3: -------------------------
3: [ OK ] ACFTest.ACFDetection (292 ms)
3: [----------] 1 test from ACFTest (292 ms total)
3:
3: [----------] Global test environment tear-down
3: [==========] 1 test from 1 test case ran. (293 ms total)
3: [ PASSED ] 1 test.
3/4 Test #3: acf_test ......................... Passed 0.51 sec
test 4
Start 4: DrishtAsanTest
4: Test command: /Users/dhirvonen/devel/drishti/_builds/osx-10-11-sanitize-address/src/lib/drishti/master/ut/Release/test-drishti-asan
4: Test timeout computed to be: 9.99988e+06
4: =================================================================
4: ==55076==ERROR: AddressSanitizer: heap-use-after-free on address 0x60200000e0b4 at pc 0x000102f8cf29 bp 0x7fff5cc73fc0 sp 0x7fff5cc73fb8
4: READ of size 4 at 0x60200000e0b4 thread T0
4: #0 0x102f8cf28 in main test-drishti-asan.cpp:38
4: #1 0x7fff8de3a5c8 in start (/usr/lib/system/libdyld.dylib+0x35c8)
4: #2 0x0 (<unknown module>)
4:
4: 0x60200000e0b4 is located 0 bytes to the right of 4-byte region [0x60200000e0b0,0x60200000e0b4)
4: freed by thread T0 here:
4: #0 0x102fd611b in wrap__ZdaPv (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/7.0.2/lib/darwin/libclang_rt.asan_osx_dynamic.dylib+0x4511b)
4: #1 0x102f8ceef in main test-drishti-asan.cpp:33
4: #2 0x7fff8de3a5c8 in start (/usr/lib/system/libdyld.dylib+0x35c8)
4: #3 0x0 (<unknown module>)
4:
4: previously allocated by thread T0 here:
4: #0 0x102fd5b5b in wrap__Znam (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/7.0.2/lib/darwin/libclang_rt.asan_osx_dynamic.dylib+0x44b5b)
4: #1 0x102f8cee4 in main test-drishti-asan.cpp:32
4: #2 0x7fff8de3a5c8 in start (/usr/lib/system/libdyld.dylib+0x35c8)
4: #3 0x0 (<unknown module>)
4:
4: SUMMARY: AddressSanitizer: heap-use-after-free test-drishti-asan.cpp:38 main
4: Shadow bytes around the buggy address:
4: 0x1c0400001bc0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
4: 0x1c0400001bd0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
4: 0x1c0400001be0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
4: 0x1c0400001bf0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
4: 0x1c0400001c00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
4: =>0x1c0400001c10: fa fa fa fa fa fa[fd]fa fa fa 00 06 fa fa 00 00
4: 0x1c0400001c20: fa fa 00 04 fa fa 00 06 fa fa 00 fa fa fa 00 00
4: 0x1c0400001c30: fa fa 00 00 fa fa 00 00 fa fa 00 00 fa fa 00 00
4: 0x1c0400001c40: fa fa 00 00 fa fa 00 00 fa fa 00 00 fa fa 00 00
4: 0x1c0400001c50: fa fa 00 00 fa fa 00 00 fa fa 00 00 fa fa 00 00
4: 0x1c0400001c60: fa fa 00 00 fa fa 00 00 fa fa 00 00 fa fa 00 00
4: Shadow byte legend (one shadow byte represents 8 application bytes):
4: Addressable: 00
4: Partially addressable: 01 02 03 04 05 06 07
4: Heap left redzone: fa
4: Heap right redzone: fb
4: Freed heap region: fd
4: Stack left redzone: f1
4: Stack mid redzone: f2
4: Stack right redzone: f3
4: Stack partial redzone: f4
4: Stack after return: f5
4: Stack use after scope: f8
4: Global redzone: f9
4: Global init order: f6
4: Poisoned by user: f7
4: Container overflow: fc
4: Array cookie: ac
4: Intra object redzone: bb
4: ASan internal: fe
4: Left alloca redzone: ca
4: Right alloca redzone: cb
4: ==55076==ABORTING
4/4 Test #4: DrishtAsanTest ...................***Exception: Other 1.78 sec
75% tests passed, 1 tests failed out of 4
Total Test time (real) = 16.17 sec
The following tests FAILED:
4 - DrishtAsanTest (OTHER_FAULT)
Errors while running CTest
Command exit with status "8": [/Users/dhirvonen/devel/drishti/_builds/osx-10-11-sanitize-address]> "ctest" "-C" "Release" "-VV"
Log: /Users/dhirvonen/devel/drishti/_logs/polly/log.txt
*** FAILED ***
from drishti.
headupinclouds
commented on May 12, 2024
1
https://cmake.org/cmake/help/v3.8/release/3.8.html
from drishti.
headupinclouds
commented on May 12, 2024
http://clang.llvm.org/docs/AddressSanitizer.html
"Simply compile and link your program with -fsanitize=address flag. The AddressSanitizer run-time library should be linked to the final executable, so make sure to use clang (not ld) for the final link step. When linking shared libraries, the AddressSanitizer run-time is not linked, so -Wl,-z,defs may cause link errors (donโt use it with AddressSanitizer). To get a reasonable performance add -O1 or higher. To get nicer stack traces in error messages add -fno-omit-frame-pointer. To get perfect stack traces you may need to disable inlining (just use -O1) and tail call elimination (-fno-optimize-sibling-calls)."
Something like this in the toolchain?
set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain//usr/lib/clang/7.0.2/lib/darwin/ -l clang_rt.asan_osx_dynamic")
from drishti.
headupinclouds
commented on May 12, 2024
UPDATED toolchain (seems to be working)
sanitize_address_libcxx.cmake
# Copyright (c) 2014-2016, Ruslan Baratov
# Copyright (c) 2016, David Hirvonen
# All rights reserved.
if(DEFINED POLLY_FLAGS_SANITIZE_ADDRESS_LIBCXX_CMAKE_)
return()
else()
set(POLLY_FLAGS_SANITIZE_ADDRESS_LIBCXX_CMAKE_ 1)
endif()
include(polly_add_cache_flag)
### Add asan linking
set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain//usr/lib/clang/7.0.2/lib/darwin/ -l clang_rt.asan_osx_dynamic" CACHE STRING "" FORCE)
polly_add_cache_flag(CMAKE_CXX_FLAGS "-fsanitize=address")
polly_add_cache_flag(CMAKE_CXX_FLAGS "-g")
polly_add_cache_flag(CMAKE_CXX_FLAGS "-D_LIBCPP_HAS_NO_ASAN")
set(
CMAKE_CXX_FLAGS_RELEASE
"-O1 -DNDEBUG"
CACHE
STRING
"C++ compiler flags"
FORCE
)
polly_add_cache_flag(CMAKE_C_FLAGS "-fsanitize=address")
polly_add_cache_flag(CMAKE_C_FLAGS "-g")
polly_add_cache_flag(CMAKE_C_FLAGS "-D_LIBCPP_HAS_NO_ASAN")
set(
CMAKE_C_FLAGS_RELEASE
"-O1 -DNDEBUG"
CACHE
STRING
"C compiler flags"
FORCE
)
from drishti.
headupinclouds
commented on May 12, 2024
Need a portable way to identify this path:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain//usr/lib/clang/7.0.2/lib/darwin/ -l clang_rt.asan_osx_dynamic
xcrun <OPTION>?
from drishti.
ruslo
commented on May 12, 2024
xcrun ?
If this toolchain is for only Xcode-based generators then it is an option. But if this toolchain is for only Xcode-based generators then it make sense to investigate XCODE_ATTRIBUTE_* approach. Probably we don't need to have _LIBCPP_HAS_NO_ASAN (or even -fsanitize=address). Need to reverse engineer *.xcodeproj file.
from drishti.
ruslo
commented on May 12, 2024
From Gitter: https://mikeash.com/pyblog/friday-qa-2015-07-03-address-sanitizer.html
from drishti.
ruslo
commented on May 12, 2024
Probably I was wrong about instrumented version of libcxx. I guess it's about memory sanitizer. Quote from docs:
MemorySanitizer requires that all program code is instrumented. This also includes any libraries that the program depends on, even libc.
Address sanitizer doesn't detect read of undefined values, right? I guess if libcxx will not be instrumented then the only error in libcxx itself will be ignored (if any). The rest will work fine.
from drishti.
ruslo
commented on May 12, 2024
But if this toolchain is for only Xcode-based generators then it make sense to investigate XCODE_ATTRIBUTE_* approach. Probably we don't need to have _LIBCPP_HAS_NO_ASAN (or even -fsanitize=address). Need to reverse engineer *.xcodeproj file.
There is no difference between *.pbxproj files with enabled and disabled ASAN. Setting saved in some other location, hence XCODE_ATTRIBUTE_* will not work.
from drishti.
ruslo
commented on May 12, 2024
I guess if libcxx will not be instrumented then the only error in libcxx itself will be ignored (if any). The rest will work fine.
Kind of: http://stackoverflow.com/a/38858905/2288008
from drishti.
ruslo
commented on May 12, 2024
_LIBCPP_HAS_NO_ASAN added to toolchain-id calculation module:
Commit is in branch toolchain.calc.update and not yet merged to master/released. If you want to try it please do it carefully because of cache.
The strange thing I see so far is that if _LIBCPP_HAS_NO_ASAN is not defined by user it will be defined by libcxx, and if it's defined by user it will be set to 1 by libcxx. If library is not libcxx the macro remains undefined. Because of such behaviour all libcxx toolchains will have new 'Toolchain-ID' even for those who doesn't use sanitizer. I will keep this branch in work-in-progress status until everything will be finished.
from drishti.
ruslo
commented on May 12, 2024
I've added toolchain based on your work however I changed osx-10-11 instead of xcode because xcode means currently active version of Xcode and osx-10-11 will activate the exact version of Xcode which is important for uploading. If your Xcode version is fresh enough there will be no difference between using xcode and osx-10-11 toolchains.
Change is in pr.osx.sanitize branch:
Testing:
Simple example will compile fine but to run executable you need to set LD_LIBRARY_PATH to the directory with libclang_rt.asan_osx_dynamic.dylib. I will think about improvement of this. Actually with Hunter we will have relocation problem since library is not static, need to figure out how to deal with RPATH, remote building and cache.
from drishti.
headupinclouds
commented on May 12, 2024
But if this toolchain is for only Xcode-based generators then it make sense to investigate XCODE_ATTRIBUTE_* approach
FWIW, this modifies the target scheme, which has the following path:
grep enable _builds/xcode/project.xcodeproj/xcuserdata/dhirvonen.xcuserdatad/xcschemes/some_target.xcscheme
enableAddressSanitizer = "YES"
This approach would not cover dependencies.
from drishti.
ruslo
commented on May 12, 2024
FWIW, this modifies the target scheme, which has the following path
Ok, I'm not sure if we can control this file with CMake.
from drishti.
ruslo
commented on May 12, 2024
I forgot that we run tests with Polly :) They fail, the reason is location of dynamic library that I've mentioned already.
from drishti.
headupinclouds
commented on May 12, 2024
Ok, I'm not sure if we can control this file with CMake.
Probably not. Having per target asan coverage seems very limited anyway. The toolchain approach seems like the right way to go.
Let's finish the Xcode asan test, but as a follow up, there are llvm packages available with home brew (probably linux apt-get as well) and the --with-asan flag will include support for all of the sanitizers, so that could be another option for full sanitizer coverage in combination with modified polly toolchains:
Homebrew/legacy-homebrew#27505
"The clang sanitizers are amazingly useful for detecting all sorts of problems with the code. And it would be cool to have this (optionally) in the standard llvm build. It's in the compiler-rt git repository and is already optionally installed by the homebrew/versions/llvm34 brew: http://compiler-rt.llvm.org/ so it's already been tested. homebrew/versions/llvm34.rb. Look at the --with-asan switch. This actually enables all sanitizers, not just asan."
- sanitize-address.cmake
- sanitize-leak.cmake
- sanitize-memory.cmake
- sanitize-thread.cmake
from drishti.
headupinclouds
commented on May 12, 2024
https://github.com/google/sanitizers/wiki/AddressSanitizer
Also ....
Q: When I link my shared library with -fsanitize=address, it fails due to some undefined ASan symbols (e.g. asan_init_v4)?
A: Most probably you link with -Wl,-z,defs or -Wl,--no-undefined. These flags don't work with ASan unless you also use -shared-libasan (which is the default mode for GCC, but not for Clang).
from drishti.
headupinclouds
commented on May 12, 2024
From your earlier test: https://travis-ci.org/ruslo/polly/builds/163364920
1: Test command: /Users/travis/build/ruslo/polly/_builds/osx-10-11-sanitize-address/Debug/simple
1: Test timeout computed to be: 9.99988e+06
1: dyld: Library not loaded: @rpath/libclang_rt.asan_osx_dynamic.dylib
1: Referenced from: /Users/travis/build/ruslo/polly/_builds/osx-10-11-sanitize-address/Debug/simple
1: Reason: image not found
1/1 Test #1: SimpleTest .......................***Exception: Other 0.02 sec
Simple example will compile fine but to run executable you need to set LD_LIBRARY_PATH to the directory with libclang_rt.asan_osx_dynamic.dylib. I will think about improvement of this. Actually with Hunter we will have relocation problem since library is not static, need to figure out how to deal with RPATH, remote building and cache.
Since this is purely a diagnostic tool on a Mac OS X host, Can we just set DYLD_LIBRARY_PATH in the travis.yml file for drishti tests? It seems like that should be functional and would support the immediate use case.
from drishti.
ruslo
commented on May 12, 2024
Since this is purely a diagnostic tool on a Mac OS X host, Can we just set DYLD_LIBRARY_PATH in the travis.yml file for drishti tests?
I don't want to store this version into Hunter cache. Because in future we have to clean it out when good version will be ready. I'm thinking about fix, we can try that if fix will take too much time.
from drishti.
headupinclouds
commented on May 12, 2024
I'm hitting issues related to address-sanitized shared libraries, but, if I skip the drishti and drishti_c targets and run with your TOOLCHAIN=osx-10-11-sanitize-address in addition to setting
export DYLD_LIBRARY_PATH=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain//usr/lib/clang/7.0.2/lib/darwin then the static drishtisdk tests seem to run fine. I guess it is working. There is probably a log file that can be configured. Here is the polly.py ... --test tail output:
[/Users/dhirvonen/devel/drishti/_builds/osx-10-11-sanitize-address]> "ctest" "-C" "Release" "-VV"
UpdateCTestConfiguration from :/Users/dhirvonen/devel/drishti/_builds/osx-10-11-sanitize-address/DartConfiguration.tcl
UpdateCTestConfiguration from :/Users/dhirvonen/devel/drishti/_builds/osx-10-11-sanitize-address/DartConfiguration.tcl
Test project /Users/dhirvonen/devel/drishti/_builds/osx-10-11-sanitize-address
Constructing a list of tests
Done constructing a list of tests
Checking test dependency graph...
Checking test dependency graph end
test 1
Start 1: DrishtiGeometryTest
1: Test command: /Users/dhirvonen/devel/drishti/_builds/osx-10-11-sanitize-address/src/lib/drishti/geometry/ut/Release/test-drishti-geometry
1: Test timeout computed to be: 9.99988e+06
1: [==========] Running 1 test from 1 test case.
1: [----------] Global test environment set-up.
1: [----------] 1 test from Ellipse
1: [ RUN ] Ellipse.EllipseLineIntersection2
1: [ OK ] Ellipse.EllipseLineIntersection2 (1 ms)
1: [----------] 1 test from Ellipse (1 ms total)
1:
1: [----------] Global test environment tear-down
1: [==========] 1 test from 1 test case ran. (1 ms total)
1: [ PASSED ] 1 test.
1/2 Test #1: DrishtiGeometryTest .............. Passed 0.17 sec
test 2
Start 2: acf_test
2: Test command: /Users/dhirvonen/devel/drishti/_builds/osx-10-11-sanitize-address/src/lib/drishti/acf/ut/Release/test-drishti-acf "/Users/dhirvonen/devel/drishti/assets/images/lena512color.png" "/Users/dhirvonen/devel/drishti/assets/images/lena512gray.png" "/Users/dhirvonen/devel/drishti/assets/drishti_face_inner_48x48.mat"
2: Test timeout computed to be: 9.99988e+06
2: [==========] Running 1 test from 1 test case.
2: [----------] Global test environment set-up.
2: [----------] 1 test from ACFTest
2: [ RUN ] ACFTest.ACFDetection
2: -------------------------
2: File: /Users/dhirvonen/devel/drishti/assets/drishti_face_inner_48x48.mat
2: MATLAB 5.0 MAT-file, Platform: MACI64, Created on: Fri Oct 2 22:17:11 2015 Variables: detector vector<vector<MatlabIOContainer>>
2: -------------------------
2: [ OK ] ACFTest.ACFDetection (302 ms)
2: [----------] 1 test from ACFTest (302 ms total)
2:
2: [----------] Global test environment tear-down
2: [==========] 1 test from 1 test case ran. (302 ms total)
2: [ PASSED ] 1 test.
2/2 Test #2: acf_test ......................... Passed 0.50 sec
100% tests passed, 0 tests failed out of 2
Total Test time (real) = 0.68 sec
Open project: /Users/dhirvonen/devel/drishti/_builds/osx-10-11-sanitize-address/drishtisdk.xcodeproj
Execute command: [
`open`
`-a`
`/Applications/Xcode.app/Contents/Developer/../..`
`/Users/dhirvonen/devel/drishti/_builds/osx-10-11-sanitize-address/drishtisdk.xcodeproj`
]
I can add an intentional asan sanity test to make sure this is behaving properly.
from drishti.
ruslo
commented on May 12, 2024
I guess it is working
Cool, can you try to add a "bug" intentionally to check that sanitizer do work actually. The best test will be to add "bug" to 3rd party code which is triggered by some Drishti test.
from drishti.
ruslo
commented on May 12, 2024
Also .... Q: When I link my shared library with -fsanitize=address, it fails due to some undefined ASan symbols (e.g. asan_init_v4)?
I don't think that this is our case, we have undefined reference because there is no library linked :)
from drishti.
headupinclouds
commented on May 12, 2024
I believe I am seeing an issue related to shared libraries. I've created a branch for testing with a DRISHTI_BUILD_SHARED_SDK=OFF option that toggles the library model for the both drishti and drishti_c modules. This will be useful in the future anyway. For now it is fine to run asan tests on static libraries. Maybe adding -shared-libasan will fix it.
from drishti.
headupinclouds
commented on May 12, 2024
Cool, can you try to add a "bug" intentionally to check that sanitizer do work actually.
I will try the internal test first. Actually, that was my last comment #90 (comment) ๐
from drishti.
ruslo
commented on May 12, 2024
I figured out how to do it however tiny peculiarity remains. Will ask mailing list: http://www.mail-archive.com/[email protected]/msg17290.html . May be I'm missing something.
from drishti.
ruslo
commented on May 12, 2024
Polly updates in master:
from drishti.
ruslo
commented on May 12, 2024
https://gitlab.kitware.com/cmake/cmake/merge_requests/140
from drishti.
headupinclouds
commented on May 12, 2024
PR #102: Nice! Looks like this is working. Shall we close this one?
from drishti.
ruslo
commented on May 12, 2024
I will remove milestone but I think we need to wait for CMake 3.8 release and update of corresponding Polly toolchain.
from drishti.
headupinclouds
commented on May 12, 2024
I see polly is now using 3.8 for CI builds:
https://github.com/ruslo/polly/blob/master/bin/install-ci-dependencies.py#L141-L158
What is left?
from drishti.
ruslo
commented on May 12, 2024
What is left?
I need to update toolchain itself and test it. My suggestion is to set 0.4 milestone for this, it will not take a long time.
from drishti.
headupinclouds
commented on May 12, 2024
Okay. Just wanted to make more room for the initial static hack. We can add it back.
from drishti.
ruslo
commented on May 12, 2024
Fixed, see ruslo/polly#137
from drishti.
Related Issues (20)
- mac os/android studio build error HOT 11
- Hide drawn curves and draw other effects/ android demo HOT 11
- Build problems because of broken glm verions + gcc 7.3 HOT 2
- Black screen for DrishtiFaceFilter on Android HOT 1
- make model file into lib HOT 5
- Drishti with Xcode 10.2 HOT 19
- Drishti does not compile with linux gcc anymore. HOT 1
- How I compile this project in android studio HOT 3
- Hide Camerapreview in android demo HOT 1
- How I compile this project in Xcode HOT 7
- Build failed-macOS HOT 2
- Example not building for android studio
- Could you give me some hints about the algorithm for this cool project๏ผ
- ABI configuration problem
- drishti works with 'armeabi-v7a', 'arm64-v8a only???
- iOS build failed. HOT 1
- iOS facefilter example error.
- SIGSEGV on System.loadLibrary HOT 1
- iOS 14.5 compile error
- Build android-studio app fail.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from drishti.