boostgsoc13 / boost.afio Goto Github PK
View Code? Open in Web Editor NEWHISTORICAL INTEREST ONLY: Post peer review AFIO v2 rewrite is now at https://github.com/ned14/boost.afio
Home Page: https://github.com/ned14/boost.afio
HISTORICAL INTEREST ONLY: Post peer review AFIO v2 rewrite is now at https://github.com/ned14/boost.afio
Home Page: https://github.com/ned14/boost.afio
This was working, but I forgot to propagate the changes to the new unit test infrastructure.
common.mkdir ......\bin.v2\libs\afio\test\async_io_torture_direct_sync_test.test\gcc-mingw-4.7.2
common.mkdir ......\bin.v2\libs\afio\test\async_io_torture_direct_sync_test.test\gcc-mingw-4.7.2\debug
common.mkdir ......\bin.v2\libs\afio\test\async_io_torture_direct_sync_test.test\gcc-mingw-4.7.2\debug\threading-multi
gcc.compile.c++ ......\bin.v2\libs\afio\test\async_io_torture_direct_sync_test.test\gcc-mingw-4.7.2\debug\threading-multi\async_io_torture_direct_sync_test.o
In file included from ....../boost/asio/detail/win_iocp_handle_service.hpp:319:0,
from ....../boost/asio/detail/win_iocp_serial_port_service.hpp:26,
from ....../boost/asio/serial_port_service.hpp:27,
from ....../boost/asio/basic_serial_port.hpp:30,
from ....../boost/asio.hpp:26,
from ../../../boost/afio/afio.hpp:33,
from test_functions.hpp:18,
from async_io_torture_direct_sync_test.cpp:1:
....../boost/asio/detail/impl/win_iocp_handle_service.ipp: In constructor 'boost::asio::detail::win_iocp_handle_service::overlapped_wrapper::overlapped_wrapper(boost::system::error_code&)':
....../boost/asio/detail/impl/win_iocp_handle_service.ipp:49:44: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
In file included from ....../boost/test/utils/runtime/cla/dual_name_parameter.hpp:19:0,
from ....../boost/test/impl/unit_test_parameters.ipp:31,
from ....../boost/test/included/unit_test.hpp:33,
from test_functions.hpp:29,
from async_io_torture_direct_sync_test.cpp:1:
....../boost/test/utils/runtime/config.hpp: In function 'void boost::runtime::putenv_impl(boost::runtime::cstring, boost::runtime::cstring)':
....../boost/test/utils/runtime/config.hpp:95:51: error: 'putenv' was not declared in this scope
In file included from async_io_torture_direct_sync_test.cpp:1:0:
test_functions.hpp: In function 'void _1000_open_write_close_deletes(std::shared_ptrboost::afio::async_file_io_dispatcher_base, size_t)':
test_functions.hpp:80:76: error: 'to_string' is not a member of 'std'
test_functions.hpp:80:76: note: suggested alternatives:
In file included from ....../boost/exception_ptr.hpp:9:0,
from ....../boost/thread/future.hpp:28,
from ../../../boost/afio/afio.hpp:35,
from test_functions.hpp:18,
from async_io_torture_direct_sync_test.cpp:1:
....../boost/exception/detail/exception_ptr.hpp:487:5: note: 'boost::to_string'
In file included from ....../boost/exception/to_string_stub.hpp:15:0,
from ....../boost/exception/info.hpp:16,
from ....../boost/exception/detail/exception_ptr.hpp:20,
from ....../boost/exception_ptr.hpp:9,
from ....../boost/thread/future.hpp:28,
from ../../../boost/afio/afio.hpp:35,
from test_functions.hpp:18,
from async_io_torture_direct_sync_test.cpp:1:
....../boost/exception/to_string.hpp:72:5: note: 'boost::to_string'
In file included from async_io_torture_direct_sync_test.cpp:1:0:
test_functions.hpp: In function 'void evil_random_io(std::shared_ptrboost::afio::async_file_io_dispatcher_base, size_t, size_t, size_t)':
test_functions.hpp:317:72: error: 'to_string' is not a member of 'std'
test_functions.hpp:317:72: note: suggested alternatives:
In file included from ....../boost/exception_ptr.hpp:9:0,
from ....../boost/thread/future.hpp:28,
from ../../../boost/afio/afio.hpp:35,
from test_functions.hpp:18,
from async_io_torture_direct_sync_test.cpp:1:
....../boost/exception/detail/exception_ptr.hpp:487:5: note: 'boost::to_string'
In file included from ....../boost/exception/to_string_stub.hpp:15:0,
from ....../boost/exception/info.hpp:16,
from ....../boost/exception/detail/exception_ptr.hpp:20,
from ....../boost/exception_ptr.hpp:9,
from ....../boost/thread/future.hpp:28,
from ../../../boost/afio/afio.hpp:35,
from test_functions.hpp:18,
from async_io_torture_direct_sync_test.cpp:1:
....../boost/exception/to_string.hpp:72:5: note: 'boost::to_string'
In file included from async_io_torture_direct_sync_test.cpp:1:0:
test_functions.hpp:426:136: error: 'to_string' was not declared in this scope
test_functions.hpp:426:136: note: suggested alternatives:
In file included from ....../boost/exception_ptr.hpp:9:0,
from ....../boost/thread/future.hpp:28,
from ../../../boost/afio/afio.hpp:35,
from test_functions.hpp:18,
from async_io_torture_direct_sync_test.cpp:1:
....../boost/exception/detail/exception_ptr.hpp:487:5: note: 'boost::to_string'
In file included from ....../boost/exception/to_string_stub.hpp:15:0,
from ....../boost/exception/info.hpp:16,
from ....../boost/exception/detail/exception_ptr.hpp:20,
from ....../boost/exception_ptr.hpp:9,
from ....../boost/thread/future.hpp:28,
from ../../../boost/afio/afio.hpp:35,
from test_functions.hpp:18,
from async_io_torture_direct_sync_test.cpp:1:
....../boost/exception/to_string.hpp:72:5: note: 'boost::to_string'
In file included from async_io_torture_direct_sync_test.cpp:1:0:
test_functions.hpp:451:55: error: 'to_string' was not declared in this scope
test_functions.hpp:451:55: note: suggested alternatives:
In file included from ....../boost/exception_ptr.hpp:9:0,
from ....../boost/thread/future.hpp:28,
from ../../../boost/afio/afio.hpp:35,
from test_functions.hpp:18,
from async_io_torture_direct_sync_test.cpp:1:
....../boost/exception/detail/exception_ptr.hpp:487:5: note: 'boost::to_string'
In file included from ....../boost/exception/to_string_stub.hpp:15:0,
from ....../boost/exception/info.hpp:16,
from ....../boost/exception/detail/exception_ptr.hpp:20,
from ....../boost/exception_ptr.hpp:9,
from ....../boost/thread/future.hpp:28,
from ../../../boost/afio/afio.hpp:35,
from test_functions.hpp:18,
from async_io_torture_direct_sync_test.cpp:1:
....../boost/exception/to_string.hpp:72:5: note: 'boost::to_string'
In file included from ....../boost/thread/shared_mutex.hpp:18:0,
from ....../boost/thread/detail/thread_group.hpp:9,
from ....../boost/thread/thread.hpp:13,
from ../../../boost/afio/afio.hpp:34,
from test_functions.hpp:18,
from async_io_torture_direct_sync_test.cpp:1:
....../boost/thread/win32/shared_mutex.hpp: In instantiation of 'T boost::shared_mutex::interlocked_compare_exchange(T*, T, T) [with T = boost::shared_mutex::state_data]':
....../boost/thread/win32/shared_mutex.hpp:124:103: required from here
....../boost/thread/win32/shared_mutex.hpp:52:99: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
....../boost/thread/win32/shared_mutex.hpp:52:99: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
....../boost/thread/win32/shared_mutex.hpp:53:52: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
....../boost/thread/win32/shared_mutex.hpp:53:52: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
afio_iocp.ipp 1385
do
{
...
BOOL ok=DeviceIoControl(p->native_handle(), FSCTL_QUERY_ALLOCATED_RANGES, buffers->data(), sizeof(FILE_ALLOCATED_RANGE_BUFFER), buffers->data(), (DWORD)(buffers->size()*sizeof(FILE_ALLOCATED_RANGE_BUFFER)), &bytesout, ol.get());
DWORD errcode=GetLastError();
if(!ok && ERROR_IO_PENDING!=errcode)
{
if(ERROR_INSUFFICIENT_BUFFER==errcode || ERROR_MORE_DATA==errcode)
{
buffers->resize(buffers->size()*2);
continue; // < ==========
}
//std::cerr << "ERROR " << errcode << std::endl;
asio::error_code ec(errcode, asio::error::get_system_category());
ol.complete(ec, ol.get()->InternalHigh);
}
else
ol.release();
} while(false); // < ==========
"continue" will terminate the loop instead of continuing it.
This is due to a bug in the WOW64 syscall translation layer. Microsoft have told me they won't fix it as I'm not supposed to be using NT kernel APIs directly, so I'll need to figure out a workaround.
Specifically:
clang-linux.link bin.v2/libs/afio/test/async_io_works_64_sync_test.test/clang-linux-3.3/debug/threading-multi/async_io_works_64_sync_test
bin.v2/libs/afio/test/async_io_works_64_sync_test.test/clang-linux-3.3/debug/threading-multi/async_io_works_64_sync_test.o: In function boost::detail::lexical_cast_do_cast<int, boost::unit_test::basic_cstring<char const> >::lexical_cast_impl(boost::unit_test::basic_cstring<char const> const&)': /home/jenkins-slave/workspace/Boost.AFIO Build Linux clang 3.3 + libc++/boost-trunk/./boost/lexical_cast.hpp:2351: undefined reference to
typeinfo for int'
bin.v2/libs/afio/test/async_io_works_64_sync_test.test/clang-linux-3.3/debug/threading-multi/async_io_works_64_sync_test.o: In function boost::detail::lexical_cast_do_cast<long, boost::unit_test::basic_cstring<char const> >::lexical_cast_impl(boost::unit_test::basic_cstring<char const> const&)': /home/jenkins-slave/workspace/Boost.AFIO Build Linux clang 3.3 + libc++/boost-trunk/./boost/lexical_cast.hpp:2351: undefined reference to
typeinfo for long'
bin.v2/libs/afio/test/async_io_works_64_sync_test.test/clang-linux-3.3/debug/threading-multi/async_io_works_64_sync_test.o: In function boost::detail::lexical_cast_do_cast<bool, boost::unit_test::basic_cstring<char const> >::lexical_cast_impl(boost::unit_test::basic_cstring<char const> const&)': /home/jenkins-slave/workspace/Boost.AFIO Build Linux clang 3.3 + libc++/boost-trunk/./boost/lexical_cast.hpp:2351: undefined reference to
typeinfo for bool'
bin.v2/libs/afio/test/async_io_works_64_sync_test.test/clang-linux-3.3/debug/threading-multi/async_io_works_64_sync_test.o: In function boost::detail::lexical_cast_do_cast<unsigned int, boost::unit_test::basic_cstring<char const> >::lexical_cast_impl(boost::unit_test::basic_cstring<char const> const&)': /home/jenkins-slave/workspace/Boost.AFIO Build Linux clang 3.3 + libc++/boost-trunk/./boost/lexical_cast.hpp:2351: undefined reference to
typeinfo for unsigned int'
bin.v2/libs/afio/test/async_io_works_64_sync_test.test/clang-linux-3.3/debug/threading-multi/async_io_works_64_sync_test.o: In function boost::exception const* boost::exception_detail::get_boost_exception<std::domain_error>(std::domain_error const*)': /home/jenkins-slave/workspace/Boost.AFIO Build Linux clang 3.3 + libc++/boost-trunk/./boost/exception/detail/exception_ptr.hpp:257: undefined reference to
__dynamic_cast'
bin.v2/libs/afio/test/async_io_works_64_sync_test.test/clang-linux-3.3/debug/threading-multi/async_io_works_64_sync_test.o: In function boost::exception const* boost::exception_detail::get_boost_exception<std::invalid_argument>(std::invalid_argument const*)': /home/jenkins-slave/workspace/Boost.AFIO Build Linux clang 3.3 + libc++/boost-trunk/./boost/exception/detail/exception_ptr.hpp:257: undefined reference to
__dynamic_cast'
bin.v2/libs/afio/test/async_io_works_64_sync_test.test/clang-linux-3.3/debug/threading-multi/async_io_works_64_sync_test.o: In function boost::exception const* boost::exception_detail::get_boost_exception<std::length_error>(std::length_error const*)': /home/jenkins-slave/workspace/Boost.AFIO Build Linux clang 3.3 + libc++/boost-trunk/./boost/exception/detail/exception_ptr.hpp:257: undefined reference to
__dynamic_cast'
bin.v2/libs/afio/test/async_io_works_64_sync_test.test/clang-linux-3.3/debug/threading-multi/async_io_works_64_sync_test.o: In function boost::exception const* boost::exception_detail::get_boost_exception<std::out_of_range>(std::out_of_range const*)': /home/jenkins-slave/workspace/Boost.AFIO Build Linux clang 3.3 + libc++/boost-trunk/./boost/exception/detail/exception_ptr.hpp:257: undefined reference to
__dynamic_cast'
bin.v2/libs/afio/test/async_io_works_64_sync_test.test/clang-linux-3.3/debug/threading-multi/async_io_works_64_sync_test.o: In function boost::exception const* boost::exception_detail::get_boost_exception<std::logic_error>(std::logic_error const*)': /home/jenkins-slave/workspace/Boost.AFIO Build Linux clang 3.3 + libc++/boost-trunk/./boost/exception/detail/exception_ptr.hpp:257: undefined reference to
__dynamic_cast'
bin.v2/libs/afio/test/async_io_works_64_sync_test.test/clang-linux-3.3/debug/threading-multi/async_io_works_64_sync_test.o:/home/jenkins-slave/workspace/Boost.AFIO Build Linux clang 3.3 + libc++/boost-trunk/./boost/exception/detail/exception_ptr.hpp:257: more undefined references to __dynamic_cast' follow bin.v2/libs/afio/test/async_io_works_64_sync_test.test/clang-linux-3.3/debug/threading-multi/async_io_works_64_sync_test.o:(.data.rel.ro._ZTIN5boost9unit_test12report_levelE[_ZTIN5boost9unit_test12report_levelE]+0x0): undefined reference to
vtable for __cxxabiv1::__enum_type_info'
bin.v2/libs/afio/test/async_io_works_64_sync_test.test/clang-linux-3.3/debug/threading-multi/async_io_works_64_sync_test.o:(.data.rel.ro._ZTIN5boost9unit_test9log_levelE[_ZTIN5boost9unit_test9log_levelE]+0x0): undefined reference to vtable for __cxxabiv1::__enum_type_info' bin.v2/libs/afio/test/async_io_works_64_sync_test.test/clang-linux-3.3/debug/threading-multi/async_io_works_64_sync_test.o:(.data.rel.ro._ZTIN5boost9unit_test13output_formatE[_ZTIN5boost9unit_test13output_formatE]+0x0): undefined reference to
vtable for __cxxabiv1::__enum_type_info'
bin.v2/libs/afio/test/async_io_works_64_sync_test.test/clang-linux-3.3/debug/threading-multi/async_io_works_64_sync_test.o:(.data.rel+0x40): undefined reference to `typeinfo for char const*'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Right now, the unit and functional testing mostly tests success. For any library which can cause data loss like this one, very robust error handling functional testing is needing. I'm think that as an absolute minimum we need to mount a filing system via a looped back file and intentionally test out of disc space error handling on it for correctness.
Other idea is to deliberately bitflip corrupt the loopback file to test how we handle weird filing system errors.
I have already mapped in the code examples per API in the documentation, but they are currently unimplemented. I'll also need the examples added to the unit test suite, might as well kill two birds with one stone.
Straight from /analyze
C6011 Dereferencing NULL pointer 'q'.
40 Enter this branch, (assume 'first_')
42 Skip this branch, (assume 'q==first_' is false)
49 Enter this loop, (assume 'p->next_')
51 Enter this branch, (assume 'p->next_==q')
53 'q' is dereferenced, but may still be NULL
Does your CI run /analyze
? ( I'm not familiar with CI )
Right now on Linux we keep an open handle to the containing directory of each open file because ext2 and some mount flags of ext3 and ext4 require you to fsync the directory every time you add or delete file entries. Such an open file handle makes enumerating that directory super quick, so might as well do so.
Note that using the kernel syscall directly is preferred - glibc's readdir() implementation is notoriously unable to scale well.
We should also add a unit test where one creates a million files, and tries enumerating them. That normally murders Linux, but our direct syscall implementation ought not to suffer from that.
I must admit, I haven't thought of a suitable quick start mini-project yet. Maybe a storage benchmarking tool or something? Maybe a benchmarker of filing system caching behaviour? I don't know.
Failed hard drive replacement is happening as I type this.
In afio.ipp
, at line 33, the definition of BOOST_AFIO_DEBUG_PRINT
is controlled by a(n always false) conditional.
Rather than hacking away at header files (which I usually set to read-only), perhaps another #define
, something like ENABLE_BOOST_AFIO_DEBUG_PRINT
would be better?
i.e. #if defined(DEBUG) && defined(ENABLE_BOOST_AFIO_DEBUG_PRINT)
Keep up the great work!
BTRFS on Linux and ZFS on Linux both provide very fast constant time filing system snapshotting. Snapshots can be used to avoid needing to use O_SYNC or fsync to enforce data writing order with really fast performance, and therefore we ought to support those. For Windows, it might be worth investigating Transactional NTFS, though I believe that is supposed to be deprecated now.
Need to rewrite the appropriate parts of Boost.ASIO to fix this problem seeing as its maintainer has left the bug report languish.
Make sure it will not overwrite one of these pseudo symlinks. Might want to add a flag to stat_t about pseudo-symlinks too.
Then the afio::future::then_XXX expands into:
afio::future::then([args...](afio::future f){ return f->XXX(args...);})
In other words, when this future signals, schedule some other operation on the newly ready future.
The Hello World example has a bug in the try
-block:
boost::afio::async_path_op_req req("example_file.txt",
boost::afio::file_flags::ReadWrite);
Another file flag is needed for correct operation: file_flags::Create
or maybe file_flags::CreateOnlyIfNotExists
. Otherwise the file is never created and the rest of the example is garbage.
This may very well be a false positive from /analyze
, but I can't rule out some subtle bug.
/analyze
says, in very many words:
C6001 Using uninitialized memory 'buffer'
75 'buffer' is not initialized
84 Skip this branch, (assume '<branch condition>' is false)
93 Skip this branch, (assume 'needInternal' is false)
100 Skip this branch, (assume 'needBasic' is false)
107 Skip this branch, (assume 'needStandard' is false)
115 Skip this branch, (assume '<branch condition>' is false)
128 Skip this branch, (assume '!!((&(wanted, ino)))' is false)
129 Skip this branch, (assume '!!((&(wanted, type)))' is false)
130 Skip this branch, (assume '!!((&(wanted, nlink)))' is false)
131 Skip this branch, (assume '!!((&(wanted, atim)))' is false)
132 Skip this branch, (assume '!!((&(wanted, mtim)))' is false)
133 Skip this branch, (assume '!!((&(wanted, ctim)))' is false)
138 Enter this branch, (assume '!!((&(wanted, birthtim)))')
138 'buffer' is used, but may not have been initialized
This very well might be just /analyze
not properly handling the unusual branch condition at 84 (if((needInternal+needBasic+needStandard)>=2)
) - as that line generates a C6323- but direntry
is complex enough that I figured that I should bring it to your attention nonetheless.
In afio.ipp
, at 2116, std::filesystem::path::value_type buffer
is initialized with the statement:
BOOST_AFIO_TYPEALIGNMENT(8) std::filesystem::path::value_type buffer[sizeof(FILE_ALL_INFORMATION)/sizeof(std::filesystem::path::value_type)+32769];
And then on 2122, it's cast to a (FILE_ID_FULL_DIR_INFORMATION *)
. Is this intended?
A static_assert( sizeof( FILE_ID_FULL_DIR_INFORMATION ) == sizeof( FILE_ALL_INFORMATION ), "different sizes!" );
fails on my end.
Looking at FILE_ALL_INFORMATION
and FILE_ID_FULL_DIR_INFORMATION
has me concerned ( but not certain due to alignment weirdness ) that this might not yield valid data. CreationTime
for example, is the third field in FILE_ID_FULL_DIR_INFORMATION
, but the first in FILE_ALL_INFORMATION
- they're both file times, so I think it's plausible that if true, this might've been missed in testing.
I've tried building from scratch, and when I got to the b2 libs/afio/build
step, I hit this:
C:\Users\Alexander Riccio\boost\boost_1_57_0>"C:\Users\Alexander Riccio\boostbuild_1_57\bin\b2.exe" libs/afio/build
warning: No toolsets are configured.
warning: Configuring default toolset "msvc".
warning: If the default is wrong, your build may not work correctly.
warning: Use the "toolset=xxxxx" option to override our guess.
warning: For more configuration options, please consult
warning: http://boost.org/boost-build2/doc/html/bbv2/advanced/configuration.html
Performing configuration checks
- symlinks supported : no (cached)
- junctions supported : yes (cached)
- hardlinks supported : yes (cached)
...patience...
...patience...
...found 2560 targets...
...updating 4 targets...
compile-c-c++ bin.v2\libs\afio\build\msvc-12.0\debug\threading-multi\afio.obj
afio.cpp
c:\users\alexander riccio\boost\boost_1_57_0\boost\afio\config.hpp(58) : fatal error C1083: Cannot open include file: 'bindlib/include/import.h': No such file or directory
call "C:\Users\ALEXAN~1\AppData\Local\Temp\b2_msvc_12.0_vcvarsall_x86.cmd" >nul
cl /Zm800 -nologo @"bin.v2\libs\afio\build\msvc-12.0\debug\threading-multi\afio.obj.rsp"
...failed compile-c-c++ bin.v2\libs\afio\build\msvc-12.0\debug\threading-multi\afio.obj...
...skipped <pbin.v2\libs\afio\build\msvc-12.0\debug\threading-multi>boost_afio-vc120-mt-gd-1_57.dll for lack of <pbin.v2\libs\afio\build\msvc-12.0\debug\threading-multi>afio.obj...
...skipped <pbin.v2\libs\afio\build\msvc-12.0\debug\threading-multi>boost_afio-vc120-mt-gd-1_57.lib for lack of <pbin.v2\libs\afio\build\msvc-12.0\debug\threading-multi>afio.obj...
...skipped <pbin.v2\libs\afio\build\msvc-12.0\debug\threading-multi>boost_afio-vc120-mt-gd-1_57.pdb for lack of <pbin.v2\libs\afio\build\msvc-12.0\debug\threading-multi>afio.obj...
...failed updating 1 target...
...skipped 3 targets...
specifically: c:\users\alexander riccio\boost\boost_1_57_0\boost\afio\config.hpp(58) : fatal error C1083: Cannot open include file: 'bindlib/include/import.h': No such file or directory
The documentation says in several places that AFIO is a "Boost.BindLib based Boost library", so I doubt this in intended. Perhaps that's just evidence of a mistake on my part?
I can't spot BindLib anywhere in Boost 1.57, or in AFIO itself, so I'm wondering if it is somehow specially configured on your machine?
Of course, the fact that you're the BindLib dev is maximally convenient ;)
ReFS doesn't support compressed files, hard links nor GUID paths, so these tests error out.
As stated in the title. The function is documented in the doxygen version of the documentation.
There is a lot of conflicting information out there regarding the need for directory fsyncing on Linux, and exactly when and when not one should fsync directories (why oh why can't Linux just be POSIX conformant! This isn't an issue on FreeBSD or any other platform).
Furthermore, directory fsyncing seems to be mostly a ext2/ext3/ext4 problem. BTRFS, XFS, JFS and others from what I know don't have this problem, but I really ought to confirm for sure.
The less toy example has a bug in the parsing of the input files. It loops, incrementing argc
instead of n
, as it should.
Even after fixing that I cannot get the program to work, and I am unsure exactly what is wrong. The output file for the concatenated inputs is created and truncate
d to the correct size, but no data ever gets written to this file. If I use large files I have seen std::invalid_argument
exceptions and even segfaults. I think the problem may be related to the buffers; I have tried using a simple std::vector<char>
for the file_buffer_type
, but this makes no difference.
UPDATE: So, I've been hacking a bit at this, and it turns out the buffer is not the problem. The problem seems to be here:
// Schedule a filling of buffer from offset o after last has completed
auto readchunk=dispatcher->read(make_async_data_op_req(last, buffer->data(),
thischunk, o));
// Schedule a writing of buffer to offset offset+o after readchunk is ready
auto writechunk=dispatcher->write(make_async_data_op_req(readchunk,
buffer->data(), thischunk, offset+o));
The precondition for the write
is the previous readchunk
future. This holds an io_op to the input file, so we are trying to write to the input file again. This file is opened for reading only, so the operation fails.
We're stuck until FreeBSD fixes their lack of __cxa_thread_atexit which causes thread local cleanup to not work.
gcc.compile.c++ bin.v2/libs/afio/build/gcc-4.7/debug/threading-multi/afio.o
/usr/include/c++/4.7/type_traits: In substitution of 'template<class _From1, class _To1> static decltype ((__test_aux<_To1>(declval<_From1>()), std::__sfinae_types::__one())) std::__is_convertible_helper<_From, _To, false>::__test(int) [with _From1 = _From1; _To1 = _To1; _From = const boost::afio::detail::async_file_io_dispatcher_op&; _To = boost::afio::detail::async_file_io_dispatcher_op] [with _From1 = const boost::afio::detail::async_file_io_dispatcher_op&; _To1 = boost::afio::detail::async_file_io_dispatcher_op]':
/usr/include/c++/4.7/type_traits:1258:70: required from 'constexpr const bool std::__is_convertible_helper<const boost::afio::detail::async_file_io_dispatcher_op&, boost::afio::detail::async_file_io_dispatcher_op, false>::value'
/usr/include/c++/4.7/type_traits:1263:12: required from 'struct std::is_convertible<const boost::afio::detail::async_file_io_dispatcher_op&, boost::afio::detail::async_file_io_dispatcher_op>'
/usr/include/c++/4.7/type_traits:116:12: required from 'struct std::_and<std::is_convertible<const unsigned int&, const unsigned int>, std::is_convertible<const boost::afio::detail::async_file_io_dispatcher_op&, boost::afio::detail::async_file_io_dispatcher_op> >'
/usr/include/c++/4.7/bits/stl_pair.h:113:38: required from 'struct std::__is_direct_constructible_impl<std::pair<const unsigned int, boost::afio::detail::async_file_io_dispatcher_op>, const std::pair<const unsigned int, boost::afio::detail::async_file_io_dispatcher_op>&>'
/usr/include/c++/4.7/type_traits:116:12: required from 'struct std::_and<std::is_destructible<std::pair<const unsigned int, boost::afio::detail::async_file_io_dispatcher_op> >, std::__is_direct_constructible_impl<std::pair<const unsigned int, boost::afio::detail::async_file_io_dispatcher_op>, const std::pair<const unsigned int, boost::afio::detail::async_file_io_dispatcher_op>&> >'
/usr/include/c++/4.7/type_traits:778:12: [ skipping 5 instantiation contexts ]
/usr/include/c++/4.7/type_traits:929:12: required from 'struct std::__is_copy_constructible_impl<std::pair<const unsigned int, boost::afio::detail::async_file_io_dispatcher_op>, false>'
/usr/include/c++/4.7/type_traits:935:12: required from 'struct std::is_copy_constructible<std::pair<const unsigned int, boost::afio::detail::async_file_io_dispatcher_op> >'
/usr/include/c++/4.7/bits/alloc_traits.h:542:12: required from 'struct std::__is_copy_insertable<std::allocator<std::pair<const unsigned int, boost::afio::detail::async_file_io_dispatcher_op> > >'
/usr/include/c++/4.7/bits/alloc_traits.h:562:63: required from 'class std::__unordered_map<unsigned int, boost::afio::detail::async_file_io_dispatcher_op, std::hash, std::equal_to, std::allocator<std::pair<const unsigned int, boost::afio::detail::async_file_io_dispatcher_op> >, false>'
/usr/include/c++/4.7/bits/unordered_map.h:264:11: required from 'class std::unordered_map<unsigned int, boost::afio::detail::async_file_io_dispatcher_op>'
libs/afio/src/afio.cpp:381:101: required from here
libs/afio/src/afio.cpp:371:3: error: 'boost::afio::detail::async_file_io_dispatcher_op::async_file_io_dispatcher_op(const boost::afio::detail::async_file_io_dispatcher_op&)' is private
In file included from libs/afio/src/../../../boost/afio/afio.hpp:19:0,
from libs/afio/src/afio.cpp:22:
/usr/include/c++/4.7/type_traits:1252:2: error: within this context
/usr/include/c++/4.7/type_traits: In substitution of 'template<class _From1, class _To1> static decltype ((__test_aux<_To1>(declval<_From1>()), std::__sfinae_types::__one())) std::__is_convertible_helper<_From, _To, false>::__test(int) [with _From1 = _From1; _To1 = _To1; _From = const boost::afio::detail::async_file_io_dispatcher_op&; _To = boost::afio::detail::async_file_io_dispatcher_op] [with _From1 = const boost::afio::detail::async_file_io_dispatcher_op&; _To1 = boost::afio::detail::async_file_io_dispatcher_op]':
/usr/include/c++/4.7/type_traits:1258:70: required from 'constexpr const bool std::__is_convertible_helper<const boost::afio::detail::async_file_io_dispatcher_op&, boost::afio::detail::async_file_io_dispatcher_op, false>::value'
/usr/include/c++/4.7/type_traits:1263:12: required from 'struct std::is_convertible<const boost::afio::detail::async_file_io_dispatcher_op&, boost::afio::detail::async_file_io_dispatcher_op>'
/usr/include/c++/4.7/type_traits:116:12: required from 'struct std::_and<std::is_convertible<const unsigned int&, const unsigned int>, std::is_convertible<const boost::afio::detail::async_file_io_dispatcher_op&, boost::afio::detail::async_file_io_dispatcher_op> >'
/usr/include/c++/4.7/bits/stl_pair.h:113:38: required from 'struct std::__is_direct_constructible_impl<std::pair<const unsigned int, boost::afio::detail::async_file_io_dispatcher_op>, const std::pair<const unsigned int, boost::afio::detail::async_file_io_dispatcher_op>&>'
/usr/include/c++/4.7/type_traits:116:12: required from 'struct std::_and<std::is_destructible<std::pair<const unsigned int, boost::afio::detail::async_file_io_dispatcher_op> >, std::__is_direct_constructible_impl<std::pair<const unsigned int, boost::afio::detail::async_file_io_dispatcher_op>, const std::pair<const unsigned int, boost::afio::detail::async_file_io_dispatcher_op>&> >'
/usr/include/c++/4.7/type_traits:778:12: [ skipping 5 instantiation contexts ]
/usr/include/c++/4.7/type_traits:929:12: required from 'struct std::__is_copy_constructible_impl<std::pair<const unsigned int, boost::afio::detail::async_file_io_dispatcher_op>, false>'
/usr/include/c++/4.7/type_traits:935:12: required from 'struct std::is_copy_constructible<std::pair<const unsigned int, boost::afio::detail::async_file_io_dispatcher_op> >'
/usr/include/c++/4.7/bits/alloc_traits.h:542:12: required from 'struct std::__is_copy_insertable<std::allocator<std::pair<const unsigned int, boost::afio::detail::async_file_io_dispatcher_op> > >'
/usr/include/c++/4.7/bits/alloc_traits.h:562:63: required from 'class std::__unordered_map<unsigned int, boost::afio::detail::async_file_io_dispatcher_op, std::hash, std::equal_to, std::allocator<std::pair<const unsigned int, boost::afio::detail::async_file_io_dispatcher_op> >, false>'
/usr/include/c++/4.7/bits/unordered_map.h:264:11: required from 'class std::unordered_map<unsigned int, boost::afio::detail::async_file_io_dispatcher_op>'
libs/afio/src/afio.cpp:381:101: required from here
libs/afio/src/afio.cpp:371:3: error: 'boost::afio::detail::async_file_io_dispatcher_op::async_file_io_dispatcher_op(const boost::afio::detail::async_file_io_dispatcher_op&)' is private
In file included from libs/afio/src/../../../boost/afio/afio.hpp:19:0,
from libs/afio/src/afio.cpp:22:
/usr/include/c++/4.7/type_traits:1252:2: error: within this context
/usr/include/c++/4.7/type_traits: In instantiation of 'constexpr const bool std::__is_convertible_helper<const boost::afio::detail::async_file_io_dispatcher_op&, boost::afio::detail::async_file_io_dispatcher_op, false>::value':
/usr/include/c++/4.7/type_traits:1263:12: required from 'struct std::is_convertible<const boost::afio::detail::async_file_io_dispatcher_op&, boost::afio::detail::async_file_io_dispatcher_op>'
/usr/include/c++/4.7/type_traits:116:12: required from 'struct std::_and<std::is_convertible<const unsigned int&, const unsigned int>, std::is_convertible<const boost::afio::detail::async_file_io_dispatcher_op&, boost::afio::detail::async_file_io_dispatcher_op> >'
/usr/include/c++/4.7/bits/stl_pair.h:113:38: required from 'struct std::__is_direct_constructible_impl<std::pair<const unsigned int, boost::afio::detail::async_file_io_dispatcher_op>, const std::pair<const unsigned int, boost::afio::detail::async_file_io_dispatcher_op>&>'
/usr/include/c++/4.7/type_traits:116:12: required from 'struct std::_and<std::is_destructible<std::pair<const unsigned int, boost::afio::detail::async_file_io_dispatcher_op> >, std::__is_direct_constructible_impl<std::pair<const unsigned int, boost::afio::detail::async_file_io_dispatcher_op>, const std::pair<const unsigned int, boost::afio::detail::async_file_io_dispatcher_op>&> >'
/usr/include/c++/4.7/type_traits:778:12: required from 'struct std::__is_direct_constructible_new_safe<std::pair<const unsigned int, boost::afio::detail::async_file_io_dispatcher_op>, const std::pair<const unsigned int, boost::afio::detail::async_file_io_dispatcher_op>&>'
/usr/include/c++/4.7/type_traits:856:12: [ skipping 4 instantiation contexts ]
/usr/include/c++/4.7/type_traits:929:12: required from 'struct std::__is_copy_constructible_impl<std::pair<const unsigned int, boost::afio::detail::async_file_io_dispatcher_op>, false>'
/usr/include/c++/4.7/type_traits:935:12: required from 'struct std::is_copy_constructible<std::pair<const unsigned int, boost::afio::detail::async_file_io_dispatcher_op> >'
/usr/include/c++/4.7/bits/alloc_traits.h:542:12: required from 'struct std::__is_copy_insertable<std::allocator<std::pair<const unsigned int, boost::afio::detail::async_file_io_dispatcher_op> > >'
/usr/include/c++/4.7/bits/alloc_traits.h:562:63: required from 'class std::__unordered_map<unsigned int, boost::afio::detail::async_file_io_dispatcher_op, std::hash, std::equal_to, std::allocator<std::pair<const unsigned int, boost::afio::detail::async_file_io_dispatcher_op> >, false>'
/usr/include/c++/4.7/bits/unordered_map.h:264:11: required from 'class std::unordered_map<unsigned int, boost::afio::detail::async_file_io_dispatcher_op>'
libs/afio/src/afio.cpp:381:101: required from here
libs/afio/src/afio.cpp:371:3: error: 'boost::afio::detail::async_file_io_dispatcher_op::async_file_io_dispatcher_op(const boost::afio::detail::async_file_io_dispatcher_op&)' is private
In file included from libs/afio/src/../../../boost/afio/afio.hpp:19:0,
from libs/afio/src/afio.cpp:22:
/usr/include/c++/4.7/type_traits:1258:70: error: within this context
All of the annotations in nt_kernel_stuff are commented out....what's up with that?
Is it a MinGW thing? I think you can include nosal.h
?
Immediately after filing #61 , I went ahead and made the change in my headers, and the test code I'm playing with failed to compile, due to an undeclared identifier for a format string, in afio_iocp.ipp at line 466.
Everything works fine when it is removed.
Usefully, this FreeBSD 10 setup is almost that for Apple Mac OS X, and therefore if it works here it almost certainly should cover OS X too.
/analyze
found a read overrun in ErrorHandling.ipp.
The call to FormatMessageA at line 31 may fail ( as SAL would describe it ), and len
equals zero.
The read at line 33 (to Remove annoying CRLF at end of message sometimes
) then accesses buffer[len-1]
, which is buffer[0-1]
.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.