russross / s3fslite Goto Github PK
View Code? Open in Web Editor NEWA file system on Amazon S3, with write-back caching of data and a persistent metadata cache
License: GNU General Public License v2.0
A file system on Amazon S3, with write-back caching of data and a persistent metadata cache
License: GNU General Public License v2.0
g++ -Wall -Wextra -Wno-unused-parameter -Werror -Os -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2 -c -o common.o common.cpp
g++ -Wall -Wextra -Wno-unused-parameter -Werror -Os -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2 -c -o fileinfo.o fileinfo.cpp
g++ -Wall -Wextra -Wno-unused-parameter -Werror -Os -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2 -c -o attrcache.o attrcache.cpp
g++ -Wall -Wextra -Wno-unused-parameter -Werror -Os -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2 -c -o filecache.o filecache.cpp
g++ -Wall -Wextra -Wno-unused-parameter -Werror -Os -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2 -c -o s3request.o s3request.cpp
s3request.cpp: In static member function ‘static void S3request::put_file(Fileinfo_, int)’:
s3request.cpp:590: error: ‘CURLOPT_SEEKDATA’ was not declared in this scope
s3request.cpp:591: error: ‘CURLOPT_SEEKFUNCTION’ was not declared in this scope
make: *_* [s3request.o] Error 1
Hi there
tcpdump shows a mass of DNS lookups (over 50/sec) for the s3.amazonaws.com bucket name when rsync is running against it. Would there be a performance benefit in s3fs caching the returned IP addresses for (say) 10sec? (shouldn't be too long as I suspect those can change) - might speed up the associated 50 HTTP requests/sec that occur next ;-)
Jason
Hi Russ!
as per title, if /etc.mime.types does not exist then the parser in s3fs.cpp ('// load the list of mime types') goes into an endless loop as eof never goes to true and the process then uses 100% cpu.
quick fix was to create an empty mime.types.
regards
charlie
dear russ,
maybe this is not intended at all, but i wanted to give it a try. i installed all dependencies using macports, but s3fslite still won't compile. i have absolutely no clue about c-coding.
(s3fs compiles, but doesn't work, so i came here. plus i mainly want to use it with rsync anyway)
this is the make output:
g++ -Wall -Wextra -Wno-unused-parameter -Werror -Os -D__FreeBSD__=10 -D_FILE_OFFSET_BITS=64 -I/opt/local/include/fuse -I/opt/local/include -I/opt/local/include -I/opt/local/include/libxml2 -c -o common.o common.cpp
g++ -Wall -Wextra -Wno-unused-parameter -Werror -Os -D__FreeBSD__=10 -D_FILE_OFFSET_BITS=64 -I/opt/local/include/fuse -I/opt/local/include -I/opt/local/include -I/opt/local/include/libxml2 -c -o fileinfo.o fileinfo.cpp
fileinfo.cpp: In constructor ‘Fileinfo::Fileinfo(std::string, stat_)’:
fileinfo.cpp:15: error: invalid use of incomplete type ‘struct stat’
fileinfo.h:16: error: forward declaration of ‘struct stat’
fileinfo.cpp:15: error: invalid use of incomplete type ‘struct stat’
fileinfo.h:16: error: forward declaration of ‘struct stat’
fileinfo.cpp:15: error: invalid use of incomplete type ‘struct stat’
fileinfo.h:16: error: forward declaration of ‘struct stat’
fileinfo.cpp:15: error: invalid use of incomplete type ‘struct stat’
fileinfo.h:16: error: forward declaration of ‘struct stat’
fileinfo.cpp:16: error: invalid use of incomplete type ‘struct stat’
fileinfo.h:16: error: forward declaration of ‘struct stat’
fileinfo.cpp: In member function ‘void Fileinfo::toStat(stat_)’:
fileinfo.cpp:31: error: invalid application of ‘sizeof’ to incomplete type ‘stat’
fileinfo.cpp:32: error: invalid use of incomplete type ‘struct stat’
fileinfo.h:16: error: forward declaration of ‘struct stat’
fileinfo.cpp:33: error: invalid use of incomplete type ‘struct stat’
fileinfo.h:16: error: forward declaration of ‘struct stat’
fileinfo.cpp:34: error: invalid use of incomplete type ‘struct stat’
fileinfo.h:16: error: forward declaration of ‘struct stat’
fileinfo.cpp:35: error: invalid use of incomplete type ‘struct stat’
fileinfo.h:16: error: forward declaration of ‘struct stat’
fileinfo.cpp:36: error: invalid use of incomplete type ‘struct stat’
fileinfo.h:16: error: forward declaration of ‘struct stat’
fileinfo.cpp:37: error: invalid use of incomplete type ‘struct stat’
fileinfo.h:16: error: forward declaration of ‘struct stat’
fileinfo.cpp:38: error: invalid use of incomplete type ‘struct stat’
fileinfo.h:16: error: forward declaration of ‘struct stat’
fileinfo.cpp:39: error: invalid use of incomplete type ‘struct stat’
fileinfo.h:16: error: forward declaration of ‘struct stat’
fileinfo.cpp:40: error: ‘S_ISREG’ was not declared in this scope
fileinfo.cpp:41: error: invalid use of incomplete type ‘struct stat’
fileinfo.h:16: error: forward declaration of ‘struct stat’
fileinfo.cpp:41: error: invalid use of incomplete type ‘struct stat’
fileinfo.h:16: error: forward declaration of ‘struct stat’
make: *** [fileinfo.o] Error 1
any idea?
cheers, yanone
Patch for trunk 20100518 version:
--- s3fs_org.cpp 2010-06-08 08:52:30.950200000 +0200 +++ s3fs.cpp 2010-06-08 13:32:55.357720457 +0200 @@ -999,7 +999,7 @@ int main(int argc, char *argv[]) { if (AWSSecretAccessKey.size() == 0) { std::string line; std::ifstream passwd("/etc/passwd-s3fs"); - while (getline(passwd, line)) { + while (!getline(passwd, line).eof()) { if (line[0]=='#') continue; size_t pos = line.find(':'); @@ -1055,7 +1055,7 @@ int main(int argc, char *argv[]) { // load the list of mime types std::string line; std::ifstream passwd("/etc/mime.types"); - while (getline(passwd, line)) { + while (!getline(passwd, line).eof()) { if (line[0] == '#') continue; std::stringstream tmp(line);
Patch for trunk 20100518 version:
--- s3request_org.cpp 2010-06-08 08:52:30.950200000 +0200 +++ s3request.cpp 2010-06-08 08:57:38.687515801 +0200 @@ -681,6 +681,9 @@ bool S3request::get_directory(std::strin bool moretocome = false; marker = ""; + if (path != "/") + prefix = path.substr(1) + "/"; + // parse the response xmlDocPtr doc = xmlReadMemory(response.c_str(), response.size(), "", NULL, 0);
If I create a folder I also get an empty file with the mime type 'application/x-directory'. If I create a file in this folder then I see this file both in the folder and in the root. This behavior confuses rsync a lot!
I have this problem on Ubuntu Lucid Lynx, but also on the Freetz platform.
Issued commands:
mkdir folder
ls -> "folder"
cd folder
touch file
cd ..
ls -Ral
.:
drwxr-xr-x 1 root root 0 May 5 12:48 .
drwxr-xr-x 4 bittorre users 4096 May 6 21:02 ..
-rw-r--r-- 1 root root 0 May 6 21:01 file
drwxr-xr-x 1 root root 0 May 6 21:01 folder
./folder:
drwxr-xr-x 1 root root 0 May 6 21:01 .
drwxr-xr-x 1 root root 0 May 5 12:48 ..
-rw-r--r-- 1 root root 0 May 6 21:01 file
Syslog:
May 6 21:01:45 fritz user.info syslog: getattr[/folder]
May 6 21:01:45 fritz user.info syslog: Filecache::new[/folder]
May 6 21:01:45 fritz user.info syslog: S3request::get_fileinfo[/folder]
May 6 21:01:45 fritz user.info syslog: getattr[/folder]: File not found
May 6 21:01:45 fritz user.info syslog: mkdir[/folder] mode[0755]
May 6 21:01:45 fritz user.info syslog: getattr[/folder]
May 6 21:01:46 fritz user.info syslog: Filecache::from_queue resurrecting file[/folder]
May 6 21:01:47 fritz user.info syslog: getattr[/]
May 6 21:01:47 fritz user.info syslog: Filecache::new[/]
May 6 21:01:47 fritz user.info syslog: getattr[/]
May 6 21:01:47 fritz user.info syslog: getattr[/]
May 6 21:01:47 fritz user.info syslog: readdir[/] offset[0]
May 6 21:01:47 fritz user.info syslog: Filecache::sync all
May 6 21:01:47 fritz user.info syslog: Filecache::fsync[/]
May 6 21:01:47 fritz user.info syslog: Filecache::fsync[/folder]
May 6 21:01:47 fritz user.info syslog: fsync: !exists || dirty_data
May 6 21:01:47 fritz user.info syslog: S3request::put_file[/folder]
May 6 21:01:47 fritz user.info syslog: S3request::get_directory[/] marker[]
May 6 21:01:47 fritz user.info syslog: getattr[/folder]
May 6 21:01:48 fritz user.info syslog: Filecache::from_queue resurrecting file[/folder]
May 6 21:01:48 fritz user.info syslog: Filecache::from_queue resurrecting file[/]
May 6 21:01:50 fritz user.info syslog: getattr[/folder]
May 6 21:01:53 fritz user.info syslog: Filecache::from_queue expiring[/]
May 6 21:01:53 fritz user.info syslog: Filecache::fsync[/]
May 6 21:01:53 fritz user.info syslog: Filecache::delete[/]
May 6 21:01:53 fritz user.info syslog: Filecache::from_queue resurrecting file[/folder]
May 6 21:01:55 fritz user.info syslog: getattr[/folder/file]
May 6 21:01:55 fritz user.info syslog: Filecache::new[/folder/file]
May 6 21:01:55 fritz user.info syslog: S3request::get_fileinfo[/folder/file]
May 6 21:01:56 fritz user.info syslog: getattr[/folder/file]: File not found
May 6 21:01:56 fritz user.info syslog: getattr[/folder/file]
May 6 21:01:56 fritz user.info syslog: getattr[/folder/file]: File not found
May 6 21:01:56 fritz user.info syslog: mknod[/folder/file] mode[0100644]
May 6 21:01:56 fritz user.info syslog: getattr[/folder/file]
May 6 21:01:56 fritz user.info syslog: open[/folder/file] flags[020002]
May 6 21:01:56 fritz user.info syslog: release[/folder/file]
May 6 21:01:58 fritz user.info syslog: Filecache::from_queue expiring[/folder]
May 6 21:01:58 fritz user.info syslog: Filecache::fsync[/folder]
May 6 21:01:58 fritz user.info syslog: Filecache::delete[/folder]
May 6 21:01:58 fritz user.info syslog: Filecache::from_queue resurrecting file[/folder/file]
May 6 21:02:02 fritz user.info syslog: getattr[/]
May 6 21:02:02 fritz user.info syslog: Filecache::new[/]
May 6 21:02:02 fritz user.info syslog: getattr[/]
May 6 21:02:02 fritz user.info syslog: getattr[/]
May 6 21:02:02 fritz user.info syslog: readdir[/] offset[0]
May 6 21:02:02 fritz user.info syslog: Filecache::sync all
May 6 21:02:02 fritz user.info syslog: Filecache::fsync[/]
May 6 21:02:02 fritz user.info syslog: Filecache::fsync[/folder/file]
May 6 21:02:02 fritz user.info syslog: fsync: !exists || dirty_data
May 6 21:02:02 fritz user.info syslog: S3request::put_file[/folder/file]
May 6 21:02:03 fritz user.info syslog: S3request::get_directory[/] marker[]
May 6 21:02:03 fritz user.info syslog: Filecache::from_queue expiring[/folder/file]
May 6 21:02:03 fritz user.info syslog: Filecache::fsync[/folder/file]
May 6 21:02:03 fritz user.info syslog: Filecache::delete[/folder/file]
May 6 21:02:03 fritz user.info syslog: Filecache::from_queue resurrecting file[/]
May 6 21:02:03 fritz user.info syslog: getattr[/]
May 6 21:02:03 fritz user.info syslog: getattr[/folder]
May 6 21:02:03 fritz user.info syslog: Filecache::new[/folder]
May 6 21:02:03 fritz user.info syslog: getattr[/folder/file]
May 6 21:02:03 fritz user.info syslog: Filecache::new[/folder/file]
May 6 21:02:03 fritz user.info syslog: readdir[/folder] offset[0]
May 6 21:02:03 fritz user.info syslog: Filecache::sync all
May 6 21:02:03 fritz user.info syslog: Filecache::fsync[/]
May 6 21:02:03 fritz user.info syslog: Filecache::fsync[/folder]
May 6 21:02:03 fritz user.info syslog: Filecache::fsync[/folder/file]
May 6 21:02:03 fritz user.info syslog: S3request::get_directory[/folder] marker[]
May 6 21:02:04 fritz user.info syslog: getattr[/folder]
May 6 21:02:04 fritz user.info syslog: getattr[/]
May 6 21:02:04 fritz user.info syslog: Filecache::from_queue resurrecting file[/folder]
May 6 21:02:05 fritz user.info syslog: Filecache::from_queue resurrecting file[/]
May 6 21:02:08 fritz user.info syslog: Filecache::from_queue expiring[/folder/file]
May 6 21:02:08 fritz user.info syslog: Filecache::fsync[/folder/file]
May 6 21:02:08 fritz user.info syslog: Filecache::delete[/folder/file]
May 6 21:02:09 fritz user.info syslog: Filecache::from_queue expiring[/folder]
May 6 21:02:09 fritz user.info syslog: Filecache::fsync[/folder]
May 6 21:02:09 fritz user.info syslog: Filecache::delete[/folder]
May 6 21:02:10 fritz user.info syslog: Filecache::from_queue expiring[/]
May 6 21:02:10 fritz user.info syslog: Filecache::fsync[/]
May 6 21:02:10 fritz user.info syslog: Filecache::delete[/]
May 6 21:02:17 fritz user.info syslog: statfs[/]
If I delete the file with the same name as the directory (using a Firefox plugin), everything is ok again:
ls -Ral
.:
drwxr-xr-x 1 root root 0 May 5 12:48 .
drwxr-xr-x 4 bittorre users 4096 May 6 21:02 ..
-rw-r--r-- 1 root root 0 May 6 21:01 file
Kia ora e te tangata, whakawhetai mo to s3fslite tuatahi, ka whakatauhia e au te raru o nga tono e rapu ana ahau.
Engari he take: I eke ahau ki te s3 ki te wordpress, he pai te mahi me nga whakaahua panui i tukuna paitia e au e s3fs i mua i roto i te wordpress(Ko taku tikanga he pai te mahi a s3fs ki te tuku me te panui), heoi, kaore i taea te tuku whakaahua ki s3 ma s3fslite, ka tio te pae tukanga tukuake ki te 100% karekau e taea te whakaoti pai.
With the help of some others I have cross-compiled s3fslite v1.0 for Freetz (http://trac.freetz.org/wiki/WikiStart.en). See here for a description of my attempts: http://trac.freetz.org/ticket/796.
I have two little requests: can bset be replaced by memset and can unistd.h be included in common.cpp, so that these not have to be patched for Freetz.
Listing files from a bucket works, however creating files results in "Software caused connection abort". What could be the cause of this? The syslog is not showing any error messages:
Apr 29 12:53:13 fritz user.info syslog: init[0814-6721-4383-test-us]
Apr 29 12:53:15 fritz user.info syslog: getattr[/s3fs.txt]
Apr 29 12:53:15 fritz user.info syslog: Filecache::new[/s3fs.txt]
Apr 29 12:53:15 fritz user.info syslog: S3request::get_fileinfo[/s3fs.txt]
Patch for trunk 20100518 version:
--- s3request_org.cpp 2010-06-09 07:23:25.812378444 +0200 +++ s3request.cpp 2010-06-09 07:27:06.173402565 +0200 @@ -166,7 +166,7 @@ size_t uploadCallback(void *data, size_t { int *fd = static_cast(userPtr); size_t len = read(*fd, data, blockSize * numBlocks); -#ifdef DEBUG_WIRE +#if defined DEBUG && defined DEBUG_WIRE if (len != blockSize * numBlocks) { syslog(LOG_INFO, "uploadCallback: short read [%u] != [%u]", (unsigned) len, (unsigned) (blockSize * numBlocks));
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.