krishnact / gaevfs Goto Github PK
View Code? Open in Web Editor NEWAutomatically exported from code.google.com/p/gaevfs
License: Apache License 2.0
Automatically exported from code.google.com/p/gaevfs
License: Apache License 2.0
If GAE ever implements a distributed CacheListener, so that all running
instances of an application are notified of memcache changes, then the
GaeMemcacheFileCache ThreadLocal cache can be replaced with a more
permanent cache that's updated based on CacheListener events.
See this thread:
http://groups.google.com/group/google-appengine-java/browse_thread/thread/542f7b
44cb3d6369/faf49ee58fc35b9c?lnk=gst&q=cachelistener#faf49ee58fc35b9c
Original issue reported on code.google.com by [email protected]
on 23 Jun 2009 at 8:22
enable cross jvm file locks via datastore as said was missing
i.e. a cross jvm file lock can be created via datastore. could be as simple as
having a locked
property on the entity. acquiring the lock would change locked property from
false to true
in a transaction and would require an initial query similar to get file where
key = ? and locked =
false then set locked you true. this is similar to how object versioning is
working and is atomic.
should get a concurrent exception if multi attempts to lock same file at same
time. with that said
may make sense to maintain a version in the file which is always incremented on
write
Original issue reported on code.google.com by [email protected]
on 15 Mar 2010 at 7:50
I've attached two patches that I have found useful while integrating gaevfs
with the Milton WebDAV server. One adds support for mark/reset on the
FileObject's random-access InputStream; another implements the Attributes
capability on the FileObject through an extra list of properties appended
to the DataStore entity.
Original issue reported on code.google.com by [email protected]
on 7 Nov 2009 at 6:52
Attachments:
As an alternate to the Commons VFS API, a "wrapper" API that mimics java.io
classes (File, FileOutputStream, FileWrite, RandomAccessFile, etc.) should be
implemented. This will make it easier for programmer who are familiar with
java.io to use GaeVFS.
Original issue reported on code.google.com by [email protected]
on 14 Aug 2009 at 9:15
The implementations of FileName.isAncestor() and FileName.isDescendant()
expect the two fils to be within the same file system. With the Combined
Local option enabled (which it is by default), this can produce unexpected
results. For example, FileObject.getChildren() will return both local and
GAE children for a GAE folder, but isAncestor() and isDescendant() will
return false.
Original issue reported on code.google.com by [email protected]
on 24 Jul 2009 at 9:17
Due to limits on GAE datastore entity size, the maximum file that can be
written to the GaeVFS virtual file system is 1.0MB. An entity chaining
mechanism needs to be implemented to remove this limit. It's probably best
to implement this in custom InputStream and OutputStream classes. It's
important that it not be necessary to have the entire file (>1.0MB)
resident in memory; for example, it should be possible to write a file to
the servlet response output stream without ever having the entire file in
memory.
Review this for possible code sharing:
http://code.google.com/p/google-file-service/
Original issue reported on code.google.com by [email protected]
on 23 Jun 2009 at 8:16
A write-behind cache based on memcached and Task Queues needs to be
implemented.
Original issue reported on code.google.com by [email protected]
on 14 Aug 2009 at 9:13
Investigate NullPointerExceptions within AbstractFileObject.finalize() due
to fs attribute being null. Are these caused by serialization changes?
Original issue reported on code.google.com by [email protected]
on 14 Jul 2009 at 1:59
Invoking FileObject.getChildren() on the root directory includes the root
directory in the returned array. This is probably happening because we're
doing a query with the key of the root directory as the ancestor, and the
root directory doesn't have a parent (so the ancestor-key query is
matching the root directory key itself).
Original issue reported on code.google.com by [email protected]
on 2 Jul 2009 at 1:47
Currently, GaeVFS needs to be initialized via the GaeVFS.setRootPath()
method. However, GAE makes the webapp root path available via the "user.dir"
system property, so GaeVFS should be able to initialize itself if the root
path isn't explicitly set.
(Note, this won't work in non-GAE servlet environments).
Original issue reported on code.google.com by [email protected]
on 14 Aug 2009 at 9:18
Provide a way within GaeVfsServlet to protect directories using the Google
Accounts API:
http://code.google.com/appengine/docs/java/users/overview.html
It should be possible to configure directories to be completely restricted
(read/write) to authorized users, or to be read-only to non-authorized
users.
Original issue reported on code.google.com by [email protected]
on 5 Jul 2009 at 1:46
When creating a new file (such as when uploading), the current
implementation puts each block entity to the datastore as the block
becomes full. It might improve performance--though take more memory--to do
a batch put of all blocks only after the file is completely written. This
may also help with error handling and recovery when a file has only been
partially written and a failure occurs.
Similarly, when a file is open for read/write random access, we should
track which blocks are modified and only put those blocks in a batch when
the file is closed. Currently, each block is put to the datastore
individually.
Original issue reported on code.google.com by [email protected]
on 21 Jul 2009 at 5:52
As a potential performance enhancement, investigate the use of protocol
buffers in place of Java serialization for creating the write-behind task
payload.
http://code.google.com/p/protobuf/
http://code.google.com/apis/protocolbuffers/docs/overview.html
http://code.google.com/p/thrift-protobuf-compare/wiki/Benchmarking
(be sure to read comments)
http://whiteboardjunkie.wordpress.com/2009/09/14/serialization-options-
compared/
Original issue reported on code.google.com by [email protected]
on 16 Sep 2009 at 10:44
It would be nice to replace the response.sendRedirect() in the
GaeVfsServlet with requestDispatcher.forward() this would make it
simpler to pass state for subsequent post-upload processing ...
Original issue reported on code.google.com by [email protected]
on 29 Oct 2009 at 7:51
What steps will reproduce the problem?
<?php
import com.newatlanta.commons.vfs.provider.gae.GaeVFS;
import org.apache.commons.io.IOUtils;
import java.lang.Long;
import java.lang.Boolean;
GaeVFS::setRootPath(quercus_servlet_request()->getSession(true)->getServletConte
xt()->getRealPath('/'));
define('VFSM', GaeVFS::getManager());
//VFSM->resolveFile('gae://gaevfs')->createFolder();
$file=VFSM->resolveFile('gae://gaevfs/tmp1');
//$file->createFile();
$text='pp';
$method=$file->getClass()->getDeclaredMethod('updateContentSize',
array(Long::TYPE, Boolean::TYPE));
$method->setAccessible(true);
$method->invoke($file, strlen($text), true); //error:argument type mismatch
$out=$file->getContent()->getOutputStream();
IOUtils::write($text, $out, 'UTF8');
$out->close();
?>
the noted error line raise error:argument type mismatch
What is the expected output? What do you see instead?
no error, and updateContentSize successfully
What version of the product are you using? On what operating system?
gaevfs-0.3.jar
Original issue reported on code.google.com by [email protected]
on 4 Nov 2011 at 10:11
The current implementation stores the block keys as a list of keys within a
metadata property field. The advantage of this approach is that it's easy
to do bulk gets of all of the block entities. The problems with this
approach are:
1) This creates an entity group for all of the block keys for a file,
causing datastore puts to be serialized and limiting to 10 puts per entity
group per second (what happens if a file has more than 10 blocks that need
to be written all at once?)
2) There's a limit of 500 elements in a property list (?) and there's also
the 1MB limit on the size of the metadata entity; both of these will limit
the number of block keys that can be stored within a single metadata
entity.
Should we instead use a unique key for each block and not use the metadata
key as the parent? This would remove the limitations of the current design,
but require searching for blocks instead of get-by-key.
Original issue reported on code.google.com by [email protected]
on 19 Oct 2009 at 7:30
Compress entities when writing to the datastore to save space. (Don't
compress entities stored in memcache).
Original issue reported on code.google.com by [email protected]
on 13 Aug 2009 at 12:54
Add junit tests.
Original issue reported on code.google.com by [email protected]
on 14 Aug 2009 at 9:08
Add a servlet filter to clear the cache as said is necessary
i.e. write a simple filter that people can use
Original issue reported on code.google.com by [email protected]
on 15 Mar 2010 at 7:35
A distributed locking mechanism needs to be implemented for GaeVFS. It should
be possible to base these on memcached counters.
Original issue reported on code.google.com by [email protected]
on 14 Aug 2009 at 9:11
What steps will reproduce the problem?
1. configure app engine application to use h2 1.2.121 and latest gaevfs
from svn
2. start application
3. error stacktrace in log
What version of the product are you using? On what operating system?
svn version 356 of gae-vfs, h2 version 1.2.121
Please provide any additional information below.
Config:
web.xml
<servlet>
<servlet-name>gaevfs</servlet-name>
<servlet-class>com.newatlanta.appengine.servlet.GaeVfsServlet</servlet-class>
<init-param>
<param-name>dirListingAllowed</param-name>
<param-value>true</param-value>
</init-param>
<init-param>
<param-name>initDirs</param-name>
<param-value>/gaevfs/data</param-value>
</init-param>
<init-param>
<param-name>uploadRedirect</param-name>
<param-value>/gaevfs/</param-value>
</init-param>
</servlet>
<servlet-mapping>
<servlet-name>gaevfs</servlet-name>
<url-pattern>/gaevfs/*</url-pattern>
</servlet-mapping>
<servlet>
<servlet-name>CachingDatastoreService</servlet-name>
<servlet-class>com.newatlanta.appengine.datastore.CachingDatastoreService</servl
et-class>
</servlet>
<servlet-mapping>
<servlet-name>CachingDatastoreService</servlet-name>
<url-pattern>/_ah/queue/write-behind-task</url-pattern>
</servlet-mapping>
<listener>
<listener-class>com.newatlanta.appengine.servlet.GaeVfsServletEventListener</lis
tener-class>
</listener>
<listener>
<display-name>h2 starter listener</display-name>
<listener-class>org.h2.server.web.DbStarter</listener-class>
</listener>
<context-param>
<param-name>db.url</param-name>
<param-value>jdbc:h2:gae://gaevfs/data/sessions;FILE_LOCK=NO;AUTO_SERVER=FALSE;D
B_CLOSE_ON_EXIT=FALSE</param-value>
</context-param>
<context-param>
<param-name>db.user</param-name>
<param-value>sa</param-value>
</context-param>
<context-param>
<param-name>db.password</param-name>
<param-value>sa</param-value>
</context-param>
Stacktrace:
java.lang.NoClassDefFoundError: java.net.URLStreamHandler is a restricted
class. Please see the Google App Engine developer's guide for more details.
at
com.google.appengine.tools.development.agent.runtime.Runtime.reject(Runtime.java
:51)
at
org.apache.commons.vfs.provider.DefaultURLStreamHandler.<init>(DefaultURLStreamH
andler.java:47)
at
org.apache.commons.vfs.provider.AbstractFileObject$1.run(AbstractFileObject.java
:430)
at java.security.AccessController.doPrivileged(Native Method)
at
org.apache.commons.vfs.provider.AbstractFileObject.getURL(AbstractFileObject.jav
a:426)
at com.newatlanta.appengine.nio.file.GaePath.toUri(GaePath.java:639)
at
com.newatlanta.h2.store.fs.FileSystemGae.accepts(FileSystemGae.java:61)
at org.h2.store.fs.FileSystem.getInstance(FileSystem.java:73)
at org.h2.util.FileUtils.normalize(FileUtils.java:128)
at org.h2.engine.ConnectionInfo.getName(ConnectionInfo.java:336)
at org.h2.engine.Engine.openSession(Engine.java:44)
at org.h2.engine.Engine.openSession(Engine.java:140)
at org.h2.engine.Engine.getSession(Engine.java:120)
at
org.h2.engine.SessionFactoryEmbedded.createSession(SessionFactoryEmbedded.java:1
7)
at
org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:245)
at org.h2.engine.SessionRemote.createSession(SessionRemote.java:223)
at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:110)
at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:94)
at org.h2.Driver.connect(Driver.java:58)
at java.sql.DriverManager.getConnection(DriverManager.java:582)
at java.sql.DriverManager.getConnection(DriverManager.java:185)
at org.h2.server.web.DbStarter.contextInitialized(DbStarter.java:39)
at
org.mortbay.jetty.handler.ContextHandler.startContext(ContextHandler.java:530)
at org.mortbay.jetty.servlet.Context.startContext(Context.java:135)
at
org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1218)
at
org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:500)
at
org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:448)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:40)
at
org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:117)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:40)
at
org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:117)
at org.mortbay.jetty.Server.doStart(Server.java:217)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:40)
at
com.google.appengine.tools.development.JettyContainerService.startContainer(Jett
yContainerService.java:181)
at
com.google.appengine.tools.development.AbstractContainerService.startup(Abstract
ContainerService.java:116)
at
com.google.appengine.tools.development.DevAppServerImpl.start(DevAppServerImpl.j
ava:217)
at
com.google.appengine.tools.development.DevAppServerMain$StartAction.apply(DevApp
ServerMain.java:162)
at
com.google.appengine.tools.util.Parser$ParseResult.applyArgs(Parser.java:48)
at
com.google.appengine.tools.development.DevAppServerMain.<init>(DevAppServerMain.
java:113)
at
com.google.appengine.tools.development.DevAppServerMain.main(DevAppServerMain.ja
va:89)
Original issue reported on code.google.com by [email protected]
on 19 Oct 2009 at 12:44
http://code.google.com/appengine/docs/java/javadoc/com/google/appengine/api
/memcache/MemcacheService.html
"Except for the increment(java.lang.Object, long) method, this service does
not offer atomicity guarantees. "
The code is using a 'put' and ADD_ONLY_IF_NOT_PRESENT to acquire the lock.
Since put is not atomic you could have two hosts in the cloud could acquire
the same lock.
Only the increment methods are atomic.
http://code.google.com/appengine/docs/java/javadoc/com/google/appengine/api
/memcache/MemcacheService.html#put(java.lang.Object, java.lang.Object,
com.google.appengine.api.memcache.Expiration)
Original issue reported on code.google.com by [email protected]
on 30 Apr 2010 at 9:32
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.