Comments (9)
monitorjbl
commented on August 23, 2024
Hmm, this is somewhat tricky. You see, the file writing is the reason memory utilization stays low. The contents of the file have to go somewhere, and the library puts it into a temp file instead of in memory.
You really can't wait to the file system at all? That seems like it would break a lot of other libraries out there.
from excel-streaming-reader.
monitorjbl
commented on August 23, 2024
This article seems to indicate that it's possible to write to a temp directory that is flushed between requests using PHP: https://gae-php-tips.appspot.com/2015/03/03/file-system-changes-in-app-engine-1-9-18/. Hopefully there's something similar for Java.
I don't think there's a way to define a specific temp directory currently in this library, but that could be added pretty easily.
from excel-streaming-reader.
slugmandrew
commented on August 23, 2024
Thanks for getting back to me. I believe the solution would be to use GCS (Google Cloud Storage) which can allow access to files as if they were on the file system.
I assume the library writes the whole file to the file system, and then reads chunks from that file into memory, right? If so, then it should be possible to specify a GCS bucket to keep the file in, instead of a location on the disk and then just read the chunks from there. It's meant to be pretty fast.
It's a PITA sometimes app engine... it's because there are multiple apps all running on the same JVM and it uses some funky scaling technologies as well as security features, blah blah... I'm going to look into GCS today and see what's possible.
Would you mind pointing me to the parts of the code doing the writing of files so I can investigate further?
from excel-streaming-reader.
monitorjbl
commented on August 23, 2024
Sure, the write happens here: https://github.com/monitorjbl/excel-streaming-reader/blob/master/src/main/java/com/monitorjbl/xlsx/StreamingReader.java#L262. It just uses the JVM's ability to write temp files. Sounds like that doesn't work in App Engine. The one caveat is that this needs to be a java.io.File object, POI requires it here: https://github.com/monitorjbl/excel-streaming-reader/blob/master/src/main/java/com/monitorjbl/xlsx/StreamingReader.java#L405
from excel-streaming-reader.
slugmandrew
commented on August 23, 2024
The strange thing is, I commented that line out and still got the error when using the other constructor! Possibly something is opening a file in Read/Write mode instead of just read-only?
I've been looking at GCS and it's quite easy to put and get documents in the form of ObjectInputStream, which when passed to your library seems to work just as expected. The issue is still actually being able to get a File instance. Do you think it could work with a connection to a remote file using the new File(new URI("http://etc-etc")) constructor?
from excel-streaming-reader.
monitorjbl
commented on August 23, 2024
Could you paste the full stacktrace you're seeing?
from excel-streaming-reader.
slugmandrew
commented on August 23, 2024
When calling read(InputStream):
[ERROR] SEVERE: javax.servlet.ServletContext log: Exception while dispatching incoming RPC call
[ERROR] com.google.gwt.user.server.rpc.UnexpectedException: Service method 'public abstract com.gwtplatform.dispatch.rpc.shared.Result com.gwtplatform.dispatch.rpc.shared.DispatchService.execute(java.lang.String,com.gwtplatform.dispatch.rpc.shared.Action) throws com.gwtplatform.dispatch.shared.ActionException,com.gwtplatform.dispatch.rpc.shared.ServiceException' threw an unexpected exception: java.lang.ExceptionInInitializerError
[ERROR] at com.google.gwt.user.server.rpc.RPC.encodeResponseForFailure(RPC.java:416)
[ERROR] at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:605)
[ERROR] at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:333)
[ERROR] at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:303)
[ERROR] at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:373)
[ERROR] at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
[ERROR] at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
[ERROR] at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
[ERROR] at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
[ERROR] at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
[ERROR] at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
[ERROR] at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
[ERROR] at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
[ERROR] at com.googlecode.objectify.ObjectifyFilter.doFilter(ObjectifyFilter.java:48)
[ERROR] at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
[ERROR] at com.gwtplatform.dispatch.rpc.server.AbstractHttpSessionSecurityCookieFilter.doFilter(AbstractHttpSessionSecurityCookieFilter.java:71)
[ERROR] at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
[ERROR] at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
[ERROR] at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
[ERROR] at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
[ERROR] at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
[ERROR] at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
[ERROR] at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
[ERROR] at com.google.appengine.api.socket.dev.DevSocketFilter.doFilter(DevSocketFilter.java:74)
[ERROR] at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
[ERROR] at com.google.appengine.tools.development.ResponseRewriterFilter.doFilter(ResponseRewriterFilter.java:128)
[ERROR] at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
[ERROR] at com.google.appengine.tools.development.HeaderVerificationFilter.doFilter(HeaderVerificationFilter.java:34)
[ERROR] at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
[ERROR] at com.google.appengine.api.blobstore.dev.ServeBlobFilter.doFilter(ServeBlobFilter.java:63)
[ERROR] at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
[ERROR] at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43)
[ERROR] at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
[ERROR] at com.google.appengine.tools.development.StaticFileFilter.doFilter(StaticFileFilter.java:125)
[ERROR] at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
[ERROR] at com.google.appengine.tools.development.DevAppServerModulesFilter.doDirectRequest(DevAppServerModulesFilter.java:366)
[ERROR] at com.google.appengine.tools.development.DevAppServerModulesFilter.doDirectModuleRequest(DevAppServerModulesFilter.java:349)
[ERROR] at com.google.appengine.tools.development.DevAppServerModulesFilter.doFilter(DevAppServerModulesFilter.java:116)
[ERROR] at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
[ERROR] at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
[ERROR] at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
[ERROR] at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
[ERROR] at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
[ERROR] at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
[ERROR] at com.google.appengine.tools.development.DevAppEngineWebAppContext.handle(DevAppEngineWebAppContext.java:98)
[ERROR] at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
[ERROR] at com.google.appengine.tools.development.JettyContainerService$ApiProxyHandler.handle(JettyContainerService.java:512)
[ERROR] at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
[ERROR] at org.mortbay.jetty.Server.handle(Server.java:326)
[ERROR] at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
[ERROR] at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:938)
[ERROR] at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:755)
[ERROR] at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
[ERROR] at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
[ERROR] at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
[ERROR] at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
[ERROR] Caused by: java.lang.ExceptionInInitializerError
[ERROR] at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
[ERROR] at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:121)
[ERROR] at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:332)
[ERROR] at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:284)
[ERROR] at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:305)
[ERROR] at com.monitorjbl.xlsx.StreamingReader.<clinit>(StreamingReader.java:59)
[ERROR] at com.monitorjbl.xlsx.StreamingReader$Builder.read(StreamingReader.java:376)
[ERROR] at com.utilitiessavings.usavappv7.server.reader.ParseBgPrices.parseFrom(ParseBgPrices.java:68)
[ERROR] at com.utilitiessavings.usavappv7.server.handler.parse.ParseBgPriceBookHandler.execute(ParseBgPriceBookHandler.java:75)
[ERROR] at com.utilitiessavings.usavappv7.server.handler.parse.ParseBgPriceBookHandler.execute(ParseBgPriceBookHandler.java:33)
[ERROR] at com.gwtplatform.dispatch.rpc.server.AbstractDispatchImpl.doExecute(AbstractDispatchImpl.java:154)
[ERROR] at com.gwtplatform.dispatch.rpc.server.AbstractDispatchImpl.execute(AbstractDispatchImpl.java:110)
[ERROR] at com.gwtplatform.dispatch.rpc.server.AbstractDispatchServiceImpl.execute(AbstractDispatchServiceImpl.java:87)
[ERROR] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[ERROR] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR] at java.lang.reflect.Method.invoke(Method.java:497)
[ERROR] at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:587)
[ERROR] ... 54 more
[ERROR] Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "createSecurityManager")
[ERROR] at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
[ERROR] at java.security.AccessController.checkPermission(AccessController.java:884)
[ERROR] at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
[ERROR] at com.google.appengine.tools.development.DevAppServerFactory$CustomSecurityManager.checkPermission(DevAppServerFactory.java:429)
[ERROR] at java.lang.SecurityManager.<init>(SecurityManager.java:299)
[ERROR] at org.slf4j.helpers.Util$ClassContextSecurityManager.<init>(Util.java:43)
[ERROR] at org.slf4j.helpers.Util$ClassContextSecurityManager.<init>(Util.java:43)
[ERROR] at org.slf4j.helpers.Util.<clinit>(Util.java:49)
[ERROR] ... 72 more
Fails because of trying to write the temp file, right? Then when calling read(File):
[ERROR] com.google.gwt.user.server.rpc.UnexpectedException: Service method 'public abstract com.gwtplatform.dispatch.rpc.shared.Result com.gwtplatform.dispatch.rpc.shared.DispatchService.execute(java.lang.String,com.gwtplatform.dispatch.rpc.shared.Action) throws com.gwtplatform.dispatch.shared.ActionException,com.gwtplatform.dispatch.rpc.shared.ServiceException' threw an unexpected exception: java.lang.ExceptionInInitializerError
[ERROR] at com.google.gwt.user.server.rpc.RPC.encodeResponseForFailure(RPC.java:416)
[ERROR] at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:605)
[ERROR] at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:333)
[ERROR] at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:303)
[ERROR] at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:373)
[ERROR] at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
[ERROR] at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
[ERROR] at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
[ERROR] at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
[ERROR] at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
[ERROR] at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
[ERROR] at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
[ERROR] at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
[ERROR] at com.googlecode.objectify.ObjectifyFilter.doFilter(ObjectifyFilter.java:48)
[ERROR] at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
[ERROR] at com.gwtplatform.dispatch.rpc.server.AbstractHttpSessionSecurityCookieFilter.doFilter(AbstractHttpSessionSecurityCookieFilter.java:71)
[ERROR] at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
[ERROR] at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
[ERROR] at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
[ERROR] at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
[ERROR] at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
[ERROR] at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
[ERROR] at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
[ERROR] at com.google.appengine.api.socket.dev.DevSocketFilter.doFilter(DevSocketFilter.java:74)
[ERROR] at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
[ERROR] at com.google.appengine.tools.development.ResponseRewriterFilter.doFilter(ResponseRewriterFilter.java:128)
[ERROR] at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
[ERROR] at com.google.appengine.tools.development.HeaderVerificationFilter.doFilter(HeaderVerificationFilter.java:34)
[ERROR] at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
[ERROR] at com.google.appengine.api.blobstore.dev.ServeBlobFilter.doFilter(ServeBlobFilter.java:63)
[ERROR] at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
[ERROR] at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43)
[ERROR] at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
[ERROR] at com.google.appengine.tools.development.StaticFileFilter.doFilter(StaticFileFilter.java:125)
[ERROR] at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
[ERROR] at com.google.appengine.tools.development.DevAppServerModulesFilter.doDirectRequest(DevAppServerModulesFilter.java:366)
[ERROR] at com.google.appengine.tools.development.DevAppServerModulesFilter.doDirectModuleRequest(DevAppServerModulesFilter.java:349)
[ERROR] at com.google.appengine.tools.development.DevAppServerModulesFilter.doFilter(DevAppServerModulesFilter.java:116)
[ERROR] at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
[ERROR] at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
[ERROR] at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
[ERROR] at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
[ERROR] at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
[ERROR] at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
[ERROR] at com.google.appengine.tools.development.DevAppEngineWebAppContext.handle(DevAppEngineWebAppContext.java:98)
[ERROR] at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
[ERROR] at com.google.appengine.tools.development.JettyContainerService$ApiProxyHandler.handle(JettyContainerService.java:512)
[ERROR] at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
[ERROR] at org.mortbay.jetty.Server.handle(Server.java:326)
[ERROR] at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
[ERROR] at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:938)
[ERROR] at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:755)
[ERROR] at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
[ERROR] at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
[ERROR] at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
[ERROR] at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
[ERROR] Caused by: java.lang.ExceptionInInitializerError
[ERROR] at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
[ERROR] at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:121)
[ERROR] at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:332)
[ERROR] at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:284)
[ERROR] at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:305)
[ERROR] at com.monitorjbl.xlsx.StreamingReader.<clinit>(StreamingReader.java:59)
[ERROR] at com.monitorjbl.xlsx.StreamingReader$Builder.findSheet(StreamingReader.java:457)
[ERROR] at com.monitorjbl.xlsx.StreamingReader$Builder.read(StreamingReader.java:418)
[ERROR] at com.utilitiessavings.usavappv7.server.reader.ParseBgPrices.parseFrom(ParseBgPrices.java:64)
[ERROR] at com.utilitiessavings.usavappv7.server.handler.parse.ParseBgPriceBookHandler.execute(ParseBgPriceBookHandler.java:75)
[ERROR] at com.utilitiessavings.usavappv7.server.handler.parse.ParseBgPriceBookHandler.execute(ParseBgPriceBookHandler.java:33)
[ERROR] at com.gwtplatform.dispatch.rpc.server.AbstractDispatchImpl.doExecute(AbstractDispatchImpl.java:154)
[ERROR] at com.gwtplatform.dispatch.rpc.server.AbstractDispatchImpl.execute(AbstractDispatchImpl.java:110)
[ERROR] at com.gwtplatform.dispatch.rpc.server.AbstractDispatchServiceImpl.execute(AbstractDispatchServiceImpl.java:87)
[ERROR] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[ERROR] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR] at java.lang.reflect.Method.invoke(Method.java:497)
[ERROR] at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:587)
[ERROR] ... 54 more
[ERROR] Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "createSecurityManager")
[ERROR] at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
[ERROR] at java.security.AccessController.checkPermission(AccessController.java:884)
[ERROR] at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
[ERROR] at com.google.appengine.tools.development.DevAppServerFactory$CustomSecurityManager.checkPermission(DevAppServerFactory.java:429)
[ERROR] at java.lang.SecurityManager.<init>(SecurityManager.java:299)
[ERROR] at org.slf4j.helpers.Util$ClassContextSecurityManager.<init>(Util.java:43)
[ERROR] at org.slf4j.helpers.Util$ClassContextSecurityManager.<init>(Util.java:43)
[ERROR] at org.slf4j.helpers.Util.<clinit>(Util.java:49)
[ERROR] ... 73 more
Fails because findSheet() tries to access the file in Read/Write mode, right?
If I change line 412 of StreamingReader.java to pkg = OPCPackage.open(f, PackageAccess.READ); (assuming that line will get called, because password == null):
[ERROR] SEVERE: javax.servlet.ServletContext log: Exception while dispatching incoming RPC call
[ERROR] com.google.gwt.user.server.rpc.UnexpectedException: Service method 'public abstract com.gwtplatform.dispatch.rpc.shared.Result com.gwtplatform.dispatch.rpc.shared.DispatchService.execute(java.lang.String,com.gwtplatform.dispatch.rpc.shared.Action) throws com.gwtplatform.dispatch.shared.ActionException,com.gwtplatform.dispatch.rpc.shared.ServiceException' threw an unexpected exception: java.lang.ExceptionInInitializerError
[ERROR] at com.google.gwt.user.server.rpc.RPC.encodeResponseForFailure(RPC.java:416)
[ERROR] at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:605)
[ERROR] at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:333)
[ERROR] at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:303)
[ERROR] at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:373)
[ERROR] at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
[ERROR] at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
[ERROR] at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
[ERROR] at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
[ERROR] at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
[ERROR] at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
[ERROR] at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
[ERROR] at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
[ERROR] at com.googlecode.objectify.ObjectifyFilter.doFilter(ObjectifyFilter.java:48)
[ERROR] at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
[ERROR] at com.gwtplatform.dispatch.rpc.server.AbstractHttpSessionSecurityCookieFilter.doFilter(AbstractHttpSessionSecurityCookieFilter.java:71)
[ERROR] at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
[ERROR] at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
[ERROR] at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
[ERROR] at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
[ERROR] at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
[ERROR] at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
[ERROR] at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
[ERROR] at com.google.appengine.api.socket.dev.DevSocketFilter.doFilter(DevSocketFilter.java:74)
[ERROR] at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
[ERROR] at com.google.appengine.tools.development.ResponseRewriterFilter.doFilter(ResponseRewriterFilter.java:128)
[ERROR] at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
[ERROR] at com.google.appengine.tools.development.HeaderVerificationFilter.doFilter(HeaderVerificationFilter.java:34)
[ERROR] at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
[ERROR] at com.google.appengine.api.blobstore.dev.ServeBlobFilter.doFilter(ServeBlobFilter.java:63)
[ERROR] at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
[ERROR] at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43)
[ERROR] at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
[ERROR] at com.google.appengine.tools.development.StaticFileFilter.doFilter(StaticFileFilter.java:125)
[ERROR] at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
[ERROR] at com.google.appengine.tools.development.DevAppServerModulesFilter.doDirectRequest(DevAppServerModulesFilter.java:366)
[ERROR] at com.google.appengine.tools.development.DevAppServerModulesFilter.doDirectModuleRequest(DevAppServerModulesFilter.java:349)
[ERROR] at com.google.appengine.tools.development.DevAppServerModulesFilter.doFilter(DevAppServerModulesFilter.java:116)
[ERROR] at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
[ERROR] at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
[ERROR] at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
[ERROR] at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
[ERROR] at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
[ERROR] at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
[ERROR] at com.google.appengine.tools.development.DevAppEngineWebAppContext.handle(DevAppEngineWebAppContext.java:98)
[ERROR] at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
[ERROR] at com.google.appengine.tools.development.JettyContainerService$ApiProxyHandler.handle(JettyContainerService.java:512)
[ERROR] at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
[ERROR] at org.mortbay.jetty.Server.handle(Server.java:326)
[ERROR] at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
[ERROR] at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:938)
[ERROR] at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:755)
[ERROR] at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
[ERROR] at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
[ERROR] at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
[ERROR] at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
[ERROR] Caused by: java.lang.ExceptionInInitializerError
[ERROR] at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
[ERROR] at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:121)
[ERROR] at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:332)
[ERROR] at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:284)
[ERROR] at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:305)
[ERROR] at com.monitorjbl.xlsx.StreamingReader.<clinit>(StreamingReader.java:59)
[ERROR] at com.monitorjbl.xlsx.StreamingReader$Builder.findSheet(StreamingReader.java:457)
[ERROR] at com.monitorjbl.xlsx.StreamingReader$Builder.read(StreamingReader.java:418)
[ERROR] at com.utilitiessavings.usavappv7.server.reader.ParseBgPrices.parseFrom(ParseBgPrices.java:64)
[ERROR] at com.utilitiessavings.usavappv7.server.handler.parse.ParseBgPriceBookHandler.execute(ParseBgPriceBookHandler.java:75)
[ERROR] at com.utilitiessavings.usavappv7.server.handler.parse.ParseBgPriceBookHandler.execute(ParseBgPriceBookHandler.java:33)
[ERROR] at com.gwtplatform.dispatch.rpc.server.AbstractDispatchImpl.doExecute(AbstractDispatchImpl.java:154)
[ERROR] at com.gwtplatform.dispatch.rpc.server.AbstractDispatchImpl.execute(AbstractDispatchImpl.java:110)
[ERROR] at com.gwtplatform.dispatch.rpc.server.AbstractDispatchServiceImpl.execute(AbstractDispatchServiceImpl.java:87)
[ERROR] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[ERROR] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR] at java.lang.reflect.Method.invoke(Method.java:497)
[ERROR] at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:587)
[ERROR] ... 54 more
[ERROR] Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "createSecurityManager")
[ERROR] at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
[ERROR] at java.security.AccessController.checkPermission(AccessController.java:884)
[ERROR] at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
[ERROR] at com.google.appengine.tools.development.DevAppServerFactory$CustomSecurityManager.checkPermission(DevAppServerFactory.java:429)
[ERROR] at java.lang.SecurityManager.<init>(SecurityManager.java:299)
[ERROR] at org.slf4j.helpers.Util$ClassContextSecurityManager.<init>(Util.java:43)
[ERROR] at org.slf4j.helpers.Util$ClassContextSecurityManager.<init>(Util.java:43)
[ERROR] at org.slf4j.helpers.Util.<clinit>(Util.java:49)
[ERROR] ... 73 more
As you can see they are all quite similar. Do you think my best bet is to learn how to roll my own solution using XSSF + SAX or SXSSF? (Not sure what the differences are between these two yet!) I'm pretty sure I will need to process an InputStream rather than a File so I think my options are limited.
Who knew this could be so hard, eh?
from excel-streaming-reader.
monitorjbl
commented on August 23, 2024
I think you have to roll your own, but to be quite honest I don't know how it would even work in App Engine. You have to have a java.io.File instance for this bit of code to work. Without that, I don't think you could even initialize the XSSFReader class which is what is needed to find sheets to read.
Theoretically, you can find the sheet and extract it from the XLSX file yourself. The XLSX file is just a ZIP archive after all. However, I don't really want to implement that in my library because I'm sure there's edge cases to Microsoft's packaging that the POI team has already figured out and I'd prefer to rely on them to maintain that bit for my own sanity. If you feel comfortable enough to implement that in your own code, you're welcome to copy/paste the bits of the StreamingReader class that actually read the XML.
It actually wouldn't be too much of a change from the code in place now, you'd just replace the Builder.read(InputStream) and Builder.findSheet(Reader) methods. I just wouldn't want that in the mainline code because it would be hard to maintain.
from excel-streaming-reader.
slugmandrew
commented on August 23, 2024
Hey buddy! I think you're right that it would need a File object somewhere. I looked into various options and the best solution I've come up with is to convert the xslx files to csv files, and use opencsv instead. It's actually lightning fast now and I can read from an InputStream without using too much memory.
The SAX stuff seemed like a lot to learn and getting dirty with xml is definitely not something I want to do if I can avoid it! JSON FTW!
Thanks again for your replies, you definitely helped me work through the issue a lot and now I understand POI a lot better too 😀
Gimme a shout if you ever need help with GWT or App Engine!
Drew
from excel-streaming-reader.
Related Issues (20)
- The supplied data appears to be in the OLE2 Format HOT 2
- Getting org.apache.poi.xssf.model.SharedStringsTable.getEntryAt(I) Error with POI 5.0.0 HOT 4
- New release planned with Apache POI 5.x (4.1.2 contains vulnerabilities)? HOT 1
- spliterator is not supported with Apache POI 5.2.0 (Unsupported operation getPhysicalNumberOfRows) HOT 1
- 内存溢出问题
- StreamingWorkbook#getSheetIndex(String) and StreamingWorkbook#getSheet(String) behavior is inconsistent with poi-ooxml HOT 6
- This cannot be used for reading which has image HOT 1
- Can I get data formateed value from a numberic cell HOT 2
- Can I get data formateed value from a numberic cell
- NoSuchMethodError on getSharedStringsTable HOT 1
- FileNotFoundException exception for MultipartFile resource file
- Unable to read xlsx file using Excel steaming reader HOT 2
- read picture HOT 5
- Unable read excell cell text if cell has line break HOT 1
- Is this project still active or should we look for alternatives? HOT 3
- High CPU utilisation spikes recorded HOT 3
- Unable to load the cells which are blank HOT 5
- Unable to read large excel files HOT 6
- XSSFWorkbook change orientation to landscape
- `org.xml.sax.SAXNotRecognizedException: http://xml.org/sax/properties/ on Android
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from excel-streaming-reader.