Comments (11)
sebastian-berlin-wmse
commented on May 26, 2024
2
I've managed to track down that 0945595 is the revision Commons stops showing up in the list of sources. It's a small change that only updates the Velocity library.
from openrefine.
vivian-rook
commented on May 26, 2024
I believe the same issue is seen in https://phabricator.wikimedia.org/T363917 I've rolled back to 3.7.9 in PAWS and the issue resolved.
from openrefine.
wetneb
commented on May 26, 2024
@sebastian-berlin-wmse @lokal-profil how does this issue fit within your development plans? Do you intend to work on it or do you see this as low priority (or out of scope)?
If any fix is required on OpenRefine's side I would be keen to include it in a 3.8.1 release, which should ideally come out soon given that we have other important fixes to release (see #6599).
If you don't intend to work on this soon I would try to find other ways to fix it (such as doing it myself).
from openrefine.
sebastian-berlin-wmse
commented on May 26, 2024
I'll have a look.
from openrefine.
sebastian-berlin-wmse
commented on May 26, 2024
For me, the extension only shows up in 3.7.7 and no later version (tried 3.7.8, 3.7.9 and 3.8.0). @wetneb, can you point me in the direction of where extensions are loaded so I can debug? I looked around a bit in the documentation and in the code, but I haven't found a good starting point.
from openrefine.
wetneb
commented on May 26, 2024
On the frontend side, extensions are loaded by including whatever Javascript files they declare into the page (project-bundle.js or index-bundle.js). On the backend side, the Jar files they include are loaded and the server-side components are registered (via the controller.js file).
To debug the frontend you could look at the developer console to see if any errors appear there, or put breakpoints on the extension JS code to see if/how it gets executed.
To debug the backend code you could first look at the server logs to see if any errors appear there. Running OpenRefine to debug an extension in Java is more involved (I am not sure how to set it up) but I generally work by writing unit tests which reproduce the bug and then debug the code via the IDE when running those tests.
We can also look at this together in our call on Thursday.
from openrefine.
sebastian-berlin-wmse
commented on May 26, 2024
Turns out that my issue was because I misunderstood the instructions for installing extensions. I put commons-extension/ in extensions/ in OpenRefine root rather than in webapp/extensions/. I think I was confused by the screenshot under Install this extension in OpenRefine, which I now realise is of the workspace directory. Now the source for Commons shows up in 3.7.9, but not 3.8.0.
from openrefine.
wetneb
commented on May 26, 2024
When loading the Commons extension just after this commit, I get:
2024-05-23T07:09:17.797150029Z pool-2-thread-7 ERROR An exception occurred processing Appender LogToConsole org.apache.logging.log4j.core.appender.AppenderLoggingException: java.lang.NoClassDefFoundError: org/apache/commons/lang3/exception/ExceptionUtils
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:164)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:133)
at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:124)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:88)
at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:705)
at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:663)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:639)
at org.apache.logging.log4j.core.config.LoggerConfig.logParent(LoggerConfig.java:696)
at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:665)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:639)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:540)
at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:67)
at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:155)
at org.apache.log4j.Category.maybeLog(Category.java:560)
at org.apache.log4j.Category.error(Category.java:291)
at org.apache.velocity.runtime.log.Log4JLogChute.log(Log4JLogChute.java:197)
at org.apache.velocity.runtime.log.LogDisplayWrapper.log(LogDisplayWrapper.java:94)
at org.apache.velocity.runtime.log.LogDisplayWrapper.error(LogDisplayWrapper.java:200)
at org.apache.velocity.runtime.VelocimacroFactory.initVelocimacro(VelocimacroFactory.java:218)
at org.apache.velocity.runtime.RuntimeInstance.init(RuntimeInstance.java:261)
at org.apache.velocity.runtime.RuntimeInstance.init(RuntimeInstance.java:589)
at org.apache.velocity.app.VelocityEngine.init(VelocityEngine.java:135)
at edu.mit.simile.butterfly.Butterfly.configureModules(Butterfly.java:862)
at edu.mit.simile.butterfly.Butterfly.configure(Butterfly.java:428)
at edu.mit.simile.butterfly.Butterfly.init(Butterfly.java:295)
at org.eclipse.jetty.servlet.ServletHolder$Wrapper.init(ServletHolder.java:1305)
at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:633)
at org.eclipse.jetty.servlet.ServletHolder.getServlet(ServletHolder.java:486)
at org.eclipse.jetty.servlet.ServletHolder.prepare(ServletHolder.java:731)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:524)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
at com.google.refine.ValidateHostHandler.handle(ValidateHostHandler.java:93)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
at org.eclipse.jetty.server.Server.handle(Server.java:563)
at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.NoClassDefFoundError: org/apache/commons/lang3/exception/ExceptionUtils
at com.google.util.logging.IndentingLayout.toSerializable(IndentingLayout.java:137)
at com.google.util.logging.IndentingLayout.toSerializable(IndentingLayout.java:54)
at org.apache.logging.log4j.core.layout.AbstractStringLayout.toByteArray(AbstractStringLayout.java:295)
at org.apache.logging.log4j.core.layout.AbstractLayout.encode(AbstractLayout.java:207)
at org.apache.logging.log4j.core.layout.AbstractLayout.encode(AbstractLayout.java:36)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:227)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:220)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:211)
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:160)
... 60 more
Caused by: java.lang.ClassNotFoundException: org.apache.commons.lang3.exception.ExceptionUtils
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
... 69 more
from openrefine.
wetneb
commented on May 26, 2024
So after a joint investigation in our weekly call with @sebastian-berlin-wmse, we found out that the problem is coming from the presence of two incompatible copies of the velocity library in the classpath, following #6190.
While #6190 updated velocity in OpenRefine, it did not update velocity in Butterfly, meaning that we ship both org.apache.velocity:velocity:jar:1.6.3 and org.apache.velocity:velocity-engine-core:jar:2.3 in our classpath.
The upgrade was motivated by a vulnerability in velocity, which does not actually have implications for OpenRefine since our velocity templates are not user-controlled, so I would be tempted to first release a 3.8.2 with this PR reverted, and then work on upgrading velocity in Butterfly and shipping this upgrade in a later version.
from openrefine.
wetneb
commented on May 26, 2024
I actually do not understand why Velocity should be a dependency of OpenRefine itself. I think it should intuitively be a dependency of Butterfly only, because I can't recall anywhere where we use it in OpenRefine. I have tried removing the dependency in OpenRefine and OR seems to still work fine without (I haven't tried running the Cypress test suite though).
However, reverting #6190 (be it by downgrading velocity or removing it entirely) does not seem to make the Commons extension work again.
Although it would still be interesting to understand the cause of the incompatibility, I think it would also be worth updating the version of OpenRefine that the Commons extension is built against:
https://github.com/OpenRefine/CommonsExtension/blob/f0fb6694d08b7a81c83b249c43a11024b0186b70/pom.xml#L23
and releasing a new version of the Commons extension. That would likely help.
from openrefine.
tfmorris
commented on May 26, 2024
This is why we need better isolation for extension dependencies. I'll try to have a look at what's going on sometime over the long weekend.
from openrefine.
Related Issues (20)
- Show Facet Spinner SOONER on any potentially slow Facets (especially Unicode char-code facet) HOT 2
- ./refine script should fail properly if npm is not available
- Uncaught TypeError causing dozens of e2e test failures HOT 4
- New GEO Extension with commonly needed functions for GIS, geography, etc. HOT 2
- Wikidata and Wikibase reconciliation services don't load in OpenRefine 3.8.0 HOT 5
- Invalid detection of empty events array in update notification
- Fixture for e2e testing against records mode in Cypress
- Quote table name in SQL exporter HOT 3
- Encoding issue regression for files imported into version 3.8.0 HOT 10
- Allow manual selection of UTF-8 BOM encoding
- forEachIndex with array containing null values throw NullPointerException
- 'Search for match' link not shown when no reconciliation candidates are present
- The dialog system uses an incorrect WAI ARIA Role attribute
- TSV import always trims white space, ignoring parse setting
- Add new GREL function to normalize characters HOT 1
- Search option has disappeared from reconciliation results (3.8.0) HOT 1
- Add new GREL function to calculate the edit distance HOT 3
- Checking running status of OpenRefine with wget will not work correctly
- Move the Wikitable importer to an extension HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from openrefine.