WInte.r is a Java framework for end-to-end data integration. The WInte.r framework implements well-known methods for data pre-processing, schema matching, identity resolution, data fusion, and result evaluation.
License: Apache License 2.0
hello
thank you for published this project.
where is inputs?
thank you in advanced.
MatchingEvaluator and DataFusionEvaluator have a different usage of verbose. Is it possible to make this consistent? Rather in the constructor or setVerbose()
When the data model contains an attribute which is a lists of objects, e.g. main entity album has the attribute tracks with each track being a separate object, the log files output the java object id rather than the actual values.
I just cloned the repository and wanted to follow the tutorial steps, but running mvn install in the winter-framework folder yields 3 failed tests.
testTypeValue(de.uni_mannheim.informatik.dws.winter.webtables.detectors.PatternbaseTypeDetectorTest) Time elapsed: 0.003 sec <<< FAILURE!
junit.framework.AssertionFailedError: expected:<null> but was:<de.uni_mannheim.informatik.dws.winter.preprocessing.units.Unit@72758afa>
at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.Assert.failNotEquals(Assert.java:329)
at junit.framework.Assert.assertEquals(Assert.java:78)
at junit.framework.Assert.assertEquals(Assert.java:86)
at junit.framework.TestCase.assertEquals(TestCase.java:253)
at de.uni_mannheim.informatik.dws.winter.webtables.detectors.PatternbaseTypeDetectorTest.testTypeValue(PatternbaseTypeDetectorTest.java:36)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at junit.framework.TestCase.runTest(TestCase.java:176)
at junit.framework.TestCase.runBare(TestCase.java:141)
at junit.framework.TestResult$1.protect(TestResult.java:122)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at junit.framework.TestResult.run(TestResult.java:125)
at junit.framework.TestCase.run(TestCase.java:129)
at junit.framework.TestSuite.runTest(TestSuite.java:252)
at junit.framework.TestSuite.run(TestSuite.java:247)
at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:86)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
testCheckUnit(de.uni_mannheim.informatik.dws.winter.preprocessing.units.UnitCategoryParserTest) Time elapsed: 0.001 sec <<< FAILURE!
junit.framework.AssertionFailedError
at junit.framework.Assert.fail(Assert.java:55)
at junit.framework.Assert.assertTrue(Assert.java:22)
at junit.framework.Assert.assertNotNull(Assert.java:256)
at junit.framework.Assert.assertNotNull(Assert.java:248)
at junit.framework.TestCase.assertNotNull(TestCase.java:417)
at de.uni_mannheim.informatik.dws.winter.preprocessing.units.UnitCategoryParserTest.testCheckUnit(UnitCategoryParserTest.java:27)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at junit.framework.TestCase.runTest(TestCase.java:176)
at junit.framework.TestCase.runBare(TestCase.java:141)
at junit.framework.TestResult$1.protect(TestResult.java:122)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at junit.framework.TestResult.run(TestResult.java:125)
at junit.framework.TestCase.run(TestCase.java:129)
at junit.framework.TestSuite.runTest(TestSuite.java:252)
at junit.framework.TestSuite.run(TestSuite.java:247)
at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:86)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
testTypeValue(de.uni_mannheim.informatik.dws.winter.preprocessing.datatypes.ValueNormalizerTest) Time elapsed: 0.029 sec <<< FAILURE!
junit.framework.AssertionFailedError: expected:<1500000.0> but was:<1.5>
at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.Assert.failNotEquals(Assert.java:329)
at junit.framework.Assert.assertEquals(Assert.java:78)
at junit.framework.Assert.assertEquals(Assert.java:86)
at junit.framework.TestCase.assertEquals(TestCase.java:253)
at de.uni_mannheim.informatik.dws.winter.preprocessing.datatypes.ValueNormalizerTest.testTypeValue(ValueNormalizerTest.java:32)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at junit.framework.TestCase.runTest(TestCase.java:176)
at junit.framework.TestCase.runBare(TestCase.java:141)
at junit.framework.TestResult$1.protect(TestResult.java:122)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at junit.framework.TestResult.run(TestResult.java:125)
at junit.framework.TestCase.run(TestCase.java:129)
at junit.framework.TestSuite.runTest(TestSuite.java:252)
at junit.framework.TestSuite.run(TestSuite.java:247)
at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:86)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Main issue: From the example in 'RDFRecordReaderTest', it seems that the schema/attributes are incorrectly assigned.
Possible causes: As can be seen from the definition of abstract method readLine() in RDFMatchableReader , the the actual attribute is not passed to the dataset population method. In the implementation (RDFRecordReader class), attribute identifier looks fine, but the attribute value is assigned to the attribute name and the values are assigned again to the record. This seems not correct to me.
Feature request: It might be nice that instead of having the dependency on log4j explicitly, using slf4j would let this project better integrate into other java projects that may or may not use a different logging framework (which I think does use log4j by default).
The method writeDebugMatchingResultsToFile of the MatchingRule writes a debug log to two separate files:
A user only supplies the path "/path/to/debugResults.csv".
The suffix "_short" is simply appended to the path to determine the path to the short debug log.
This results in a file of type .csv_short, which is no common file type.
Requirement:
The file path of the short debug log should be "/path/to/debugResults_short.csv".
Hi ,
i want to perform some Identity Resolution where the entities have many attributes, but some of the attribute values can be null. Is there any way to handle this from winter ?
Running the intersection method, causes a null pointer, if the two sets do not contain any value.
Union does not have the problem, as the list is initialized, which does not happen in Intersection.java.
Would be good if its possible to instanciate the DataFusionEvaluator without a RecordGroupFactory. If null is given, the evaluation breaks. Either checking for null or internal initiate on your internally with new RecordGroupFactory.
What happened:
I use the Median conflict resolution in one of my AttributeValueFusers and get an IndexOutOfBounds exception on some of my RecordGroups.
What I expect to happen:
There should be no exception thrown in the library.
Root cause:
The linked list that is used internally covers all cases that are 0 and > 1, but in case the list is of size one the exception is thrown.
https://github.com/olehmberg/winter/blob/master/winter-framework/src/main/java/de/uni_mannheim/informatik/dws/winter/datafusion/conflictresolution/numeric/Median.java#L48
A possible fix would include an update to the if statement like this:
boolean isEven = list.size() % 2 == 0;
if (list.size() == 0) {
return new FusedValue<>((Double) null);
} else if (list.size() == 1) { // Return the only element in the list as median if length == 1
return new FusedValue<>(list.get(0));
} else if (isEven) {
double middle = ((double) list.size() + 1.0) / 2.0;
double median1 = list.get((int) Math.floor(middle) - 1);
double median2 = list.get((int) Math.ceil(middle) - 1);
return new FusedValue<>((median1 + median2) / 2.0);
} else {
int middle = list.size() / 2;
return new FusedValue<>(list.get(middle - 1)); // Throws indexOutOfBoundsException if middle = 0
}Another possibility would be to round the list.size() / 2 correctly. In the current implementation all decimal places will just be removed.
See https://stackoverflow.com/a/2654897/6059889 on how to get a correctly rounded integer value.
When I try loading a XML in without making an extra model, like described in the documentation. There will not be a schema designed to the dataset.
Loading itself works. You can access the data loaded in the dataset. I tried to find our what is going wrong.
When load a csv-file a schema is there. When constructing an extra model for the xml-File. The schema is there, too.
When looking into the XMLMatchableReader there is a schema there for defaultLoading but it disappears when going back to the original starting class.
That makes it impossible to use any matching tools.
After creating a fused dataset, the density printing is not possible as the attributes are not given
Workaround:
addAttribute for each attribute
Hi Oli,
Great work. Can you add the/a overview figure for the classes for fusion and IR? In the former release, we had one of those right?
Cheers,
Robert
Hello,
Have you run your program to test the result of approximate FD? I generate same result as in the original paper if the error threshold is 0. But for the approximate FD result with error threshold 0.01, 0.05, etc, I cannot obtain same result as in the published experiment. Have you repeated the same experimental result for approximate FDs?
Thank you!
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.