prestodb / presto-hadoop-apache2 Goto Github PK
View Code? Open in Web Editor NEWShaded version of Apache Hadoop 2.x for Presto
License: Apache License 2.0
Shaded version of Apache Hadoop 2.x for Presto
License: Apache License 2.0
com.facebook.presto.spi.PrestoException: java.io.IOException: Filesystem closed
at com.facebook.presto.hive.orc.OrcPageSource.getNextPage(OrcPageSource.java:159)
at com.facebook.presto.hive.HivePageSource.getNextPage(HivePageSource.java:204)
at com.facebook.presto.operator.ScanFilterAndProjectOperator.getOutput(ScanFilterAndProjectOperator.java:248)
at com.facebook.presto.operator.Driver.processInternal(Driver.java:378)
at com.facebook.presto.operator.Driver.processFor(Driver.java:301)
at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:622)
at com.facebook.presto.execution.TaskExecutor$PrioritizedSplitRunner.process(TaskExecutor.java:534)
at com.facebook.presto.execution.TaskExecutor$Runner.run(TaskExecutor.java:670)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.io.IOException: Filesystem closed
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at com.facebook.presto.hive.orc.OrcPageSource.close(OrcPageSource.java:176)
at com.facebook.presto.hive.orc.OrcPageSource.getNextPage(OrcPageSource.java:137)
... 10 more
Caused by: java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:822)
at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:672)
at java.io.FilterInputStream.close(FilterInputStream.java:181)
at com.facebook.presto.hive.orc.HdfsOrcDataSource.close(HdfsOrcDataSource.java:42)
at com.facebook.presto.orc.OrcRecordReader.close(OrcRecordReader.java:281)
at com.facebook.presto.hive.orc.OrcPageSource.close(OrcPageSource.java:173)
... 11 more
Windows tries to load the native hadoop binaries which do not exist. It tries to load them from a wide range of folder because Windows returns Windows-<Version> from os.name. After changing this to use 'Windows' as the os.name for all Windows versions and the addition of the native libraries to /nativelib/Windows_amd64, HadoopNative correctly loads the libraries from the temp files that are created. Unfortunately, when loadAllCodecs runs it subsequently fails with the following error:
java.lang.RuntimeException: failed to load Hadoop native library
at com.facebook.presto.hadoop.HadoopNative.requireHadoopNative(HadoopNative.java:60)
at com.facebook.presto.hadoop.TestHadoopNative.testCodecRoundTrip(TestHadoopNative.java:51)
Caused by: java.lang.RuntimeException: native snappy library not available: SnappyCompressor has not been loaded.
at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:68)
at org.apache.hadoop.io.compress.SnappyCodec.getDecompressorType(SnappyCodec.java:189)
at com.facebook.presto.hadoop.HadoopNative.loadAllCodecs(HadoopNative.java:73)
at com.facebook.presto.hadoop.HadoopNative.requireHadoopNative(HadoopNative.java:55)
Changing HadoopNative.requireHadoopNative to create a temporary directory and making loadLibrary copy the embedded files with the original names into this directory allows the libraries to correctly loaded.
ENV:
arch=ppc64le
os=RHEL7.6
As mentioned in the issue prestodb/presto#14549, in order to get the tests for presto-hive, presto-geospatial, presto-spark-base, etc. to PASS and to get the presto-server execute successfully with hive catalog, I needed to replace the libhadoop.so, libsnappy.so in hadoop-apache2-2.7.4-7.jar.
So, in order to get that fixed at the source, I need to get the libraries which are compatible with RHEL/CentOS 7.6 added to this repo. I have a query regarding that:
@electrum as a part of this commit, the two libraries were added to this repo (maybe compatible with ubuntuxx.xx, but the don't work for RHEL 7.6). I'd like to request some guidance from you as to what would be the best way to get the libraries compatible with RHEL/CentOS 7.6 added to this repo?
When Hadoop KMS is used, Presto Hive plugin calls org.apache.hadoop.crypto.key.kms.KMSClientProvider.createURL which fails with the current master.
Hadoop KMS calls use URIBuilder class which is missing in shaded version of hadoop libraries.
I have been able to fix this issue in https://github.com/vlkodev/presto-hadoop-apache2/commit/06b7b29060c40ec22318860696b6ab1677b5f84d
Steps I have done:
Would you consider pulling my change to include missing libraries?
URIBuilder class missing:
2016-06-20T12:13:24.190Z ERROR remote-task-callback-7 com.facebook.presto.execution.StageStateMachine Stage 20160620_121321_00003_nfw2i.1 failed
java.lang.NoClassDefFoundError: com/facebook/presto/hadoop/$internal/org/apache/http/client/utils/URIBuilder
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createURL(KMSClientProvider.java:433)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:773)
at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388)
at org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1395)
at org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1465)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:305)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:299)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:312)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767)
at org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:108)
at org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67)
at com.facebook.presto.hive.HiveUtil.lambda$createRecordReader$2(HiveUtil.java:166)
at com.facebook.presto.hive.RetryDriver.run(RetryDriver.java:136)
at com.facebook.presto.hive.HiveUtil.createRecordReader(HiveUtil.java:166)
at com.facebook.presto.hive.GenericHiveRecordCursorProvider.createHiveRecordCursor(GenericHiveRecordCursorProvider.java:47)
at com.facebook.presto.hive.HivePageSourceProvider.getHiveRecordCursor(HivePageSourceProvider.java:129)
at com.facebook.presto.hive.HivePageSourceProvider.createPageSource(HivePageSourceProvider.java:107)
at com.facebook.presto.spi.connector.classloader.ClassLoaderSafeConnectorPageSourceProvider.createPageSource(ClassLoaderSafeConnectorPageSourceProvider.java:44)
at com.facebook.presto.split.PageSourceManager.createPageSource(PageSourceManager.java:48)
at com.facebook.presto.operator.TableScanOperator.createSourceIfNecessary(TableScanOperator.java:268)
at com.facebook.presto.operator.TableScanOperator.isFinished(TableScanOperator.java:210)
at com.facebook.presto.operator.Driver.processInternal(Driver.java:375)
at com.facebook.presto.operator.Driver.processFor(Driver.java:301)
at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:618)
at com.facebook.presto.execution.TaskExecutor$PrioritizedSplitRunner.process(TaskExecutor.java:529)
at com.facebook.presto.execution.TaskExecutor$Runner.run(TaskExecutor.java:665)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: com.facebook.presto.hadoop.$internal.org.apache.http.client.utils.URIBuilder
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at com.facebook.presto.server.PluginClassLoader.loadClass(PluginClassLoader.java:102)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
Jackson dependencies missing:
java.lang.NoClassDefFoundError: org/codehaus/jackson/map/ObjectMapper
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.writeJson(KMSClientProvider.java:216)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:513)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:504)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:779)
at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388)
at org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1395)
at org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1465)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:305)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:299)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:312)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767)
at org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:108)
at org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67)
at com.facebook.presto.hive.HiveUtil.lambda$createRecordReader$2(HiveUtil.java:166)
at com.facebook.presto.hive.RetryDriver.run(RetryDriver.java:136)
at com.facebook.presto.hive.HiveUtil.createRecordReader(HiveUtil.java:166)
at com.facebook.presto.hive.GenericHiveRecordCursorProvider.createHiveRecordCursor(GenericHiveRecordCursorProvider.java:47)
at com.facebook.presto.hive.HivePageSourceProvider.getHiveRecordCursor(HivePageSourceProvider.java:129)
at com.facebook.presto.hive.HivePageSourceProvider.createPageSource(HivePageSourceProvider.java:107)
at com.facebook.presto.spi.connector.classloader.ClassLoaderSafeConnectorPageSourceProvider.createPageSource(ClassLoaderSafeConnectorPageSourceProvider.java:44)
at com.facebook.presto.split.PageSourceManager.createPageSource(PageSourceManager.java:48)
at com.facebook.presto.operator.TableScanOperator.createSourceIfNecessary(TableScanOperator.java:268)
at com.facebook.presto.operator.TableScanOperator.isFinished(TableScanOperator.java:210)
at com.facebook.presto.operator.Driver.processInternal(Driver.java:375)
at com.facebook.presto.operator.Driver.processFor(Driver.java:301)
at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:618)
at com.facebook.presto.execution.TaskExecutor$PrioritizedSplitRunner.process(TaskExecutor.java:529)
at com.facebook.presto.execution.TaskExecutor$Runner.run(TaskExecutor.java:665)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.codehaus.jackson.map.ObjectMapper
at com.facebook.presto.server.PluginClassLoader.loadClass(PluginClassLoader.java:106)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 32 more
com.facebook.presto.spi.PrestoException: Error reading from hdfs://.... at position 3
at com.facebook.presto.hive.orc.HdfsOrcDataSource.readInternal(HdfsOrcDataSource.java:76)
at com.facebook.presto.orc.AbstractOrcDataSource.readFully(AbstractOrcDataSource.java:105)
at com.facebook.presto.orc.CachingOrcDataSource.readCacheAt(CachingOrcDataSource.java:78)
at com.facebook.presto.orc.CachingOrcDataSource.readFully(CachingOrcDataSource.java:96)
at com.facebook.presto.orc.CachingOrcDataSource.readFully(CachingOrcDataSource.java:85)
at com.facebook.presto.orc.StripeReader.readStripeFooter(StripeReader.java:365)
at com.facebook.presto.orc.StripeReader.readStripe(StripeReader.java:109)
at com.facebook.presto.orc.OrcRecordReader.advanceToNextStripe(OrcRecordReader.java:446)
at com.facebook.presto.orc.OrcRecordReader.advanceToNextRowGroup(OrcRecordReader.java:399)
at com.facebook.presto.orc.OrcRecordReader.nextBatch(OrcRecordReader.java:348)
at com.facebook.presto.hive.orc.OrcPageSource.getNextPage(OrcPageSource.java:135)
at com.facebook.presto.hive.HivePageSource.getNextPage(HivePageSource.java:197)
at com.facebook.presto.operator.ScanFilterAndProjectOperator.processPageSource(ScanFilterAndProjectOperator.java:265)
at com.facebook.presto.operator.ScanFilterAndProjectOperator.getOutput(ScanFilterAndProjectOperator.java:229)
at com.facebook.presto.operator.Driver.processInternal(Driver.java:337)
at com.facebook.presto.operator.Driver.lambda$processFor$6(Driver.java:241)
at com.facebook.presto.operator.Driver.tryWithLock(Driver.java:614)
at com.facebook.presto.operator.Driver.processFor(Driver.java:235)
at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:622)
at com.facebook.presto.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
at com.facebook.presto.execution.executor.LegacyPrioritizedSplitRunner.process(LegacyPrioritizedSplitRunner.java:23)
at com.facebook.presto.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:485)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Suppressed: java.lang.RuntimeException: java.io.IOException: Filesystem closed
at com.google.common.base.Throwables.propagate(Throwables.java:240)
at com.facebook.presto.hive.orc.OrcPageSource.close(OrcPageSource.java:177)
at com.facebook.presto.hive.orc.OrcPageSource.closeWithSuppression(OrcPageSource.java:200)
at com.facebook.presto.hive.orc.OrcPageSource.getNextPage(OrcPageSource.java:154)
... 14 more
Caused by: java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808)
at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:710)
at java.io.FilterInputStream.close(FilterInputStream.java:181)
at com.facebook.presto.hive.orc.HdfsOrcDataSource.close(HdfsOrcDataSource.java:55)
at com.facebook.presto.orc.CachingOrcDataSource.close(CachingOrcDataSource.java:125)
at com.facebook.presto.orc.OrcRecordReader.close(OrcRecordReader.java:318)
at com.facebook.presto.hive.orc.OrcPageSource.close(OrcPageSource.java:174)
... 16 more
Caused by: java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808)
at org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1419)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1410)
at org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:78)
at org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:107)
at com.facebook.presto.hive.orc.HdfsOrcDataSource.readInternal(HdfsOrcDataSource.java:64)
... 24 more
Hi , guys
I found with patch "File system finalizer #23" , when I start presto server first and do some query , It always happened exception like above . The reason I guess when privateCredentials not equals , the filesystem become PhantomReference and do clean , so i think we should cache filesystem .
This is needed in issue prestodb/presto#19981
I used macbook air(m1) and got an error: library not found: /nativelib/Mac_OS_X-aarch64/libhadoop.dylib
What should i do, thank you!
Hi,
Current Presto master requires GlibC to be at least 2.12. However SLES11 and probably some other commercial distros still have GlibC 2.11 or lower. Would you mind if I compile GlibC statically into libhadoop and libsnappy so that no GlibC problems occur?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.