instaclustr / cassandra-sstable-tools Goto Github PK

Cannot install from branch 3.11
No pom.xml for 3.11 branch

To Reproduce
$ git checkout cassandra-3.11
sudo mvn clean install

Not able to find ic-sstable-tools.jar to be copied in cassandra home and also seeing Error: Could not find or load main class com.instaclustr.sstabletools.SummaryCollecto error.

Describe the bug
Hello,
We are facing an issue with the ic-pstats on Cassandra V2 where the output of ic-pstats is showing table sizes at about 10x than what is displayed using "du -sh" on our servers.

To Reproduce
For example,

Sizes from linux shell:
[user@server]$ du -sh * | grep Data.db
5.3G k-t-ka-3566231-Data.db
43G k-t-ka-3566232-Data.db
8.7G k-t-ka-3566235-Data.db
9.2G k-t-ka-3566236-Data.db
281M k-t-ka-3566237-Data.db
2.3G k-t-ka-3566238-Data.db
79G k-t-ka-3566239-Data.db
697M k-t-ka-3566242-Data.db
194M k-t-ka-3566247-Data.db
496M k-t-ka-3566256-Data.db
159M k-t-ka-3566261-Data.db
48M k-t-ka-3566262-Data.db

Sizes from ic-pstats:
SSTables:
+----------------------------+----------+-------------------------+-------------------------+-------+------------+--------------------+--------------------+
| SSTable | Size | Min Timestamp | Max Timestamp | Level | Partitions | Avg Partition Size | Max Partition Size |
+----------------------------+----------+-------------------------+-------------------------+-------+------------+--------------------+--------------------+
| k-t-ka-3566231-Data.db | 47.1 GB | 2018-06-28 13:43:37.017 | 2022-12-07 18:05:09.294 | 0 | 1169960 | 40.3 kB | 76.2 MB |
| k-t-ka-3566232-Data.db | 394.7 GB | 2018-05-02 14:44:33.379 | 2022-10-31 18:34:23.681 | 0 | 8845889 | 44.6 kB | 276.7 MB |
| k-t-ka-3566235-Data.db | 79.8 GB | 2018-05-02 15:45:37.477 | 2023-04-03 18:35:59.361 | 0 | 1843560 | 43.3 kB | 34.5 MB |
| k-t-ka-3566236-Data.db | 84.1 GB | 2018-05-02 14:44:47.370 | 2023-02-04 18:16:15.122 | 0 | 1983864 | 42.4 kB | 51.2 MB |
| k-t-ka-3566237-Data.db | 2.2 GB | 2018-11-29 05:14:52.502 | 2023-03-24 12:42:03.975 | 0 | 79023 | 27.8 kB | 6.2 MB |
| k-t-ka-3566238-Data.db | 20.2 GB | 2018-07-05 04:13:49.724 | 2023-04-19 00:55:43.314 | 0 | 505155 | 40.1 kB | 11.1 MB |
| k-t-ka-3566239-Data.db | 681.2 GB | 2018-05-02 14:44:18.494 | 2021-12-15 15:19:41.094 | 0 | 21152939 | 32.2 kB | 1.1 GB |
| k-t-ka-3566242-Data.db | 6.0 GB | 2018-05-02 14:44:41.400 | 2023-04-20 18:01:43.291 | 0 | 165167 | 36.2 kB | 8.1 MB |
| k-t-ka-3566247-Data.db | 1.6 GB | 2018-09-06 13:53:53.353 | 2023-04-22 18:43:14.779 | 0 | 52510 | 29.9 kB | 4.1 MB |
| k-t-ka-3566256-Data.db | 3.9 GB | 2018-06-26 17:39:24.745 | 2023-04-24 17:43:05.252 | 0 | 125750 | 31.2 kB | 16.0 MB |
| k-t-ka-3566261-Data.db | 1.2 GB | 2019-02-08 05:14:38.296 | 2023-04-25 13:05:16.766 | 0 | 46096 | 25.8 kB | 2.7 MB |
+----------------------------+----------+-------------------------+-------------------------+-------+------------+--------------------+--------------------

Expected behavior
We were expecting the size of the data file will be the same from the linux shell and the ic-pstats output.

System and versions (please complete the following information):

• OS: Ubuntu
• Cassandra version: V2.1
• Java version: Java 8
• SSTable tools version: 2.1

I have a feeling I'm missing something obvious, but whenever I run an ic-* command I'm getting an error about missing jamm jar similar to

Error opening zip file or JAR manifest missing : /x/y/z/../lib/jamm-0.3.0.jar
Error occurred during initialization of VM
agent library failed to init: instrument

Here's what I've done so far:
git clone https://github.com/instaclustr/cassandra-sstable-tools.git
cd cassandra-sstable-tools
apt-get install ant
ant -Dcassandra.version=3.0.12 (built fine)
cp dist/3.0.12/ic-sstable-tools.jar /usr/share/cassandra/
restart cassandra
bin/ic-summary

I see a jamm jar
cassandra-sstable-tools/cassandra/3.0.12/lib/jamm-0.3.0.jar

The project uses a bunch of other external libraries, but those dependencies are not specified in the build.xml file. It would be great if you could update the build file so that a fat jar that includes all external dependencies can be built using ANT or maven.

I execute ic-cfstats,ic-pstats,ic-purge got the error

Error retrieving snapshot for keyspace.columnFamily

Hi InstaClustr!

Trying to get work sstable-tools, but running into java.lang.AssertionError.
cassandra-sstable tools were built using ant -Dcassandra.version=3.0.11 command

ic-summary

java.lang.AssertionError: []
        at org.apache.cassandra.db.lifecycle.Tracker.getMemtableFor(Tracker.java:312)
        at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1185)
        at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:573)
        at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:421)
        at org.apache.cassandra.db.Mutation.apply(Mutation.java:210)
        at org.apache.cassandra.db.Mutation.apply(Mutation.java:215)
        at org.apache.cassandra.db.Mutation.apply(Mutation.java:224)
        at org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:566)
        at org.apache.cassandra.cql3.statements.ModificationStatement.executeInternal(ModificationStatement.java:556)
        at org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:295)
        at org.apache.cassandra.db.SystemKeyspace.updateSchemaVersion(SystemKeyspace.java:724)
        at org.apache.cassandra.config.Schema.updateVersion(Schema.java:563)
        at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:138)
        at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:126)
        at com.instaclustr.sstabletools.cassandra.CassandraBackend.<clinit>(CassandraBackend.java:33)
        at com.instaclustr.sstabletools.SummaryCollector.main(SummaryCollector.java:50)

when i try to run ic-summary i get the following :
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/cli/ParseException
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.cli.ParseException
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more

The column 'droppable' is reported with a decimal place in ic-sstable-tools sstables, but the value appears that it should be an integer. The decimal place is confusing as it suggest the number maybe is in KB or something similar.

Also, it might be helpful to add 'TS' as 'droppableTS:'

i.e.,
"avgPartitionSize": "23.3 kB",
"maxPartitionSize": "454.8 kB",
"avgColumnCount": 1642,
"maxColumnCount": 29521,
"droppable": "0.0"

To Reproduce
Steps to reproduce the behavior:

Expected behavior
A clear and concise description of what you expected to happen.

System and versions (please complete the following information):

• OS, Cassandra version, Java version, SSTable tools version

Additional context
Add any other context about the problem here.

Could you please take a look?

10:53:47.725 [main] ERROR o.a.c.config.DatabaseDescriptor - Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: Expecting URI in variable: [cassandra.config]. Please prefix the file with file:/// for local files or file:/// for remote files. Aborting. If you are executing this from an external tool, it needs to set Config.setClientMode(true) to avoid loading configuration.
at org.apache.cassandra.config.YamlConfigurationLoader.getStorageConfigURL(YamlConfigurationLoader.java:73) ~[apache-cassandra-2.1.18.jar:2.1.18]
at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:84) ~[apache-cassandra-2.1.18.jar:2.1.18]
at org.apache.cassandra.config.DatabaseDescriptor.loadConfig(DatabaseDescriptor.java:161) ~[apache-cassandra-2.1.18.jar:2.1.18]
at org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDescriptor.java:136) ~[apache-cassandra-2.1.18.jar:2.1.18]
at com.instaclustr.sstabletools.cassandra.CassandraBackend.(CassandraBackend.java:32) [ic-sstable-tools.jar:na]
at com.instaclustr.sstabletools.SSTableMetadataCollector.main(SSTableMetadataCollector.java:65) [ic-sstable-tools.jar:na]
Expecting URI in variable: [cassandra.config]. Please prefix the file with file:/// for local files or file:/// for remote files. Aborting. If you are executing this from an external tool, it needs to set Config.setClientMode(true) to avoid loading configuration.
Fatal configuration error; unable to start. See log for stacktrace.

Thanks