naveenrudra / rttm Goto Github PK

Hello,
I followed to instructions for the tool setup.
The last step is to run "/opt/RTTM run command java -jar scraptool/target/scraptool-1.0-SNAPSHOT-standalone.jar -t test -c /home/n0r00ij/RTS/scrapper_config/" from inside the container.
I did so and received a hava error as follows:
root@c23541f35546:/opt/RTTM/RTTM# java -jar scraptool/target/scraptool-1.0-SNAPSHOT-standalone.jar -t test -c /home/n0r00ij/RTS/scrapper_config
Initializing of scrapper has started!!
java.nio.file.NoSuchFileException: /home/n0r00ij/RTS/scrapper_config/scanner-configuration.properties
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.Files.newByteChannel(Files.java:361)
at java.nio.file.Files.newByteChannel(Files.java:407)
at java.nio.file.Files.readAllBytes(Files.java:3152)
at org.rts.base.ScrapperImpl.initialize(ScrapperImpl.java:37)
at org.scraptool.ScrapperTool.initializeScrappers(ScrapperTool.java:40)
at org.scraptool.ScrapperTool.startScrappers(ScrapperTool.java:80)
at org.scraptool.ScrapperTool.main(ScrapperTool.java:127)
2019-12-22 10:27:05 ERROR ScrapperTool:90 - Something wrong happened in scrappers
java.nio.file.NoSuchFileException: /home/n0r00ij/RTS/scrapper_config/consumer.properties
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.Files.newByteChannel(Files.java:361)
at java.nio.file.Files.newByteChannel(Files.java:407)
at java.nio.file.Files.readAllBytes(Files.java:3152)
at org.kafkaparser.base.NotificationConsumerThread.getConfig(NotificationConsumerThread.java:64)
at org.kafkaparser.base.NotificationConsumerThread.(NotificationConsumerThread.java:51)
at org.kafkaparser.base.NotificationConsumerGroup.(NotificationConsumerGroup.java:20)
at org.scraptool.ScrapperTool.startScrappers(ScrapperTool.java:86)
at org.scraptool.ScrapperTool.main(ScrapperTool.java:127)
java.nio.file.NoSuchFileException: /home/n0r00ij/RTS/scrapper_config/consumer.properties
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.Files.newByteChannel(Files.java:361)
at java.nio.file.Files.newByteChannel(Files.java:407)
at java.nio.file.Files.readAllBytes(Files.java:3152)
at org.kafkaparser.base.NotificationConsumerThread.getConfig(NotificationConsumerThread.java:64)
at org.kafkaparser.base.NotificationConsumerThread.(NotificationConsumerThread.java:51)
at org.kafkaparser.base.NotificationConsumerGroup.(NotificationConsumerGroup.java:20)
at org.scraptool.ScrapperTool.startScrappers(ScrapperTool.java:86)
at org.scraptool.ScrapperTool.main(ScrapperTool.java:127)

Why is this happening? all previous steps passed successfully.

Thanks

Currently there are fixed no. of sites in config file. We need to give an option to dynamically add sites to the config.

Hi,

Step 12/24 : ADD script/run.sh /usr/local/bin/run.sh
ADD failed: stat /var/snap/docker/common/var-lib-docker/tmp/docker-builder151664376/script/run.sh: no such file or directory

Currently there is only one worker node which is handling git, gist, pasties etc. If the github has massive data then it takes hours to analyze for secrets.

The performance needs to be optimized.

I have build the Dockerfile .

but do not konw how to run from the image.

In instruction you should mention about how to run the docker image: docker run -d -t -i <image_id> /bin/bash

Hi,
I have two problems with search mechanism:

1. If I config domain as search term, for example test.example.com the analyzer will look for the existence of the separate words in the whole post: test, example, com and not as one term.

2. If I config multiple keywords as one search term, the same issue will occurs, for example the search term "google cloud" will find posts that includes the different words and even they are not closed.

SearchTerms columns are being saved as ArrayList objects. The SearchTerms should be saved as values.

Design

• SearchTerms should be a separate table.
• Since one searchterm can be in many gist urls and one gist url can have many search team hence its many - many relationship. Hence there should be a mapper table between Result and SearchTerms.

Hi,

I'm trying to use my own profile for Paste.com sites:

I just changed for my own at: scrapper_config/scanner-configuration.properties but appear an error when I execute that.

2019-12-18 13:22:36 ERROR ScrapperTool:90 - Something wrong happened in scrappers
java.lang.RuntimeException: ScannerProfile my-own-profile is not defined
at org.rts.base.profileregistry.ScrapperProfileRegistry.newScrapProfile(ScrapperProfileRegistry.java:28)
at org.rts.base.ScrapperImpl.initialize(ScrapperImpl.java:51)
at org.rts.base.ScrapperImpl.initialize(ScrapperImpl.java:38)
at org.scraptool.ScrapperTool.initializeScrappers(ScrapperTool.java:40)
at org.scraptool.ScrapperTool.startScrappers(ScrapperTool.java:80)
at org.scraptool.ScrapperTool.main(ScrapperTool.java:127)
java.lang.RuntimeException: ScannerProfile coolpixgnu is not defined
at org.rts.base.profileregistry.ScrapperProfileRegistry.newScrapProfile(ScrapperProfileRegistry.java:28)
at org.rts.base.ScrapperImpl.initialize(ScrapperImpl.java:51)
at org.rts.base.ScrapperImpl.initialize(ScrapperImpl.java:38)
at org.scraptool.ScrapperTool.initializeScrappers(ScrapperTool.java:40)
at org.scraptool.ScrapperTool.startScrappers(ScrapperTool.java:80)
at org.scraptool.ScrapperTool.main(ScrapperTool.java:127)