[Connector] [Kafka] Improve code coverage in Kafka Modules about gimel HOT 8 CLOSED

@Dee-Pac if there is no one working on it, yes. But though I'm looking on code yet, so it will take bit time.
If no one is working with #63 , I'll try on it too.

from gimel.

Hello,
Is test running on kafka module? I couldn't find in log. Or maybe I'm doing something wrong...

from gimel.

@djKooks I realized the scalatest plugin is missing in the pom.xml of project gimel-dataapi/gimel-connectors/gimel-kafka-0.1.0

By adding the snippet below, the tests would execute.

      <plugin>
        <groupId>org.scalatest</groupId>
        <artifactId>scalatest-maven-plugin</artifactId>
        <version>1.0</version>
        <configuration>
          <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
          <junitxml>.</junitxml>
          <filereports>WDF TestSuite.txt</filereports>
        </configuration>
        <executions>
          <execution>
            <id>test</id>
            <goals>
              <goal>test</goal>
            </goals>
          </execution>
        </executions>
      </plugin>

Please let me know if you'd be ok to add the snippet and work on your fixes.

Once you add above snippet to the pom.xml in gimel-dataapi/gimel-connectors/gimel-kafka-0.1.0/pom.xml , you'd start seeing the test results along these lines ...

build/gimel -pl gimel-dataapi/gimel-connectors/gimel-kafka-0.10 -am -T10

.......
.....
..

[INFO] --- scalatest-maven-plugin:1.0:test (test) @ gimel-kafka-0.10 ---
Discovery starting.
Discovery completed in 372 milliseconds.
Run starting. Expected test count is: 12
DataSetUtilsSpec:
- should throw exception for null input.
- should throw exception for malformed str input.
- should throw exception for str input key-values are not separated by semicolon.
- should return empty Map for the empty string input.
- should return empty for empty Map inputs.
- should return the right value for valid str input.
- should return the right value for valid map input.
KafkaConvertersTests:
- should convert array of offset ranges to a single parsable string
- should converr offset Range to a single parsable checkPoint String
- should convert a single parsable CheckPoint string to a valid offset Range
- should convert composite `CheckPoint (Array[String])` to a valid Array(Offset Range)
- should convert a json string of custom partition information to an array of offset ranges
Run completed in 764 milliseconds.
Total number of tests run: 12
Suites: completed 3, aborted 0
Tests: succeeded 12, failed 0, canceled 0, ignored 0, pending 0
All tests passed.

from gimel.

@Dee-Pac yes, it works now. Thanks!
Is there a reason not putting this setting for kafka module?

Also, it seems DataSetUtilsSpec test is not related with kafka. Am I right?

from gimel.

@djKooks Yes, you are right. DataSetUtilsSpec is not.
Also, there is an issue open to add the missing scalatest plugin.

#63

Are you planing to add tests??

from gimel.

@Dee-Pac as I commented, I'm trying to update test for APIs in kafka connector. But I'm still studying on module, so hope you could help.

I just added test for NewOffsetRangesProvider, but not quite sure for the purpose. Could you simply describe the use/purpose of this module? Give a comment in this issue, or you could write in PR.
Thanks.

from gimel.

@djKooks

Please have a look at the method toNewOffsetRanges. It takes two sets of OffsetRange, does a diff between the right & left OffsetRange - provides a New OffsetRange for the consumer.
https://github.com/paypal/gimel/search?q=toNewOffsetRanges&unscoped_q=toNewOffsetRanges

Let's say a new KafkaConsumer is initiated, this consumer picks up the Left(OffsetRange) from Last Saved Checkpoint, Picks the Right(OffsetRange) from Latest Offsets in Kafka or what is passed. Using the two OffsetRange - the new OffsetRange for Current Consumption is calculated.

Please let me know if you need more clarification.

from gimel.

@Dee-Pac thanks for help! I'll keep make updates.

from gimel.