Giter Club home page Giter Club logo

Comments (10)

leonchen83 avatar leonchen83 commented on July 29, 2024 1

@billcrook
refer to implement formatter service
this API not only for json format, you can define your own output format. so I can't add emitZSetJson , emitString method to this API. also can't change visibility from private to protected . you could copy these methods to your own formatter class.

from redis-rdb-cli.

leonchen83 avatar leonchen83 commented on July 29, 2024 1

@billcrook
fixed
download v0.5.0

from redis-rdb-cli.

leonchen83 avatar leonchen83 commented on July 29, 2024

@billcrook
It' a good idea to support custom formatters
We can use java SPI to implement this need like redis-sink-api in readme.md

for example I will release an api jar to maven central.
you can dependent that jar and implement your own formatter

        <dependency>
            <groupId>com.moilioncircle</groupId>
            <artifactId>redis-rdb-cli-api</artifactId>
            <version>1.2.0</version>
            <scope>provided</scope>
        </dependency>

after build your own project ,you can copy your jar to lib folder. redis-rdb-cli will found your implement dynamicly. no need use fully qualified path to the new formatter

from redis-rdb-cli.

leonchen83 avatar leonchen83 commented on July 29, 2024

this feature will release v0.5.0 after 7 days.

from redis-rdb-cli.

billcrook avatar billcrook commented on July 29, 2024

awesome! You might consider relaxing method visibility from private to protected in the base Formatter classes. For instance, emitZSetJson or emitString. In doing so, custom formatters can leverage functionality in your base visitor classes. I had to do this in order to extend your base classes. For example: billcrook@b3cce93#diff-05443b96f34d75b1a17b1f445e61afef18a5c5ba96740bcd05925d31844dd556R43-R91

from redis-rdb-cli.

billcrook avatar billcrook commented on July 29, 2024

Thanks again for the great tool!

from redis-rdb-cli.

leonchen83 avatar leonchen83 commented on July 29, 2024

@billcrook
this API for now is not stable. please use it next week.

from redis-rdb-cli.

leonchen83 avatar leonchen83 commented on July 29, 2024

@billcrook
dependent following api jar

        <dependency>
            <groupId>com.moilioncircle</groupId>
            <artifactId>redis-rdb-cli-api</artifactId>
            <version>1.4.1</version>
            <scope>provided</scope>
        </dependency>

or just use following maven pom template

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    
    <groupId>com.your.company</groupId>
    <artifactId>your-sink-service</artifactId>
    <version>1.0.0</version>
    
    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
    </properties>

    <dependencies>
        <dependency>
            <groupId>com.moilioncircle</groupId>
            <artifactId>redis-rdb-cli-api</artifactId>
            <version>1.4.1</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>com.moilioncircle</groupId>
            <artifactId>redis-replicator</artifactId>
            <version>3.4.0</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-api</artifactId>
            <version>1.7.25</version>
            <scope>provided</scope>
        </dependency>
        
        <!-- 
        <dependency>
            your other dependencies
        </dependency>
        -->
        
    </dependencies>
    
    <build>
        <plugins>
            <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>3.1.0</version>
                <configuration>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.1</version>
                <configuration>
                    <source>${maven.compiler.source}</source>
                    <target>${maven.compiler.target}</target>
                    <encoding>${project.build.sourceEncoding}</encoding>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

more detail please see test TestFormatterService and FormatterRdbVisitorTest

from redis-rdb-cli.

leonchen83 avatar leonchen83 commented on July 29, 2024

be notice that method name perfix with apply in FormatterService

for example.

applyListZipList
applyListQuickList
...

applyListZipList can split 3 parts

  1. apply
  2. List
  3. ZipList

part 2 means redis type
part 3 means redis storage structure.
so if you want parse redis list type, you must implement 3 methods : applyListZipList , applyListQuickList, applyList

if there is a hash key contains 1g or more data. you could not use following implemention

    public Event applyHash(Replicator replicator, RedisInputStream in, int version, byte[] key, int type, ContextKeyValuePair context) throws IOException {
        Map<byte[], byte[]> map = new DefaultRdbValueVisitor(replicator).applyHash(in, version);
        // your business
        return context;
    }

because new DefaultRdbValueVisitor(replicator).applyHash(in, version); will cosume too much memory to parse huge key
you should parse this hash key step by step

    public Event applyHash(Replicator replicator, RedisInputStream in, int version, byte[] key, int type, ContextKeyValuePair context) throws IOException {
        BaseRdbParser parser = new BaseRdbParser(in);

        long len = parser.rdbLoadLen().len;
        while (len > 0) {
            byte[] field = parser.rdbLoadEncodedStringObject().first();
            byte[] value = parser.rdbLoadEncodedStringObject().first();
            // your business to handle field and value step by step
            len--;
        }
        return context;
    }

from redis-rdb-cli.

billcrook avatar billcrook commented on July 29, 2024

Fantastic, thank you!

from redis-rdb-cli.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.