Comments (10)
@billcrook
refer to implement formatter service
this API not only for json format, you can define your own output format. so I can't add emitZSetJson
, emitString
method to this API. also can't change visibility from private to protected . you could copy these methods to your own formatter class.
from redis-rdb-cli.
@billcrook
fixed
download v0.5.0
from redis-rdb-cli.
@billcrook
It' a good idea to support custom formatters
We can use java SPI to implement this need like redis-sink-api in readme.md
for example I will release an api jar to maven central.
you can dependent that jar and implement your own formatter
<dependency>
<groupId>com.moilioncircle</groupId>
<artifactId>redis-rdb-cli-api</artifactId>
<version>1.2.0</version>
<scope>provided</scope>
</dependency>
after build your own project ,you can copy your jar to lib
folder. redis-rdb-cli
will found your implement dynamicly. no need use fully qualified path to the new formatter
from redis-rdb-cli.
this feature will release v0.5.0 after 7 days.
from redis-rdb-cli.
awesome! You might consider relaxing method visibility from private to protected in the base Formatter classes. For instance, emitZSetJson
or emitString
. In doing so, custom formatters can leverage functionality in your base visitor classes. I had to do this in order to extend your base classes. For example: billcrook@b3cce93#diff-05443b96f34d75b1a17b1f445e61afef18a5c5ba96740bcd05925d31844dd556R43-R91
from redis-rdb-cli.
Thanks again for the great tool!
from redis-rdb-cli.
@billcrook
this API for now is not stable. please use it next week.
from redis-rdb-cli.
@billcrook
dependent following api jar
<dependency>
<groupId>com.moilioncircle</groupId>
<artifactId>redis-rdb-cli-api</artifactId>
<version>1.4.1</version>
<scope>provided</scope>
</dependency>
or just use following maven pom
template
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.your.company</groupId>
<artifactId>your-sink-service</artifactId>
<version>1.0.0</version>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>com.moilioncircle</groupId>
<artifactId>redis-rdb-cli-api</artifactId>
<version>1.4.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.moilioncircle</groupId>
<artifactId>redis-replicator</artifactId>
<version>3.4.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.25</version>
<scope>provided</scope>
</dependency>
<!--
<dependency>
your other dependencies
</dependency>
-->
</dependencies>
<build>
<plugins>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.1.0</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.1</version>
<configuration>
<source>${maven.compiler.source}</source>
<target>${maven.compiler.target}</target>
<encoding>${project.build.sourceEncoding}</encoding>
</configuration>
</plugin>
</plugins>
</build>
</project>
more detail please see test TestFormatterService and FormatterRdbVisitorTest
from redis-rdb-cli.
be notice that method name perfix with apply
in FormatterService
for example.
applyListZipList
applyListQuickList
...
applyListZipList
can split 3 parts
- apply
- List
- ZipList
part 2 means redis type
part 3 means redis storage structure.
so if you want parse redis list
type, you must implement 3 methods : applyListZipList
, applyListQuickList
, applyList
if there is a hash
key contains 1g or more data. you could not use following implemention
public Event applyHash(Replicator replicator, RedisInputStream in, int version, byte[] key, int type, ContextKeyValuePair context) throws IOException {
Map<byte[], byte[]> map = new DefaultRdbValueVisitor(replicator).applyHash(in, version);
// your business
return context;
}
because new DefaultRdbValueVisitor(replicator).applyHash(in, version);
will cosume too much memory to parse huge key
you should parse this hash key step by step
public Event applyHash(Replicator replicator, RedisInputStream in, int version, byte[] key, int type, ContextKeyValuePair context) throws IOException {
BaseRdbParser parser = new BaseRdbParser(in);
long len = parser.rdbLoadLen().len;
while (len > 0) {
byte[] field = parser.rdbLoadEncodedStringObject().first();
byte[] value = parser.rdbLoadEncodedStringObject().first();
// your business to handle field and value step by step
len--;
}
return context;
}
from redis-rdb-cli.
Fantastic, thank you!
from redis-rdb-cli.
Related Issues (20)
- 在prometheus 可以新增监控bigkey ? HOT 3
- [Feature Request] Add a way to set dest db as rmt option HOT 10
- merge two rdb files with repeated key HOT 2
- Problem installing HOT 1
- password can't have "%" HOT 1
- NOAUTH despite authPassword and/or ACL style on Redis 6.2.6 HOT 4
- Missing required option: '--source <source>' HOT 3
- Redis remote backup import failing HOT 19
- Feature Request: rmt sync from cluster to single redis HOT 1
- java.lang.RuntimeException: java.lang.NumberFormatException: null HOT 6
- expiry is not right HOT 2
- More information about applications HOT 4
- Sync data between KeyDB v6 - Redis v7 HOT 26
- Terminating due to java.lang.OutOfMemoryError: Java heap space HOT 7
- Check migrated data consistency HOT 5
- Bloom keys not synced to target redis instance HOT 1
- rst's password maybe can't contain # HOT 2
- Adding any -k parameter returns an empty list HOT 4
- Publish to maven central HOT 5
- Compatibility of redis-rdb-cli with Redis and KeyDB versions HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from redis-rdb-cli.