Giter Club home page Giter Club logo

easy-es's Introduction

East-Es-Logo

Born To Simplify Development

maven code style

What is Easy-Es?

Easy-Es is a powerfully enhanced toolkit of RestHighLevelClient for simplify development. This toolkit provides some efficient, useful, out-of-the-box features for ElasticSearch. By using Easy-Es, you can use MySQL syntax to complete Es queries. Use it can effectively save your development time.

Official website

easy-es website https://en.easy-es.cn/

easy-es gitee https://gitee.com/dromara/easy-es

easy-es github https://github.com/dromara/easy-es

dromara website https://dromara.org/

dromara gitee homepage https://gitee.com/dromara/

Links

Features

  • Automatically create and update indexes, automatically migrate data, and process zero downtime
  • Auto configuration on startup
  • Out-of-the-box interfaces for operate es
  • Powerful and flexible where condition wrapper
  • Lambda-style API
  • Automatic paging operation
  • Support high-level syntax such as highlighting and weighting and Geo etc
  • ...

Compare

Demand: Query all documents with title equals "Hi" and author equals "Guy"

// Use Easy-Es to complete the query with only 1 lines of code
List<Document> documents = documentMapper.selectList(EsWrappers.lambdaQuery(Document.class).eq(Document::getTitle, "Hi").eq(Document::getCreator, "Guy"));
// Query with RestHighLevelClient requires 11 lines of code, not including parsing JSON code
String indexName = "document";
        SearchRequest searchRequest = new SearchRequest(indexName);
        BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
        TermQueryBuilder titleTerm = QueryBuilders.termQuery("title", "Hi");
        TermsQueryBuilder creatorTerm = QueryBuilders.termsQuery("creator", "Guy");
        boolQueryBuilder.must(titleTerm);
        boolQueryBuilder.must(creatorTerm);
        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
        searchSourceBuilder.query(boolQueryBuilder);
        searchRequest.source(searchSourceBuilder);
        try {
        SearchResponse searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
        // Then parse the DocumentList from searchResponse in various ways, omitting these codes...
        } catch (IOException e) {
        e.printStackTrace();
        }

The above is just a simple query demonstration. The more complex the actual query scene, the better the effect, which can save 3-5 times the amount of code on average.

Getting started

  • Latest Version: Maven Central

  • Add Easy-Es dependency

    • Maven:
      <dependency>
        <groupId>org.dromara.easy-es</groupId>
        <artifactId>easy-es-boot-starter</artifactId>
        <version>Latest Version</version>
      </dependency>
    • Gradle
      compile group: 'org.dromara.easy-es', name: 'easy-es-boot-starter', version: 'Latest Version'
  • Add mapper file extends BaseEsMapper interface

    public interface DocumentMapper extends BaseMapper<User> {
    }
  • Use it

    LambdaEsQueryWrapper<Document> wrapper = new LambdaEsQueryWrapper<>();
    wrapper.eq(Document::getTitle,"Hello World")
           .eq(Document::getCreator,"Guy");
    List<Document> documentList = documentMapper.selectList();

    Easy-Es will execute the following Query:

    {"query":{"bool":{"must":[{"term":{"title":{"value":"Hello World","boost":1.0}}},{"term":{"creator":{"value":"Guy","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}}

    The syntax of this query in MySQL is:

     SELECT * FROM document WHERE title = 'Hello World' AND creator = 'Guy'

This showcase is just a small part of Easy-Es features. If you want to learn more, please refer to the documentation.

Architecture

Architecture

MySQL Easy-Es and Es syntax comparison table

MySQL Easy-Es Es-DSL/Es java api
and and must
or or should
= eq term
!= ne boolQueryBuilder.mustNot(queryBuilder)
> gt QueryBuilders.rangeQuery('es field').gt()
>= ge .rangeQuery('es field').gte()
< lt .rangeQuery('es field').lt()
<= le .rangeQuery('es field').lte()
like '%field%' like QueryBuilders.wildcardQuery(field,value)
not like '%field%' notLike must not wildcardQuery(field,value)
like '%field' likeLeft QueryBuilders.wildcardQuery(field,*value)
like 'field%' likeRight QueryBuilders.wildcardQuery(field,value*)
between between QueryBuilders.rangeQuery('es field').from(xx).to(xx)
notBetween notBetween must not QueryBuilders.rangeQuery('es field').from(xx).to(xx)
is null isNull must not QueryBuilders.existsQuery(field)
is notNull isNotNull QueryBuilders.existsQuery(field)
in in QueryBuilders.termsQuery(" xx es field", xx)
not in notIn must not QueryBuilders.termsQuery(" xx es field", xx)
group by groupBy AggregationBuilders.terms()
order by orderBy fieldSortBuilder.order(ASC/DESC)
min min AggregationBuilders.min
max max AggregationBuilders.max
avg avg AggregationBuilders.avg
sum sum AggregationBuilders.sum
order by xxx asc orderByAsc fieldSortBuilder.order(SortOrder.ASC)
order by xxx desc orderByDesc fieldSortBuilder.order(SortOrder.DESC)
- match matchQuery
- matchPhrase QueryBuilders.matchPhraseQuery
- matchPrefix QueryBuilders.matchPhrasePrefixQuery
- queryStringQuery QueryBuilders.queryStringQuery
select * matchAllQuery QueryBuilders.matchAllQuery()
- highLight HighlightBuilder.Field
... ... ...

Advertising provider

ad


ad
ad
ad

Donate

Donate Easy-Es

License

Easy-Es is under the Apache 2.0 license. See the Apache License 2.0 file for details.

easy-es's People

Contributors

xpc1024 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

easy-es's Issues

多个索引查询问题

我们有这样的场景,每天按照日期生成一个索引 例如:loginlog-2022.07.01、loginlog-2022.07.02 。。。
有的逻辑要查询整个6月份的,就需要查整个6月份中的所有索引。 原生的api可以根据loginlog-2022.06*去查询。
easy-es想实现这样的效果会支持吗?

文档示例 @IndexField 更新一下

md是新的,使用示例还是旧的,还是@TableField。可能怪我没仔细看第一行的注释,不过快速上手不是先扫下示例代码吗?
排查了1个小时的错误... 哎...

Add ability to configure TLS certificate and set ApiCompatibilityMode to support ElasticSearch 8

注解标记高亮,导致空指针异常

在insert保存的时候,由于当前需要存储的字段被标记为高亮,在insert时,被标记为高亮字段所在列被忽略,导致序列化过程中出现空指异常。为什么需要插入的时候需要把高亮字段忽略,没太明白作者的意图,我理解不应该忽略该内容,如果需要忽略也是按照具体序列化方式,或者把序列化方式的操作抽象为接口,可以让使用者自己选择

eq 查询不到对应的数据

 LambdaEsQueryWrapper<Demo> wrapper = new LambdaEsQueryWrapper<>();
 wrapper.like(Demo::getMessage, demoDto.getMessage())
               .eq(Demo::getUserName, demoDto.getUserName())
              ...
demoMapper.selectList(wrapper);

比如这个 demoDto.getUserName() 的值为张三,es中是存在userName为张三的数据的,但是无法查询出来

自动平滑模式创建的新索引没有别名

吃了个线上bug 望帮忙看一下

步骤

  1. 已经存在索引document

2.注解配置
@IndexName(value = "document", shardsNum = 3)

3.应用启动

现象: 发现原document索引被删除 新建了一个document_s0索引 索引迁移成功 但是查看索引别名发现没有指向任何索引
此时集群环境导致其他机器查询找不到索引Elasticsearch exception [type=index_not_found_exception, reason=no such index [document]]

GET document/_alias
返回
{
"document_s0" : {
"aliases" : {
"ee_default_alias" : { }
}
}
}

在索引实体类中加入另外一个实体类会报错

两个实体:

@DaTa
@IndexName(value = "content_index",shardsNum = 1,replicasNum = 2)
public class Content {

@IndexId(type = IdType.CUSTOMIZE)
private String indexId;

/**
 * 这个字段没有存进去,在研究
 */
@IndexField("id")
private Long id;

@IndexField(value = "title")
private String title;

/**
 * 将时间格式化,并转为**时区
 */
@JsonFormat(pattern = "yyyy-MM-dd HH:mm:ss", timezone = "GMT+8")
private Date planEndTime;

private Integer tplId;


@IndexField(value = "student",fieldType = FieldType.OBJECT)
private Student student;

// @IndexField(value = "students",fieldType = FieldType.ARRAY)
// private List students;

public void setId(Long id) {
    this.id = id;
    this.indexId = id.toString();
}

}

@DaTa
@AllArgsConstructor
@NoArgsConstructor
public class Student {

private String name;

private Integer age;

@JsonFormat(pattern = "yyyy-MM-dd HH:mm:ss", timezone = "GMT+8")
private Date dirthday;

}

报错内容:
12:56:51.371 [http-nio-8080-exec-1] ERROR o.a.c.c.C.[.[.[.[dispatcherServlet] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.reflect.UndeclaredThrowableException] with root cause
java.lang.NullPointerException: null
at com.alibaba.fastjson.serializer.SerializeWriter.writeFieldValueStringWithDoubleQuoteCheck(SerializeWriter.java:1869)
at com.alibaba.fastjson.serializer.SerializeWriter.writeFieldValue(SerializeWriter.java:1854)
at com.alibaba.fastjson.serializer.ASMSerializer_2_Student.writeNormal(Unknown Source)
at com.alibaba.fastjson.serializer.ASMSerializer_2_Student.write(Unknown Source)
at com.alibaba.fastjson.serializer.ASMSerializer_1_Content.writeNormal(Unknown Source)
at com.alibaba.fastjson.serializer.ASMSerializer_1_Content.write(Unknown Source)
at com.alibaba.fastjson.serializer.JSONSerializer.write(JSONSerializer.java:312)
at com.alibaba.fastjson.JSON.toJSONString(JSON.java:769)
at com.alibaba.fastjson.JSON.toJSONString(JSON.java:707)
at cn.easyes.core.conditions.BaseEsMapperImpl.buildJsonIndexSource(BaseEsMapperImpl.java:1132)
at cn.easyes.core.conditions.BaseEsMapperImpl.buildIndexRequest(BaseEsMapperImpl.java:855)
at cn.easyes.core.conditions.BaseEsMapperImpl.doInsert(BaseEsMapperImpl.java:546)
at cn.easyes.core.conditions.BaseEsMapperImpl.lambda$insert$7(BaseEsMapperImpl.java:266)
at java.base/java.util.stream.ReferencePipeline$4$1.accept(ReferencePipeline.java:212)
at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.IntPipeline.reduce(IntPipeline.java:491)
at java.base/java.util.stream.IntPipeline.sum(IntPipeline.java:449)
at cn.easyes.core.conditions.BaseEsMapperImpl.insert(BaseEsMapperImpl.java:267)
at cn.easyes.core.conditions.BaseEsMapperImpl.insert(BaseEsMapperImpl.java:257)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at cn.easyes.core.proxy.EsMapperProxy.invoke(EsMapperProxy.java:28)
at com.sun.proxy.$Proxy57.insert(Unknown Source)
at com.hhf.controller.TestController.t1(TestController.java:56)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:655)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:895)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1732)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:834)

SearchResponse结果与直接执行DSL中获取的结果不一致

  1. 版本: easy-es-boot-starter.version 版本为1.0.3 ,elasticsearch版本为:7.2.0
  2. 使用Suggest推荐词条,DSL为: {"_source":{"includes":["ayCompletion"],"excludes":[]},"suggest":{"ayCompletion":{"prefix":"合","completion":{"field":"ayCompletion","size":10,"skip_duplicates":true}}}}
  3. SearchResponse结果中条目为两条,但是DSL的options中为5条
  4. java代码为:java代码
  5. DSL执行结果为: DSL结果

贡献使用者 丨 Contributing user

为了更好地建立开源互信体系,若贵公司在项目中有使用Easy-Es,推荐您在下方进行登记。
我们将在官网公示,每月可享EE官网免费几十万流量,非商业化运作,不用担心承担任何法律责任。
格式:公司名 + 公司网址
例如: 老汉有限责任公司 https://www.easy-es.cn
请如实填写,不要为了蹭流量而填写,实际项目中有用到EE再填写,感谢每一位支持EE的coder,respect!
image

请求支持索引数据从DB中同步

请求支持场景:
业务数据使用MySQL存储,ES只做搜索,这样就需要支持MySQL数据同步到ES;
MySQL设计表结构一般遵循三范式、ES索引结构却支持多层级JSON,故一般SQL主表一条记录对应ES一个文档、SQL主表的关联表对应ES文档下的一个JSON,如:

create table student
(
    id   bigint unsigned auto_increment
        primary key,
    name varchar(32) not null comment '名称'
);
create table course
(
    id   bigint unsigned auto_increment
        primary key,
    name varchar(32) not null comment '名称'
);
create table student_course
(
    id         bigint unsigned auto_increment
        primary key,
    student_id bigint not null comment '学生id',
    course_id  bigint not null comment '课程id'
);

对应ES的数据应该是:

{
  "_id": 1,
  "name": "学生名称",
  "course": [
    {
      "course_id": 1,
      "name": "课程名称"
    }
  ]
}

请求支持此种数据同步,业界方案一般是使用读binlog、表和文档字段映射关系配置来实现,但自己实现起来会遇到各种问题,如:拓展性不够、容灾性不高、数据会丢失、同步会卡住、同步延迟大、SQL改动后会涉及到重建索引
一直没有一个好用的、开源的、易入手的框架工具来解决这个事情,看了作者大大的EE索引维护方案,希望作者大大能够考虑这种场景,出一个解决方案

id是索引中的一个字段怎么办,又想把id设置为主键

@DaTa
@IndexName(value = "rmrb_live_info_test_s1")
public class Content {

@IndexField("id")
@IndexId(type = IdType.CUSTOMIZE)
private Long id;

private String title;

private Date planEndTime;

private Integer tplId;

}

我实体类是这样写的,id即使主键又是索引数据的一个字段,我这样写,只能把id设置成主键,索引数据中没有id这个字段,请问怎么设置呢

NullPointerException BaseEsMapperImpl json 过滤器导致空指针异常。

Caused by: java.lang.NullPointerException
at com.alibaba.fastjson.serializer.SerializeWriter.writeFieldValueStringWithDoubleQuoteCheck(SerializeWriter.java:1869)
at com.alibaba.fastjson.serializer.SerializeWriter.writeFieldValue(SerializeWriter.java:1854)
at com.alibaba.fastjson.serializer.ASMSerializer_2_UserDetail.writeNormal(Unknown Source)
at com.alibaba.fastjson.serializer.ASMSerializer_2_UserDetail.write(Unknown Source)
at com.alibaba.fastjson.serializer.ASMSerializer_1_ZMessage.writeNormal(Unknown Source)
at com.alibaba.fastjson.serializer.ASMSerializer_1_ZMessage.write(Unknown Source)
at com.alibaba.fastjson.serializer.JSONSerializer.write(JSONSerializer.java:312)
at com.alibaba.fastjson.JSON.toJSONString(JSON.java:793)
at com.alibaba.fastjson.JSON.toJSONString(JSON.java:731)
at cn.easyes.core.conditions.BaseEsMapperImpl.buildJsonIndexSource(BaseEsMapperImpl.java:1153)
at cn.easyes.core.conditions.BaseEsMapperImpl.buildIndexRequest(BaseEsMapperImpl.java:876)
当去除掉项目中的 json 过滤器则可以解析成功。
String json = JSON.toJSONString(zMessage, SerializerFeature.DisableCircularReferenceDetect,SerializerFeature.WriteMapNullValue);

public static void main(String[] args) {
    ZMessage zMessage = new ZMessage();
    zMessage.setAck(true);
    zMessage.setCmd(1);
    UserDetail userDetail = new UserDetail();
    userDetail.setType("ceshi");
    userDetail.setSex("男");

    zMessage.setBody(userDetail);
    zMessage.setUser(userDetail);
    zMessage.setMetadata(new HashMap<>());

    // 字段过滤器
    EntityInfo entityInfo = EntityInfoHelper.getEntityInfo(ZMessage.class);
    List<SerializeFilter> serializeFilters = new ArrayList<>();
    Optional.ofNullable(entityInfo.getClassSimplePropertyPreFilterMap().get(ZMessage.class))
            .ifPresent(serializeFilters::addAll);
    Set<String> excludeColumn = new HashSet<>();
    excludeColumn.add("id");
    // 主类中的字段策略过滤
    SimplePropertyPreFilter simplePropertyPreFilter = FastJsonUtils.getSimplePropertyPreFilter(ZMessage.class, excludeColumn);
    Optional.ofNullable(simplePropertyPreFilter).ifPresent(serializeFilters::add);
    //String json = JSON.toJSONString(zMessage, serializeFilters.toArray(new SerializeFilter[0]), SerializerFeature.DisableCircularReferenceDetect,SerializerFeature.WriteMapNullValue);
    String jsonString = JSON.toJSONString(zMessage, serializeFilters.toArray(new SerializeFilter[0]), SerializerFeature.DisableCircularReferenceDetect,SerializerFeature.WriteMapNullValue);
    //System.out.println(json);
    System.out.println(jsonString);
}

ZMessage {
@indexid(type = IdType.NONE)
@ApiModelProperty("nosql主键")
private String id;
@ApiModelProperty("指令 对应数据字典type值 cmd")
private Integer cmd;
@ApiModelProperty("消息体")
@JSONField(name="body")
private Object body;
@ApiModelProperty(" true是回执消息,false为非回执消息")
private Boolean ack = false;
@ApiModelProperty("metadata 元数据自定义 K,V")
public Map<String,String> metadata;
@ApiModelProperty("用户信息")
private UserDetail user;
@ApiModelProperty("是否进行回调通知 true 是 false 否")
private Boolean ackNotify = false;
/**
* 高亮返回值被映射的字段
*/
@ApiModelProperty("高亮返回值被映射的字段")
private String highlightContent;

}

UserDetail {
/**
* 头像
/
@ApiModelProperty("头像")
@JSONField(name="avatarUrl")
private String avatarUrl;
/
*
* 昵称
/
@ApiModelProperty("昵称")
@JSONField(name="nickname")
private String nickname;
/
*
* 个性签名
/
@ApiModelProperty("个性签名")
@JSONField(name="signature")
private String signature;
/
*
* 通讯账号
/
@ApiModelProperty("通讯账号")
@JSONField(name="userAccount")
private String userAccount;
/
*
* 性别
*/
@ApiModelProperty("性别")
@JSONField(name="sex")
private String sex;

@ApiModelProperty("设备ID")
@JSONField(name="deviceId")
private String deviceId;

@ApiModelProperty("sdk类型")
@JSONField(name="type")
private String type;

}

版本 esay-es 1.1.0 fastjson 1.2.83

如果文档有嵌套对象,序列化时会出现空指针异常

版本:0.9.8
异常场景:新增数据
异常原因:com.xpc.easyes.core.toolkit.EntityInfoHelper#addNameFilter。获取类下所有属性时,不包含子对象的属性。导致从mappingColumnMap中获取映射字段会出现空指针异常。
image
测试代码:
image

Failed to resolve org.junit.platform:junit-platform-launcher:1.5.2

easy-es-test模块需要加上依赖依赖:否则编译会不通过

    <dependency>
        <groupId>org.junit.platform</groupId>
        <artifactId>junit-platform-launcher</artifactId>
        <scope>test</scope>
        <version>1.8.0</version>
    </dependency>

   <dependency>
        <groupId>org.junit.platform</groupId>
        <artifactId>junit-platform-engine</artifactId>
        <scope>test</scope>
        <version>1.8.2</version>
    </dependency>

根据嵌套对象里的字段排序

除了利用原生的语句,可以用wrapper的方式根据嵌套对象里的字段排序吗,没有找到对应的方法

OrderByParam order = new OrderByParam();
order.setOrder("student.age");
wrapper.orderByDesc("student.age");
用这种方式没有效果,不能排序

兼容低版本

公司使用的es版本为6.8.12。easy-es能做兼容吗

Lambda 缓存

LambdaWrapper 构建条件的时候,Lambda 表达式对应的字段名称缓存。

@Test
public void testSelect() {
    // 测试查询
    String title = "老汉";
    LambdaEsQueryWrapper<Document> wrapper = new LambdaEsQueryWrapper<>();
    wrapper.eq(Document::getTitle,title);
    Document document = documentMapper.selectOne(wrapper);
    System.out.println(document);
    Assert.assertEquals(title,document.getTitle());
}

e.g. Document::getTitle -> title 第二次使用的时候,直接使用第一次生成好的缓存。

完全按照官方文档写的,但是SpringBoot3.0.3启动报错

image
SpringBoot版本:3.0.3
Elasticsearch版本:7.14.0
依赖引入如下:
<parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>3.0.3</version> <relativePath/> <!-- lookup parent from repository --> </parent> <groupId>com.aoxqwl</groupId> <artifactId>eedemo</artifactId> <version>1</version> <name>eedemo</name> <description>eedemo</description> <properties> <java.version>17</java.version> </properties> <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> <!-- lombok插件依赖 --> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <scope>provided</scope> </dependency> <!-- Easy-Es --> <dependency> <groupId>cn.easy-es</groupId> <artifactId>easy-es-boot-starter</artifactId> <version>1.1.1</version> <exclusions> <exclusion> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> </exclusion> <exclusion> <groupId>org.elasticsearch.client</groupId> <artifactId>elasticsearch-rest-high-level-client</artifactId> </exclusion> <exclusion> <groupId>org.elasticsearch</groupId> <artifactId>elasticsearch</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.elasticsearch.client</groupId> <artifactId>elasticsearch-rest-high-level-client</artifactId> <version>7.14.0</version> </dependency> <dependency> <groupId>org.elasticsearch</groupId> <artifactId>elasticsearch</artifactId> <version>7.14.0</version> </dependency> </dependencies>

1.1.0版本 使用groupBy 查询报错

使用的es版本为 7.14.0

java代码:
LambdaEsQueryWrapper queryWrapper = new LambdaEsQueryWrapper<>();
queryWrapper.groupBy(ImportElasticDataVO::getCustNo);
SearchResponse search = elasticMapper.search(queryWrapper);

报错:
Caused by: ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=maxSize must be >= 0 and < 2147483631; got: 2147483647]]

dsl语句:
{"size":10000,"query":{"bool":{"adjust_pure_negative":true,"boost":1.0}},"track_total_hits":2147483647,"aggregations":{"custNoTerms":{"terms":{"field":"custNo","size":2147483647,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},{"_key":"asc"}]}}}}

使用or()发现DSL被篡改了

  • 目的:实现类似这样的sql:where (id = 1) and (id = 1 and id in (1, 2)) and (id = 1 or id in (1, 2, 3))

  • 单元测试如下:

        @Test
        void m1() {
    
            // String sql = "where (id = 1) and (id = 1 and id in (1, 2)) and (id = 1 or id in (1, 2, 3))";
    
            LambdaEsQueryWrapper<User> w = lambdaQuery(User.class)
                    .eq(User::getId, 1L)
                    .and(i -> i
                            .eq(User::getId, 1L)
                            .in(User::getId, newArrayList(1L, 2L)))
                    .and(i -> i
                            .eq(User::getId, 2L)
                            .or()
                            .in(User::getId, newArrayList(1L, 2L, 3L)));
    
            List<User> users = userMapper.selectList(w);
            System.err.println(users);
        }
  • 控制台打印的DSL(json较长,去掉了不影响说明问题的部分),下方的should和must错乱了,出问题的地方下方我使用了注释标记

    {
      "query": {
        "bool": {
          "must": [
            {
              "term": {
                "_id": {
                  "value": 1,
                  "boost": 1.0
                }
              }
            },
            {          
              "bool": {
                // 理论上此处应该是must
                "should": [
                  {
                    "term": {
                      "_id": {
                        "value": 1,
                        "boost": 1.0
                      }
                    }
                  },
                  {
                    "terms": {
                      "_id": [
                        1,
                        2
                      ],
                      "boost": 1.0
                    }
                  }
                ],
                "adjust_pure_negative": true,
                "boost": 1.0
              }
            },
            {
              "bool": {
                // 理论上此处应该是should
                "must": [
                  {
                    "term": {
                      "_id": {
                        "value": 1,
                        "boost": 1.0
                      }
                    }
                  },
                  {
                    "terms": {
                      "_id": [
                        1,
                        2,
                        3
                      ],
                      "boost": 1.0
                    }
                  }
                ],
                "adjust_pure_negative": true,
                "boost": 1.0
              }
            }
          ]
        }
      }
    }
  • 假如我把wrapper中的第一个and删掉,变成如下,那打印出来的DSL语句又变成正确的了

        @Test
        void m2() {
    
            // String sql = "where (id = 1) and (id = 1 or id in (1, 2, 3))";
    
            LambdaEsQueryWrapper<User> w = lambdaQuery(User.class)
                    .eq(User::getId, 1L)
                    .and(i -> i
                            .eq(User::getId, 2L)
                            .or()
                            .in(User::getId, newArrayList(1L, 2L, 3L)));
    
            List<User> users = userMapper.selectList(w);
            System.err.println(users);
        }

Date 类型字段格式化问题

设置了日期字段类型,并且设置dateFormat=“yyyy-MM-dd HH:mm:ss”

image

插入数据报错。

org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=mapper_parsing_exception, reason=failed to parse field [operTime] of type [date] in document with id 'OX-rLYQBzWmLI-OUHflm'. Preview of field's value: '2022-10-31T10:51:39.764']
        at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177)
        at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1793)
        at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1770)
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1527)
        at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1484)
        at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1454)
        at org.elasticsearch.client.RestHighLevelClient.index(RestHighLevelClient.java:871)
        at cn.easyes.core.conditions.BaseEsMapperImpl.insert(BaseEsMapperImpl.java:306)
        at cn.easyes.core.conditions.BaseEsMapperImpl.insert(BaseEsMapperImpl.java:294)
        ... 17 common frames omitted
        Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://elasticsearch:9200], URI [/tutor_operlog/_doc?timeout=1m], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse field [operTime] of type [date] in document with id 'OX-rLYQBzWmLI-OUHflm'. Preview of field's value: '2022-10-31T10:51:39.764'"}],"type":"mapper_parsing_exception","reason":"failed to parse field [operTime] of type [date] in document with id 'OX-rLYQBzWmLI-OUHflm'. Preview of field's value: '2022-10-31T10:51:39.764'","caused_by":{"type":"illegal_argument_exception","reason":"failed to parse date field [2022-10-31T10:51:39.764] with format [yyyy-MM-dd HH:mm:ss]","caused_by":{"type":"date_time_parse_exception","reason":"Text '2022-10-31T10:51:39.764' could not be parsed at index 10"}}},"status":400}
                at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:283)
                at org.elasticsearch.client.RestClient.performRequest(RestClient.java:261)
                at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
                at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1514)
                ... 22 common frames omitted
Caused by: org.elasticsearch.ElasticsearchException: Elasticsearch exception [type=illegal_argument_exception, reason=failed to parse date field [2022-10-31T10:51:39.764] with format [yyyy-MM-dd HH:mm:ss]]
        at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:496)
        at org.elasticsearch.ElasticsearchException.fromXContent(ElasticsearchException.java:407)
        at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:437)
        at org.elasticsearch.ElasticsearchException.failureFromXContent(ElasticsearchException.java:603)
        at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:169)
        ... 25 common frames omitted
Caused by: org.elasticsearch.ElasticsearchException: Elasticsearch exception [type=date_time_parse_exception, reason=Text '2022-10-31T10:51:39.764' could not be parsed at index 10]
        at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:496)
        at org.elasticsearch.ElasticsearchException.fromXContent(ElasticsearchException.java:407)
        at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:437)
        ... 29 common frames omitted

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.