Giter Club home page Giter Club logo

go-solr's Introduction

go-solr

GoDoc Build Status codebeat badge

Solr client in golang

Usage

  • install the cli using go get -u github.com/at15/go-solr/cmd/solrgo, it can create core and index json document
package main

import (
	"context"
	"fmt"
	"log"
	"os"

	"github.com/at15/go-solr/solr"
)

const coreName = "job"

func main() {
    c := solr.Config{}
    solrClient, err := solr.NewClient(c)
    if err != nil {
        log.Fatal(err)
        return
    }
    if err := solrClient.IsUp(context.Background()); err != nil {
        log.Fatalf("Solr is not up %v", err)
        return
    }
    log.Println("Solr is up")
    solrClient.UseCore(coreName)
    if status, err := solrClient.DefaultCore.Status(context.Background(), false); err != nil {
        log.Fatalf("Check core status failed %v", err)
        return
    } else {
        log.Printf("Got status for core %s %v\n", coreName, status)
    }
}

Features

  • standalone command line util (single binary)
  • query builder
  • auto convert struct to JSON when ingest document

Not implemented

Roadmap

  • using managed schema
    • generate schema xml based on golang struct (w/o?) annotation
    • support schema less
  • query using JSON API
  • v2 API
  • collect metrics about upstream
  • client side load balancing

Alternatives

go-solr's People

Contributors

at15 avatar

Stargazers

 avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

go-solr's Issues

[cmd] Command line for generator and util

Related #7

  • gosolr create core or gosolr core create to create a core in command line (without using bin/solr
  • generate schema migration file
  • generate pogo (plain old golang object) based on json? could be used for go generate

[internal] nil pointer when fetch system info

/Users/piguo/app/go/bin/go run /Users/piguo/workspace/src/github.com/at15/go-solr/example/job/main.go
example for storing job log in solr
panic: runtime error: invalid memory address or nil pointer dereference
creating solr client
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x11e3d84]

goroutine 1 [running]:
net/http.(*Request).WithContext(0x0, 0x139a860, 0xc420010228, 0xc4200c0600)
	/Users/piguo/app/go/src/net/http/request.go:332 +0x44
github.com/at15/go-solr/pkg/internal.(*Client).Do(0xc4200146e0, 0x139a860, 0xc420010228, 0x0, 0x1225080, 0xc4200cc580, 0x0, 0x0, 0x0)
	/Users/piguo/workspace/src/github.com/at15/go-solr/pkg/internal/client.go:74 +0x7e
github.com/at15/go-solr/pkg/internal.(*Client).Get(0xc4200146e0, 0x139a860, 0xc420010228, 0x1293415, 0x17, 0x1225080, 0xc4200cc580, 0xc42000e038, 0xc420049eb8, 0x12078e4)
	/Users/piguo/workspace/src/github.com/at15/go-solr/pkg/internal/client.go:70 +0xe8
github.com/at15/go-solr/pkg/admin.(*Service).SystemInfo(0xc42000e038, 0x139a860, 0xc420010228, 0xc42000e018, 0xc42000e038, 0xc4200146e0)
	/Users/piguo/workspace/src/github.com/at15/go-solr/pkg/admin/service.go:20 +0x86
github.com/at15/go-solr/pkg.(*SolrClient).IsUp(0xc420016480, 0x139a860, 0xc420010228, 0xc420016480, 0x0)
	/Users/piguo/workspace/src/github.com/at15/go-solr/pkg/client.go:65 +0x47
main.main()
	/Users/piguo/workspace/src/github.com/at15/go-solr/example/job/main.go:19 +0x1e7
exit status 2

Client, Request, Response

We need to design a standard struct/interface for client, request, response to use across all the operations, schema changes, updating documents etc. It is possible we do the converting directly and user don't need to call json.Unmarshal directly

TODO

  • checkResponse does not work for certain APIs because
    • they don't use status code to indicate error
    • 404 returns html page
    • schema management has its own error response format

Ref

[tracker] Core

Admin

(move those operations under the admin package, and leave the core package for dealing with documents)

  • create
  • create if not exists #6
  • exist
  • unload
  • delete

Core itself (it should be put into document, or query etc.)

  • insert
  • commit
  • query

[schema] Infer schema based on type and tag using reflect

TODO

%v     the value in a default format
       when printing structs, the plus flag (%+v) adds field names
%#v    a Go-syntax representation of the value
%T     a Go-syntax representation of the type of the value
%%     a literal percent sign; consumes no value

[core] CreateCoreIfNotExist need to do real check instead of create and fail

ERRO[0004] can't create core solr: can't create core http://solr:8983/solr/admin/cores?action=CREATE&configSet=data_driven_schema_configs&instanceDir=demo&name=demo&wt=json: solr: 400: Error CREATEing SolrCore 'demo': Unable to create core [demo] Caused by: Lock held by this virtual machine: /opt/solr/server/solr/demo/data/index/write.lock pkg=github.com/at15/go-solr/pkg/core
  • use Status API to fetch core
  • duplicated can't create core in error message
  • use a external script to setup the test environment instead of doing it in TestMain

Test with Solr

Since it is a solr client, it should test against real solr (w/ & w/o SolrCloud)

We can also use docker-compose to run test locally, though it is more convenient to use a local Solr

  • docker-compose setup (single node, no zk)
  • code coverage
  • unit test helpers (create core etc.) #7
  • e2e tests

[cmd] version is not set when using go get

run go get -u github.com/at15/go-solr/cmd/solrgo to install solrgo and the version etc. won't be set through ldflag and solrgo version would return nothing, we should have some default value

admin/info/system returns NaN for systemCpuLoad on Mac

  • it works on ubuntu
  • it broke on mac "systemCpuLoad":"NaN"

Solutions

  • use interface{} for this field, since we don't use it
  • custom unmarshaler
  • it could be a problem from solr itself
"system":{"name":"Mac OS X","arch":"x86_64","availableProcessors":4,"systemLoadAverage":2.24951171875,"version":"10.12.6","committedVirtualMemorySize":4545413120,"freePhysicalMemorySize":257499136,"freeSwapSpaceSize":1625554944,"processCpuLoad":0.5227272727272727,"processCpuTime":7047363000,"systemCpuLoad":"NaN","totalPhysicalMemorySize":17179869184,"totalSwapSpaceSize":2147483648,"maxFileDescriptorCount":10240,"openFileDescriptorCount":132,"uname":"Darwin LM-SJN-21001657 16.7.0 Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64 x86_64\n","uptime":"13:08  up 2 days,  1:28, 6 users, load averages: 2.25 2.26 2.20\n"}}

[schema][index] Can't index document after create core

  • it works locally
  • might because the core is not created?
  • file permission
INFO[0000] Core job already exists pkg=gosolr 
cd example/film; solrgo core index film films.json
FATA[0000] can't index films.json: solr: can't update document http://solr:8983/solr/film/update?commit=true&wt=json: solr: 500: Error persisting managed schema /opt/solr/server/solr/configsets/film/conf/managed-schema pkg=gosolr 

[refactor] Flatten packages

Current packages are pretty annoying when using, over subpackaged, especially when the schema part is almost unusable except create core due to #15

  • rename pkg folder to solr so user don't need rename when import
  • move core, search, common into top level, not sure about admin though

[test] Use external script to setup test environment

Related #6

When golang run test against multiple packages, multiple TestMain is executed, so when run CreateIfNotExist there may have race condition, thus we got error as Unable to create core [demo] Caused by: Lock held by this virtual machine: /opt/solr/server/solr/demo/data/index/write.lock besides writing CreateIfNotExist in the right way, we should also write scripts to setup the test environment before all the tests start running

  • re-create local environment
    • rm old solr installation
    • extract tar
  • create demo core

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.