https://questdb.io/docs/reference/api/ilp/columnset-types#long256

This sample is a bit silly: https://github.com/questdb/go-questdb-client/blob/2190485745fb270465fabf6af364bcb7dee3b61a/examples/from-conf/main.go

I assume this was taken from Java samples which I am guilty of using in the past.

We should use samples related to QuestDB use-cases. Ideally the same samples across clients. Since the samples are imported into Cloud and shown in a console.

It looks like the go client doesn't support something like BinaryColumn, which is supported by the core db.

But it looks like the ILP parser doesn't support binary types yet...

There are several issues and pull requests throughout this repo that discuss the fact that the LineSender is not fully safe, i.e. there's issues in auto-reconnection, there's no way to retrieve the failed sent buffer to retry if there is a connection issue, etc.

@puzpuzpuz has said that this can't be implemented without data deduplication on the server side.

According to the QuestDB Docs site, data deduplication is implemented on the DB since QuestDB 7.3: https://questdb.io/docs/concept/deduplication/

Can we get this MAJOR ISSUE fixed (at the very least as an option on the client that we can enable) so we can have some better safety in what will happen if there's a connection issue, so we don't lose data and we don't have to implement reconnection logic on our own?

Hello and thank you for all the great efforts put into the questdb project. I have been using the client for a few days now and it came to my attention that the client does not seem to reconnect on TCP disconnections. The way it works now is that you will need to take the following steps

1. Establish a connection
2. Call LineSender.Table... then LineSender.At
3. Flush lines
4. Close connection

The above will need to be repeated every time there is a need to send the data to questdb. This is most likely by design and is intended to work this way. Though there is an overhead for establishing a connection every time one wants to send data or reconnect during a flush error.

Would it be desirable to have the client auto-connect and maintain the connection with the server?

See questdb/questdb#4385

Hi,

thanks for your library and awesome db! I generate line protocol entry via reflection and I got all my lines ready to be pushed. I want to use the library so it handles batch, connection, tls, etc. Is there a way where I can write the line directly to the buffer?

It would be nice to have something like https://pkg.go.dev/github.com/jackc/pgx/v5/pgxpool so we could instantiate a pool of senders to be concurrently reused across go routines.

There are a few issues with the current approach:

• If flush fails due to a network issue, you are forced to create a new sender... meaning all the messages you have buffered to be flushed get lost.
• There's no re-connection logic (automatic or manual) and there's no way to check if the connection is alive.

Actually maybe it's not even a good idea to connect on NewLineSender() init, as there may be substantial delay between initialization and flushing, causing the connection to timeout (or disconnect due to any network error) before flush is even called.

The Bug

If you try to create a sender with a bad URL the code hangs indefinitely. I think it should return an error, perhaps after a configurable timeout.

To Replicate

You should be able to replicate this behavior by running the following code:

package main

import (
	"context"
	"encoding/json"
	"fmt"
	qdb "github.com/questdb/go-questdb-client/v2"
	"io"
	"log"
	"net/http"
	"os"
)

func main() {
	fmt.Printf("Attempting to connect...")
	ctx := context.TODO()

	sender, err := qdb.NewLineSender(
		ctx,
		qdb.WithAddress("trial-instance-xxxxxxxx.ilp.xxxx.questdb.com:3xxxx"), // <== If this is wrong the program will hang
		qdb.WithBufferCapacity(4096),
	)
	if err != nil {
		log.Fatal(fmt.Errorf("error: Failed to connect to QDB: %w", err))
		return
	}
	//defer sender.Close()
	// Make sure to close the sender on exit to release resources.
	defer func(sender *qdb.LineSender) {
		err := sender.Close()
		if err != nil {
			log.Printf("Error closing sender to QuestDB - may cause resource leak: %v", err)
		}
	}(sender)

	fmt.Printf("Connection made")
}

For Others Looking For Connection URL:

For anyone else trying to connect to your cloud trial instance, poke around your trial instance webpage a bit, but not in the web-console. You'll find the correct URL in places where they talk about adding data to the DB. The port is not 9009, unlike the local hello-world install.

Currently you can send timestamps as long (int64), but this assumes a table was already pre-created. It wont work for auto-created table. It would be nice to support timestamp columns explicitly.

Java client counterpart PR: questdb/questdb#2257