Giter Club home page Giter Club logo

ydb's Introduction


License Release Discord

YDB

Website | Documentation | Official Repository | Blog | YouTube | Discord | Telegram | LinkedIn | X

YDB is an open source Distributed SQL Database that combines high availability and scalability with strict consistency and ACID transactions.

YDB Product Video

Main YDB Advantages

YDB is designed from scratch as a response to growing demand for scalable interactive web services. Scalability, strict consistency and effective cross-row transactions were a must for such OLTP-like workload. YDB is built by people with strong background in databases and distributed systems, who had an experience of developing No-SQL database and the Map-Reduce system for one of the largest search engines in the world. We found that YDB's flexible design allows us to build more services on top of it including persistent queues and virtual block devices.

Basic YDB features:

  • Fault-tolerant configuration that survive disk, node, rack or even datacenter outage;
  • Horizontal scalability;
  • Automatic disaster recovery with minimum latency disruptions for applications;
  • SQL dialect (YQL) for data manipulation and scheme definition;
  • ACID transactions across multiple nodes and tables with strict consistency.

Fault-tolerant Configurations

YDB could be deployed in three availability zones. Cluster remains available for both reads and writes during complete outage of a single zone. Availability zones and regions are covered in more detail in documentation.

Horizontal Scalability

Unlike traditional relational databases YDB scales out providing developers with capability to simply extend cluster with computation or storage resources to handle increasing load. YDB has desaggregated storage and compute layers which allow you to scale storage and compute resources independently.

Current production installations have more than 10,000 nodes, store petabytes of data and handle millions distributed transactions per second.

Automatic Disaster Recovery

YDB has built-in automatic recovery support to survive a hardware failure. After unpredictable disk, node, rack or even datacenter failure YDB remains fully available for reads and writes and restores required data redundancy automatically.

Multitenant and Serverless Database

YDB has support for multitenant and serverless setups. A user can run a YDB cluster and create several databases that share one pool of storage and have different compute nodes. Alternatively a user can run several serverless databases that share one pool of compute resources to utilize them effectively.

Supported Platforms

Minimal system requirements

YDB runs on x86 64bit platforms with minimum 8 GB of RAM.

Operating Systems

We have major experience running production systems on 64-bit x86 machines working under Ubuntu Linux.

For development purposes we test that YDB could be built and run under latest versions of MacOS and Microsoft Windows on a regular basis.

Getting Started

  1. Install YDB using pre-built executables, build it from source or use Docker container.
  2. Install command line interface tool to work with scheme and run queries.
  3. Start local cluster or container and run YQL query via YDB CLI.
  4. Access Embedded UI via browser for schema navigation, query execution and other database development related tasks.
  5. Run available example application.
  6. Develop an application using YDB SDK.

How to Build from Source Code

How to Deploy

How to Contribute

We are glad to welcome new contributors!

  1. Please read the contributor's guide.
  2. We can accept your work to YDB after you have signed contributor's license agreement (aka CLA).
  3. Please don't forget to add a note to your pull request, that you agree to the terms of the CLA.

Success Stories

Take a look at YDB web site for the latest success stories and user scenarios.

ydb's People

Contributors

4ertus2 avatar adameat avatar alexnick88 avatar alexv-smirnov avatar alexvru avatar arcadia-devtools avatar cyberrofl avatar dcherednik avatar dorooleg avatar enjection avatar georgthegreat avatar gridnevvvit avatar hor911 avatar ivanmorozov333 avatar kruall avatar nepal avatar nikitka avatar resetius avatar robot-piglet avatar rvu1024 avatar snaury avatar snermolaev avatar ssmike avatar stekperepolnen avatar tony-romanov avatar ugninesirdis avatar uzhastik avatar va-kuznecov avatar vitstn avatar zverevgeny avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ydb's Issues

[feature] JDBC driver

Well, that's what is wanted, would like to get the point when, where and how to get the thing ?
Хотелось бы понять, есть ли, планируется, когда и как ждать?

Non-primitive data types in table schema.

Ability to use non-primitive data types in table column definition.

As i understand current workaround is to use Pickle, Unpickle functions with String data type, but it's inconvenient.

For example, i use this construction for value updating and it's not really human readable.

text_list = Pickle(ListExtend(Unpickle(List<Utf8>, unwrap(NVL(text_list, Pickle(CAST(asList() as List<Utf8>))))),[unwrap(text)]))

Jepsen testing

Hi! I’ve been wondering if there are any plans to perform Jepsen testing of YDB? Or maybe someone has already done that and can share the results?

And thank you for open-sourcing YDB 🔥

Problem deploy cluster on prem

На одной машине все успешно работает, при сборе в кластер на 3ех не собирается
Ставлю на 3 машины Compute в Yandex Cloud
Если в host и node_id указываю IP как внутренние, как и внешние то такая ошибка

`2022-05-05T20:48:33.875257Z :INTERCONNECT NOTICE: Proxy [1:7094356886080632466:1] [node 2] ICP25 outgoing handshake failed, temporary: 1 explanation: outgoing handshake Peer# 10.129.0.10(10.129.0.10:19001) error from peer: ReceiverHostName# 10.129.0.10 mismatch, expected# localhost incoming: [0:0:0] held: no
2022-05-05T20:48:33.875276Z :INTERCONNECT NOTICE: Proxy [1:7094356886080632466:1] [node 2] ICP32 transit to hold-by-error state Explanation# outgoing handshake Peer# 10.129.0.10(10.129.0.10:19001) error from peer: ReceiverHostName# 10.129.0.10 mismatch, expected# localhost LastSessionDieTime# 1970-01-01T00:00:00.000000Z
2022-05-05T20:48:36.210727Z :INTERCONNECT NOTICE: Handshake [1:7094357040699463145:4107] [node 2] ICH03 handshake failed, explanation# incoming handshake Peer# localhost(::ffff:51.250.25.185) ReceiverHostName# 51.250.96.80 mismatch, expected# localhost
2022-05-05T20:48:40.421824Z :INTERCONNECT NOTICE: Handshake [1:7094357057879333233:8186] [node 3] ICH03 handshake failed, explanation# incoming handshake Peer# localhost(::ffff:51.250.28.84) ReceiverHostName# 51.250.96.80 mismatch, expected# localhost

Если указываю Внутренний FQDN (ydb1.ru-central1.internal,ydb2.ru-central1.internal,ydb3.ru-central1.internal)
Caught exception: ydb/core/driver_lib/cli_utils/cli_cmds_server.cpp:349: cannot detect node ID for ydb1:19001
`
Если указываю hostname (имя машин ydb1,ydb2,ydb3)
ICP32 transit to hold-by-error state Explanation# outgoing handshake Peer# ydb2() DNS resolve error: Could not contact DNS servers LastSessionDieTime# 1970-01-01T00:00:00.000000Z

telnet и nslookup успешно отрабатывают

#2006 Kikimr cluster or one of its subsystems is overloaded. (S_ERROR)

При тестировании из java-client судя про ошибки проблемы шарда
com.yandex.ydb.core.UnexpectedResultException: expected success result, code: OVERLOADED, issues: [#1060 Execution (S_ERROR)
#2006 Kikimr cluster or one of its subsystems is overloaded. (S_ERROR)
Error executing transaction (ProxyShardOverloaded): One or more of affected datashards are overloaded, try again (S_ERROR)
Data shard errors: [WRONG_SHARD_STATE] Rejecting data TxId 281482397066193 because datashard 72075186256745705: decided to reject due to given RejectProbability
(S_ERROR)]
at com.yandex.ydb.core.Result$Fail.newException(Result.java:217) ~[ydb-sdk-core-1.10.5.jar!/:na]
at com.yandex.ydb.core.Result$Fail.expect(Result.java:169) ~[ydb-sdk-core-1.10.5.jar!/:na]
at com.griphon.service.ArticleServices.saveArticleTransaction(ArticleServices.java:123) ~[classes!/:0.0.1-SNAPSHOT]

при вставке может отсекать чуть ли не половину запросов, возникает как при YDB в Cloud, так и при собственной установки. Такой троттлинг может и понятен, но необходимо либо понимать как ограничивать тогда нагрузку, либо в идеале, чтобы транзакция выполнялась но дольше.
Сценарий теста, вставка в пустую таблицу от 0 до N строк с уникальными не пересекающимися ключами

Can't use 'SET' in ALTER TABLE or 'WITH' in CREATE TABLE

Hi,
I've deployed a test cluster as per documentation, consisting of 3 nodes and 3 hdds each. 3 static nodes and 3 dynamic nodes to serve a test DB.
Now, according to manual I should be able to create a table like this:

CREATE TABLE my_table (
    id Uint64,
    title Utf8,
    PRIMARY KEY (id)
)
WITH (
    AUTO_PARTITIONING_BY_SIZE = ENABLED,
    AUTO_PARTITIONING_PARTITION_SIZE_MB = 512
);

To set AUTO_PARTITIONING values, or be able to modify existing table this way:

ALTER TABLE series SET (AUTO_PARTITIONING_BY_SIZE = DISABLED);

Neither of these two examples are working, when I try to run ALTER query I'm getting
'Unexpected token SET: cannot match to any predicted input.

When I'm trying to create the table, I'm getting:
Unexpected token 'WITH' : syntax error...

What's the correct syntax to use? The only way to change these parameters I found was to dump the table scheme, update parameters manually in the dump file and load them back into DB.

Thanks.

Feature request: ListPushBack/ListPushFront

Ability to add single element to list without wrapping it to a list:

SELECT ListPushBack(aaa, val) FROM (SELECT [1,2,3] AS aaa, 5 as val);

[1,2,3,5]

Current approach:

SELECT ListExtend(aaa, [val]) FROM (SELECT [1,2,3] AS aaa, 5 as val);

[1,2,3,5]


SELECT ListPushFront(aaa, val) FROM (SELECT [1,2,3] AS aaa, 5 as val);

[5,1,2,3]

Current approach:

SELECT ListExtend([val], aaa) FROM (SELECT [1,2,3] AS aaa, 5 as val);

[5,1,2,3]

It can save quite a bit of cpu on avoiding casting of value to a list.

How to force YDB to use table index, not to read all rows.

Hello. I have this table:

CREATE TABLE `order_items`
(
    `id` String,
    `order_id` String,
    `product_id` String,
    `quantity` Uint32,
    PRIMARY KEY (`id`)
);

with index idx_order_id_quantity -> order_id, quantity

I'd like to get quantity by order_id, so I use this query:

SELECT quantity
FROM order_items
WHERE order_id=""

But it scans all rows in table instead of using my index.

Explanation
{
    "plan": {
        "meta": {
            "type": "script",
            "version": "0.2"
        },
        "queries": [
            {
                "tables": [
                    {
                        "name": "/ru-central1/b1gnjh1pn5tbeqdb2lqt/etn4ge82u3bfv9qldrbk/order_items",
                        "reads": [
                            {
                                "columns": [
                                    "id",
                                    "order_id"
                                ],
                                "scan_by": [
                                    "id"
                                ],
                                "type": "FullScan"
                            }
                        ]
                    }
                ]
            }
        ]
    }
}

Исправление start.sh

если память не выделилась, например, не хватило места на девайсе, файл всё равно создан, и дальше если мы освободили память, мы должны ручками удалить ydb.data, иначе словим exception. Кажется логичным, чтобы если память не выделилась start.sh сам удалил ydb.data

Add ARM arch support

Hi!

As far as I see the only officially supported hardware architecture for now is x86-64.

We have a case when we want to build cloud-native application, which will be deployed to AWS EC2 instances with AWS Graviton CPUs (just because they are cheaper for the same performance in our workload). Not only AWS has ARM-based CPU - e.g. Alibaba Cloud also has their own ARM-based CPUs.

Is it possible to add ARM architecture support to your roadmap?

Thanks in advance!

Add CI

Hi!

Currently, no CI is avaiable (at least in public). And it is not easy to understand, does a PR breaks something or not.

Please add a CI to the repo. E.g. based on GitHub Actions CI or maybe integrate somehow reports from your internal CI system.

Thanks.

Support NOT_NULL constraint

Currently there is no support for NOT_NULL constraint for columns so all columns are nullable.
It would be great to have non-nullable columns.

An example from the documentation doesn't work

Hello!

I'm trying to install and test a ydb instance from the binary file according https://cloud.yandex.ru/docs/ydb/getting_started/self_hosted/ydb_local

  1. I downloaded and installed the ydb binary
  2. I downloaded and installed the ydb cli utility
  3. I satarted ydb daemon:
[oleg@oleg2 test]$ ./start.sh disk
Data file ydb.data not found, creating ...
Starting storage process...
Initializing storage ...
Registering database ...
Starting database process...

Database started. Connection options for YDB CLI:

-e grpc://localhost:2136 -d /Root/test
  1. When I trying to test it with ydb, I receive an error:
[oleg@oleg2 test]$ ydb -e grpc://localhost:2136 -d /Root/test scheme ls
Status: UNAVAILABLE
Issues: 
<main>: Error: Database nodes resolve failed with no certain result, code: 200400
<main>: Error: Endpoint list is empty for database /Root/test, cluster endpoint localhost:2136.
[oleg@oleg2 test]$ 

What did I make wrong? How to localize the problem?

Sometimes executeQuery throws query not found error

Hello. I'm using serverless YDB.
I prepared this query:

DECLARE $items AS List<Struct<id: String, order_id: String, product_id: String, quantity: Uint32>>;
DECLARE $order_id AS String;
DECLARE $user_id AS String;
        
UPSERT INTO order_items(id, order_id, product_id, quantity) 
SELECT id, order_id, product_id, quantity FROM AS_TABLE($items);
        
$table = (
  SELECT $order_id, false, false, $user_id, SUM(order_item.quantity * product.price)
  FROM AS_TABLE($items) AS order_item
  INNER JOIN products AS product
  ON (order_item.product_id==product.id)
);
        
UPSERT INTO orders(id, hasPaid, isCompleted, user_id, price)
SELECT column0, column1, column2, column3, column4 FROM $table;
        
SELECT column4 FROM $table;

and saved it in memory.

When I call it using session.executeQuery, the method sometimes throws:
NotFound [Error]: NotFound: [
{
"message": "Query not found: 23e79871-3b42b18f-8a7dde08-5099c5b8",
"severity": 1
}
]

Node.JS code:

const result = await this.client.withSessionRetry(async (session) => {
return await session.executeQuery(
await this.queries.insertOrder(session),
this.queries.createInsertOrderParams(params.products, context.userID, id)
)
})

insertOrder() {
if (this._insertOrder) {
return this._insertOrder
}

    // language=SQL
    this._insertOrder = await session.prepareQuery(`
    DECLARE $items AS List<Struct<id: String, order_id: String, product_id: String, quantity: Uint32>>;
    DECLARE $order_id AS String;
    DECLARE $user_id AS String;
    
    UPSERT INTO order_items(id, order_id, product_id, quantity) 
    SELECT id, order_id, product_id, quantity FROM AS_TABLE($items);
    
    $table = (
        SELECT $order_id, false, false, $user_id, SUM(order_item.quantity * product.price)
        FROM AS_TABLE($items) AS order_item
        INNER JOIN products AS product
        ON (order_item.product_id==product.id)
    );
    
    UPSERT INTO orders(id, hasPaid, isCompleted, user_id, price)
    SELECT column0, column1, column2, column3, column4 FROM $table;
    
    SELECT column4 FROM $table;
    `)
    return this._insertOrder

}

How to fix this?

JsonDocument modification

Не нашел в документации способа изменения JsonDocument без полной его перезаписи. Это возможно?

bug: using Like in secondary index don't use index

YDB serverless

Use this table:

CREATE TABLE my_table (
    k1 Uint32,
    k2 Uint64,
    desc Utf8,
    index my_tbl_ind global on(desc),
    PRIMARY KEY (k1,k2)
);
commit;
insert into my_table (k1,k2,desc)
values
(1u,1ul,'desc 11'u),
(2u,2ul,'desc 22'u),
(3u,3ul,'desc 33'u),
(4u,4ul,'desc 44'u),
(5u,5ul,'desc 55'u);
commit;

This query don't using secondary index (as described in docs) and generate FullScan

SELECT * FROM my_table view my_tbl_ind  WHERE desc like 'desc 1%'u

Unable to build ydb with debug symbols

I can build ydb with successfully in several hours.

cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_TOOLCHAIN_FILE=/home/oleg/work/ydb/Ydb/clang.toolchain /home/oleg/work/ydb/Ydb /home/oleg/work/ydb/Ydb
ninja

But when I change -DCMAKE_BUILD_TYPE=Release to -DCMAKE_BUILD_TYPE=Debug, I receive an error at the linkinng stage:

cmake -G Ninja -DCMAKE_BUILD_TYPE=Debug -DCMAKE_TOOLCHAIN_FILE=/home/oleg/work/ydb/Ydb/clang.toolchain /home/oleg/work/ydb/Ydb /home/oleg/work/ydb/Ydb
ninja
...
ld.lld: error: undefined symbol: NKikimr::NPDisk::YdbDefaultPDiskSequence

referenced by node_warden.h:52 (/home/oleg/work/ydb/Ydb/ydb/core/blobstorage/nodewarden/node_warden.h:52)
node_warden_pdisk.cpp.o:(NKikimr::TNodeWardenConfig::CreatePDiskKey() const) in archive ydb/core/blobstorage/nodewarden/libcore-blobstorage-nodewarden.a

ld.lld: error: undefined symbol: NKikimr::CreateBlobStorageGroupProxyMockActor()

referenced by node_warden_proxy.cpp:21 (/home/oleg/work/ydb/Ydb/ydb/core/blobstorage/nodewarden/node_warden_proxy.cpp:21)
node_warden_proxy.cpp.o:(NKikimr::NStorage::TNodeWarden::StartLocalProxy(unsigned int)) in archive ydb/core/blobstorage/nodewarden/libcore-blobstorage-nodewarden.a
clang-12: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.


The full listing is attached [linkerr.txt](https://github.com/ydb-platform/ydb/files/8764504/linkerr.txt)

Create FreeBSD port

FreeBSD is a widely used server OS, a port for clickhouse exists, it would be great to also have a port for ydb to facilitate adoption.

kubernetes create Storage errorinvalid type for tech.ydb.v1alpha1.Storage.spec.dataStore: got "map", expected "array", ValidationError(Storage.spec): missing required field "erasure" in tech.

error: error validating "STDIN": error validating data: [ValidationError(Storage.spec.dataStore): invalid type for tech.ydb.v1alpha1.Storage.spec.dataStore: got "map", expected "array", ValidationError(Storage.spec): missing required field "erasure" in tech.ydb.v1alpha1.Storage.spec]; if you choose to ignore these errors, turn validation off with --validate=false

[bug] At the first start TypeError: 'bytes' object is not callable

At the first start or start with clean volumes of ydb docker image TypeError: 'bytes' object is not callable happens.

Run:

docker run -it --rm  \
 --name ydb-local  -h 127.0.0.1  -p 2135:2135   -p 8765:8765   -p 2136:2136 \
 -e YDB_DEFAULT_LOG_LEVEL=NOTICE   -e GRPC_TLS_PORT=2135  \
-e GRPC_PORT=2136   -e MON_PORT=8765  \
 cr.yandex/yc/yandex-docker-local-ydb:latest

Logs:

Starting periodic command scheduler: cron.
Starting enhanced syslogd: rsyslogdrsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.
rsyslogd: activation of module imklog failed [v8.2102.0 try https://www.rsyslog.com/e/2145 ]
.
Traceback (most recent call last):
  File "contrib/tools/python3/src/Lib/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "contrib/tools/python3/src/Lib/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "ydb/public/tools/local_ydb/__main__.py", line 140, in <module>
    arguments.command(arguments)
  File "ydb/public/tools/lib/cmds/__init__.py", line 327, in deploy
    recipe.write_certificates_path(configuration.grpc_tls_ca())
TypeError: 'bytes' object is not callable

Image: cr.yandex/yc/yandex-docker-local-ydb@sha256:39915662b73c1e4bd4aa87a0405f50db8a8246c1da84fb29a66270fdc484288e

feature request: need a spark library to build data files offline and loaded directly into YDB

The use case is to serve read only large amount of data, which is a snapshot of machine learning results. The snapshot needs to update daily. Need a spark library to generate the files for YDB tables and indexes.

Ideally the files can be used by YDB directly, without any ingestion process. Preferably not CSV files because the CSV files need additional processing, impacting request latency.

No error message displayed in the Run Script with the built-in Web UI

I can execute some YQL script with ydb. If a statement fails, ydb returns an error message:

[oleg@oleg2 ~]$ ydb yql -s "CREATE TABLE tab2 (id Uint64, value1 json, primary key(id));"
[oleg@oleg2 ~]$ ydb yql -s "CREATE TABLE tab2 (id Uint64, value2 json, primary key(id));"
Status: GENERIC_ERROR
Issues: 
<main>: Error: Type annotation, code: 1030
    <main>:1:56: Error: At function: KiCreateTable!
        <main>:1:56: Error: Table name conflict: kikimr.[/Root/test/tab2] is used to reference multiple tables.

But if I try to execute the same YQL statement with the built-in Web UI, I receive the result Failed without any error message.

I'd like to see the same error message as I can see with ydb.

Entity Framework Core.

Добрый день.
Хотелось бы предложить вам добавить возможность взаимодействия с БД с помощью EF Core.

In English:
I'd like to propose adding capability of working with the database via EF Core.

bug: CrashLoopBackOff after Storage creation

Bug Report

YDB Kubernetes Operator version:

NAME        	NAMESPACE	REVISION	UPDATED                             	STATUS  	CHART                        	APP VERSION
prom        	default  	1       	2022-04-20 22:49:30.209714 +0300 MSK	deployed	kube-prometheus-stack-34.10.0	0.55.0     
ydb-operator	default  	1       	2022-04-20 22:49:57.786045 +0300 MSK	deployed	operator-0.4.5               	0.4.9      

Environment

Minikube (MacOS 10.15.7), GKE, GKE autopilot

Current behavior:

This happens after any cluster setup based on
https://ydb.tech/en/docs/getting_started/self_hosted/ydb_minikube
or
https://ydb.tech/en/docs/deploy/orchestrated/yc_managed_kubernetes

panic: assignment to entry in nil map after initial Storage setup

kubectl get storages.ydb.tech shows always PENDING
Screenshot 2022-04-21 at 11 58 53

ydb-operator get into CrashLoopBackOff
Screenshot 2022-04-21 at 11 59 10

Steps to reproduce:

follow this guide
https://ydb.tech/en/docs/getting_started/self_hosted/ydb_minikube

After storage create step

kubectl create -f - <<EOF
apiVersion: ydb.tech/v1alpha1
kind: Storage
metadata: 
  name: storage-sample
spec: 
  dataStore: 
    - 
      accessModes: 
        - ReadWriteOnce
      resources: 
        requests: 
          storage: 10Gi
      volumeMode: Block
  erasure: none
  nodes: 4
  version: "21.4.30"
EOF

Related code:

Full error code

panic: assignment to entry in nil map

goroutine 361 [running]:
github.com/ydb-platform/ydb-kubernetes-operator/internal/configuration.Build(0xc0005234c8, 0xc0005234c8)
	/workspace/internal/configuration/configuration.go:76 +0xea
github.com/ydb-platform/ydb-kubernetes-operator/internal/resources.(*StorageClusterBuilder).GetResourceBuilders(0xc0000b2498)
	/workspace/internal/resources/storage.go:59 +0xd9
github.com/ydb-platform/ydb-kubernetes-operator/internal/controllers/storage.(*StorageReconciler).handleResourcesSync(0xc0000b9810, {0x19c3838, 0xc000996e70}, 0xc0000b2498)
	/workspace/internal/controllers/storage/sync.go:168 +0x7a
github.com/ydb-platform/ydb-kubernetes-operator/internal/controllers/storage.(*StorageReconciler).Sync(0xc0000b94f0, {0x19c3838, 0xc000996e70}, 0xc000250879)
	/workspace/internal/controllers/storage/sync.go:69 +0xc5
github.com/ydb-platform/ydb-kubernetes-operator/internal/controllers/storage.(*StorageReconciler).Reconcile(0xc0000b9810, {0x19c3838, 0xc000996e70}, {{{0xc000250879, 0x1682520}, {0xc000250860, 0xc0002a7300}}})
	/workspace/internal/controllers/storage/controller.go:64 +0x1eb
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0xc00024d680, {0x19c3838, 0xc000996de0}, {{{0xc000250879, 0x1682520}, {0xc000250860, 0xc0004bc8c0}}})
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114 +0x222
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00024d680, {0x19c3790, 0xc0005bdd80}, {0x1608f40, 0xc00097a020})
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311 +0x2f2
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00024d680, {0x19c3790, 0xc0005bdd80})
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266 +0x205
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:223 +0x354

Screenshot 2022-04-21 at 11 58 00

Screenshot 2022-04-21 at 11 59 57

Feature Request: Alternative to ON DUPLICATE KEY UPDATE?

Hello.
I'd like to rewrite simple MySQL query:

INSERT INTO users(id, phone, sms_code, sms_code_expiration)
VALUES (?id, ?phone, ?sms_code, ?sms_code_expiration)
ON DUPLICATE KEY
UPDATE sms_code=?sms_code, sms_code_expiration=?sms_code_expiration;

I see that I need to execute 2 (or 1) requests to YDB (and use more Request Units) and I also need to handle exceptions on DB client.
Pseudocode:

try:
    """
    DECLARE $id AS String;
    DECLARE $sms_code AS Uint32;
    DECLARE $sms_code_expiration AS Datetime;
    DECLARE $phone AS Utf8;

    INSERT INTO users(id, phone, sms_code, sms_code_expiration)
    VALUES ($id, $phone, $sms_code, $sms_code_expiration);
    """
except PreconditionFailed:
    """
    DECLARE $phone AS Utf8;
    DECLARE $sms_code AS Uint32;
    DECLARE $sms_code_expiration AS Datetime;

    UPDATE users 
    SET sms_code=$sms_code, sms_code_expiration=$sms_code_expiration
    WHERE phone=$phone;
    """

And I also need to make phone a primary key (because there aren't UNIQUE fields).
But it's ok for me, if I want to change phone I can just remove field and add it again leaving previous ID.
Yes, there are any duplicate chance on ID field (it's used to references) but it's pretty small (due to UUID4).

zlib missing in contrib folder

I tried to follow the build procedure at my ubuntu 20.04 workstation but build failed due to absence of contrib/libs/zlib/zlib.h which is referenced here

Obviously there's no zlib in contrib folder

ninja build output
[7980/12163] Building CXX object library/cpp/actors/http/CMakeFiles/cpp-actors-http.dir/http_compress.cpp.o
FAILED: library/cpp/actors/http/CMakeFiles/cpp-actors-http.dir/http_compress.cpp.o 
/usr/bin/clang++-12 -DCARES_STATICLIB -DCATBOOST_OPENSOURCE=yes -I/data/ydbwork/ydb -I/data/ydbwork/build -I/data/ydbwork/ydb/contrib/libs/cxxsupp/libcxx/include -I/data/ydbwork/ydb/contrib/libs/double-conversion -I/data/ydbwork/ydb/contrib/libs/libc_compat/include/readpassphrase -I/data/ydbwork/ydb/contrib/libs/libc_compat/include/random -I/data/ydbwork/ydb/contrib/libs/protobuf/src -I/data/ydbwork/ydb/contrib/libs/brotli/include -I/data/ydbwork/ydb/contrib/libs/snappy/include -isystem /home/sasha/.conan/data/zlib/1.2.12/_/_/package/cf142e38e022cf2a99aadebfd8f37ead3e935ad7/include -isystem /home/sasha/.conan/data/openssl/1.1.1l/_/_/package/cf142e38e022cf2a99aadebfd8f37ead3e935ad7/include -isystem /home/sasha/.conan/data/libiconv/1.15/_/_/package/c4e0fbaccfc6985428b092ccc7aa4bcd74617546/include -isystem /home/sasha/.conan/data/c-ares/1.17.2/_/_/package/c2627225292f8d772a40f1bfef7180540cbf7f0d/include -m64   -fexceptions   -fno-common   -fuse-init-array   -fcolor-diagnostics   -faligned-allocation   -ffunction-sections   -fdata-sections   -Wall   -Wextra   -Wno-parentheses   -Wno-implicit-const-int-float-conversion   -Wno-unknown-warning-option   -D_THREAD_SAFE   -D_PTHREADS   -D_REENTRANT   -D_LARGEFILE_SOURCE   -D__STDC_CONSTANT_MACROS   -D__STDC_FORMAT_MACROS   -D_FILE_OFFSET_BITS=64   -D_GNU_SOURCE   -D_YNDX_LIBUNWIND_ENABLE_EXCEPTION_BACKTRACE   -D__LONG_LONG_SUPPORTED   -DSSE_ENABLED=1   -DSSE3_ENABLED=1   -DSSSE3_ENABLED=1   -DSSE41_ENABLED=1   -DSSE42_ENABLED=1   -DPOPCNT_ENABLED=1   -DCX16_ENABLED=1   -D_libunwind_   -DLIBCXX_BUILDING_LIBCXXRT   -msse2   -msse3   -mssse3   -msse4.1   -msse4.2   -mpopcnt   -mcx16   -Woverloaded-virtual   -Wimport-preprocessor-directive-pedantic   -Wno-undefined-var-template   -Wno-return-std-move   -Wno-address-of-packed-member   -Wno-defaulted-function-deleted   -Wno-pessimizing-move   -Wno-range-loop-construct   -Wno-deprecated-anon-enum-enum-conversion   -Wno-deprecated-enum-enum-conversion   -Wno-deprecated-enum-float-conversion   -Wno-ambiguous-reversed-operator   -Wno-deprecated-volatile    -O3 -DNDEBUG -nostdinc++ -DLIBCXX_BUILDING_LIBCXXRT -D_LIBCPP_ENABLE_CXX17_REMOVED_FEATURES -D_libunwind_ -std=c++20 -MD -MT library/cpp/actors/http/CMakeFiles/cpp-actors-http.dir/http_compress.cpp.o -MF library/cpp/actors/http/CMakeFiles/cpp-actors-http.dir/http_compress.cpp.o.d -o library/cpp/actors/http/CMakeFiles/cpp-actors-http.dir/http_compress.cpp.o -c /data/ydbwork/ydb/library/cpp/actors/http/http_compress.cpp
/data/ydbwork/ydb/library/cpp/actors/http/http_compress.cpp:3:10: fatal error: 'contrib/libs/zlib/zlib.h' file not found
#include <contrib/libs/zlib/zlib.h>

Minus zero values

Signed zero values should be additionally checked:

  1. Minus zero values are not supported for float datatype
create table t1_1 (id Uint64, a Float, primary key(id));
insert into t1_2(id,a) values (1,-0.0);
Status: GENERIC_ERROR
Issues: 
<main>: Error: Type annotation, code: 1030
    <main>:1:13: Error: At function: KiWriteTable!
        <main>:1:21: Error: Failed to convert type: Struct<'a':Double,'id':Int32> to Struct<'a':Float?,'id':Uint64?>
        <main>:1:21: Error: Failed to convert input columns types to scheme types, code: 2031

create table t1(id Uint64, a Double, primary key(id));
insert into t1(id,a) values (1,0),(2,-0.0);
select * from t1;
┌────┬────┐
| a  | id |
├────┼────┤
| 0  | 1  |
├────┼────┤
| -0 | 2  |
└────┴────┘
  1. Division by zero returns error when using YDB CLI, +inf/-inf values are not supported
select 1/-0.0;
library/cpp/json/writer/json.cpp:259: JSON writer: invalid float value: -inf

select 1/0.0;
library/cpp/json/writer/json.cpp:259: JSON writer: invalid float value: inf

select 1/0;
┌─────────┐
| column0 |
├─────────┤
| null    |
└─────────┘
  1. Result of min/max functions might be inconsistent (should be +0)
create table t1(id Uint64, a Double, primary key(id));
insert into t1(id,a) values (1,0),(2,-0.0);
select min(a) from t1;
┌─────────┐
| column0 |
├─────────┤
| -0      |
└─────────┘

select max(a) from t1;
┌─────────┐
| column0 |
├─────────┤
| -0      |
└─────────┘

select min_of(0.0,-0.0);'
┌─────────┐
| column0 |
├─────────┤
| 0       |
└─────────┘

select max_of(0.0,-0.0);'
┌─────────┐
| column0 |
├─────────┤
| 0       |
└─────────┘

Добавление рекурсивных запросов

Планируете ли вы добавить рекурсивные запросы? Насколько я понял, на данный момент yql не поддерживает рекурсивные запросы даже через ACTION, лямбда функции или subquerries.

Where is a ydb-related discussion forum?

A lot of others open source projects on Github have Discussion pages.

Is it possible to enable the Discussion pages for this project in the repository settings too?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.