blockworks-foundation / bankingstageerrorstrackingsidecar Goto Github PK
View Code? Open in Web Editor NEWThis is a side car which will track banking stage errors from validator over geyser.
This is a side car which will track banking stage errors from validator over geyser.
(not a problem - just an observation)
acc_id is growing very fast and produces gaps:
select count(*),min(acc_id),max(acc_id),count(distinct account_key) from banking_stage_results_2.accounts;
select *, acc_id-previd FROM (select acc_id, LAG(acc_id) OVER (ORDER BY acc_id) previd from banking_stage_results_2.accounts) as list where acc_id-previd>1;
The gaps are by design. serial with INSERT ON CONFLICT
burns generated ids when not inserting something:
https://stackoverflow.com/questions/37204749/serial-in-postgres-is-being-increased-even-though-i-added-on-conflict-do-nothing
Database log shows this -- while the sidecar gets stuck:
2024-02-08 14:15:32.131 CET [10326] bankingstage_sidecar@banking_stage ERROR: deadlock detected
2024-02-08 14:15:32.131 CET [10326] bankingstage_sidecar@banking_stage DETAIL: Process 10326 waits for Share
Lock on transaction 2245819; blocked by process 10328.
Process 10328 waits for ShareLock on transaction 2245818; blocked by process 10326.
Process 10326:
INSERT INTO banking_stage_results_2.accounts(account_key) SELECT key from temp_table_2_22322
ON CONFLICT DO NOTHING
Process 10328:
INSERT INTO banking_stage_results_2.accounts(account_key) SELECT key from temp_table_4_22308
ON CONFLICT DO NOTHING
same happens for most other tables too !
Add redundant table (next to accounts_map_transaction) which stores the most recent 1000 transactions per account.
array_dedup_append('{8,3,2,1}', '{5,3}', 4) -> {2,1,5,3}
SELECT
array_dedup_append('{8,3,2,1}', '{5,3}', 4),-- {2,1,5,3}
array_dedup_append('{8,3,2,1}', '{5,3}', 10000) -- {8,2,1,5,3}
CREATE TABLE banking_stage_results_2.accounts_map_transaction_latest_parted(
acc_id BIGINT PRIMARY KEY,
tx_ids BIGINT[]
) PARTITION BY HASH (acc_id);
DO $$
DECLARE part_no integer;
DECLARE num_parts integer := 4;
BEGIN
for part_no in 0..num_parts-1 loop
EXECUTE format(
'
CREATE TABLE banking_stage_results_2.accounts_map_transaction_latest_part%s
PARTITION OF banking_stage_results_2.accounts_map_transaction_latest_parted
FOR VALUES WITH (MODULUS %s, REMAINDER %s)
WITH (FILLFACTOR=80)
', part_no, num_parts, part_no);
end loop;
END; $$
INSERT INTO banking_stage_results_2.accounts_map_transaction_latest_parted
SELECT * FROM banking_stage_results_2.accounts_map_transaction_latest;
SELECT count(*) FROM banking_stage_results_2.accounts_map_transaction_latest;
ALTER TABLE banking_stage_results_2.accounts_map_transaction_latest RENAME TO accounts_map_transaction_latest_unparted_old;
ALTER TABLE banking_stage_results_2.accounts_map_transaction_latest_parted RENAME TO accounts_map_transaction_latest;
SELECT count(*) FROM banking_stage_results_2.accounts_map_transaction_latest;
DROP TABLE banking_stage_results_2.accounts_map_transaction_latest_unparted_old;
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.