Comments (4)
rafiss
commented on August 16, 2024
A pebble invariant was violated. I am not sure where to start with debugging this -- can a Storage engineer help?
=== RUN TestLeaseManagerDrain
test_log_scope.go:170: test logs captured to: outputs.zip/logTestLeaseManagerDrain3734620821
test_log_scope.go:81: use -show-logs to present logs inline
test_server_shim.go:157: automatically injected a shared process virtual cluster under test; see comment at top of test_server_shim.go for details.
F240404 10:20:19.909945 37965 3@pebble/version_set.go:617 [n2,s2,pebble] 1 MANIFEST apply failed: pebble: internal error: L6 files 000687 and 000682 have overlapping ranges: [/Table/13/1/2024-04-04T10:20:18.264898Z/957335530272030721/0/1712226018.369845783,0#0,SET-/Table/39/2/"\x80"/"\x01\x01\x80\xf2g\xa1-\xa3VK\x93\xac\xd1V\x04\xbe\xee3m"/0/1712225993.873559494,0#0,SET] vs [/Table/20/0,0#19999,RANGEKEYDEL-/Table/21/0,0#inf,RANGEDEL]
F240404 10:20:19.909945 37965 3@pebble/version_set.go:617 [n2,s2,pebble] 1 !goroutine 37965 [running]:
F240404 10:20:19.909945 37965 3@pebble/version_set.go:617 [n2,s2,pebble] 1 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0, 0x5?, 0x0})
F240404 10:20:19.909945 37965 3@pebble/version_set.go:617 [n2,s2,pebble] 1 ! github.com/cockroachdb/cockroach/pkg/util/allstacks/allstacks.go:32 +0x8d
F240404 10:20:19.909945 37965 3@pebble/version_set.go:617 [n2,s2,pebble] 1 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F240404 10:20:19.909945 37965 3@pebble/version_set.go:617 [n2,s2,pebble] 1 ! github.com/cockroachdb/cockroach/pkg/util/allstacks/allstacks.go:19
F240404 10:20:19.909945 37965 3@pebble/version_set.go:617 [n2,s2,pebble] 1 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}}, ...})
F240404 10:20:19.909945 37965 3@pebble/version_set.go:617 [n2,s2,pebble] 1 ! github.com/cockroachdb/cockroach/pkg/util/log/clog.go:282 +0xe5
F240404 10:20:19.909945 37965 3@pebble/version_set.go:617 [n2,s2,pebble] 1 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0xb1056d8, 0xc0011d30b0}, 0x3, 0x4, 0x3, 0x0, {0x82b0a85, 0x2}, {0xc01e241710, 0x1, ...})
F240404 10:20:19.909945 37965 3@pebble/version_set.go:617 [n2,s2,pebble] 1 ! github.com/cockroachdb/cockroach/pkg/util/log/channels.go:109 +0x7a5
F240404 10:20:19.909945 37965 3@pebble/version_set.go:617 [n2,s2,pebble] 1 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepth(...)
F240404 10:20:19.909945 37965 3@pebble/version_set.go:617 [n2,s2,pebble] 1 ! github.com/cockroachdb/cockroach/pkg/util/log/channels.go:39
F240404 10:20:19.909945 37965 3@pebble/version_set.go:617 [n2,s2,pebble] 1 !github.com/cockroachdb/cockroach/pkg/util/log.loggerStorage.FatalfDepth(...)
F240404 10:20:19.909945 37965 3@pebble/version_set.go:617 [n2,s2,pebble] 1 ! github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/util/log/log_channels_generated.go:2223
F240404 10:20:19.909945 37965 3@pebble/version_set.go:617 [n2,s2,pebble] 1 !github.com/cockroachdb/cockroach/pkg/storage.pebbleLogger.Fatalf({{0xb1056d8?, 0xc0011d30b0?}, 0xc01e241710?}, {0x82b0a85, 0x2}, {0xc01e241710, 0x1, 0x1})
F240404 10:20:19.909945 37965 3@pebble/version_set.go:617 [n2,s2,pebble] 1 ! github.com/cockroachdb/cockroach/pkg/storage/pebble.go:846 +0xc5
F240404 10:20:19.909945 37965 3@pebble/version_set.go:617 [n2,s2,pebble] 1 !github.com/cockroachdb/pebble.(*versionSet).logAndApply(0xc00baf9908, 0x24e, 0xc018f67000, 0xc01afd8d50, 0x0, 0xc00d37b7c0)
F240404 10:20:19.909945 37965 3@pebble/version_set.go:617 [n2,s2,pebble] 1 ! github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/version_set.go:617 +0xd38
F240404 10:20:19.909945 37965 3@pebble/version_set.go:617 [n2,s2,pebble] 1 !github.com/cockroachdb/pebble.(*DB).compact1.func2(0xc00758e008, 0xc0169c0308, 0x24e, 0xc018f67000)
F240404 10:20:19.909945 37965 3@pebble/version_set.go:617 [n2,s2,pebble] 1 ! github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:2461 +0x215
F240404 10:20:19.909945 37965 3@pebble/version_set.go:617 [n2,s2,pebble] 1 !github.com/cockroachdb/pebble.(*DB).compact1(0xc00758e008, 0xc0169c0308, 0x0)
F240404 10:20:19.909945 37965 3@pebble/version_set.go:617 [n2,s2,pebble] 1 ! github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:2464 +0x40b
F240404 10:20:19.909945 37965 3@pebble/version_set.go:617 [n2,s2,pebble] 1 !github.com/cockroachdb/pebble.(*DB).compact.func1({0xb1056d8?, 0xc0128d3050?})
F240404 10:20:19.909945 37965 3@pebble/version_set.go:617 [n2,s2,pebble] 1 ! github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:2402 +0xf1
F240404 10:20:19.909945 37965 3@pebble/version_set.go:617 [n2,s2,pebble] 1 !runtime/pprof.Do({0xb105508, 0xfbb4a80}, {{0xc000e28260?, 0x2006bb7?, 0xc019691890?}}, 0xc016ad8f78)
F240404 10:20:19.909945 37965 3@pebble/version_set.go:617 [n2,s2,pebble] 1 ! GOROOT/src/runtime/pprof/runtime.go:51 +0x118
F240404 10:20:19.909945 37965 3@pebble/version_set.go:617 [n2,s2,pebble] 1 !github.com/cockroachdb/pebble.(*DB).compact(0xc00758e008, 0xc0169c0308, 0x0)
F240404 10:20:19.909945 37965 3@pebble/version_set.go:617 [n2,s2,pebble] 1 ! github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:2399 +0xc8
F240404 10:20:19.909945 37965 3@pebble/version_set.go:617 [n2,s2,pebble] 1 !created by github.com/cockroachdb/pebble.(*DB).tryScheduleAutoCompaction in goroutine 37961
F240404 10:20:19.909945 37965 3@pebble/version_set.go:617 [n2,s2,pebble] 1 ! github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:2120 +0x776
from cockroach.
itsbilal
commented on August 16, 2024
I'll need to dive deeper into the logs to confirm but this is possibly just a SQL repro of cockroachdb/pebble#3491 .
from cockroach.
itsbilal
commented on August 16, 2024
We actually dont' have Pebble logs from this, only the panic. So it's hard to conclusively say this is cockroachdb/pebble#3491 but it should almost certainly be just that.
from cockroach.
itsbilal
commented on August 16, 2024
Fixed in both master and 24.1.
from cockroach.
Related Issues (20)
- pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed HOT 2
- : failed
- roachtest: asyncpg failed
- roachtest: weekly/tpcc/headroom failed HOT 3
- pkg/sql/logictest/tests/local-legacy-schema-changer/local-legacy-schema-changer_test: TestLogic_fk failed HOT 1
- roachtest: unoptimized-query-oracle/disable-rules=all/rand-tables failed HOT 1
- streamproducer: infinite retry on unrecoverable error HOT 2
- pkg/sql/logictest/tests/fakedist/fakedist_test: TestLogic_alter_table failed HOT 1
- ccl/multiregionccl: TestMrSystemDatabase failed HOT 4
- kvserver: lease shedding counter-productive when multiple stores are similarly IO-overloaded HOT 3
- pkg/sql/logictest/tests/cockroach-go-testserver-23.2/cockroach-go-testserver-23_2_test: TestLogic_mixed_version_sequence_per_node_cache failed HOT 1
- o11y: unavailable ranges in console is coloured green HOT 2
- kv: closed timestamp regression on drt-large HOT 5
- pkg/sql/logictest/tests/cockroach-go-testserver-23.1/cockroach-go-testserver-23_1_test: TestLogic_upgrade_skip_version failed HOT 1
- release-23.1: MODIFYSQLCLUSTERSETTING cannot be used in mixed version clusters HOT 1
- storage: benchmark zstd vs snappy HOT 1
- acceptance: TestDockerCLI_test_distinguished_name_validation failed HOT 29
- server/debug: TestAdminDebugPprof failed HOT 2
- ARM64 support in GCE is limited: one US region (`us-central1`), no local SSD HOT 2
- acceptance: TestDockerCLI_test_distinguished_name_validation failed HOT 14
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cockroach.