Comments (12)
Yes it failed, yes I got email, yes our alerts went off, yes we are continuing to investigate.
from planet-dump-ng.
As you can see if you look at #25 which is the actual ticket where we're currently dealing with this...
from planet-dump-ng.
We had one good run but it looks like last week locked up again :-(
from planet-dump-ng.
Weird that this is happening so often now. 🤔 I wonder if something changed?
Thanks very much for the thread backtraces, they were very helpful. It looks like one of the output writer threads has died at some point while outputting the relations and this isn't handled properly.
It looks like the incron script that runs planet-dump-ng
should send me any output, but the last time it sent me anything was July 2020... So either the output isn't being written to disk, or the script isn't mailing it. If it happens again, please look if there's a /tmp/planetdump.log.XXX
file. If it's non-empty, then it might give a better clue what's happening.
from planet-dump-ng.
Unfortunately the wrapper script deletes the log after it has mailed it so it's no longer there.
I assume it must have been empty though, or it should have been emailed to you as you say and I see no signs of that.
from planet-dump-ng.
Yeah, I thought the same, but the log file should never be empty. Until July 2020, I was getting weekly confirms of the form:
Writing changesets...
Writing nodes...
Writing ways...
Writing relations...
Donereal 3530m46.156s
user 31670m53.427s
sys 972m26.327s
But then I got a few errors around the 14-16th July, and nothing since. (Clearly I wasn't doing an awesome job noticing these emails, or I'd have realised they'd stopped coming before now.)
I think I made a change that could help (assuming this is how the thread exits...). Please could you try version 1.2.2 and see if that helps? https://github.com/zerebubuth/planet-dump-ng/releases/tag/v1.2.2
from planet-dump-ng.
I've deployed that, and I think I've figured out the email problem and fixed it. I think you've broken something though because I get a stream of errors now if I try and start the dump:
pg_restore: error: one of -d/--dbname and -f/--file must be specified
I assume this is something to do with e2d9c70?
from planet-dump-ng.
Ooops, fail. Looks like the machine I was testing on had a truly ancient version PostgreSQL (9!). I reverted that commit and pushed a new version, v.1.2.3.
https://github.com/zerebubuth/planet-dump-ng/releases/tag/v1.2.3
from planet-dump-ng.
I ran a quick test to validate that a simulated runtime_error in pbf_writer::relations
is handled ok now:
10000 = very first relation id in dump:
if (r.id == 10000) { BOOST_THROW_EXCEPTION(std::runtime_error("bla")); }
Result on 3e48263:
Writing changesets...
Writing nodes...
Writing ways...
Writing relations...
EXCEPTION: writer_thread(1): pbf_writer.cpp(613): Throw in function virtual void pbf_writer::relations(const std::vector<relation>&, const std::vector<relation_member>&, const std::vector<old_tag>&)
Dynamic exception type: boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<std::runtime_error> >
std::exception::what: bla
<stuck>
Result on b190303:
Finishing thread with 24 bytes
Writing changesets...
Writing nodes...
Writing ways...
Writing relations...
EXCEPTION: writer_thread(1): pbf_writer.cpp(613): Throw in function virtual void pbf_writer::relations(const std::vector<relation>&, const std::vector<relation_member>&, const std::vector<old_tag>&)
Dynamic exception type: boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<std::runtime_error> >
std::exception::what: bla
. Trying to continue...
Done
echo $? returns 0 in case of an issue. This may not be ideal for monitoring...
from planet-dump-ng.
I was wondering if the issue was reproducible, i.e. when processing the same db dump twice, it would show exactly the same behavior as with the previous run.
Given that this happened in relations (which tend to be much larger than ways and nodes), I could imagine that a certain combination of large relations might trigger some rare bug in the pbf writer code, e.g. due to lack of space in a pbf block.
from planet-dump-ng.
So, do the mails with errors arrive now at least?
It seems planet dump has failed again? Today is 2022-09-20, and last one is planet-220905.osm.pbf (created on 2022-09-10).
from planet-dump-ng.
Which is actually a totally separate issue - it's a crash rather than a lockup.
from planet-dump-ng.
Related Issues (20)
- PBF doesn't use DenseNodes HOT 3
- Changeset dump file contains no comments after February 23rd HOT 3
- Reference failures in planet file HOT 1
- Check references in current output
- Update github description HOT 1
- Set osmosis_replication_base_url HOT 1
- Build instructions fail on `make` HOT 5
- PBF History file contains too large blobs HOT 1
- handle directory-based input for parallel pgdump HOT 2
- Invalid xml data in planet osm HOT 15
- Exception dumping recent planets HOT 10
- .
- OSM AWS Athena Tables Not Updated HOT 1
- No separate changesets file. HOT 3
- v1.2.4 fails to compile on Debian 11 HOT 4
- All member types of relation objects are from type="relation" HOT 1
- Changeset discussion dumps HOT 1
- Planet state file HOT 4
- Add comments_count to changesets dump HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from planet-dump-ng.