Comments (16)
yep, re: 6)
as I expected, your shell is reading the whole stream into memory because you're running it in a sub shell using $()
.
the same thing happens if you take all the data away and just do:
/dev/null < $(cat /dev/urandom)
you'll see your system memory filling up until eventually your shell runs out of heap
the same doesn't apply to:
cat /dev/urandom > /dev/null
or
cat /dev/urandom | cat > /dev/null
from interpolation.
hi @kylebarron it sounds like there is a problem with the .0sv
file, could you please post the file size and first 10 lines?
the pbf streets
command wasn't originally written to support such large extracts, I just added missinglink/pbf#15 this morning which should make it possible to cut your us-midwest
extract on a laptop.
I'll also change that fatal error to a warning, if it's only 1 or 2 streets that are missing their names then we can probably just skip over those and warn the user they are malformed, rather than fatally erroring.
from interpolation.
Thanks @missinglink; sorry, which is the .0sv
file? I don't seem to have any files with that extension, or is that the same as the .osm.pbf
file?
from interpolation.
Sorry, I mean the .polylines
file A.K.A. .0sv
(nullbyte separated file) which has a nicer ring to it :)
from interpolation.
Closing issue, still interested to hear what happened. I suspect the .polylines
file is empty or contains invalid lines.
You can also run the import again now with the latest code from master
and it'll spit out the offending polyline for debugging purposes.
from interpolation.
I read my initial bug report again, and I should've specified that the error arises from a specific line around ~128000. Above I cut out some of the output to be concise; the full output to console is below. The file size of us-midwest-latest.polylines
is 121MB. With your PR changing the response from an error to a warning, this should proceed fine.
> ./interpolate polyline street.db < us-midwest-latest.polylines
0 0/sec
1000 1000/sec
3000 2000/sec
4000 1000/sec
10000 6000/sec
14000 4000/sec
16000 2000/sec
20000 4000/sec
24000 4000/sec
28000 4000/sec
32000 4000/sec
32000 0/sec
35000 3000/sec
38000 3000/sec
41000 3000/sec
45000 4000/sec
48000 3000/sec
48000 0/sec
50000 2000/sec
54000 4000/sec
57000 3000/sec
60000 3000/sec
64000 4000/sec
64000 0/sec
66000 2000/sec
69000 3000/sec
73000 4000/sec
76000 3000/sec
79000 3000/sec
80000 1000/sec
82000 2000/sec
85000 3000/sec
87000 2000/sec
90000 3000/sec
93000 3000/sec
96000 3000/sec
96000 0/sec
98000 2000/sec
101000 3000/sec
103000 2000/sec
106000 3000/sec
109000 3000/sec
112000 3000/sec
112000 0/sec
113000 1000/sec
116000 3000/sec
118000 2000/sec
121000 3000/sec
123000 2000/sec
125000 2000/sec
127000 2000/sec
/disk/agedisk1/medicare.work/doyle-DUA18266/barronk/npi-geocode/lib/interpolation/lib/Street.js:24
if( !Array.isArray( names ) || !names.length ){ throw new Error( 'invalid names array' ); }
^
Error: invalid names array
at Street.setNames (/disk/agedisk1/medicare.work/doyle-DUA18266/barronk/npi-geocode/lib/interpolation/lib/Street.js:24:57)
at DestroyableTransform._transform (/disk/agedisk1/medicare.work/doyle-DUA18266/barronk/npi-geocode/lib/interpolation/stream/street/augment.js:25:12)
at DestroyableTransform.Transform._read (/disk/agedisk1/medicare.work/doyle-DUA18266/barronk/npi-geocode/lib/interpolation/node_modules/through2/node_modules/readable-stream/lib/_stream_transform.js:184:10)
at DestroyableTransform.Transform._write (/disk/agedisk1/medicare.work/doyle-DUA18266/barronk/npi-geocode/lib/interpolation/node_modules/through2/node_modules/readable-stream/lib/_stream_transform.js:172:83)
at doWrite (/disk/agedisk1/medicare.work/doyle-DUA18266/barronk/npi-geocode/lib/interpolation/node_modules/through2/node_modules/readable-stream/lib/_stream_writable.js:418:64)
at writeOrBuffer (/disk/agedisk1/medicare.work/doyle-DUA18266/barronk/npi-geocode/lib/interpolation/node_modules/through2/node_modules/readable-stream/lib/_stream_writable.js:407:5)
at DestroyableTransform.Writable.write (/disk/agedisk1/medicare.work/doyle-DUA18266/barronk/npi-geocode/lib/interpolation/node_modules/through2/node_modules/readable-stream/lib/_stream_writable.js:334:11)
at DestroyableTransform.ondata (/disk/agedisk1/medicare.work/doyle-DUA18266/barronk/npi-geocode/lib/interpolation/node_modules/through2/node_modules/readable-stream/lib/_stream_readable.js:619:20)
at emitOne (events.js:116:13)
at DestroyableTransform.emit (events.js:211:7)
> head us-midwest-latest.polylines
cmeuoAnmljgD?udAcBi{IaBisI�West State Park Road
in_eqAv`gviD}LnCiM~@ke@v@�Sellery Street
mzknmA|eifvDpSJ�Bison Circle
ufqhnAprk`oD|FaKvCqFhG}R~Ho]dGsd@x@oObAgX^}u@RsgFZuyH{AazBQiVgAmoEy@of@Yk[a@aSc@aS}@oz@Mua@_@ohAFojAz@ifAn@wlBEmaBy@ieBcBmzAoEkbCEkk@Fo|@A}b@_@i_AKqfA?w`AOk\u@avA`@cdCi@{aCUmaDCoj@JyvCy@cnEEeg@ZicCu@{l@}Cky@e@gLgJubAkVu}ActCkjOkJ{m@wGkq@kC_\gBk`@s@c[?oKCen@VgbC@kUBuoA?kjBBky@nC_uCRcTAwc@AcwAEqmBEkyEoAcaAu@aTkCq[oH_f@iRg|@mTkk@{Qu_@q\gi@{MmSgXwa@a@q@cTo_@iHcOiAeDiKo[oC{IcJs^cEeVaDuT_C{YyA}b@M{a@vAk~FGmpAz@etA@iJfAmrAn@kxAAwPh@ic@x@umA~@gzCj@gx@tBufEYsnBa@mp@Qo\cAktBaBexCeCk|DwB{dBqB{wEY{sESspABiw@@o[MyZsPsp@cfBskIucAo|EeF}U{RifA{_@ioBs]wlBcb@szBq[q`B}TonAyUkpAy[gaBkN_w@qG_^eFqa@eCye@_@e\Ewx@C_KI}k@EihCCsc@}@mmAiAkoBMm`@Gwv@CasCPgiHN_o@^miALsc@z@i_CE}aAoE}fFqAgwA?iU~@gS~@kNl@uG`@cEfBkLlCeQxCyMxE_PdlAcfEhBsGbLud@`FuZhCmX|AwSr@iSJySEiUcEyfC_PixJeA_YyCa[qD{ViE{UqEeRsFuRaFgPyDwMoQ_n@yHgVyK_^kEmRiCeNqC{PkBsOeBkSsA{Ri@iVu@u}GAeYcAanBg@ctB{BywGqA_vDG{T[kO{BipAKwPE_HSmd@?a}@g@}mC@ym@y@s|Ai@gvD?ys@J_Wg@eFaA{CaBqAgB{@uFSo\e@oq@]eSQ{EmAmEmCwCsCsBeEkBmFcCqKcIka@{Oez@gNsw@uB}NoDg`@qFwv@eCkZqFqi@oAeLg@yLg@ax@W_qBA_aEU_jDAw`BGga@UqbCi@aiGQwTAeOv@KtJcCrpDe@rHSlG}AlC_CxCyDpB{G`@wLAcnDVg_AAoUAeDGiRMe}BJ{cCSaf@I_jALqr@U_z@?i`ATuHj@cE|@qCbBiD~C_CnFuAp_@Wj{@Bvr@o@`ZAlOArGKrC_BlCaEl@uINm`@k@uoFSgyAMmjCTioAEwYKyr@CyNCyWI_l@Gua@Gwe@EaTKir@CiYcBctEw@auCn@avBEk]kCsbJi@i_FF_fEEq[i@gzDa@}xCYifDEcq@i@af@gAaLiE}WqJ}]}JgU_RkW{FgFsEeE_[sQci@aX{FwD}YoRyHeIaMqSoIuQgJwZ{Fu_@eBoVe@k^Mar@HglAKioAc@mv@g@_YuA{MyBmNuBcLk@uCyHsV}HcQ{M}QyMyPsu@yy@iq@mv@}MsPcLsWeC_GkJga@gDu]yAyZgDmiAuCmx@uB{Z_Fc]iEkVqAiFkQwr@iO{k@{C{LoAcFgSmw@_\}nAaSiw@yDqOoPkp@oc@_cBqLsg@qGgd@eD{YqE{y@_Aka@eFumCgDw}Aa@_OM}IIwE]eSqBi~@gBw{@GyRDeTx@}UrBaXdEg\VkBfHo\|Jq_@|Lk[pK_SxPmXzQ_TpzCmvCtWwXbFmFj^c]fm@mk@ba@sa@xNkQtOsVnIuPbEgI|H_UrGsVtFsVbEyWnEm]vT{qB~Iyw@`Hgp@pBeTpAeNfAwTD_HG{PQaQOsGsAaSi@aFoMmnAsBaRkJa}@iDo]_C_ZiAgScAqRaHcjAeL{qByFudAGcA}KokB_Cob@cTcrD}H}uA{Gy_AiEeg@sGsp@cScpAsWm|A_m@clDuVexAuKkn@sGy]aK}j@_Kyk@gLar@aDuQgOqz@cCsMeB}MeEg`@mBwYmAyTg@mScAcm@YmQ_D}nBeCgeByCa}Bm@kXGa]\iX`AoTfBoTtBaOpCcQnGi[xIoYpU_r@rJ{YnN{a@xN_b@fPqg@pv@{}B|EiObUuq@f_@aiAnZ{~@nIe\`FaZrEi[|Dkb@`Ckf@Xc\wBspCO_Rs@az@y@_`@_BkTiC{X_G}e@eFqW{EqSuI{Y{JkWuHgR}MyXmUs_@u_Am~AqKeUuJmU{KmYeUqt@wKof@sG__@sJmq@aZqyCcQmsAwJae@sBaJwO}i@yVyv@muAcdEaPyc@gZcy@sGeQunA}cD�County Highway F52
guxknAnrk}qDdEnApDrBfCbCbCnDjBjExSnn@l@jBrCvKhBdKpAbLt@`NdAb_@hBnn@n@pNn@zIjAvKjKp{@l@jDz@fDnBfF~BbEvn@`~@fC|DvFjKdE|JpCrHbM|\jDdMrEjP|FnT�Wilden Drive
kfvknAz`i}qDwHaI_IaKeGoJa@s@qFkKiF}LsGkPqGcNkFuJ�Wilden Drive
khsknAhjy{qDsEgOgBgJsAmJ{@uJUiEKsEEiE?oDFeERqETaENkGnAkK�Wilden Drive
{xrknAl{ywqDt@uD`@gF?cGa@yEu@}DwAeE_BsCo]mg@_BoC_BcEa@yAc@{Bm@aFoCu`@i@iEkAgEmBsDqB}BmCiB{CaAkDU�Wilden Drive
i`vknAhcfwqDyDpIiDvKgAtFoAdOIfG?vL?xB?jBArM?bITjHn@jHbA`H|AvGnBfGtCxG|^fk@|JvOlD|Fn`@vo@�Wilden Drive
{owknA~hs|qDrDrCvG~EvD~BlC~@rEhAnKxC|Dx@jCt@hA~@x@tBv@dA|@~@v@J~@?d@[l@cAJaAD{@Y_Bu@mAaAUcA?qA\wBh@w@P�Wilden Drive
from interpolation.
I updated from master and it did spit out the warning:
street has no valid names, check your 0sv file:
oee}mAthsagDN|Jd@hDhBpB|CfAvHxAnJxChHdDzE|DxF|G|FhJlGlPhEbT~@bJb@`LB~Fb@pHdA`IhLd]
It seems to be progressing fine otherwise.
To clarify since I couldn't find this in the documentation, is there a way to append to the sqlite db, or should I be doing
./interpolate polyline street.db < us-midwest-latest.polylines \
us-northeast-latest.polylines \
us-south-latest.polylines \
us-west-latest.polylines
to get a full US interpolation dataset.
from interpolation.
ok cool, looks like the commit I pushed is now allowing the importer to skip over the offending line(s).
could you please copy->paste the output of this grep command so I can get a copy of the exact line that's causing it?
grep -aC5 'oee}mAthsagDN' *.polylines
I don't think the syntax you posted is doing what you expect (try https://explainshell.com).
My understanding is that the first (and only the first) filename after the <
will be piped to stdin of the ./interpolate
script, I'm not sure what happens to the rest of them.
What you can do instead is to use the cat
command to concatenate them and pipe them to ./interpolate
as such:
cat us-midwest-latest.polylines \
us-northeast-latest.polylines \
us-south-latest.polylines \
us-west-latest.polylines \
| ./interpolate polyline street.db
from interpolation.
you could also write it like this (assuming you want to import all the .polylines
files):
./interpolate polyline street.db <(cat *.polylines)
or
cat *.polylines | ./interpolate polyline street.db
AFAIK these two syntaxes are equivalent
from interpolation.
Yes you're right. I use |
and >
often, but rarely <
.
> grep -aC5 'oee}mAthsagDN' *.polylines
us-midwest-latest.polylines-m~v~iAzvchrDQpuAm@lfC?tPCjtABj~@Lvb@I~yBI`m@\fZ?z_C?~ESjr@?`ZQpy@Lfm@]n_@Efq@]npAIxo@MnEm@pE{@pEoAvEqBzEkCjFqj@|dAcYbh@yPbXkEzF{KlMwQ|RgIlKaFzJaCbJsAnLwDbi@�West 126th Street
us-midwest-latest.polylines-szfeiAlqs|sDRiHjBwHpDeGdDmDv[iVbC{ChB{CzB{Hr@wDToJ?_dA?iPg@oHoAmF{BqF_JgOyCaF{BqF_AmF[aHp@gEpAqCNqBm@cBoAo@qAJs@pAOhBT|An@~Bx@xD�West 126th Street
us-midwest-latest.polylines-c`qfkAjzomcDg@cDa@gBa@gCE]c@sCi@kEYeCQkDKoCCoCCkFOmbBc@ueDe@gcC]i{Bw@_fAQswAcA{~BMy|@o@euAe@oxAQo]DsANqBb@yCTeArAuFLi@�West 126th Street
us-midwest-latest.polylines-et{~kAbljnyCyApL_Hhj@�Statler Addition
us-midwest-latest.polylines-_w{~kAtyjnyCfpAfa@�Statler Addition
us-midwest-latest.polylines:oee}mAthsagDN|Jd@hDhBpB|CfAvHxAnJxChHdDzE|DxF|G|FhJlGlPhEbT~@bJb@`LB~Fb@pHdA`IhLd]�\
us-midwest-latest.polylines-_q~zeArjikwDm^cFk}AhDanC?a{@oTumAnKkpLfD�265th Road
us-midwest-latest.polylines-yk{akAhdzxtDS~dCxDlkb@�265th Road
us-midwest-latest.polylines-gd{akA|soutD_C_G{@}H`@cvOSoG{AiG�265th Road
us-midwest-latest.polylines-si{akA`|kxtDgA}g]OwnEg@gtV�265th Road
us-midwest-latest.polylines-okeamApcfc|D}AymCm@ep@GyGuAylCcAym@YyfAm@mbAMeGWiMsAk{BQmZsAe`BmA}qCkA{tBy@o{A_@qV�265th Road
from interpolation.
@kylebarron I'm still very confused what's happening, there should be a nullbyte (\0
) character marking the delimiter between the encoded polyline and the name(s).
For example here is another .0sv
file on my computer (I'm replacing the nullbyte with underscores so we can see it):
$ head -n1 file.0sv | sed 's/\x0/___/g'
kgionAlvsegCdb@oJpaAkU___10th Avenue
but when I try this on the output you posted there is no null byte:
$ head -n1 error.0sv | sed 's/\x0/___/g'
m~v~iAzvchrDQpuAm@lfC?tPCjtABj~@Lvb@I~yBI`m@\fZ?z_C?~ESjr@?`ZQpy@Lfm@]n_@Efq@]npAIxo@MnEm@pE{@pEoAvEqBzEkCjFqj@|dAcYbh@yPbXkEzF{KlMwQ|RgIlKaFzJaCbJsAnLwDbi@�West 126th Street
so either there is a copy->paste error or something on your computer is removing them?
edit: I do see a �
character, what operating system are you using?
from interpolation.
The format is:
encodedpolyline\0name\0name\0name\n
from interpolation.
> cat /etc/*-release
LSB_VERSION=base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Scientific Linux release 6.9 (Carbon)
Scientific Linux release 6.9 (Carbon)
from interpolation.
And that screenshot in text form:
> grep -aC5 'oee}mAthsagDN' *.polylines | sed 's/\x0/___/g'
us-midwest-latest.polylines-m~v~iAzvchrDQpuAm@lfC?tPCjtABj~@Lvb@I~yBI`m@\fZ?z_C?~ESjr@?`ZQpy@Lfm@]n_@Efq@]npAIxo@MnEm@pE{@pEoAvEqBzEkCjFqj@|dAcYbh@yPbXkEzF{KlMwQ|RgIlKaFzJaCbJsAnLwDbi@___West 126th Street
us-midwest-latest.polylines-szfeiAlqs|sDRiHjBwHpDeGdDmDv[iVbC{ChB{CzB{Hr@wDToJ?_dA?iPg@oHoAmF{BqF_JgOyCaF{BqF_AmF[aHp@gEpAqCNqBm@cBoAo@qAJs@pAOhBT|An@~Bx@xD___West 126th Street
us-midwest-latest.polylines-c`qfkAjzomcDg@cDa@gBa@gCE]c@sCi@kEYeCQkDKoCCoCCkFOmbBc@ueDe@gcC]i{Bw@_fAQswAcA{~BMy|@o@euAe@oxAQo]DsANqBb@yCTeArAuFLi@___West 126th Street
us-midwest-latest.polylines-et{~kAbljnyCyApL_Hhj@___Statler Addition
us-midwest-latest.polylines-_w{~kAtyjnyCfpAfa@___Statler Addition
us-midwest-latest.polylines:oee}mAthsagDN|Jd@hDhBpB|CfAvHxAnJxChHdDzE|DxF|G|FhJlGlPhEbT~@bJb@`LB~Fb@pHdA`IhLd]___\
us-midwest-latest.polylines-_q~zeArjikwDm^cFk}AhDanC?a{@oTumAnKkpLfD___265th Road
us-midwest-latest.polylines-yk{akAhdzxtDS~dCxDlkb@___265th Road
us-midwest-latest.polylines-gd{akA|soutD_C_G{@}H`@cvOSoG{AiG___265th Road
us-midwest-latest.polylines-si{akA`|kxtDgA}g]OwnEg@gtV___265th Road
us-midwest-latest.polylines-okeamApcfc|D}AymCm@ep@GyGuAylCcAym@YyfAm@mbAMeGWiMsAk{BQmZsAe`BmA}qCkA{tBy@o{A_@qV___265th Road
from interpolation.
Just FYI I'm not sure that
cat *.polylines | ./interpolate polyline street.db
actually uses all files. At the least, on the following step,
> cat ../../data/openaddresses/us/ak/anchorage.csv | ./interpolate oa address3.db full-us-street.db
0 0/sec
> cat ../../data/openaddresses/us/**/*.csv | ./interpolate oa address4.db full-us-street.db
0 0/sec
> md5sum address3.db address4.db
541954acebe2cbe74a3533795b02e5a1 address3.db
541954acebe2cbe74a3533795b02e5a1 address4.db
When I do
./interpolate oa address.db street.db < $(cat ../../data/openaddresses/us/**/*.csv)
I get an error zsh: fatal error: out of heap memory
, so that is at least doing something different.
from interpolation.
heya, so...
-
regarding the
�
chars, this seems to be some sort of copy->paste error. -
looking at that line ending in
Ld]___\
I can see that there is no valid name, it's called '\'. I tried running the import with your data and still couldn't get the error message, anyway I'm going to let this one go as a one-off. -
I'm pretty sure the
cat *.polylines
command works as expected, you can confirm with a command such asls *.polylines
to check the glob pattern is correctly expanded. -
You can query the sqlite files directly to see what's in them and get a count of the records in order to confirm if your globbing pattern is inserting more records:
$ sqlite3 street.db
SQLite version 3.22.0 2018-01-22 18:45:57
Enter ".help" for usage hints.
sqlite> .tables
names rtree rtree_parent
polyline rtree_node rtree_rowid
sqlite> select count(*) from polyline;
10
sqlite> select count(*) from names;
10
sqlite> .exit
- using the
cat
command on a file format which has a header (like.csv
) will cause the header to be included multiple times in the output stream, this will cause issues.
I have included a script in this repo to handle concatenation of openaddresses csv files:
try this:
export OAPATH=../../data/openaddresses/us
./script/concat_oa.sh
you should get a stream of all the files combined, deduplicated and only containing the header line once.
- the error
zsh: fatal error: out of heap memory
is specific to zsh, I'm not 100% sure on why this is happening but could be something to do with the<
operator, try using a pipe on the left hand side instead to ensure that you're ingesting a stream incrementally and not trying to buffer the whole stream in to memory before sending it to the process.
from interpolation.
Related Issues (20)
- could not locate bindings file HOT 2
- address.db is not populating for openaddresses HOT 5
- An in-range update of pelias-config is breaking the build 🚨 HOT 2
- Interpolation planet openaddresses data issue build without docker HOT 1
- total openaddresses records imported in address.db is only 56000 for AUS HOT 2
- TIGER download hangs on unzip error HOT 3
- Update package.json to address security vulnerabilities of dependency libraries HOT 3
- normalization mismatch for search/extract
- An in-range update of unzipper is breaking the build 🚨 HOT 2
- Prepare interpolation is very slow HOT 6
- Combine street.db and address.db into one database HOT 1
- Tiger 2016 data being used instead of TIGER 2019 HOT 3
- Parallelize interpolation build HOT 1
- Street Address Interpolation - Interactive Demo - PAGE NOT FOUND HOT 1
- support of .pbf files
- enable 'unsafe' mode of better-sqlite HOT 1
- reduce verbosity of `*.{err, skip}` logs
- Hard-coded GeocodeEarth API key HOT 3
- Data loaded with the csv-import module is not used on the interpolation module HOT 5
- Skip importing OSM HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from interpolation.