Comments (3)
jsvine
commented on June 2, 2024
Hi @rajathsalegame, and thanks for the suggestion. I agree that this could be a useful feature. Unfortunately for your multiprocessing use-case, a lot of the heavy processing load currently is handled by the pdfminer.six dependency, which I don't believe supports pickling.
To better understand the request, could you provide a bit more detail about your goals with multiprocessing and at what specific stage in your pipeline you'd be pickling/unpickling?
from pdfplumber.
XiYuan68
commented on June 2, 2024
Hi, in my case, I need to apply page.dedupe_chars().extract_words(keep_blank_chars=True) on every pages in a PDF file. For one file with 20 pages, it takes about 16 sec, taking up most of the time consumtion in my pipline. It will be great for pdfplumber to support pickle and multiprocessing.
BTW, if I do something like:
import pandas as pf
from joblib import Parallel, delayed
def parse_page(page):
df_page = pd.DataFrame(page.dedupe_chars().extract_words(keep_blank_chars=True))
with pdfplumber.open(pdf_file_or_path) as pdf:
result_parse = Parallel(-1, 'threading')(delayed(parse_page)(page) for page in pdf.pages)
It seems the error info is not about page being unpickable, but:
File "<ipython-input-50-192c80c0ccbc>", line 207, in parse_page
df_page = pd.DataFrame(page.dedupe_chars().extract_words(keep_blank_chars=True))
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfplumber/page.py", line 403, in dedupe_chars
p._objects = {kind: objs for kind, objs in self.objects.items()}
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfplumber/page.py", line 215, in objects
self._objects: Dict[str, T_obj_list] = self.parse_objects()
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfplumber/page.py", line 275, in parse_objects
for obj in self.iter_layout_objects(self.layout._objs):
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfplumber/page.py", line 161, in layout
interpreter.process_page(self.page_obj)
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdfinterp.py", line 997, in process_page
self.render_contents(page.resources, page.contents, ctm=ctm)
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdfinterp.py", line 1016, in render_contents
self.execute(list_value(streams))
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdfinterp.py", line 1021, in execute
parser = PDFContentParser(streams)
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdfinterp.py", line 251, in __init__
PSStackParser.__init__(self, None) # type: ignore[arg-type]
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/psparser.py", line 545, in __init__
PSBaseParser.__init__(self, fp)
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/psparser.py", line 193, in __init__
self.seek(0)
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdfinterp.py", line 263, in seek
self.fillfp()
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdfinterp.py", line 256, in fillfp
strm = stream_value(self.streams[self.istream])
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdftypes.py", line 217, in stream_value
x = resolve1(x)
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdftypes.py", line 118, in resolve1
x = x.resolve(default=default)
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdftypes.py", line 106, in resolve
return self.doc.getobj(self.objid)
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdfdocument.py", line 866, in getobj
obj = self._getobj_parse(index, objid)
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdfdocument.py", line 840, in _getobj_parse
(_, obj) = self._parser.nextobject()
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/psparser.py", line 656, in nextobject
self.do_keyword(pos, token)
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdfparser.py", line 79, in do_keyword
(objid, genno) = (int(objid), int(genno)) # type: ignore[arg-type]
TypeError: int() argument must be a string, a bytes-like object or a real number, not 'PSKeyword'
"""
Does this make multiprocess with pdfplumber a bit easier?
from pdfplumber.
XiYuan68
commented on June 2, 2024
For anyone who's interested, a work-around for multiprocessing for pdfplumber is to start multiprocessing before opening the pdf file with pdfplumber:
from joblib import Parallel, delayed
import pdfplumber
import pandas as pd
def plumber_parsepage(pdf_path: str,
idx_page: int = 0,
) -> pd.DataFrame:
with pdfplumber.open(pdf_path) as pdf:
page = pdf.pages[idx_page]
df_page = pd.DataFrame(page.dedupe_chars().extract_words(keep_blank_chars=True))
return df_page
with pdfplumber.open(pdf_file_or_path) as pdf:
num_page = len(pdf.pages)
result = Parallel(-1)(delayed(plumber_parsepage)(pdf_file_or_path, i) for i in range(num_page))
from pdfplumber.
Related Issues (20)
- suggest page.extract_words() word sequence same as page.extract_text()
- Lots of whitespaces in between words
- page.to_image() causes error "PermissionError: [WinError 32] The process cannot access the file because it is being used by another process " HOT 5
- can' resolve pdf encoded in ETenms-B5-H HOT 1
- debug_tablefinder is weirdly offset HOT 2
- Extracting devnagiri text. HOT 3
- `Page.to_image()` leaks file descriptors HOT 7
- Extracting table with no vertical lines (only horizontal lines) doesn't work
- page.search("text", regex = True) is magnitudes slower in 0.10.4 compared to 0.10.3. HOT 5
- The position of numbers and punctuation marks is incorrect HOT 2
- bottom-to-top text in cell rendered in wrong location in extract_text() HOT 6
- Any way to detect formatting? HOT 1
- Add `autodetect_direction` option to text-extraction methods
- Table extraction bug when lines are just barely end-to-end
- About paragraph recognition
- Custom deduppe_chars char properties HOT 4
- Got different result of "page.to_image()" on MacOS and Linux HOT 7
- TypeError: argument of type 'PDFObjRef' is not iterable HOT 5
- table_settings support draw line per page
- function "extract_words" extract words that don't exist in a pdf. HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pdfplumber.