Comments (22)
Issued is solved, Please close. This, thank you
Good to hear!
Can you explain what the problem was, and how you solved it?
Knowing that might be helpful for others, if they encounter a similar issue.
from fpdf2.
I guess the text shaping in line 9 needs to be specific regarding script and language. This worked for me with similar problems. The shape engine does not automatically detect the font and language used. It might be implemented for one of the 173 scripts, but certainly not for one of the 634 possible languages used by these scripts. Therefore in most cases (if it's more than ligatures in English documents) this information needs to be added. My solution was
pdf.set_text_shaping(use_shaping_engine=True, script="khmr", language="khm")But let's see what @kuth-chi answers.
When you don't specify the script and language, fpdf2 will call harfbuzz with the option to guess the text properties. Reading the documentation of the guess function it applies the script of the first character if finds with a valid unicode script.
In this example the string has english + Khmer, so it's not applying the Khmer shaping because it's guessing the text is latin script since the english text is sent first.
from fpdf2.
I updated with
pip install git+https://github.com/py-pdf/fpdf2.git@master
and still get the same result.pip show fpdf2
just shows the version number 2.7.9. I hope that the updated library is used (no error message from installation) but is there a way to know since the version number is still the same?Simon Cozens replied to the notofonts/notofonts.github.io#46
get_string_width
issue and mentions advance width of mark attached glyphs that should be taken into account. I don't have a glyph editor or viewer yet so look for these parameters. Maybe I'll try glyphsapp with the 30 days trial. And he was right - all the other fonts I used for Khmer were created by Dan Hong!
Try uninstalling fpdf2 with pip uninstall fpdf2
, make sure it's uninstalled on pip freeze
and then install from github
from fpdf2.
I've seen the text_shaping working with pdf.cell() and pdf.write() (like in this example here). But I wanted to achieve exact positioning on the page for each string, and used pdf.text(). The shaping engine seems not to work with pdf.text().
fpdf.text()
is deprecated and only kept in the codebase for backwards compatibility. It doesn't support text shaping, markdown and many other features.
from fpdf2.
Text shaping works now, but I discovered two challenges. One is related to
pdf.get_string_width(text)
for the above Khmer examples. I use it to render some strings with right align, but the results are inconsistent. See below:The second problem is that the Unicode sequence originally representing the Khmer text is replaced with the character sequence that gives the correct result for the rendering engine - but no longer represents a Khmer Unicode text. If I highlight the text in my PDF viewer, the copied sequence contains unintelligible codepoints. Can this be separated in the source code - rendering instructions and what it should represent in Unicode?
Thank you for reporting these issues. I will take a look as soon as possible.
from fpdf2.
Thanks for researching this, @kreier !
It would be interesting to figure out in more detail which glyph combinations can cause unexpected width determinations, and why.
Unfortunately, it is quite possible that we will be unable to create a fix within fpdf2, since we are dependent on the harfbuzz library to do the actual work, and harfbuzz is again dependent on the information it finds in the font file.
- fpdf2 accepts a sequence of characters, and passes it to pyharfbuzz.
- pyharfbuzz converts the python string to a C structure and passes it to harfbuzz.
- harfbuzz consults the font file, combines character sequences into glyph clusters, and adds the width information given in the font file to each cluster.
- pyharfbuzz converts the result back into python data
- fpdf2 uses the returned width information for line wrapping, and adds the resulting line data into the PDF stream.
- A PDF viewer reads that stream, and needs to figure out where to place the glyphs on the page.
Theoretically, at any of the above steps something could go wrong. In practise, the font file, harfbuzz' interpretation of its contents, and the rendering by the viewer are most error prone. The python components primarily just pass on the information they receive.
The khmer script uses many ligatures, conjuncts, and diacritics, so harfbuzz needs to perform many substitutions. If a font file contains contains slightly incorrect width information for just one of the combined glyphs, then that may result in the effect you are observing.
Did you try the same experiment with a different font?
Do you think you can figure out which specific character combination(s) is/are causing the positional shift?
from fpdf2.
You can install the most recent version on
pip install git+https://github.com/py-pdf/fpdf2.git@master
from fpdf2.
I updated with pip install git+https://github.com/py-pdf/fpdf2.git@master and still get the same result.
Works for me:
![Screenshot 2024-06-06 at 09 14 57](https://private-user-images.githubusercontent.com/106728/337180690-4d46e9b9-c5c8-4cff-9504-c7df367397b3.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjE3NDg5NjIsIm5iZiI6MTcyMTc0ODY2MiwicGF0aCI6Ii8xMDY3MjgvMzM3MTgwNjkwLTRkNDZlOWI5LWM1YzgtNGNmZi05NTA0LWM3ZGYzNjczOTdiMy5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzIzJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcyM1QxNTMxMDJaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0yYjBhYjBkODM0ZjNjNDRlNDM0OTdjZjY5ZjExNmUyYzk1OWNhNTc5Njk1YWNkZWU1M2ZmYjlkNGVjMTIyYzQwJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.Tt0h2K_btTmQahEOJ6LXlzLLppNbs_e08BLzuQphDb8)
from fpdf2.
Thank you @andersonhc , that was it! The fix worked. Here my newly rendered test line with the same NotoSansKhmer-Regular.ttf
:
And my initial problem is solved, too. Right align looks good:
from fpdf2.
Issued is solved, Please close. This, thank you
from fpdf2.
I guess the text shaping in line 9 needs to be specific regarding script and language. This worked for me with similar problems. The shape engine does not automatically detect the font and language used. It might be implemented for one of the 173 scripts, but certainly not for one of the 634 possible languages used by these scripts. Therefore in most cases (if it's more than ligatures in English documents) this information needs to be added. My solution was
pdf.set_text_shaping(use_shaping_engine=True, script="khmr", language="khm")
But let's see what @kuth-chi answers.
from fpdf2.
from fpdf2.
I've seen the text_shaping working with pdf.cell() and pdf.write() (like in this example here). But I wanted to achieve exact positioning on the page for each string, and used pdf.text(). The shaping engine seems not to work with pdf.text(). Here the example code:
# example rendering Khmer
from fpdf import FPDF
pdf = FPDF(orientation="P", unit="mm", format="A4")
pdf.add_page()
pdf.add_font("noto", style="", fname="../../fonts/NotoKhmer.ttf")
pdf.set_font('noto', size=32)
pdf.cell(text="King - ស្តេច", new_x="LMARGIN", new_y="NEXT")
pdf.cell(text="Prophet - ហោរា", new_x="LMARGIN", new_y="NEXT")
pdf.set_font("Helvetica", size=12)
pdf.cell(h = 20,text="Now using __text_shaping__ with **uharfbuzz** in pdf.cell() and pdf.write():", markdown=True, new_x="LMARGIN", new_y="NEXT")
pdf.set_font("noto", size=32)
pdf.set_text_shaping(use_shaping_engine=True, script="khmr", language="khm")
pdf.cell(text="King - ស្តេច", new_x="LMARGIN", new_y="NEXT")
pdf.cell(text="Prophet - ហោរា", new_x="LMARGIN", new_y="NEXT")
pdf.write(text="King - ស្តេច, Prophet - ហោរា")
pdf.text(10, 110, "King - ស្តេច")
pdf.text(10, 121, "Prophet - ហោរា")
pdf.set_font("Helvetica", size=12)
pdf.text(10, 95, "Does not work if you use pdf.text() instead of pdf.cell()")
pdf.output("example_fpdf.pdf")
from fpdf2.
I used to get the exact location with pdf.text() pdf.text(x_value, y_value, textstring)
. Textshaping does not work with pdf.text. The solution with pdf.cell() needs a few additional steps:
pdf.set_margin(0) # avoid overflow to next page if text is to close to any border
pdf.c_margin = 0 # removes the 2.83pt or 1mm margin from the start of the string
pdf.set_xy(x_value, y_value)
pdf.cell(textstring)
I could not find the .c_margin = 0
in the documentation. Might be a possible update. Text shaping works now, but I discovered two challenges. One is related to pdf.get_string_width(text)
for the above Khmer examples. I use it to render some strings with right align, but the results are inconsistent. See below:
If I deactivate the shape engine pdf.set_text_shaping(use_shaping_engine=False
the returned values are correct:
The second problem is that the Unicode sequence originally representing the Khmer text is replaced with the character sequence that gives the correct result for the rendering engine - but no longer represents a Khmer Unicode text. If I highlight the text in my PDF viewer, the copied sequence contains unintelligible codepoints. Can this be separated in the source code - rendering instructions and what it should represent in Unicode?
from fpdf2.
I guess the text shaping in line 9 needs to be specific regarding script and language. This worked for me with similar problems. The shape engine does not automatically detect the font and language used. It might be implemented for one of the 173 scripts, but certainly not for one of the 634 possible languages used by these scripts. Therefore in most cases (if it's more than ligatures in English documents) this information needs to be added. My solution was
pdf.set_text_shaping(use_shaping_engine=True, script="khmr", language="khm")But let's see what @kuth-chi answers.
When you don't specify the script and language, fpdf2 will call harfbuzz with the option to guess the text properties. Reading the documentation of the guess function it applies the script of the first character if finds with a valid unicode script.
In this example the string has english + Khmer, so it's not applying the Khmer shaping because it's guessing the text is latin script since the english text is sent first.
I have abstract with uharfbuzz shaping, but it's not a good way to do.
We would to solve the issue with can use generally in FPDF2.
from fpdf2.
Text shaping works now, but I discovered two challenges. One is related to
pdf.get_string_width(text)
for the above Khmer examples. I use it to render some strings with right align, but the results are inconsistent. See below:
The second problem is that the Unicode sequence originally representing the Khmer text is replaced with the character sequence that gives the correct result for the rendering engine - but no longer represents a Khmer Unicode text. If I highlight the text in my PDF viewer, the copied sequence contains unintelligible codepoints. Can this be separated in the source code - rendering instructions and what it should represent in Unicode?Thank you for reporting these issues. I will take a look as soon as possible.
I created an example script to demonstrate the two problems. With some selected examples its easier to see. The measurements for pdf.get_string_width(text)
and the updated x-position of pdf.get_x
seem to match. But they are actually not correct. I added another symbol after the rendered cell, and in some cases it actually overlaps with the rendered string. See pictures below
The example also demonstrates the second problem with the replaced copy/paste values of the rendered string. The three example strings in the pdf are initially all the same. One could easily compare and test them with Google Translate and copy/paste:
# example rendering Khmer
from fpdf import FPDF
pdf = FPDF(orientation="P", unit="mm", format="A4")
pdf.add_page()
pdf.c_margin = 0
pdf.add_font("noto", style="", fname="../../fonts/NotoKhmer.ttf")
# teststring = ["ពុម្ពអក្សរស្មុគស្មាញ", "គឺជាលក្ខណៈ", "នៃភាសាខ្មែរ។"]
# teststring = ["អំរី (ម្នាក់ឯង)", "យេរ៉ូបោម", "យ៉ូសៀស"]
teststring = ["ស៊ីមរី (៧ ថ្ងៃ)", "អេឡា (២ ឆ្នាំ)", "អំរី ( ម្នាក់ ឯង ) (៨ ឆ្នាំ)"]
pdf.set_font("noto", size=12)
pdf.cell(text="The following text consists of three cells. We determine the width before and after rendering.")
pdf.ln()
pdf.set_font('noto', size=24)
teststringlength = []
teststringlength_measured = []
for i in range(len(teststring)):
teststringlength.append(pdf.get_string_width(teststring[i]))
start = pdf.get_x()
pdf.cell(h=17, text=teststring[i])
end = pdf.get_x()
pdf.cell(h=17, text="—")
teststringlength_measured.append(end - start)
pdf.ln()
pdf.set_font("noto", size=12)
for i in range(len(teststringlength)):
pdf.cell(text=f"Before: {teststringlength[i]} - after: {teststringlength_measured[i]}")
pdf.ln()
pdf.ln()
pdf.cell(text="Now activating the shape engine and try this again:")
pdf.ln()
pdf.set_text_shaping(use_shaping_engine=True, script="khmr", language="khm")
pdf.set_font('noto', size=24)
teststringlength = []
teststringlength_measured = []
for i in range(len(teststring)):
teststringlength.append(pdf.get_string_width(teststring[i]))
start = pdf.get_x()
pdf.cell(h=17, text=teststring[i])
end = pdf.get_x()
pdf.cell(h=17, text="—")
teststringlength_measured.append(end - start)
pdf.ln()
pdf.set_font("noto", size=12)
for i in range(len(teststringlength)):
pdf.cell(text=f"Before: {teststringlength[i]} - after: {teststringlength_measured[i]}")
pdf.ln()
pdf.ln()
pdf.c_margin = 1
pdf.set_font('noto', size=24)
for i in range(len(teststring)):
pdf.cell(text=teststring[i])
pdf.cell(text="—")
pdf.ln()
pdf.cell(h=17, text=teststring[0]+"—"+teststring[1]+"—"+teststring[2]+"—")
pdf.output("example_fpdf2_strignwidth.pdf")
The strange overlap for the middle rendering is a result of reducing the cell margin pdf.c_margin = 0
to zero. With the expected value of 1 millimeter it looks better, but only the combined string rendered in a single run (seen in the last line) is correct:
from fpdf2.
I updated my comment with other strings and an additional dash after the three example strings to visualize that both the results of pdf.get_string_width(text)
and the updated x-position pdf.get_x
are incorrect after applying the shape engine.
from fpdf2.
Hi @gmischler,
Thanks for the detailed breakdown of the steps involved! I'm certainly have to come back to these steps at one point in the future. As with Khmer, I neither know the script nor language, so it's difficult to see which Unicode codepoints combined give a letter, consonant, conjunct, ligature or add diacritics to form a word, sentence or idiom. I'm planning to visit Cambodia at the end of the month to talk to some native speakers.
But I just found the solution for problem Nr. 1: It's the font (or typeface)! Both Noto Khmer sans and serif produce the problem (see below). Replacing the font with the Google font for Khmer solved the problem immediately. Here my new test code and the correct result:
from fpdf import FPDF
pdf = FPDF(orientation="P", unit="mm", format="A4")
pdf.add_page()
pdf.c_margin = 0
# pdf.add_font("noto", style="", fname="../../fonts/NotoKhmer.ttf")
pdf.add_font("noto", style="", fname="../../fonts/Khmer-Regular.ttf")
# strings: years, alone, 1st year, sword, fork, objects
teststrings = ["ឆ្នាំ", "ម្នាក់ ឯង", "ឆ្នាំទី១", "ដង្កាវ", "ង្គ្រា", "វត្ថុ"]
def render_strings(teststrings):
pdf.set_font('noto', size=24)
pdf.set_draw_color(160)
pdf.set_line_width(0.3)
for string in teststrings:
pdf.rect(pdf.get_x(), pdf.get_y()+2, pdf.get_string_width(string), 13, style="D")
pdf.cell(h=17, text=string + " ")
pdf.ln()
def info(text):
pdf.set_font("Helvetica", size=12)
pdf.cell(text=text)
pdf.ln()
info("Rendering without shape engine:")
render_strings(teststrings)
info("Now activating the shape engine and try this again:")
pdf.set_text_shaping(use_shaping_engine=True, script="khmr", language="khm")
render_strings(teststrings)
render_strings([''.join(teststrings)])
pdf.output("fpdf2_stringwidth.pdf")
For comparison, this is the result with NotoSansKhmer-Regular.ttf with under-measurement of pdf.string_width
:
from fpdf2.
The error is now reported at notofonts/notofonts.github.io#46
from fpdf2.
I just merged #1193. It should solve:
- Automatically detecting the script, so you don't have to force the script and language when you enable text shaping.
- Error in the text width calculation.
I still don't have a final solution for the third problem—an extra character being added when you copy/paste from your PDF. The issue arises in scenarios where the text shaper resolves two glyphs with a single Unicode code (normally, ligatures are one glyph per two codes). fpdf2 creates a mapping associating the first glyph with the Unicode code and doesn't associate anything with the second glyph. However, the reader doesn't handle this well and copies the glyph ID when no associated character is found.
I will probably have to implement Tagged pdf /ActualText to handle this case.
from fpdf2.
I updated with pip install git+https://github.com/py-pdf/fpdf2.git@master
and still get the same result. pip show fpdf2
just shows the version number 2.7.9. I hope that the updated library is used (no error message from installation) but is there a way to know since the version number is still the same?
Simon Cozens replied to the notofonts/notofonts.github.io#46 get_string_width
issue and mentions advance width of mark attached glyphs that should be taken into account. I don't have a glyph editor or viewer yet so look for these parameters. Maybe I'll try glyphsapp with the 30 days trial. And he was right - all the other fonts I used for Khmer were created by Dan Hong!
from fpdf2.
Theoretically, at any of the above steps something could go wrong. In practise, the font file, harfbuzz' interpretation of its contents, and the rendering by the viewer are most error prone. The python components primarily just pass on the information they receive.
As it turns out, passing on information can also go wrong... 😉
Thanks @simoncozens for chiming in, not surprisingly your diagnosis was much more accurate than my own. 👍
from fpdf2.
Related Issues (20)
- Unexpected HTML end tag </span>, start tag was <p> HOT 2
- Word wrapping for Thai text (zero-width space) may not work as intended // wrapmode==WORD should also use zero-width space? HOT 2
- Add Cell Level Controll for Table Borders HOT 1
- Wrong signature for `pdf.table` HOT 4
- OSS-Fuzz Integration HOT 8
- multipage table using fpdf HOT 2
- Get rid of FPDF class/instance attributes getting passed to HTML2PDF. HOT 5
- Proposal for Generic Robust Substitution Mechanism HOT 1
- `local_context()` blocks appear to "leak" into one another HOT 6
- HTML: indent of lists on new line not flush HOT 5
- Line spacing for a long line that is displayed in several lines. HOT 5
- Text columns issue with top margin HOT 1
- Markdown not escaping MD special characters HOT 3
- Solution needed: Typographical Margins, Line Height, Leading HOT 1
- write_html() ignores font changes HOT 4
- call .write_html in footer cause RecursionError HOT 1
- pdf.font_style = "B" and pdf.emphasis = "BOLD" should behave the same way as pdf.set_font(style="B") HOT 1
- Lowercase hex (e.g. #fff) do not convert properly HOT 5
- Stampapdf HOT 3
- FPDF.unbreakable() drops table instead of adding page break beforehand HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fpdf2.