[Question] Type2 hintmask and cntmask about typography HOT 14 CLOSED

(note:
Haha, Typography.OpenFont.CFF.CffEvaluationEngine reads hint instructions but the engine does not implement the those hint instructions yet=> it skips all hint instructions.)

Your question: "How to determine which amount of bytes should be read after we detect hintmask or cntmask operator"?

early help : please see the implementation here

hintmask =>
https://github.com/LayoutFarm/Typography/blob/master/Typography.OpenFont/Tables.CFF/Type2CharStringParser.cs#L1017

cntmask=>
https://github.com/LayoutFarm/Typography/blob/master/Typography.OpenFont/Tables.CFF/Type2CharStringParser.cs#L1174

from typography.

Ref: https://adobe-type-tools.github.io/font-tech-notes/pdfs/5177.Type2.pdf

Note 1

3.1 Type 2 Charstring Organization

The sequence and form of a Type 2 charstring program may be represented as:

      w? {hs* vs* cm* hm* mt subpath}? {mt subpath}* endchar

Where:

w = width
hs = hstem or hstemhm command
vs = vstem or vstemhm command
cm = cntrmask operator
hm = hintmask operator
mt = moveto (i.e. any of the moveto) operators
subpath = refers to the construction of a subpath (one complete closed contour), which may include hintmask operators where appropriate.

and the following symbols indicate specific usage:
* zero or more occurrences are allowed
? zero or one occurrences are allowed
+ one or more occurrences are allowed
{ } indicates grouping

Note 2

2. Hints: zero or more of each of the following hint operators, in exactly the following order:
   hstem, hstemhm, vstem, vstemhm, cntrmask, hintmask.
   Each entry is optional, and each may be expressed by one or more occurrences of the operator.

The hint operators cntrmask and/or hintmask must not occur if the charstring has no stem hints.

Note 3

IMPORTANT

from page24 of 5177.Type2.pdf (above)

    hintmask | -hintmask(19 + mask) | -
    The mask data bytes are defined as follows:

    • The number of data bytes is exactly the number needed, one
    bit per hint, to reference the number of stem hints declared
    at the beginning of the charstring program.

    • Each bit of the mask, starting with the most-significant bit of
    the first byte, represents the corresponding hint zone in the
    order in which the hints were declared at the beginning of
    the charstring.

    • For each bit in the mask, a value of ‘1’ specifies that the
    corresponding hint shall be active. A bit value of ‘0’ specifies
    that the hint shall be inactive.

    • Unused bits in the mask, if any, must be zero.

Question:

"How to determine which amount of bytes should be read after we detect hintmask or cntmask operator"?

see Note 3

You need to count number of stem hints declared at the beginning of the charstring program

In my implementation=>

1. I store "counting-mode" in a field "_doStemCount" (indicate that we are counting stem or not).
   _doStemCount is set to true at the begin of parsing steps.

2. And I store number of hint stem in a field __hintStemCount.
   

   Typography/Typography.OpenFont/Tables.CFF/Type2CharStringParser.cs

   Line 517 in b0d8517

   int _hintStemCount = 0;

3. My Type2-Cff StringParser will stop stem-counting when found some path instructions (check it with StopStemCount())
   you will see when it stop stem counting. (_doStemCount is set to false).

4. Please note that (from spec)

    If hstem and vstem hints are both declared at the beginning of
    a charstring, and this sequence is followed directly by the
    hintmask or cntrmask operators, ...
    the vstem hint operator **need not be included ***
   

see

5. then we can calculate number of bytes of

   one bit per hint

see ..(hintmask)

and ..(cntmask)

@QuantumDeveloper

That's it!
Feel free to ask me more.

:)

from typography.

I tried to implement behaviour as described in docs and I still facing a parsing issues.
I will describe you what I did step by step. Maybe you will find what I am missing here.

1. I am accumulating stem count based on operands count:

                       default:
                            switch (token)
                            {
                                case (ushort)OperatorsType.vstem:
                                case (ushort)OperatorsType.hstem:
                                case (ushort)OperatorsType.hstemhm:
                                case (ushort)OperatorsType.vstemhm:
                                    stemCount+=operands.Count;
                                    break;
                            }
                            
                            var command = new Command();

                            command.@operator = bytesToOperatorMap[token];
                            command.operands = operands;

                            commands.Add(command);
                            break;

2. When I am facing hintmask or cntmask I do the following:

switch(token)
...
                        case (ushort)OperatorsType.hintmask:
                        case (ushort)OperatorsType.cntrmask:
                            stemCount /= 2;
                           
                            if (stemCount == 0) stemCount = 1;
                            var bytesToRead = Math.Ceiling((double) stemCount / 8);
                            for (int i = 0; i < bytesToRead; ++i)
                            {
                                var maskByte = mainStack.Pop();
                            }

                            stemCount = 0;
                            break;

And even with this algorithm I am facing with issue.
I am takштп into account the width, which could be the first argument for hints by dividing by 2. It will round to the lowest integer value.

I debug your code and you somehow calculating that you need to read exactly 2 bytes (for curtain glyphs), where I am always read only 1 byte.
Obviously, your code is working correctly, but I cannot understand where and what I am missing.
If you could help me with it and point what is worng, I would be very appreacite.
@prepare

from typography.

1. please show some code region of my implementation about ..

".. you somehow calculating that you need to read exactly 2 bytes (for curtain glyphs), where I am always read only 1 byte."

2. Did you stop stem counting correctly? (after enter into shape instructions region)

from typography.

from my implementation when _hintStemCount ==0

   if (_hintStemCount == 0)
            {
                if (!_foundSomeStem)
                {
                    _hintStemCount = (_current_integer_count / 2);
                    if (_hintStemCount == 0)
                    {
                        return;
                    }
                    _foundSomeStem = true;//?
                }
                else
                {
                    throw new NotSupportedException();
                }
            }

please check your code

switch(token)
...
                        case (ushort)OperatorsType.hintmask:
                        case (ushort)OperatorsType.cntrmask:
                            stemCount /= 2;
                           
                            if (stemCount == 0) stemCount = 1;
                            var bytesToRead = Math.Ceiling((double) stemCount / 8);
                            for (int i = 0; i < bytesToRead; ++i)
                            {
                                var maskByte = mainStack.Pop();
                            }

                            stemCount = 0;
                            break;

from typography.

1. Did you stop stem counting correctly? (after enter into shape instructions region)

I think yes. Because As far as I can see, you stop stem count almost on all operators except stem operators. This is what I am doing exactly. All operands before stem commands in accumulating in array and then I add them to the stem count. I do this only if I faced with one of stem operators.

switch (token)
                            {
                                case (ushort)OperatorsType.vstem:
                                case (ushort)OperatorsType.hstem:
                                case (ushort)OperatorsType.hstemhm:
                                case (ushort)OperatorsType.vstemhm:
                                    stemCount+=operands.Count;
                                    break;
                            }

And after I found hintmask/cntmask operator, I set stem count to 0.

Also, I didnt see that you ever fall in case when you when this condition is executing:

                   if (_hintStemCount == 0)
            {
                if (!_foundSomeStem)
                {
                    _hintStemCount = (_current_integer_count / 2);
                    if (_hintStemCount == 0)
                    {
                        return;
                    }
                    _foundSomeStem = true;//?
                }
                else
                {
                    throw new NotSupportedException();
                }
            }

At least not in fonts which I have tested.

But maybe I am missing something and I need to calculate stem count somehow differently?
Maybe I can provide my test font and glyph index that is failing and you can give me more info then?

from typography.

Yes, please give me the test font and glyph index.

from typography.

I downloaded the font and it removed it.

Thank you. give me a time to test it.
(Glyph index 554)

from typography.

Also, I would like to know why you are doing decrement in callsubr/callgsubr operatos for _current_integer_count--
I didnt find anything about this in docs.

from typography.

Hello, This is your glyph-554 of Glametrix font

pic 1: glyph 554 Glametrix (cff font)

and this is a (part of) raw byte[] of a glyph-554 instructions (Type2 CFF CharString)

pic 2: begin part of raw byte buffer of glyph 554

After the raw buffer is parsed, the parser translate it to these instructions...

[0] GlyphWidth 162
[1] 0 [2] 50 [3] 170 [4] 50 [5] 190 [6] 20 [7] 90 [8] 60 [9] hstemhm
[10] 55 [11] 65 [12] 48 [13] 60 [14] 74 [15] 68 [16] -62 [17] 60 [18] 57 [19] 65 [20] vstem
[21] hintmask2 10101000000000000000000000000
[22] 308 [23] 570 [24] rmoveto
[25] 49 [26] hlineto
[27] 8.900391
[28] 1.099609
[29] 1.099609
[30] 8.900391
[31] hvcurveto
[32] 49 [33] -49 [34] vlineto
[35] -9.900391
[36] -2.099609
[37] -2.099609
[38] -9.900391
[39] hvcurveto
[40] -140 [41] -49 [42] rmoveto
[43] 49 [44] hlineto
[45] 8.900391
[46] 1.099609
[47] 1.099609
[48] 8.900391
[49] hvcurveto
[50] 49 [51] -49 [52] vlineto
[53] -9.900391
[54] -2.099609
[55] -2.099609
[56] -9.900391
[57] hvcurveto
[58] hintmask2 11101010100000000000000000000000
[59] -113 [60] -619 [61] rmoveto
[62] 136 [63] hlineto
[64] 107.4004
[65] 70.59961
[66] 54 [67] 81 [68] 81 [69] -71.59961
[70] 54 [71] -108.4004
[72] hvcurveto
[73] -71 [74] 210 [75] -65 [76] hlineto
[77] 65 [78] -430 [79] rmoveto
[80] 170 [81] 75 [82] vlineto
[83] 64.2002
[84] 41.7998
[85] -34 [86] -51 [87] -51 [88] -42.7998
[89] -34 [90] -65.2002
[91] hvcurveto
[92] 230 [93] -50 [94] rmoveto
[95] 65 [96] 480 [97] -65 [98] hlineto
[99] endchar

glyph 554 instructions (translated)

Your problem is How to find number of bytes for hint mask?
so lets check only first 22 instructions

from raw buffer in pic2, at array index 12 must be hstemhm.

pic 3: raw byte buffer, hstemhm (18) at index12

at this point the CFF instructions in the instruction-list look like the pic 4. (below)

pic 4: instructions, 0=> glyph width, index 1-8 => hstemhm paramters

9 instructions (odd), before detect 1st hstemhm.
since (from spec) all stem-instructions use even number

            //|- y dy {dya dyb}*  hstemhm (18) |-
            //2.
            //|- x dx {dxa dxb}* vstemhm (23) |-
            //3.
            //|- y dy {dya dyb}*  hstem (1) |-
            //4. 
            //|- x dx {dxa dxb}*  vstem (3) |- 

the 1st instruction (index-0) is converted to glyph-width info (162)
then the instructions are ...

[0] GlyphWidth 162
[1] 0 [2] 50 [3] 170 [4] 50 [5] 190 [6] 20 [7] 90 [8] 60 [9] hstemhm

instructions 0=> glyph width, index 1-8 => hstemhm paramters

at this point _hintStemCount =4

Next ... index 23 of raw byte buffer is hintmask

pic5: raw byte buffer, index 23 is hintmask (19)

at this point the instructions are...

IMPORTANT

"...the vstem hint operator need not be included" -a vstem hint is (auto) generated and is inserted here.

from spec (5177.pdf) ...

        If hstem and vstem hints are both declared at the beginning of
        a charstring, and this sequence is followed directly by the
        hintmask or cntrmask operators, ...
        the vstem hint operator **need not be included ***

Instruction 10-19 are parameters of vstem hint

[0] GlyphWidth 162
[1] 0 [2] 50 [3] 170 [4] 50 [5] 190 [6] 20 [7] 90 [8] 60 [9] hstemhm
[10] 55 [11] 65 [12] 48 [13] 60 [14] 74 [15] 68 [16] -62 [17] 60 [18] 57 [19] 65 [20] vstem

instructions 10-19 are parameters of (auto inserted) vstem

And at this point _hintStemCount = 4 + 5 = 10 stems

And proper number of mask bytes=>

int properNumberOfMaskBytes = (_hintStemCount + 7) / 8; 

properNumberOfMaskBytes = (10+7)/8 => 2 bytes
so read next 2 bytes and stop stem count

[0] GlyphWidth 162
[1] 0 [2] 50 [3] 170 [4] 50 [5] 190 [6] 20 [7] 90 [8] 60 [9] hstemhm
[10] 55 [11] 65 [12] 48 [13] 60 [14] 74 [15] 68 [16] -62 [17] 60 [18] 57 [19] 65 [20] vstem
[21] hintmask2 10101000000000000000000000000

Next are parameters of rmoveto ...

[0] GlyphWidth 162
[1] 0 [2] 50 [3] 170 [4] 50 [5] 190 [6] 20 [7] 90 [8] 60 [9] hstemhm
[10] 55 [11] 65 [12] 48 [13] 60 [14] 74 [15] 68 [16] -62 [17] 60 [18] 57 [19] 65 [20] vstem
[21] hintmask2 10101000000000000000000000000
[22] 308 [23] 570 [24] rmoveto
...
(read until complete ...)
..
[95] 65 [96] 480 [97] -65 [98] hlineto
[99] endchar

@QuantumDeveloper
That's it.
You must check your implementation step by step.

( @QuantumDeveloper please check latest highlight on important parts )

from typography.

Another question:

..why you are doing decrement in callsubr/callgsubr operatos for _current_integer_count--

Ans: it is my implementation => because callsubr/callgsubr use 1 parameter , so I remove its parameter from..

Type2Instruction inst = _insts.RemoveLast();

so under _doStemCount => I need to do reduce 1 _current_integer_count.

 if (_doStemCount)
 {
        _current_integer_count--;
  }

from typography.

You can dump CFF instructions of a glyph by=>

 if (dbugCurrentGlyphIndex == 554)
            {
                _insts.dbugDumpInstructionListToFile("glyph_554.txt");
            }

from typography.

@prepare
Thanks a lot. Your answers were really helpful. I think I found all bugs in parser. At least I hope.
But I still have some questions:

1. Where in your code I can find Intrerpretation of vvcurveto, hhcurveto and other such operators? I am not sure that my implementation of interpreter process these operators fully correctly.
2. Do you know where I can find otf fonts with CFF Type1, CFF2 Type1, CFF Type2, OTF with TTF inside? Because for now all fonts I can find is CFF with Type2.
   Would be very appreciate for help.

from typography.

1. Where in your code I can find Intrerpretation of vvcurveto, hhcurveto and other such operators?

see: https://github.com/LayoutFarm/Typography/blob/master/Typography.OpenFont/Tables.CFF/CffEvaluationEngine.cs

for example vvcurveto >

for example hhcurveto >

2. Do you know where I can find otf fonts with CFF Type1, CFF2 Type1, CFF Type2, OTF with TTF inside? Because for now all fonts I can find is CFF with Type2

I don't have a complete set. Most otf fonts (I have) are CFF Type2

please check these further

• "Adobe Type Tools" https://github.com/adobe-type-tools
• "Adobe Font Development Kit for OpenType" https://github.com/adobe-type-tools/afdko
• "FreeType" https://www.freetype.org/ , https://github.com/freetype/freetype

from typography.