Comments (61)
@PerBothner @christianparpart @wez I was hoping we could all agree on how to implement cursor placement after emitting a sixel. As far as I can tell, foot, DomTerm, Contour and Wezterm all place the cursor on the same row as the last sixel. But do you follow the DEC algorithm, and place it on the same row as the upper pixel of the last sixel, or do you place it on the last row touched by the sixel (i.e. the row containing the bottom pixel of the last sixel).
I know at least some of you have been following the discussions between @hackerb9 j4james, but I don't know what you ended up implementing. From an application point of view, I think it would be beneficial if we all implemented the same cursor placement algorithm...
Foot currently implements the DEC algorithm, but I think it would be easier for applications if I changed it to just place the cursor on the last row. Then, to print text under the sixel, you know all that's needed is (always) a single newline. Not one or two.
But, I think it's a bad idea to change foot if all other sixel terminals implement the DEC algorithm, and don't want to change.
from chafa.
from chafa.
Won't it affect drawing an image with P2=0 over another?
That's a valid question. Not sure if @hackerb9 has any insights on what the real VT340 does? I would kind of make sense to only trim when
P2=1
.
Definitely a good question, though straying a bit from the issue nominally at hand (newlines: graphical and otherwise).
I just ran a test of p2 effects on overlaying graphics and the results surprised me.
![](https://raw.githubusercontent.com/hackerb9/vt340test/main/sixeltests/p2effect.png)
The rules for overlaying graphics seem to be:
- If transparency is on (P2 = 1), everything is composited as expected regardless of the setting of RA.
- If transparency is off (P2 = 0) and a size is specified in RA, a rectangle of that size is cleared and the cursor is moved back to the starting corner (top left of image) before drawing sixels. This does not affect the final text cursor.
- If transparency is off (P2 = 0) and a size is not specified by RA, then a rectangle of the background color is cleared from the cursor position to the bottom right corner of the screen. This also does not affect the text cursor.
№ 3 was the most surprising to me, but I guess it makes sense for a sixel parser: if you don't have any guess what size the graphics actually are, but you know there's an opaque background that must be cleared first, set the RA size to maximum.
This behaviour also fits with how the documentation talks about the RA size parameter not being the actual geometry of the sixel image but rather an easy way to clear a rectangle. (You can see that in my test because I made the 20x20 image have a 60x60 RA size, which matters when transparency is off, P2=0).
It was also interesting to me that the Raster Attribute size had no effect on the final cursor position. I'm not sure what the benefit is, but I think perhaps it makes sense since multiple Raster Attributes are allowed in a single sixel DCS string.
If you want to test your terminal emulator of choice, you can get my script from here: https://raw.githubusercontent.com/hackerb9/vt340test/main/sixeltests/p2effect.sh . I'm curious to know the results.
Footnote
Footnote: I think of the VT340 as lacking the rectangle operations that existed in later terminals like the VT4x0. I'm not sure if I ever quite grasped before that that there is actually an easy way to clear rectangles on the VT340. (And the rectangle doesn't even have to align to the character cell! --- not sure if that's a bug or a feature.)
from chafa.
@PerBothner Imho xterm.js currently keeps the text cursor at the row of the bottom-most pixel drawn from last sixel band. Means if the last band contains only "fiftel" (6th pixel never set), the 5th pixel would be the last one, not the sixth anymore.
I did this to allow to print pictures in non 6-multiple px height and still properly align them at the bottom w'o nonsense excess row or excess space at the bottom.
(There is still a bug attached to it, where empty sixel bands at the end might get truncated - jerch/node-sixel#58)
from chafa.
following along for notcurses, good to see this effort taking place
from chafa.
@j4james I don't believe there is a "standard version" of Sixel. That is part of the problem: Different implementations act differently. Is "standard Sixel" whatever DEC implemented in their terminals? Are all such terminals consistent? What about the specifications (manuals) from DEC? What about corner cases not convered in the manuals? What about xterm - and which version of xterm? If all of these were consistent, I'd consider that as "standard sixel" - but I'm pretty certain that is not the case,
from chafa.
maybe i'm misunderstanding the need, but in notcurses i handle what i believe to be your problem by getting the terminal size in pixels, dividing that out by the number of rows and cols, and using those as the cell pixel dimensions. doesn't this provide you enough?
It's sufficient, but not ideal (click to expand summary):
- The final sixel band can fall entirely within the final cell row, or it can fall in multiple cell rows (most likely the final two - but technically if your cells are <= 4px tall, it could be more). When the terminal leaves the cursor at the row containing the topmost pixel of the final sixel band, it means you'll have to move the cursor down by one or two (or perhaps more) rows to get clear of the image. This seems more complex than necessary.
- You need an interactive terminal session that can report its pixel size (ioctl or control sequences). A tool like
convert
can't produce a sixel image occupying a consistent cell extent. - There's probably a race condition where the application gets the terminal dimensions, and while it prepares the image, the terminal's cell size changes. "Zoom" accelerators like the ones VTE implement (C-S-+ and C-S--) allow the user to trigger this easily during animations.
To be clear, I'm not asking for anything in particular to be done about this, just that it's taken into account if terminal maintainers are making changes anyway. IMO, a broad consensus is more important than any of these concerns. Also, as @hackerb9 suggested, we should probably leave points 2 and 3 for a separate issue :-)
from chafa.
It may be a good idea to get @arakiken and the other mlterm developers on board too. I've been testing with it, since it had one of the first implementations, and is still one of the fastest. It currently (as of version 3.9.3) places the cursor on the row immediately after (that is, the first character row not touched by any sixel, transparent or not).
My main concerns as an application writer are a) consistency between terminals and b) simplicity of design. I'll happily support any consensus terminal developers arrive at.
Imho xterm.js currently keeps the text cursor at the row of the bottom-most pixel drawn from last sixel band. Means if the last band contains only "fiftel" (6th pixel never set), the 5th pixel would be the last one, not the sixth anymore.
I favored this approach at first, but it has the minor annoyance of deliberate image transparency being cut off. It also means applications must inspect the image data in order to know where the cursor'll end up, which is a slightly bigger problem. Correct me if I'm wrong and there's a way around this.
from chafa.
Thanks (@dnkl, @dankamongmen and @jerch) for the clarifications and suggestions. I guess I can work with those.
EDIT: ... as regards cursor horizontal placement.
from chafa.
related, but perhaps worth its own issue; chafa currently ends the sixel with a GNL ('-'). Is this intentional? It adds an extra, empty, graphical row. I think it would be better to use a textual newline instead. Fwiw, this behavior has a (very) minor performance impact on foot, for sixels with an explicit width/height, as we're forced to reallocate and enlarge the backing image buffer, and then initialize it to the background color. I'm not really bothered by it, but thought it might be worth mentioning at least. Be happy to move this to a separate issue if you'd prefer that.
I don't remember exactly how intentional it was, but when I wrote most of the encoder back in 2018 I had to work around issues in existing decoders. For instance, I specify the raster dimensions but still make sure to pad every sixel row to the full width, since I noticed a case where the terminal would have garbage in the image buffer otherwise. It's possible the GNL was required by a decoder at some point.
That said, after testing it again now, it seems e.g. mlterm behaves the same with and without the GNL; I think it opens a new sixel row only when its pixel data starts arriving. I don't know of anything that needs the final GNL anymore, so I'll remove it.
I'm also partial to the idea that raster attributes should preempt dynamic resizing. It makes things more predictable for everyone.
from chafa.
I'm glad to see all the terminal developers here working together!
If I can summarize, it sounds like everyone is in agreement that modern terminals should allow what I will refer to as splat-nl-print: Applications may send sixels to a screen and simply send a newline before any text if they do not wish to overlap the graphics. Although VT340 compatibility is not the highest priority, I can add that my tests show splat-nl-print as the algorithm of choice even on a real VT340 as the occasional glitch is vanishingly rare in actual usage.
Additional points brought up:
-
Should the width and height specified in the Raster Attributes (RA) be used as a clipping box despite DEC's documentation explicitly stating RA does not limit the size of the image? Personally, I think, "Yes". It is a reasonable optimization for modern terminals when there is exactly one RA present in the sixel data stream. However, I would also hope modern terminals would be robust enough to fall back to unoptimized rendering when necessary — for example, no RA in the image, multiple RAs, an RA with zero width / height, or data where the program doesn't know the size ahead of time. (Sidenote: I do not expect any modern emulator to be able to handle @jerch's endless scrolling sixels.)
-
Should the text after a new line overwrite transparent pixels at the bottom of the graphic? I believe so unless the RA width and height specify otherwise.
-
Should Graphic New Line scroll the screen immediately before pixel data is received? Yes, I think that is correct. And applications encoding sixels should not output a final Graphic New Line at the end of the stream.
-
How can positioning of text to the right of a sixel image be made easier for application developers? I agree that a new issue should be created to discuss this. (If someone does, please @ me in the discussion as I'm curious about possible solutions.)
from chafa.
Have you tested recent versions of xterm? I think it is desirable to be compatible with xterm. It may be a good idea to contact Thomas E. Dickey, the maintainer of xterm. He has tweaked the handling of Sixels in the past, and may be open to (if necessary) doing so to match the "saner" behavior.
from chafa.
Looking at Xterm's source code reminded me of a minor point: while a text newline ('\n'
) is the preferred separator between an image and text, there are other options.
-
Index (EscD) after a sixel image behaves the same as a newline except that the column is not reset to zero.
-
Cursor down (Esc[B) after a sixel image behaves the same as Index in that it keeps the column position, however Down differs in that it will not cause the terminal to scroll when it hits the bottom of the screen.
-
Graphic Newline (a - before the DCS String Terminator Esc\) is not as useful, but it is commonly seen in encoders, such as ImageMagick's convert. One nice feature of DEC's algorithm for handling Graphic Newlines on the VT340 is that when an image is a multiple of 20 pixels high (the line height), a single GNL has the exact same effect as a single text newline. That is, the text that follows is put directly underneath the image with no gap and no overlap.
-
Nothing is also a possibility, but I do not know of what utility it is, if any. DEC clearly had some idea, though, as its behavior is not as simple as "always overwrite from the top left".
Again, this is minor. Reverse engineering and implementing the "correct" behavior of any of these is low-priority compared to getting text newlines working the same everywhere.
UPDATE: Thank you to @j4james for pointing out that "newline" is ambiguous. I mean the C '\n'
character which moves the cursor down and to the leftmost column. UNIX systems encode newline as ASCII LF, 0x0a
, but other operating systems may be different.
UPDATE: Another thank you to j4james for correcting me that Down does not scroll the screen and that one must use Index. I have updated the information in the list above.
from chafa.
from chafa.
- Cursor down (Esc[1B) after a sixel image behaves the same as a newline except that the column is not reset to zero. Likely any terminal which has implemented Newline will already have Down working as well.
Just a point of correction here. CUD
won't trigger a scroll at the bottom of the page, but NEL
will (assuming that's what you mean by Newline). If you want to move down and scroll without changing the column offset, you can use IND
, LF
, VT
, or FF
(although the latter three would require that LNM
is reset).
from chafa.
@ all and some in particular:
Idk why the discussions on these things always have to heat up. Its also tiresome to get the important bits filtered out of the rants, and it waters completely the goal to find a model, that works for both sides, current TEs with their technical limitations and app side for screen state control and convenience.
from chafa.
As I understand it, the controversial issue is: How should applications and terminals handle this common case: How to print an image (with no mixing of text and image on the same line), and then move the cursor to the first line below the image (that does not overlap any of the pixels)?
In "standard sixel" you can so this if the application knows how many sixel image lines there are per text line - the text height in "logical pixels". But this adds some complication and isn't necessarily portable. The "standard" has the awkward (somewhat useless) behavior that the text cursor is moved to match the line of the top sixel line of the last set of sixels, which means the application may need to send one or two newlines.
One idea: Define a "fresh-line" escape sequence (defined below). Then the application writes the image, followed by CR, then LF, then "fresh-line".
"Fresh-line" has no effect if the cursor is at the first column of an empty line, where "empty" means no characters and no overlap with a previous image. Otherwise, it is equivalent to CR+LF.
"Fresh-line" should be an escape sequence that behaves like CR+LF in terminals that are not aware of it. One possibility is:
\r\e[1;9D
The 9
is a modifier that tells the terminal to ignore the command if at the start of an empty line.
The fresh-line sequence is useful in other applications besides sixel output. For example a shell (or other REPL) can emit it before a prompt if it is unsure whether the previous command properly ended with a CR+LF sequence. (Though in this case the default is probably wrong: Assuming most commands correctly end with CR-LR, we want fresh-line to be ignored on terminals that don't recognize it. Perhaps we want to specify two alternate encodings for fresh-line: one that is ignored if not recognized; one that CR+LF if it is unrecognized. DomTerm implements \e[20u
for the former.)
from chafa.
As I understand it, the controversial issue is: How should applications and terminals handle this common case: How to print an image (with no mixing of text and image on the same line), and then move the cursor to the first line below the image (that does not overlap any of the pixels)?
Fortunately, there is not much genuine controversy on that point: just send a single newline, '\n'
. Easypeasy.
Click to read Hackerb9's humble opinion
The reason there seems to be controversy is that the vt340 can have an extremely rare quirk where the text will overwrite a few pixels at the bottom of the image. Depending upon your goal for the terminal emulator you are writing, this may or may not be important. It's not a major graphical problem and it almost never happens. Still a terminal which wants to be a faithful clone of the VT340's behaviours would of course care about this nuance. The cost of attempting to replicate it are high, causing other design trade-offs and adding complexity not just for the terminal developers but also for application programmers. Terminals which aim to be useful in modern times would be well advised to skip the quirk.
I believe it was not a design goal but a compromise. The VT340's "top pixel" heuristic is a quick approximation for a calculation that was too expensive at the time: "bottom pixel". Fortunately -- or perhaps by design -- the VT340's character cell height of 20 pixels makes that heuristic work just like "bottom pixel" in nearly every case.
I had thought this glitch was a bug in the VT340 but, after looking into it deeply, I am actually extremely impressed with the engineers from DEC. They came up with a clever solution that nobody noticed at the time was any different from the correct calculation. DEC's lack of documentation on this point would be surprising given how thorough the manuals are until one realizes it was probably omitted on purpose. If people knew the trick the VT340 was using, they might start relying on the quirky behaviour and future terminals would be obligated to support it.
Modern terminals have no need to approximate the calculation of the bottom-most opaque pixel as processors are not as limited as they were in the 1980s.
Even if they wanted to, there is no benefit to trying to extrapolate what the VT340 heuristic would be in modern times. Whatever it is, it is certainly not just picking the top pixel as that doesn't work for other character cell heights. Trying to salvage it by presuming all character cells are 10x20 regardless of the font size causes a cascade of other problems, the worst being that high res images require making the font size imperceptibly small.
And, even if some terminal did implement a heuristic that worked at any font size, it would be useless to application programmers. Calculating when to send two newlines is unnecessarily complicated and sometimes not even possible.
Consider the case where a program wants to display files that contain sixel screen dumps, perhaps captured by the VT340's MediaCopy. Since each file can contain an arbitrarily sized region, the program doesn't know ahead of time how high the image is in pixels. The only sane thing to do would be to send a single newline and presume one is enough to get the text cursor to a free line. This works on a VT340 so close to always that it isn't worth it to try to work around the occasional glitch.
In summary: We're talking about a very minor and rare graphical glitch that can occur on the VT340. While interesting from a historical perspective, only a precise VT340 emulator needs to care about such quirks. There is no benefit to copying this behaviour of the VT340 to modern terminals and much harm.
@PerBothner: Although not appropriate for sixel graphics, I could see your fresh-line proposal being useful for other situations, such as to make sure the prompt is located correctly after a program dies abruptly.
@hpjansson: To not return to the first column after displaying sixels, use IND, '\eD'
, instead of newline. If a terminal has newline working correctly, then IND should work, too.
from chafa.
Just emit IND once, same as a newline. This conserves DEC's sixel design.
Okay - I'll do that (and unless I've misunderstood something, accept that a few pixels may get cut off). I'll get out of your hair now so you can discuss the other aspects (e.g. should raster attributes define a clipping rectangle? :-) Enjoying the conversation.
from chafa.
should raster attributes define a clipping rectangle?
Good question. I've already said I think it's a reasonable, if not ideal, optimization even though it clearly violates both DEC's documentation and actual hardware behaviour.
I should ask, though, does anyone have a good hypothesis for why DEC repeatedly stated that sixel images can extend beyond the rectangle defined by RA? What is lost by taking this optimization?
My working theory had been that DEC probably wanted RA to define a clipping box but their hardware wasn't up to the task. However, that kinda falls apart when I look into it as their "GPU" (DRAGON) actually featured multiple viewports that might have done the job in no time. And, if jerch's results apply, using clipping could have actually made the VT340 run quite a bit faster, not slower. But do they apply? Would the VT340 have seen a significant speed benefit?
@jerch, when you say you get a 40% speed boost, what exactly was the bottleneck? Memory pressure from dynamic allocation of large rectangles?
from chafa.
@hackerb9 how does trailing GNLs interact with last transparent rows being clipped? One way of looking at final, trailing GNL, is that it is a completely transparent sixel row (and thus that it should be removed). But perhaps it's more correct to say that a GNL should be treated as a fully opaque row, until you start printing sixels; then you start tracking the bottom-most opaque pixel.
when you say you get a 40% speed boost, what exactly was the bottleneck? Memory pressure from dynamic allocation of large rectangles?
I can obviously not speak for @jerch , and I, too, am very curious. However, for me, there's no 40% speed boost just from allowing the raster attributes to act like a clipping region. Foot allocates the entire backing memory when the raster attributes is set. We still have to check for "overflows" (either increase image size if the sixel cursor goes beyond the raster attributes, or ignore the sixel). Thus, it makes very little difference while processing sixel characters.
There would be a small performance gain, in that we wouldn't have to reallocate the backing image when we encounter "sloppy" encoders that emit a trailing GNL, that triggers a vertical resize.
Treating it as a clipping region does simplify things though. And, almost removes the need to scan for last-opaque sixel row ;)
But, I'm fine with either way.
from chafa.
@hackerb9 how does trailing GNLs interact with last transparent rows being clipped?
Before I get into the weeds about a trailing graphic newline, I do want to say that I think GNL is not as important as getting the text newline behaviour consistent across modern terminals.
One way of looking at final, trailing GNL, is that it is a completely transparent sixel row (and thus that it should be removed). But perhaps it's more correct to say that a GNL should be treated as a fully opaque row, until you start printing sixels; then you start tracking the bottom-most opaque pixel.
Click to see hackerb9's pondering of GNL
@j4james is most knowledgeable of precise VT340 behaviour and may even know the exact algorithm for 20 pixel tall fonts off-hand.
For modern terminals, I think perhaps a better question would be why did DEC choose the algorithm they did for the VT340? We've already seen that sometimes they developed fast but inexact algorithms to overcome hardware limitations, so what benefits did the algorithm they chose for the VT340's GNL provide to programmers and users at that time?
With the caveat that I haven't thought this out as deeply as I have text newlines, here's my current take on GNL:
EFFECT OF A TRAILING GRAPHIC NEW LINE ON TEXT CURSOR POSITION
Previous image height | Behaviour |
---|---|
Exact multiple of text height | Cursor is moved to the blank line under the image |
Anything else | Has no effect (usually) |
It seems that a trailing GNL is practically useless to current programmers as the following text will almost always overlap. The one case it is sure to give a fresh line is not terribly useful since a text newline works the same and is more general.
I don't know the design parameters DEC was constrained by, but it looks an awful lot like an attempt at backwards compatibility. Historically, sixels were designed for printers and teletypewriters in which GNL represented advancing the paper by a fraction of the usual line height.
Excerpt from LJ250 Printer Programmer's Reference Manual
6.3.2.4 Graphic New Line (
-
) The graphic new line (GNL) control code (2/13) sets the active column to the [graphic] left margin and advances the paper by the current sixel height.
Since the fractions can add up, it makes sense that some programmers may have relied on printing images at a multiple of the line height and sending a final GNL to move the printhead to the next (whole) text line instead of using an explicit LF. Perhaps this was a common programming idiom and DEC wanted to make sure it still worked on video terminals.
A possible critique and response
One problem with this theory is that printer-terminals, unlike video-terminals, might have been able to print a fraction of a line down so sizing images to a multiple of the line-height might not matter. Response: It's also possible that being aligned to whole lines was important if not 100% necessary. For example, the manual for the DEC LA100 printer has a caution about using Partial Line Down:
The PLD sequence does not modify the active line. To avoid losing the top of form reference send an equal number of PLU sequences to the terminal.
Another possible reason aligning to whole lines may have been important back in those days was that green bar paper was common, but that seems weak to me.
Even if my above theory is correct, one thing I don't get is why not always advance the text cursor? What, if any, benefit is there to have a trailing GNL stay on the same line?
My first thought was that perhaps the fractional page motion was saved and would be used to align any following sixel images, but no, they overwrite the previous image just like text does. Speed of calculation is likely part of it, but what exactly were they trying to calculate? I suspect this is a historical mystery which won't be solved until someone documents the actual behaviour of something even older than a VT340, perhaps a DECwriter IV printer-terminal.
from chafa.
@hpjansson thanks for the explanation. i work around these three issues, but they're all valid concerns.
from chafa.
If we choose to truncate images with raster attributes, we could also choose to not trim trailing transparent rows.
Honestly, I think this approach results in the most reliable/consistent/predictable behaviour and is technically the most straightforward and efficient to implement... both for TE and app developers.
from chafa.
Wow! Ain't that something... Now, i kinda regret asking.
from chafa.
@hackerb9 thanks! That's some interesting results. I'll be doing a couple of changes in foot to better match the VT340. I'm also inclined to not make RA truncate images, but instead continue allowing images to extend beyond their RA. But combine that with trimming trailing transparent sixel rows.
from chafa.
Hi @dnkl. The VT340's algorithm should not be followed too strictly in modern terminals. Despite what the documentation implied, it uses a fast heuristic which relies on the character cell being 20 pixels tall. That algorithm was faster but included a glitch which should not be copied.
If you are designing a terminal that uses characters that are not 20 pixels tall, the algorithm does not apply and will have to be adapted in one of two ways:
- I suggest using the simpler algorithm which I believe you referred to as "bottom pixel". Such a terminal will be compatible with the way that programmers presumed the VT340 always worked and that DEC's documentation would lead a reasonable person to believe. All known original programs and sixel images for the VT340 will work correctly with that algorithm. Additionally, it is easily programmed for as one knows exactly how to draw text under any graphic: just send a newline.
- j4james has suggested limiting the sixel image resolution so that, regardless of the font resolution, each character cell shows only 10x20 pixels. This might be useful for someone who wants to run hypothetical VT340 software from thirty years ago which knows about and works around the VT340's cursor positioning quirk. One major downside is that graphical resolution is limited and the only way to increase it is to make the font so tiny it is unreadable.
I strongly believe the first method is the correct one for most modern terminals. It lets programmers easily create software that integrates graphics with character cell text interfaces, which is to me what makes sixels useful.
If you've read my discussion with j4james about whether this VT340 behavior is a "glitch", you'll see that even though he believes it is the historical behavior and thus correct for any terminal that claims to emulate a VT340, neither of us could come up with an easy solution for application programmers who want to just splat a sixel image on the screen and show some text underneath it. Since a workaround requires the application to model the internal state of the VT340, no sane program will ever intentionally use this odd behavior, whether it is technically a glitch or not.
from chafa.
@hackerb9 I don't mind changing foot to always put the cursor on the last row touched by the sixel (i.e. the bottom pixel of the last sixel).
What I don't want is slightly different behavior in modern terminals, and I was under the impression that the other "correct" terminals also followed the DEC algorithm? If not, I'd be more than happy to update foot.
from chafa.
That said, it looks like chafa isn't emitting a newline at all, so even with the tweaked cursor placement (always put it on the last row touched by the sixel), the image is sometimes cut off.
from chafa.
I agree putting the cursor on the row containing the bottom sixel row makes more sense, and I can certainly change it if that the consensus. I prefer to match xterm.js for various reasons. https://github.com/jerch - what do you think?
from chafa.
I'm open to tweaking wezterm to be more sane, assuming that there are a couple of test cases with examples of where the cursor should end up.
FWIW, I think the current cursor placement in wezterm may well be a bit of a fluke arising from re-using the iterm2 image protocol logic that preceded it rather than a conscious effort to implement the vt340 algorithm.
wezterm's logic for this (shared by iterm2, kitty and sixel handling) can be found here:
https://github.com/wez/wezterm/blob/22424c3280cb21af43317cb58ef7bc34a8cbcc91/term/src/terminalstate/image.rs#L65
the vertical position:
https://github.com/wez/wezterm/blob/22424c3280cb21af43317cb58ef7bc34a8cbcc91/term/src/terminalstate/image.rs#L166-L170
the horizontal position:
https://github.com/wez/wezterm/blob/22424c3280cb21af43317cb58ef7bc34a8cbcc91/term/src/terminalstate/image.rs#L233-L246
from chafa.
Sorry to intrude....
I just want to add that if it's possible to also consider the horizontal cursor position, it'd be really good (from the perspective of an application developer).
A unified vertical position is good enough for aligning images with text or other images vertically but not horizontally (i.e side-by-side).
Yes, it's probably possible to workaround this using absolute cursor positioning or save/restore but these are not always viable options, plus I believe the purpose of a consensus includes eliminating the need for workarounds in applications anyways.
Thank you all.
from chafa.
from chafa.
from chafa.
My understanding is the text cursor's horizontal position isn't changed at all. It only moves vertically. Put another way, it is positioned "at the beginning of the sixel", i.e in the bottom left corner of the sixel.
i went and looked at what we do in notcurses, and we do a hard cursor position after emission of any sixel. i imagine any application wanting to be portably correct will have to do the same thing, no? since they might be dealing with old terminals, or noncompliant ones, and it's not indicated via term queries? i don't want to disrupt unification, but from an app/toolkit author's perspective, i don't see how this helps...?
from chafa.
from chafa.
I favored this approach at first, but it has the minor annoyance of deliberate image transparency being cut off. It also means applications must inspect the image data in order to know where the cursor'll end up, which is a slightly bigger problem. Correct me if I'm wrong and there's a way around this.
No you are right. This "bottom-most colored pixel" behavior cuts a fully transparent line of pixels at the bottom as not being part of the original image. If an image has that line intentionally, it will get stripped. Thats for level 1 sixel.
Correct me if I'm wrong and there's a way around this.
Well I put another warning into the docs not to use level 1 sixel on encoder side anymore, but to go with level 2 with explicit raster attributes denoting width and height extend. DEC STD 070 also tells us, that the graphics extends in raster attributes should never be exceeded by encoders, thus my decoder uses these to trim the graphics, which also solves the issue of non multiple-of-6 image heights in a more deterministic way.
We already had several discussions about the worth of the sixel chapter in DEC STD 070 and how that deviates even from DEC's own machines. Imho DEC STD 070 is the only lengthy source from DEC, thats tries to sound normative, e.g. by implying certain limits on the sixel format, like height and width, or 256 color slots rule. Maybe they did that to get it in line with other industry standards of that time (I guess 256 colors support was at the top notch end of the 80s hardware caps), but it kinda never came into life as they soon stopped the whole sixel line.
I just want to add that if it's possible to also consider the horizontal cursor position, it'd be really good (from the perspective of an application developer).
Thats not possible with sixel level 1, it has no width idea. Every sixel band can have different sixel cursor width to the right (an image might be ragged to the right) - which one to choose from?
Sixel level 2 brings width&height with its raster attributes, so yes that could be used for a right border.
To support both conformance levels - only the start cursor offset in a line is determined, which basically leads to the VT340 cursor mode.
Btw xterm.js also uses the VT340 cursor for IIP as the only supported cursor mode to level out image sequence differences. While it is more annoying to deal with that cursor mode as app dev, if you want to place text right of the image, its handling is always the same:
- know initial cursor pos, either by tracking in your own buffer state or do a explicit CPR
- place image of x*y pixels, either with sixel or IIP
- deduct image output size in cols;rows from TEs grid resolution (to get grid resolution of the TE, either do ioctl or CSI 14/18 t)
- move text cursor by image rows/cols up/right
- write your text
from chafa.
from chafa.
I'd be more than happy to change it, to instead truncate the image to the width/height specified in the raster attributes. It'd just make everything simpler on the terminal side.
Yepp, it reduces code complexity alot, and on perf side - it is actually ~40% faster during sixel decoding because of known upper bounds prehand in my decoder.
from chafa.
If you're going to define your own version of Sixel, can you please make it something that apps can opt into or out of with a mode. Worst case, if you don't want to implement both standard and non-standard cursor placement, you could still report the mode as permanently set, and then apps can at least tell what behavior to expect from the terminal.
from chafa.
Have you tested recent versions of xterm? I think it is desirable to be compatible with xterm. It may be a good idea to contact Thomas E. Dickey, the maintainer of xterm. He has tweaked the handling of Sixels in the past, and may be open to (if necessary) doing so to match the "saner" behavior.
UPDATE I have determined that I was mistaken about Xterm's behavior regressing. In fact, it is now almost precisely correct. The one thing it is missing, however, is moving the text cursor down on Graphic New Lines, which just happens to be the default output from ImageMagick's convert
tool. @ThomasDickey.
from chafa.
Here is a new script, textcursor2.sh, which shows how a TEXT NEW LINE (or, equivalently, CURSOR DOWN) separates a sixel image from any following text on a VT340 with its 20 pixel high character cell.
It also shows what happens when GRAPHIC NEW LINE is used; the most important feature of which seems to be that it acts exactly like a single text new line whenever the image height is a multiple of the character cell height.
from chafa.
Should the text after a new line overwrite transparent pixels at the bottom of the graphic? I believe so unless the RA width and height specify otherwise.
Not sure I agree this is a good idea. Is that what the VT340 does? Or is there another reason for doing it this way?
If we choose to truncate images with raster attributes set, I don't really see the need for special casing the last line of transparent sixel bands - the encoder can simply emit a sixel with raster attributes to get an exact sized image.
from chafa.
Yes, that is the way the VT340 does it, or tries to, anyhow. To not do it that way would cause peculiar artifacts.
Consider a terminal with a text height of 20 pixels. Two images that end on the same line ought to have following text that is also on the same line. Place a 60px high rectangle at row 0 and a 40 px high rectangle at row 1 (the 20th pixel). If the terminal does not pay attention to transparent bands at the bottom, it would display the text after the 40px rectangle a line lower than the 60px rectangle.
#!/bin/bash
DCS=$'\eP'; ST=$'\e\\'; DOWN=$(tput cud 1)
clear
echo -n ${DCS}q'"1;1;10;60#1!10~-!10~-!10~-!10~-!10~-!10~-!10~-!10~-!10~-!10~'${ST}
echo "${DOWN}This is 60px high (starting at 0)"
tput cup 1 40
echo -n ${DCS}q'"1;1;10;40#2!10~-!10~-!10~-!10~-!10~-!10~-!10N'${ST}
echo "${DOWN}This is 40px high (starting at 20)"
Xterm works the same as the VT340 and, while it took some time to get it right, hopefully that means it'll be easier for other terminals which can use it as a reference implementation.
I completely understand not wanting to special case the last band of transparent sixels. It seems yucky to me, too. I believe DEC engineers had the same thoughts and went to the extreme of inventing a wacky heuristic that works swiftly (if not perfectly) by presuming character cells are exactly 20 pixels high. That algorithm does not work in general, so we may have to just OR together every sixel in the last row to see what the lowest opaque pixel actually was. I believe technology has advanced sufficiently since 1988 that it should be possible.
If ORing a few thousands bytes takes too long one could write CUDA code to do it in parallel on the GPU. 😆 .
Using Raster Attributes to truncate images seems a reasonable optimization for current hardware, but it is not ideal. And, since Sixel images aren't required to set the geometry, it is not always applicable.
from chafa.
Is "standard Sixel" whatever DEC implemented in their terminals?
@PerBothner Yes. I can't think of anyone better suited to define the standard than the people that invented the protocol.
Are all such terminals consistent?
To the best of my knowledge, yes. The VT125 and VT240 don't support sixel scrolling, so they behave as if DECSDM
is permanently set, which means the cursor doesn't move. And from the screenshots I've seen of the VT382, I believe it behaves the same way as the VT340 when DECSDM
is reset, although it has a different cell size, and doesn't support color. But I'd be happy to learn otherwise if you have more information on the subject.
What about the specifications (manuals) from DEC?
The programming manuals don't go into that detail, and the latest revision of DEC STD 070 that is publicly available doesn't cover terminal cursor positioning - only the non-scrolling sixel terminals and sixel printers. But if you're aware of a more recent version, I'd love to have access to that.
What about corner cases not convered in the manuals?
Do you have something particular in mind? As mentioned above, the manuals don't cover details like that, but I'm not aware of any corners cases that haven't already been tested on the VT340.
What about xterm - and which version of xterm?
I'm assuming you're not expecting the standard sixel behavior to somehow match every different version of xterm, but I do know that xterm is at least trying to move closer to the standard. I haven't tested recently, though, so for all I know it may already match the standard behavior.
If all of these were consistent, I'd consider that as "standard sixel"
Well they're at least damn close to being consistent, and you certainly aren't going to find another "standard sixel" that does better than that.
I just don't get what you're trying to achieve by coming up with yet another cursor positioning algorithm that doesn't match any of the actual DEC terminals, and then also hiding your incompatibility from applications so they can't compensate for it. But I'm not going to interfere any more if you all really want to go ahead with this. I just find it baffling that anyone would think this is a good idea.
from chafa.
I wouldn't mind if there were some way for the client to attach a hint indicating what kind of sixel regime it's expecting for a given image, possibly chosen from a set advertised by the terminal emulator. Ideally not just for cursor positioning, but cell size too. Then TEs could default to a cell size (e.g. 10x20) depending on their emulation mode, but allow printing of sixels emitted for a different resolution while maintaining a correct character extent by applying a scaling factor. This is clearly extra work on the TE side, but if you support changing the font size at runtime you're probably keeping track of image-to-cell scaling factors already.
Practical use cases for this come up now and then. I think someone in the MS Windows sixel support thread mentioned issues with sixel-based monitoring software still running in older power plants that had to be migrated from DEC to software TEs. So a "clean" break may be preferable. Plus it just makes sense for the image to be able to indicate its cell extent somehow.
A solution to this would have to come from TE maintainers though - I don't know how practical it'd be, or how much appetite there is for it.
Edit: I can assist with fast image scaling with no external deps, and the integration work for it, if needed.
from chafa.
I wouldn't mind if there were some way for the client to attach a hint indicating what kind of sixel regime it's expecting for a given image, possibly chosen from a set advertised by the terminal emulator. Ideally not just for cursor positioning, but cell size too.
Interesting idea, but I think we're getting a bit far afield. If someone starts an issue to discuss this, I'd like to be included on it.
Practical use cases for this come up now and then. I think someone in the MS Windows sixel support thread mentioned issues with sixel-based monitoring software still running in older power plants that had to be migrated from DEC to software TEs.
Click here to read hackerb9's digression on this.
Perhaps I'm misunderstanding here, but power plants should be fine. If they are presuming sixel graphics always extend the exact same number of character cells as a VT340 in 80 column mode, they can set their font to be a 10x20 bitmap. That already works great in Xterm.
Then TEs could default to a cell size (e.g. 10x20) depending on their emulation mode, but allow printing of sixels emitted for a different resolution while maintaining a correct character extent by applying a scaling factor. This is clearly extra work on the TE side, but if you support changing the font size at runtime you're probably keeping track of image-to-cell scaling factors already.
Again, I'm likely misunderstanding, but I hope modern TEs do not try to decouple the resolution of sixel graphics from the resolution of the font. The behaviour of sixel graphics when the terminal changes font size is undefined, but if you want to copy the VT340, the graphics should simply be erased as that's what happens when switching to 132 column mode. App programmers (at least on UNIX) will get a WINCH signal and redraw the screen. Since programmers can already determine the number of pixels per character cell, there is no extra work TE devs need to do. Why make it any more complicated?
As for coming up with a way to allow for sixels emitted at different resolution, I will briefly mention that the sixel protocol actually already has support for that. In addition to pixel aspect ratio settings which you may know about, device independent scale is defined by the ANSI SSU escape sequence (select size units) and the grid parameter Pn3. Although only DEC's printers actually obeyed it, the VT340 always generates sixel images with the correct real-world scale embedded in them so that hardcopy on any device would be exactly the same size as the screen.
Click to view excerpt from DEC-070 section 7.8
Level 2 Sixel Devices -
ASSUMPTIONS: Level 2 sixel devices support the Set Raster
Attribute command, Background Select, Horizontal Grid Size and
Macro Parameter commands.
Sixel control strings are sent as follows:
ESC P Ps1 ; Ps2 ; Pn3 q " Pn4 ; Pn5 ; Pn6 ; Pn7 ****** ESC \
\___/ \_______________/ \_____________________/ \____/ \___/
DCS Protocol Selector Raster Attributes Picture ST
data
\______________________/ \_____________________________/
DCS Introducer Sequence sixel data
Where:
-
Ps1 Is the Macro Parameter and is always ZERO.
-
Ps2 Background Select
- 1 if Background Printing is disabled in Set-Up
- 2 if Background Printing is enabled in Set-Up
-
Pn3 Horizontal Grid Size, given in units specified
by ANSI SSU (default is decipoints, 1/720 inch).
For default size units, the grid size should be
6 for COMPRESSED and 9 for EXPANDED or ROTATED print.
Since the host can change the printer between accesses,
SSU should be sent once before each sixel dump.ESC [ 2 SP I (Set Size Unit 1/11 5/11 3/2 2/0 4/9 to Decipoints)
-
Pn4 Pixel aspect ratio numerator, 1
-
Pn5 Pixel aspect ratio denominator, 1
-
Pn6 Horizontal extent
(number of pixels in image horizontally) -
Pn7 Vertical extent
(number of pixels in image vertically)
While it'd be pretty cool if some modern terminal actually implemented device independent graphics resolution based on SSU, I think developer time would be better spent doing the reverse: MediaCopy to host. This was a graphics feature the VT340 supported and I'd find it immediately useful on terminal emulators. The escape sequence requests that the terminal send an image of the screen back to the host as a sixel file. You can probably imagine how handy this would be from the command line, though of course a modern TE would want to address privacy by making it something the user has to explicitly allow.
But I digress from my digression... 😄
from chafa.
To everyone...
Just want to say I think digressions are okay (though, should be avoided when possible) but I've noticed and would like to encourage @hackerb9's style of puting them (and replies to them) within <details>
(with <summary>
) tags in order to ease the filtering of important bits.
Thank you all.
from chafa.
That's an interesting proposal, though ideally I'd wish for something that can be used without returning to the first column, and which doesn't require a blank row. Chafa (by request) has a --relative
switch that's supposed to print output at the current cursor location, leaving it immediately below the bottom-left corner. This works for symbol graphics, but the promise is hard to uphold for other image protocols.
from chafa.
@hpjansson: To not return to the first column after displaying sixels, use IND, '\eD', instead of newline. If you have newline working correctly, then IND should work, too.
Right - but assuming a DEC-faithful TE, I would have to emit IND once or twice, depending - or rely on some extension such as @PerBothner's suggestion. The central question is "can we conserve DEC sixels but do something else to obviate the need to know where the last sixel band fell in relation to text cells?"
from chafa.
Right - but assuming a DEC-faithful TE, I would have to emit IND once or twice, depending - or rely on some extension such as @PerBothner's suggestion. The central question is "can we conserve DEC sixels but do something else to obviate the need to know where the last sixel band fell in relation to text cells?"
Just emit IND once, same as a newline. This conserves DEC's sixel design.
from chafa.
Right - but assuming a DEC-faithful TE, I would have to emit IND once or twice, depending - or rely on some extension such as @PerBothner's suggestion. The central question is "can we conserve DEC sixels but do something else to obviate the need to know where the last sixel band fell in relation to text cells?"
maybe i'm misunderstanding the need, but in notcurses i handle what i believe to be your problem by getting the terminal size in pixels, dividing that out by the number of rows and cols, and using those as the cell pixel dimensions. doesn't this provide you enough?
from chafa.
does anyone have a good hypothesis for why DEC repeatedly stated that sixel images can extend beyond the rectangle defined by RA? What is lost by taking this optimization?
Infinite scroll would be the most obvious example (I'm sure we discussed this somewhere before but I can't find it in your repo right now). You'd also lose some bandwidth saving tricks that could be beneficial when working with non-rectangular output. You can see the sort of thing I mean in the raster dimension tests.
from chafa.
Alright, I now have three open PRs for foot, addressing the following:
-
Place cursor on the last character row touched by the sixel: this is the one I started out this ticket with.
-
limit image size to the one specified in the raster attributes: changes foot from allowing images to grow beyond the dimensions in the raster attributes. This is mostly to sync with @jerch. In the end, it didn't really offer any major benefits, and I would be just as happy to continue supporting dynamically growing the image beyond the raster attributes' dimensions. If I were to decide all on my own, I wouldn't merge this PR, and instead continue supporting dynamic resizes. Note that even with this PR, dynamically sized images are still supported, as long as they omit the raster attributes.
-
trim trailing, fully transparent sixel rows: does what it says. We haven't really discussed the nitty gritty details on this one, so, I chose to do this for all sixels, regardless of the background color mode (i.e. the P2 parameter), and regardless of whether there are any raster attributes present or not.
Is this something you all (though I guess it's pretty clear where @j4james stands on this) would consider implementing in your TEs?
Just to make it clear. I don't intend to merge any of the above (1640 being the exception) unless we can reach at least some level of consensus here.
@hackerb9 thanks again for your detailed explanation. What I ended up doing (in 1640), is to let trailing GNLs move the text cursor as if you had at least one fully opaque 6-pixel sixel on that row, but as soon as you start printing sixels, I switch to tracking whatever the actual bottom pixel is. In other words, a trailing GNL will not be trimmed out when we remove trailing, transparent sixel rows.
from chafa.
Considering images actually having trailing transparent rows, I have a couple concerns/questions as regards trimming trailing transparent rows:
- Won't it affect vertical cursor placement?
- Won't it affect drawing an image with
P2=0
over another?
from chafa.
Won't it affect vertical cursor placement?
Yes, and that's kind of the whole idea. If we don't trim, all images will be forced to have a height that is a multiple of 6.
If we choose to truncate images with raster attributes, we could also choose to not trim trailing transparent rows. But if we don't truncate the image, I think trimming should be done regardless of whether the image has raster attributes or not. Otherwise, an image with raster attributes would still be forced to have a height that is a multiple of 6.
Won't it affect drawing an image with P2=0 over another?
That's a valid question. Not sure if @hackerb9 has any insights on what the real VT340 does? I would kind of make sense to only trim when P2=1
.
from chafa.
I ran the p2effect.sh script on wezterm and xterm.
Looks a bit wonky in wezterm(!)
from chafa.
@wez I think it's your system that is a bit wonky! Maybe an incompatibility with the shell? Because even the Xterm image has a whole bunch of mistakes that I'm not seeing when I run the script myself. For example, where are all those $
characters coming from? Why are the red blocks offset one column to the right? And why is the cursor
text not lined up with the ↑
character? Xterm does have a few issues, but it's not nearly as bad as it appears in your screenshot.
from chafa.
Hey @wez, I think @j4james is right about your shell. Did you get it figured out? If not, please let me know the output of bash --version
as my script should definitely not be doing that. If I recall correctly, MacOS comes with an ancient version of bash.
from chafa.
Ah, I think I was lazy and didn't chmod the script and just ran it with sh
. Running with bash explicitly gives:
xterm:
wezterm -n:
from chafa.
@wez Your Xterm screenshot still looks wrong to me. Are you sure you're using the latest version? This is what I get:
It doesn't get the cursor position right when raster attributes are set, and it doesn't set the opaque background correctly when raster attributes are not set, but otherwise it seems OK to me.
And note that you need to use a 10x20 font if you want to fully emulate the VT340, otherwise the cursor position tests are going to be misleading.
from chafa.
Related Issues (20)
- Set up Cygwin builds HOT 11
- Possible memory leak HOT 1
- xterm sixel support not detected automatically HOT 5
- Infinite redraw while previewing a pdf HOT 4
- Erratic behavior of `chafa -f iterm image.jpg` on Windows HOT 2
- kitty with tmux passthrough only works if `kitty +icat` was invoked before HOT 10
- Iterm support not detected HOT 6
- Eat 0.9 terminal supports Sixel HOT 3
- Add support for Framebuffer graphics HOT 4
- Unicode 16.0 support HOT 4
- chafa 1.14.1 fails to build on i686 HOT 9
- The images not showing correctly in Konsole terminal HOT 4
- symbols mode not working since 1.14.1 HOT 16
- Segfault when doing --symbols on WSLg HOT 3
- Chafa no longer showing images on urxvt HOT 3
- Feature request: display cover art embedded in media file HOT 1
- Crash with `-f sixels` HOT 17
- Sixels are too small in Windows Terminal HOT 26
- Aspect-preserving calculations are off in some cases
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from chafa.