Feature request: ANSI color output in doctests about documenter.jl HOT 6 OPEN

Do doctests with ANSI codes work in the REPL? Although, even if they do, they would not be very readable in the source code. I agree that it would look great in the rendered documentation, but I'm unsure about how to reconcile it with the rest of the doctest architecture.

from documenter.jl.

The doctests work, yes. And the way they work is rather nice: the ANSI codes are ignored, so colored output is compared to the plain text output. I assume this is using an IOContext with color=false.

What I'm proposing is to keep that behavior, just color the actual rendered HTML. There isn't any good way to include the colors in the "answer" part of the test anyway, so I think the part it's already doing is the right thing, and then going on to color the actual rendered output should be harmless, and generally what the user would prefer, with ansicolor=false available if they don't

from documenter.jl.

There isn't any good way to include the colors in the "answer" part of the test anyway, so I think the part it's already doing is the right thing, and then going on to color the actual rendered output should be harmless

Documenter doesn't evaluate the doctests when it's rendering them (just when it checks them). So with the current architecture, the color information needs to go into the jldoctest blocks.

from documenter.jl.

Ah, that makes sense, and it would entail a bigger change than I naively thought it would. Although it could probably reuse the most of the logic from @repl blocks, right?

Putting the color in the jldoctest doesn't strike me as a possible solution, because any markup chosen (raw ANSI strings, HTML tags, the new StyledStrings) also corresponds to a literal output which could be the result of show/repr. There's a use/mention distinction which isn't possible to reconcile in full generality.

When I filed the issue I had hoped it would be as simple as changing the IOContext color flag between testing and rendering, but this sounds more involved than that. It'd be nice anyway, but using @repl blocks when I expect color output isn't such a hardship, really. Anything which would meaningfully change the output is supposed to be covered by not-doc-tests anyhow.

from documenter.jl.

Yeah, I think technically re-executing all doctests as if they're at-repl blocks basically, would be doable (although it might involve some refactoring of the doctest code). But the semantics get weird. E.g. what would you do with doctest filters?

I'd say we could try some opt-in experiment, but it's not 100% obvious to me what this change should look like.

from documenter.jl.

That raises a good point. I haven't used doctest filters, but I just gave the docs a fresh read, and it doesn't look like anything would need to change in terms of how they function.

If I'm understanding the feature correctly, doctests are run on both the output in the .md and the output from running the code. Then when the docs are rendered, the version from the .md file is used, without any filtering being applied. This is already being done on uncolored output, and since nothing in this proposal changes the doctests, nothing can break there.

So if there's a random number being generated, the HTML will contain the exact number given in the documentation. So this feature would change behavior, since if Documenter is now using the output from running the code, the number would be whatever it happened to be while building the docs. This could definitely lead to unexpected results, for instance, that number would be different building locally and building remotely.

It's unclear to me whether this should be considered a breaking change. I'm sure Documenter makes no guarantees that the HTML will stay identical, that's far too restrictive to be practical. But this is a genuine behavior change, and it's easy for me to imagine it being unwanted. For instance, a user might have made that random number an injoke, their zipcode, 1337, or something like that, and wouldn't appreciate Documenter suddenly changing that number to be truly random.

So I would suggest that it should be considered breaking, and introduced on an opt-in, experimental basis. My sense is that the Documenter maintainers are reaching the point where a 2.0 will be forthcoming, and that would be an opportunity to reverse the default and make "your contents will be the result of executing the code, including color" the opt-out, rather than the opt-in. I don't personally have a strong opinion on which default is better, actually, just which one I would choose.

from documenter.jl.