Feature proposal: Allow specifying out as list about pytest-mypy-plugins HOT 7 OPEN

I like it!

But, I think I need to explain why current format exists (and will be supported in the future).

It main feature is that you can copy-paste "Actual" output from mypy there. I sometime do this:

1. I write some test
2. I leave out: empty
3. I run test
4. I copy-paste "Actual" output and check it to be correct
5. Done!

With this format, it will be easier to read, but harder to write.

from pytest-mypy-plugins.

@sobolevn Right, I think that makes sense too, there wouldn't be any need to replace the current format.

Just piling on with some more thoughts here:

Another possible "win" with introducing a higher level of abstraction is that the library could "take the hit" for unstable output from mypy. It would sure take work to implement, but it could possibly be a way forward for addressing issues like #45.

from pytest-mypy-plugins.

This is a nice idea, but I have a few concerns.

This probably requires more knowledge of the internals than I currently have, but I think it might make sense to allow omitting line numbers. I'm proposing that this example would match if there's revealed type on line nine, followed by an error message on any line after that (strictly greater than line 9).

This shouldn't be hard, but at the cost of further cementing match-in-order logic, which is already pretty blunt tool (#77). Also, but it is probably a lesser concern, it could result in false negatives, when

• Expected error doesn't happen.
• Unexpected, but matching error, happens at another line.

Another possible "win" with introducing a higher level of abstraction is that the library could "take the hit" for unstable output from mypy.

At the moment, dealing with changing mypy output is primarily a responsibility of the end users. It is a little bit of pain, but manageable, since you can control mypy version.

In contrast, making more sense out of mypy output within library, might require version specific parser ‒ something that can be hard to maintain, when checker output is not really part of the contract.

Not a big concern in case of the core idea, but can quickly get out of hand if you keep adding features.

from pytest-mypy-plugins.

Another possible "win" with introducing a higher level of abstraction is that the library could "take the hit" for unstable output from mypy. It would sure take work to implement, but it could possibly be a way forward for addressing issues like #45.

@antonagestam I'm also a victim of mypy changing output string over and over. Well, it's kind of expected when people have to deal with plain text output, but still. Pinning mypy version as suggested by @zero323 is a way to deal with such problem.

If you are interested in seeing more fine-grained mypy output, you can take a look at python/mypy#10816 and python/mypy#11396. However don't hold your breath for it — given the people involved in the discussion over time, I am fairly confident that mypy core dev has completely no interest and no intention to make output more accessible.

from pytest-mypy-plugins.

Well, since I am mypy core dev myself, I am pretty interested in this :)

But, this is a complex project that requires quite a lot of details to get it right.
If you have any proposals, please feel free to share them.

from pytest-mypy-plugins.

@abelcheung Oh cool, structured output from mypy directly seems like a much more viable approach.

from pytest-mypy-plugins.

@sobolevn I have a proposal in python/mypy#11396, but there has been zero core dev interaction on that PR. If you can take a look at it, not only will we get structured output but it'll also unlock future endeavours like being able to define a custom output format with a plugin, or to add new fields to the output like exact issue codes.

from pytest-mypy-plugins.