Comments (12)
Here is the signature of AIOKafkaProducer:
def __init__(self, *, loop=None, bootstrap_servers='localhost', ...)
The bootstrap_servers
parameter has a default value of localhost
, so the type inference system infers it as a string (and throws an error when a list of strings is passed in).
This can be confusing for many users who use the type system. Can we prioritize adding type annotations for these classes?
from aiokafka.
#981 should I close this PR?
I hope to find time on weekend to look through it.
from aiokafka.
Hi @odysa, thank you for your willingness to contribute! Probably we need more communication with interested people to discuss problems. Let me highlight some difficult places:
- It would be nice to have some way of verifying that typing is correct.
mypy
doesn't work well with partially typed code, as it assumeAny
everywhere instead of type inference. Probably some way of runtime verification is needed. - It's better to start from low-level modules, that don't rely on other untyped parts of aiokafka. One of such packages is
ptotocol
package. Right now it has with methods in subclasses not following initial signature, which doesn't work when type checked. - One of features, an ability to provide custom serializer/deserializer requires making high-level classes generic, which is highly unwanted. We have to decide, what to do with this.
from aiokafka.
from aiokafka.
Yes, we're interested in adding type hints. I'd like to highlight some issues here.
- Incorrect types are often worse that missing types
- Gradual approach doesn't work good here: you never sure types are correct and consistent, until everything is annotated
- Some features like custom serialization functions don't allow us to provide good annotations. So, we may want to consider deprecating such features first.
from aiokafka.
Thank you for bringing up those concerns. I understand the importance of ensuring that type hints are accurate and consistent. Regarding the Gradual approach doesn't work good here
point, I believe we can still make progress by prioritizing certain areas for annotation while acknowledging that some parts may remain untyped initially.
By leveraging tools like MyPy, we can focus on annotating sections of the codebase where types are clearer, thereby incrementally improving the overall typing coverage. We can then gradually address the more complex or ambiguous sections, such as the custom serialization functions, as we gain more clarity on their typings.
from aiokafka.
#981 should I close this PR?
from aiokafka.
Beartype could be of interest here. I've successfully used it in unit tests to increase confidence that type hints are accurate. It can be injected by using an early stage pytest hook.
https://github.com/beartype/beartype
from aiokafka.
@antonagestam, thanks for the suggestion! It looks interesting, definitely worth to try. Do you use it via pytest-beartype or in some other way?
from aiokafka.
@ods Currently I use the decorator, mostly because the code-base is too large for it to be feasible to introduce it everywhere at once, and we don't want to enable it for all test suites. After some initial acclimatization I hope we'll be able to use one of the import hooks.
from aiokafka.
As a user of aiokafka, I like typing hints. I want to contribute to the typing feature. Where should I start from?
from aiokafka.
@ods please reopen this issue :)
from aiokafka.
Related Issues (20)
- Inconsistent producer start/stop
- AIOKafkaAdminClient.create_topics fails randomly because it selects a node at random HOT 1
- performance degradation of producer when having many topics HOT 1
- Can I use Azure Event Hubs with `aiokafka`? HOT 1
- 0.9.0 zstd codec depends on cramjam but missing in documentation HOT 1
- Let's put `aiokafka` under the `aio-libs` org on PyPI HOT 2
- Add create_acls function for kafka admin client
- AIOKafkaProducer failed to produce message with headers HOT 3
- I keep getting MessageSizeTooLargeError, error message gives size much bigger than actual message were given to producer. HOT 9
- Add delete_records to the admin client HOT 5
- Can't connect to kafka docker HOT 1
- asyncio.exceptions.CancelledError
- Regarding Kafka Connection
- invalid Type AioKafkaAdminClient create_partitions
- Consumer stopped consuming, task Fetcher._fetch_task has finished HOT 8
- High Incoming request sum on Azure Event Hub
- [QUESTION] Unable connect to node with id: X: [Errno 111]: Connection refused
- admin client - failure to create topics (error code 41) HOT 3
- [QUESTION] How to get old messages from topics? HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aiokafka.