Broken Docker image about ngsi-timeseries-api HOT 3 CLOSED

So we should really move away from Python 3.8, but that's too much effort. An avenue we can explore is to upgrade pyyaml from version 5 to version 6. With this upgrade I could make some progress locally, it remains to be seen if that works on CircleCI too. (I still need to upgrade other deps though.) The reason we can't upgrade to version 6 without pain is PR #516 which added a ~=5.4 constraint to the Pipenv dev deps

• https://github.com/orchestracities/ngsi-timeseries-api/pull/516/files#diff-230078d672f10d17463a8a6265cad825b790885898256a3365be90685caac58dR40

But I think that was a useless thing to do anyway. In fact, the same PR installs the latest pyyaml version on every CircleCI test session run:

• https://github.com/orchestracities/ngsi-timeseries-api/pull/516/files#diff-78a8a19706dbd2a4425dd72bdab0502ed7a2cef16365ab7030a5a0588927bf47

which means at the moment we get version 6 on CircleCI. So what's the use of adding the ~=5.4 constraint to the Pipenv dev deps? Also the main section of the Pipenv file requires pyyaml >= 4.2. So it looks like something fell through the cracks here. My gut is that we should undo this change

• https://github.com/orchestracities/ngsi-timeseries-api/pull/516/files#diff-230078d672f10d17463a8a6265cad825b790885898256a3365be90685caac58dR40

and pin pyyaml to version 6 in the main section of Pipfile.

from ngsi-timeseries-api.

Ideally we should upgrade requests and redis too b/c of security vulnerabilities---see #729 and #724.

from ngsi-timeseries-api.

Notice that the upgrade path isn't straightforward. The problem is that, like us, other Pythonistas out there also have dependency hell issues. Here's an example. If you delete the Pipenv lock file and try regenerating it (or just upgrade deps) you'll see that all the QL tests will break. One of the errors is this

ImportError: cannot import name 'soft_unicode' from 'markupsafe'

(see pallets/markupsafe#282)

Why is that happening? Well, our deps declaration in the Pipfile lead to the following dependency tree---just showing the paths that contain the markupsafe package.

connexion 2.14.2 Connexion - API first applications with OpenAPI/Swagger and Flask
├── flask >=1.0.4,<2.3
|   ├── jinja2 >=2.10.1,<3.0
|       └── markupsafe >=0.23
├── swagger-ui-bundle >=0.0.2,<0.1
    └── jinja2 >=2.0
        └── markupsafe >=0.23

flask 1.1.4 A simple framework for building complex web applications.
├── jinja2 >=2.10.1,<3.0
    └── markupsafe >=0.23

pytest-flask 1.2.0 A set of py.test fixtures to test Flask applications.
├── flask *
    ├── jinja2 >=2.10.1,<3.0
        └── markupsafe >=0.23

With these dependency constraints, the dependency solver can pick any version of the jinja2 package greater or equal to 2.10.1 but less than 3 and any markupsafe version greater or equal to 0.23. So the solver picks

• jinja2 2.11.3
• markupsafe 2.1.3

But in actual fact, jinja2 2.11.3 is incompatible with markupsafe 2.1.3 because it tries to import soft_unicode from markupsafe but soft_unicode isn't in markupsafe 2.1.3. The last version containing soft_unicode is markupsafe 2.0.1. So the jinja2 maintainers should've specified an upper version bound of 2.0.1 for markupsafe in their deps, instead of just a lower bound of 0.23:

• https://github.com/pallets/jinja/blob/2.11.3/setup.py#L53C23-L53C40

Anyways, what can we do about it? One option would be to add an explicit dependency on markupsafe 2.0.1 in our Pipfile. This isn't great for all the obvious reasons, but it's an option nonetheless. Another option would be to upgrade connexion and flask to recent versions which depend on jinja2 > 3.0---recent jinja2 versions actually work with the latestmarkupsafe version, 2.1.3. But Python being that wonderful dynamic/untyped soup that it is, I'm a bit scared of upgrading flask without extensive testing.

from ngsi-timeseries-api.