Remove dev dependencies from PyPI constraints.#53631
Conversation
potiuk
added
the
all versions
boring-cyborg
bot
added
area:dev-tools
backport-to-v3-1-test
|
note - this is experimental.. I want to run it to see the impact it will bring, to be able to probably start discussion on devlist whether we can accept the risks involved (but I would like to have some real examples of consequences). |
|
If there is a risk that we hat dev-dependencies not contained in constraints... would it help to split these out in a separate constraint file and keep this just for CI purposes? Don't know otherwise if the diff is easy to be calculated... |
OMG, sorry, now reading the email thread... okay sorry - understand this is a bit more complex... |
Yeah. It's not separate file - we really want to keep dev deps in pyproject.toml - and So we want to keep that workflow - it's nice and useful (for example when you And in this case it's not even that - the problem is that those development dependencies might have their own dependnecies - and they might (but do not have to be) be production dependencies of other dependencies. Yes ... the word "dependency" is used far too often in the previous sentence 😱 In short. Yes. It's complex. |
Historically, `PyPI` constraints of airflow contained also the development dependencies of Airflow - because it was not very easy to separate and remove them from the "core" constraints. This however might not be really needed and those dev tools could be removed from PyPI constraints. There are however certain consequences: a) currently when we generate PyPi constraints we generally *know* that the dependencies of ours have been tested at some point of time at least when we released the provider - because potentially some of our dev dependencies might limit some 3rd-party deps b) if we remove devel deps when we generate the PyPI constraints this **might** change - i.e. removal of development dependencies might cause some of the transient dependencies to be upgraded to newer - potentially incompatible versions. It's not very likely but possible. This means that if we remove devel deps, there is a higher risk that constraints published for users might contain dependency versions that has never been tested in our CI. Another option considered is to use a) and b) and remove the dependencies missing in b) from a) - leaving the versions that were matching the versions used during CI. This however has limited impact on the users (just removes the "devel noise" from PyPI constraints - and the noise that should not matter when installing airflow). And it has anothe risk that some edge cases might kick-in. For example there might be a case where our dependencies add or remove their transient dependencies, which might change status of those from devel to prod or the other way round. And it's quite a bit more complex.
potiuk
force-pushed
the
remove-dev-dependencies-from-pypi-constraints
branch
from
2959e09 to
afcd0a2
Compare
jscheffl
removed
the
backport-to-v3-1-test
Historically, `PyPI` constraints of airflow contained also the development dependencies of Airflow - because it was not very easy to separate and remove them from the "core" constraints. This however might not be really needed and those dev tools could be removed from PyPI constraints. There are however certain consequences: a) currently when we generate PyPi constraints we generally *know* that the dependencies of ours have been tested at some point of time at least when we released the provider - because potentially some of our dev dependencies might limit some 3rd-party deps b) if we remove devel deps when we generate the PyPI constraints this **might** change - i.e. removal of development dependencies might cause some of the transient dependencies to be upgraded to newer - potentially incompatible versions. It's not very likely but possible. This means that if we remove devel deps, there is a higher risk that constraints published for users might contain dependency versions that has never been tested in our CI. Another option considered is to use a) and b) and remove the dependencies missing in b) from a) - leaving the versions that were matching the versions used during CI. This however has limited impact on the users (just removes the "devel noise" from PyPI constraints - and the noise that should not matter when installing airflow). And it has anothe risk that some edge cases might kick-in. For example there might be a case where our dependencies add or remove their transient dependencies, which might change status of those from devel to prod or the other way round. And it's quite a bit more complex.
Historically, `PyPI` constraints of airflow contained also the development dependencies of Airflow - because it was not very easy to separate and remove them from the "core" constraints. This however might not be really needed and those dev tools could be removed from PyPI constraints. There are however certain consequences: a) currently when we generate PyPi constraints we generally *know* that the dependencies of ours have been tested at some point of time at least when we released the provider - because potentially some of our dev dependencies might limit some 3rd-party deps b) if we remove devel deps when we generate the PyPI constraints this **might** change - i.e. removal of development dependencies might cause some of the transient dependencies to be upgraded to newer - potentially incompatible versions. It's not very likely but possible. This means that if we remove devel deps, there is a higher risk that constraints published for users might contain dependency versions that has never been tested in our CI. Another option considered is to use a) and b) and remove the dependencies missing in b) from a) - leaving the versions that were matching the versions used during CI. This however has limited impact on the users (just removes the "devel noise" from PyPI constraints - and the noise that should not matter when installing airflow). And it has anothe risk that some edge cases might kick-in. For example there might be a case where our dependencies add or remove their transient dependencies, which might change status of those from devel to prod or the other way round. And it's quite a bit more complex.
2 participants
Historically,
PyPIconstraints of airflow contained also the development dependencies of Airflow - because it was not very easy to separate and remove them from the "core" constraints.This however might not be really needed and those dev tools could be removed from PyPI constraints.
There are however certain consequences:
a) currently when we generate PyPi constraints we generally know that the dependencies of ours have been tested at some point of time at least when we released the provider - because potentially some of our dev dependencies might limit some 3rd-party deps
b) if we remove devel deps when we generate the PyPI constraints this might change - i.e. removal of development dependencies might cause some of the transient dependencies to be upgraded to newer - potentially incompatible versions. It's not very likely but possible.
This means that if we remove devel deps, there is a higher risk that constraints published for users might contain dependency versions that has never been tested in our CI.
Another option considered is to use a) and b) and remove the dependencies missing in b) from a) - leaving the versions that were matching the versions used during CI. This however has limited impact on the users (just removes the "devel noise" from PyPI constraints
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.