The inconsistency for the `package_data` configuration in sdists
when `include_package_data=True` in #1461 have been causing some
problems for the community for a while, as also shown in #2835.
As pointed out by
[@jaraco](https://github.com/pypa/setuptools/issues/1461#issuecomment-749092366),
this was being caused by a mechanism to break the recursion between the
`egg_info` and `sdist` commands.
In summary the loop is caused by the following behaviour:
- the `egg_info` command uses a subclass of `sdist` (`manifest_maker`)
to calculate the MANIFEST,
- the `sdist` class needs to know the MANIFEST to calculate the data files when
`include_package_data=True`
Previously, the mechanism to break this loop was to simply ignore
the data files in `sdist` when `include_package_data=True`.
The approach implemented in this change was to replace this mechanism,
by allowing `manifest_maker` to override the `_safe_data_files` method
from `sdist`.
---
Please notice [an extensive experiment]
(https://github.com/abravalheri/experiment-setuptools-package-data)
was carried out to investigate the previous confusing behaviour.
There is also [a simplified theoretical analysis]
(https://github.com/pyscaffold/pyscaffold/pull/535#issuecomment-956296407)
comparing the observed behavior in the experiment and the expected
one. This analysis point out to the same offender indicated by
[@jaraco](https://github.com/pypa/setuptools/issues/1461#issuecomment-749092366)
(which is being replaced in this change).