Copying Python repositories

Posted on by

While many Python packages are provided by mirrors of traditional Linux distributions like Debian, Arch, or Ubuntu, using the official PyPI.org mirror directly provides access to a greater number of packages as well as the most recent versions of standard packages.

There are no plans to copy the entire Python mirror, as this would require tens of TB, most of which would never be used. It is therefore possible to select the desired packages with the usual version constraints. For example, retrieving the TOP 8000 most downloaded packages consumes about 200 GB, with an additional few dozen GB for weekly updates.

Permission management allows users to easily suggest new packages to retrieve, which will be added to the full list after approval by an administrator.

Properly handling all Python package formats is very complex because the archives can contain many different elements and may rely on executing a setup.py file to calculate the necessary metadata. Unfortunately, running this file in InterDiode is not feasible for obvious security reasons.