Scaling

As detailed in the architecture overview Global architecture, most processes can be multiplied to handle a higher load. Only three processes cannot be multiplied in case of increased load:

• the scheduler, which only triggers tasks on the different workers,

• the sending and receiving process, which can be blocking in case of too substantial transfers. If they do not allow transferring data quickly enough, it is recommended to use a dedicated unidirectional transfer system with file transfer mode.

The other processes can be multiplied to handle a higher load by increasing the number of dedicated containers.

However, if many machines need to use the software repositories made available by InterDiode, it is probably simpler to use dedicated mirrors for this purpose.

• These mirrors are specialized for this task and can more easily absorb a significant load,

• The “transfer from the internet to the internal network” function will be more cleanly decoupled from the “internal mirror” function,

• These mirrors can also serve as internal mirrors.

Many solutions exist for this purpose. The following list is provided for informational purposes only.

• Pulp:

  • Docker containers

  • Debian-based (Debian, Ubuntu) distributions

  • Red Hat-based (Red Hat, CentOS) distributions

  • Ruby gem packages

  • Maven (Java and Java-like) packages

  • NPM packages

  • Python packages

  • Cargo (Rust) packages

• Artifactory from JFrog:

  • Ansible

  • Docker containers

  • Debian-based (Debian, Ubuntu) distributions

  • Red Hat-based (Red Hat, CentOS) distributions

  • Ruby gem packages

  • Maven (Java and Java-like) packages

  • NPM packages

  • Python packages

  • PHP Composer packages,

  • Cargo (Rust) packages

  • Git (with LFS) repositories

  • Go modules

  • Helm charts

  • NuGet (.Net) packages

• Nexus Repository from Sonatype:

  • Ansible

  • Docker containers

  • Debian-based (Debian, Ubuntu) distributions

  • Red Hat-based (Red Hat, CentOS) distributions

  • Ruby gem packages

  • Maven (Java and Java-like) packages

  • NPM packages

  • Python packages

  • PHP Composer packages,

  • Cargo (Rust) packages

  • Git (with LFS) repositories

  • Go modules

  • Helm charts

  • NuGet (.Net) packages

• apt-mirror, debmirror, aptly for Debian/Ubuntu packages

• Docker Distribution, Harbor or regsync for Docker containers.