Wikipedia projects

InterDiode can download ZIM archives, such as Wikipedia or StackOverflow dumps, and make them available on the internal network. Users on the internal network can either search directly within the transferred archive or download the full archive and use their own viewer such as Kiwix.

Downloads can be scheduled regularly, including using regularly updated links, such as Wikipedia copies.

Copies of the various Wikipedia projects are available and updated regularly. Other projects such as StackExchange can also be downloaded.

Creation

Usage

If the link is dynamic, you can use wildcards such as * (matching any string of any length) and character ranges with [a-z] in the URL.

For example, if the provided link is https://dumps.wikimedia.org/other/kiwix/zim/wikipedia/wikipedia_en_100_mini_*-*.zim, the page https://dumps.wikimedia.org/other/kiwix/zim/wikipedia/ will first be retrieved and the last link matching the expression wikipedia_en_100_mini_*-*.zim will then be selected.

The supported expressions follow the standard UNIX glob format:

  • * matches everything,

  • ? matches any single character,

  • [seq] matches any character in seq,

  • [!seq] matches any character not in seq.

Intermediate pages will be downloaded to obtain the final link, and this can be done recursively: https://example.com/downloads/zim/202?/*/archive-202?-[0-1][0-9]-*.zim.

Updating data

By default, data update must be triggered manually and the retrieved data will be automatically transferred to the internal network at the end of this synchronization. In the settings panel, it is also possible to schedule this synchronization to be triggered automatically. The synchronization can be scheduled from every minute to once a year, including very fine-grained control to choose precisely at which minute or hour this synchronization will be triggered.

Note

Each worker can only process one synchronization at a time. If many projects need to be synchronized frequently, the number of workers must be adapted accordingly.

Access Control

Several permission levels are available and can be assigned to users individually or by groups. The accesses granted by default to anonymous or simply authenticated users can also be selected.

  • no access: has no access at all..

  • reader: can only read pages..

  • admin: can change settings and access controls..

Only local administrators can change access controls, and default permissions for anonymous and simply authenticated users are transferred to the red side only during the first synchronization.

User preferences

Each user can set their own preferences, like pining it as a favorite source or creating a authentication token specific to it.

Resolve transfer issues

If a problem occurred during a transfer and both sides are out of sync, it is possible to force a resynchronization by sending all or part of the data again.

Deletion

If you want to delete this source, you must delete on both sides, starting with the black side to avoid any risk of data transfer to the red side after deletion. Once deleted, the source will no longer be available, and the related data will be removed during background cleanup tasks.

File integrity verification

Each file is stored along with its SHA256 digest, allowing to verify the integrity of the file and to detect any changes or corruption.

When you download a file, you can check its digest against the expected value to ensure that the file has not been tampered with or corrupted during transfer. This digest is provided in the HTTP headers, both Content-Digest (following the RFC 9530) and Digest. This can be done using various tools, such as command-line utilities or programming libraries that support digest verification. The digest is calculated using the SHA-256 algorithm and is encoded in Base64 format.

curl -L -v -I [URL]
# the -I option retrieves only the headers, and the -v option shows the details of the request and response, including the headers
Digest: sha-256=VqiR0u1OsX0ZGPCDVG9mlRjei0DC76I6r1+FXEhmuIs=
Content-Digest: sha-256=:VqiR0u1OsX0ZGPCDVG9mlRjei0DC76I6r1+FXEhmuIs=:

Propositions

A user with the appropriate permission can make a proposition, which must be validated by an administrator.