Find URLs in a webpage

Download a root page and search for links matching a set of specified regular expressions.All these links are then downloaded and made available to the internal network.

Creation

Usage

  1. The root page is first downloaded and then analyzed to look for links. If the page is HTML, links are searched in the href attribute of a tags and must match one of the provided regular expressions. Otherwise, the regular expressions must be precise enough to capture only links.

  2. Any relative links are resolved against the original page.

  3. If multiple links match the same regular expression, you can choose to keep only the first one, the last one, or all of them.

  4. All collected links are then downloaded and sent over the internal network.

  5. If one of these links has already been downloaded, several strategies are available:

    1. always download it,

    2. download it only if it has changed,

    3. download it only once, even if it has changed since,

    4. download it only if it has changed, or if it is impossible to determine.

    Link validity is determined using the Expires, Last-Modified, and ETag headers.

Updating data

By default, data update must be triggered manually and the retrieved data will be automatically transferred to the internal network at the end of this synchronization. In the settings panel, it is also possible to schedule this synchronization to be triggered automatically. The synchronization can be scheduled from every minute to once a year, including very fine-grained control to choose precisely at which minute or hour this synchronization will be triggered.

Note

Each worker can only process one synchronization at a time. If many projects need to be synchronized frequently, the number of workers must be adapted accordingly.

Access Control

Several permission levels are available and can be assigned to users individually or by groups. The accesses granted by default to anonymous or simply authenticated users can also be selected.

  • no access: has no access at all..

  • reader: can only read pages..

  • admin: can change settings and access controls..

Only local administrators can change access controls, and default permissions for anonymous and simply authenticated users are transferred to the red side only during the first synchronization.

User preferences

Each user can set their own preferences, like pining it as a favorite source or creating a authentication token specific to it.

Resolve transfer issues

If a problem occurred during a transfer and both sides are out of sync, it is possible to force a resynchronization by sending all or part of the data again.

Deletion

If you want to delete this source, you must delete on both sides, starting with the black side to avoid any risk of data transfer to the red side after deletion. Once deleted, the source will no longer be available, and the related data will be removed during background cleanup tasks.

File integrity verification

Each file is stored along with its SHA256 digest, allowing to verify the integrity of the file and to detect any changes or corruption.

When you download a file, you can check its digest against the expected value to ensure that the file has not been tampered with or corrupted during transfer. This digest is provided in the HTTP headers, both Content-Digest (following the RFC 9530) and Digest. This can be done using various tools, such as command-line utilities or programming libraries that support digest verification. The digest is calculated using the SHA-256 algorithm and is encoded in Base64 format.

curl -L -v -I [URL]
# the -I option retrieves only the headers, and the -v option shows the details of the request and response, including the headers
Digest: sha-256=VqiR0u1OsX0ZGPCDVG9mlRjei0DC76I6r1+FXEhmuIs=
Content-Digest: sha-256=:VqiR0u1OsX0ZGPCDVG9mlRjei0DC76I6r1+FXEhmuIs=:

Propositions

A user with the appropriate permission can make a proposition, which must be validated by an administrator.

License limits

Free

Standard

Premium

Propose a search for downloadable links on a page

❌

❌

✅

Download links found in a page

✅

✅

✅

URL finders in web pages

10

1,000

100,000

patterns searched in web pages

100

10,000

1,000,000