Complete transfer process

On the black side

Each source is independent from the other ones. In the following, $transfer_path is the transfer_path value in the [transfer] section of the configuration file. When the transfer (of all data created with the specified date range) is required on a source, the following things happen:

• this transfer is an action associated to a unique identifier (action_uid),

• a temp directory $transfer_path/tmp/action_uid is created,

• a control.json file is created in this directory with enough info to identify this transfer (and the uid of the previous transfer),

• required database export is written in the database.db directory,

• new files are written to the tracked_files directory, and their metadata are written to the tracked_files.db directory,

• a signatures.sign file is created with the digital signature of some important files (ensuring the integrity and the author),

• the temp directory is atomically moved to $transfer_path/out/action_uid

if you decided to use hairgap, then a hairgap transfer is required:

• a lock file $transfer_path/hairgaps/send.lock is created, ensuring that a single file is sent at a time,

• a new DiodeTransfer is created and data are moved from $transfer_path/out/action_uid to $transfer_path/hairgaps/action_uid,

• an index file is created at $transfer_path/hairgaps/action_uid.ini,

• all data are then sent to the red side using a UDP connection,

• if [transfer]keep_transfers = true, then the directory is kept as long as the total size of $transfer_path/hairgaps does not exceed the limit.

otherwise:

• you must transfer $transfer_path/out/action_uid to the red side of your network and remove it from the black side,

On the red side

if you use hairgap:

• raw data are received to $transfer_path/hairgapr,

• completed data are moved to $transfer_path/out/transfer_id,

otherwise:

• the transferred data must be atomically moved to $transfer_path/out/unique_id,

• $transfer_path/out is regularly scanned to detect newly added directories,

• if any file inside the tracked_files directory is removed but a new file is created with the same name suffixed by “.removed”, the import does not raise an error since this removal is assumed to be deliberate.

In both cases:

• a lock file is created at $transfer_path/out/unique_id/lockfile to ensure that a single process will import these data,

• $transfer_path/out/unique_id/control.json is read to retrieve the action_uid,

• if this transfer must be imported after another transfer (not imported yet, but present in $transfer_path/out), then the import process is cancelled and the lock file removed,

• if data have been corrupted during the transfer process, then the directory is moved to $transfer_path/corrupted/action_uid (if [transfer]keep_corrupted_imports is set to yes in the config file),

• if an error happens during the import process, then the directory is moved to $transfer_path/error/action_uid (if [transfer]keep_failed_imports is set to yes in the config file),

• if everything goes well, then the directory is moved to $transfer_path/success/action_uid (if [transfer]keep_imports is set to yes in the config file).

Desynchronization

If an error happens and a transfer is broken, then any subsequent transfer will also fail. You can try to send again this broken transfer by going to the “Recent actions” tab. In the “setting” tab, you can also force the transfer of all data.

Analyzing imported files

You can set [transfer]imported_file_analyzer to the dotted path of a Python function. This function must be in the PYTHONPATH of the InterDiode’s Python process. In this case, this function will be called on every imported file with the following arguments: (file_absolute_path: Path, file_object) -> bool.

If this function returns False, no check is performed after the import, neither checksum or even file existence. If this function returns True, the file must be left unmodified or hashes and size must be updated and the file_object must be saved:

class FileObject:
    mime_type: str ="MIME type"
    sha3_512: str = "SHA3-512 hash"
    sha3_384: str = "SHA3-384 hash"
    sha3_256: str = "SHA3-256 hash"
    sha3_224: str = "SHA3-224 hash"
    sha512: str = "SHA512 hash"
    sha256: str = "SHA256 hash"
    sha1: str = "SHA1 hash"
    md5: str = "MD5 hash"
    size: int = file_size

    def save(self):
        """must be called in case of file modification"""

If you do not use the integrated transfer protocol and you prefer use your own transfer solution, you can remove a suspicious file and still import data from the tracked_files directory if you replace it by an empty file with the .removed suffix.