Transferring data

On the black Side

Each source is independent from the other ones. When the transfer (of all data created with the specified date range) is required on a source, the following things happen:

  • this transfer is an action associated to a unique identifier (transfer_id),

  • a temp directory <LOCAL_CACHE_DIR>/tmp/<transfer_id> is created,

  • a control.json file is created in this directory with the metadata required to identify this transfer (and the uid of the previous transfer),

  • required database export is written in the database.db directory,

  • new files are written to the tracked_files directory, and their metadata are written to the tracked_files.db directory,

  • a signatures.sign file is created with the digital signature of some important files (ensuring the integrity and the author),

  • the temp directory is atomically moved to <LOCAL_CACHE_DIR>/transfer/<transfer_id>

if you decided to use TCP or UDP, then a network transfer is required:

  • a lock file <LOCAL_CACHE_DIR>/sending/send.lock is created, ensuring that a single file is sent at a time,

  • a new DiodeTransfer is created and data are moved from <LOCAL_CACHE_DIR>/transfer/<transfer_id> to <LOCAL_CACHE_DIR>/sending/<transfer_id>,

  • an index file is created at <LOCAL_CACHE_DIR>/sending/<transfer_id>.ini,

  • all data are then sent to the red side using a TCP or UDP connection,

  • if TRANSFER_KEEP_TRANSFERS, then the directory is kept as long as the total size of <LOCAL_CACHE_DIR>/sending does not exceed the limit.

otherwise, if you use the file transfer mode:

  • you must transfer <LOCAL_CACHE_DIR>/transfer/transfer_id to the red side of your network and remove it from the black side,

On the red Side

if you use the TCP or UDP transfer mode:

  • raw data are received to <LOCAL_CACHE_DIR>/receiving,

  • completed data are moved to <LOCAL_CACHE_DIR>/transfer/<transfer_id>,

otherwise:

  • the transferred data must be atomically moved to <LOCAL_CACHE_DIR>/transfer/<unique_id>,

  • <LOCAL_CACHE_DIR>/transfer/ is regularly scanned to detect newly added directories,

  • if any file inside the tracked_files directory is removed but a new file is created with the same name suffixed by “.removed”, the import does not raise an error since this removal is assumed to be deliberate.

In both cases:

  • a lock file is created at <LOCAL_CACHE_DIR>/transfer/<unique_id>/lockfile to ensure that a single process will import these data,

  • <LOCAL_CACHE_DIR>/transfer/<unique_id>/control.json is read to retrieve the transfer_id,

  • if this transfer must be imported after another transfer (not imported yet, but present in <LOCAL_CACHE_DIR>/transfer/), then the import process is cancelled and the lock file removed,

  • if data have been corrupted during the transfer process, then the directory is moved to <LOCAL_CACHE_DIR>/corrupted/<transfer_id> (if TRANSFER_KEEP_CORRUPTED_IMPORTS is set to yes in the config file),

  • if an error happens during the import process, then the directory is moved to <LOCAL_CACHE_DIR>/error/<transfer_id> (if TRANSFER_KEEP_FAILED_IMPORTS is set to yes in the config file),

  • if everything goes well, then the directory is moved to <LOCAL_CACHE_DIR>/success/<transfer_id> (if TRANSFER_KEEP_IMPORTS is set to yes in the config file).

In case of desynchronization

If an error happens and a transfer is broken, then any subsequent transfer will also fail. You can try to send again this broken transfer by going to the “Recent actions” tab. In the “setting” tab, you can also force the transfer of all data.

If the export data are still present on the black side, then you can also try to transfer with interdiode-ctl source transfer [--background]  input_abspath.

You can also export and transfer data from a single source or from all sources with interdiode-ctl source export [--last LAST_DATETIME] [--first FIRST_DATETIME] [--background] <UUID|*>. This is typically the case when a network problem happened during several days and you want to transfer all data created during this period. More information is available in the documentation.

These actions can also be performed from the web interface.

Connection Test

The command line interface offer to create a test connection, as documented. The web interface also offer to create a test message, which is a random number sent from the black side to the red side.