Monitoring and Logging

Prometheus Metrics Export

Prometheus metrics can be exported by all processes and can be collected and visualized using tools like Prometheus and Grafana.

docker run --rm -p 8000:8000 [-p PROMETHEUS_PORT:PROMETHEUS_PORT] interdiode:latest interdiode-ctl run [--prometheus-bind PROMETHEUS_BIND] [--prometheus-port PROMETHEUS_PORT] [--prometheus-ssl-key PROMETHEUS_SSL_KEY] [--prometheus-ssl-cert PROMETHEUS_SSL_CERT] http
docker run --rm [-p PROMETHEUS_PORT:PROMETHEUS_PORT] interdiode:latest interdiode-ctl run [--prometheus-bind PROMETHEUS_BIND] [--prometheus-port PROMETHEUS_PORT] [--prometheus-ssl-key PROMETHEUS_SSL_KEY] [--prometheus-ssl-cert PROMETHEUS_SSL_CERT] background
docker run --rm [-p PROMETHEUS_PORT:PROMETHEUS_PORT] interdiode:latest interdiode-ctl run [--prometheus-bind PROMETHEUS_BIND] [--prometheus-port PROMETHEUS_PORT] [--prometheus-ssl-key PROMETHEUS_SSL_KEY] [--prometheus-ssl-cert PROMETHEUS_SSL_CERT] beat

Since processes are run in a Docker container, the default bind address for Prometheus metrics is 0.0.0.0 and specifying the Prometheus port is enough to activate the Prometheus export.

If you use SSL/TLS certificates, do not forget to specify the --prometheus-ssl-key and --prometheus-ssl-cert options and to mount these files into the containers.

docker run --rm -p 8012:8012 interdiode:latest interdiode-ctl run --prometheus-port 8012 beat
curl http://localhost:8012/

Logging Configuration

By default, Interdiode logs messages to standard output (warnings and below on stdout and errors and above on stderr). LOG_DIR environment variable can be defined to log messages to files instead. Do not forget to mount the log directory into the containers and to set appropriate permissions for the user running Interdiode processes.

The logging level can be configured using the LOG_LEVEL environment variable. By default, it is set to WARN. Setting it to INFO or even to DEBUG will log more detailed information, which can be useful for troubleshooting.

LOG_REMOTE_URL can be used to specify a remote logging service URL where logs will be sent. Currently, Interdiode supports sending logs to Grafana Loki or Syslog. Syslog can be used with the following URL: syslog://host:port (syslog over UDP) or syslog+tcp://host:port (syslog over TCP), where host and port should be replaced with the appropriate values for your Syslog server. Loki can be used with the following URL: lokis://username:password@host:port?query (Loki over HTTP) or lokis://username:password@host:port?query (Loki over HTTPS), where username, password, host, and port should be replaced with the appropriate values for your Loki instance.

If LOG_REMOTE_ACCESS is set to true and LOG_REMOTE_URL is defined, logs about HTTP requests will be sent to the specified remote logging service.

Exceptions are also sent to the administrators defined by the ADMINS environment variable. EMAIL_FROM and EMAIL_HOST_URL environment variables must be defined.

LOG_SLOW_QUERY_DURATION_IN_S can be set to a positive number to log slow database queries that take longer than the specified duration in seconds. This can help identify performance bottlenecks in the database.

System Checks

Interdiode performs various checks on the configuration and environment to ensure that everything is set up correctly. If any issues are detected, warnings or errors will be logged with specific IDs that can be used to identify the problem. See the CLI doc for more details on how the check the configuration with command line.

Warnings in all processes can also be displayed in the web interface of InterDiode, in the “Admin/Healthchecks” section.

Here is the list of warnings and errors that can be displayed by InterDiode processes:

  • df_config.W002: File or directory does not exist or is not of the expected type.

  • df_config.W003: File or directory does not exist.

  • df_config.W004: File or directory does not exist but can be fixed by running the configuration apply command.

  • df_config.W005: Invalid remote logging scheme in LOG_REMOTE_URL: only syslog, syslog+tcp, loki and lokis (Loki over HTTPS) schemes are supported.

  • df_config.W008: The specified LOG_DIR directory is missing.

  • df_config.W009: InterDiode cannot write to the LOG_DIR directory.

  • df_websockets.E001: Unable to use the Redis cache database, the Redis URL REDIS_URL is invalid.

  • df_websockets.E002: Unable to use the Redis cache database, the Redis URL REDIS_URL is invalid.

  • interdiode.W001: The current default GPG key is invalid. Check the documentation to select a valid GPG key.

  • interdiode.W003: Unable to check the ARP cache of the host machine. This check is only performed if the TRANSFER_MODE is set to udp.

  • interdiode.W004: The MAC address of the internal (red) server is not found in the ARP cache of the host machine. This check is only performed if the TRANSFER_MODE is set to udp. Check the documentation to fix this problem.

  • interdiode.W006: A lock file exists but is a link instead of a regular file. You should remove it.

  • interdiode.W007: A lock file exists but is a directory instead of a regular file. You should remove it.

  • interdiode.W008: A lock file exists but is not writeable by InterDiode. You should remove it or fix the rights.

  • interdiode.W009: A required directory for local secrets is missing. You should create it or run the configuration apply command to fix this problem.

  • interdiode.W011: A required directory for local secrets is not writeable. You should fix the rights or run the configuration apply command to fix this problem.

  • interdiode.W012: A lock file exists but is quite old. Be sure that everything is normal (maybe the transfer has been interrupted) and that it should not be removed.

  • interdiode.W013: A directory is waiting in the transfer path but should be quickly moved to the import path. See the documentation for more details on used directories.

  • interdiode.W014: A transfer is still waiting in the transfer path (documentation). You can mark it as failed and try again to send it.

  • interdiode.W015: Invalid permissions for secrets data. You should fix them manually or run the configuration apply command.

  • interdiode.W016: Invalid owner for secrets data. You should fix it manually or run the configuration apply command.

  • interdiode.W017: Unable to connect to a service (e.g., SQL database, Redis, etc.). Check the logs for more details and ensure that the service is running and accessible.

  • interdiode.W018: A PID file exists but does not contain a valid PID. You should remove it to fix the problem.

  • interdiode.W019: A PID file exists but does not contain a running PID. You should remove it to fix the problem.

  • interdiode.W021: An background action has been interrupted and needs to be marked as cancelled with the interdiode-ctl configuration reset –actions command.

  • interdiode.W022: You should never use a SQLite database. Please switch to a more robust database like PostgreSQL, MySQL, or MariaDB and set DATABASE_URL.

  • interdiode.W023: A transfer directory is still present in the transfer path but should be deleted. You can safely remove it to fix the problem.

  • interdiode.W026: Red destination IP and MAC addresses are not configured. Please set the RED_DESTINATION_MAC and RED_DESTINATION_IP environment variables to use the integrated air gap transfer protocol.

  • interdiode.W027: Your database is out of date. You should run the configuration apply command to update it and fix this problem.

  • interdiode.W028: Your database is out of date with missing migrations. You should run the configuration apply command to update it and fix this problem.

  • interdiode.W029: Periodic tasks have been run in the future. This may indicate a clock problem in your network. Check the logs for more details and ensure that the system clocks of all servers are synchronized.

  • interdiode.W030: The version of a required software is too old. You should update it to the required minimum version.

  • interdiode.W031: Your PostgreSQL version is too old (should be at least 14). You should update it to a more recent version.

  • interdiode.W032: The SSL PEM certificate file is missing. You should provide a valid SSL PEM certificate file in HTTP_SSL_CERTFILE and mount it in the Docker containers.

  • interdiode.W033: The SSL PEM private key file is missing. You should provide a valid SSL PEM private key file in HTTP_SSL_KEYFILE and mount it in the Docker containers.

  • interdiode.W034: Unable to load the SSL PEM certificate file in HTTP_SSL_CERTFILE. You should check the file and ensure that it is a valid SSL PEM certificate file and that it is correctly mounted in the Docker containers.

  • interdiode.W035: Unable to connect to the SQL database or to Redis. You should check the database connection parameters and ensure that the database server is running and accessible from the InterDiode server, and that the SSL/TLS certificates are correctly configured if an SSL/TLS connection is used.

  • interdiode.W036: GPG key is not set. You should use the gpg command to create a new one, check the documentation to select a valid GPG key.

  • interdiode.W037: A required directory is missing. You should create it or run the configuration apply command to fix this problem.

  • interdiode.W038: A required file is missing. You should create it or run the configuration apply command to fix this problem.

  • interdiode.W039: No license is configured. You should provide a valid license key to enable all features.

  • interdiode.W042: Unable to load a plugin.

  • interdiode.W043: Unable to load a plugin and its settings.

  • interdiode.W044: The <LOCAL_CACHE_DIR>/transfer/ directory is not writeable by InterDiode.

  • interdiode.W045: The <LOCAL_CACHE_DIR>/transfer/ directory is not readable by InterDiode.

  • interdiode.W046: The <LOCAL_CACHE_DIR>/transfer/ directory not shared between containers. See documentation about transfers and directories for more details on directories used in transfers.

  • interdiode.W047: A required directory used in importing transfers is not writeable. You should fix the permission.

  • interdiode.W048: A required directory used in transfers is not writeable. You should fix the permission.

  • interdiode.W049: A required directory used in transfers is not writeable. You should fix the permission.

  • interdiode.W050: A required directory is not writeable. You should fix the permission.

  • interdiode.W051: Unable to load the SSL PEM certificate file in HTTP_SSL_CERTFILE. You should check the file and ensure that it is a valid SSL PEM certificate file and that it is correctly mounted in the Docker containers.

  • interdiode.W052: <LOCAL_CACHE_DIR>/tmp/ and <LOCAL_CACHE_DIR>/transfer/ must be on the same partition to allow atomic file move operations (see Local Files for more details). You should ensure that these directories are on the same partition and that they are correctly configured in the environment variables.