Request Demo

Server Administration Last updated: June 14, 2024 15:25

This page covers a wide range of topics that may be interesting to Dwarfguard server administrator. Server administrator means the person responsible for Dwarfguard deployment and operation and having root-level commandline access to the server. The information applies both to the self-managed servers and VMs.

Basic operations

Getting Dwarfguard service status

  • Using systemctl integration (unless you made custom manual install it's there)
    • systemctl status dwarfg_<your_domain>
  • Using cmdline integration (no need to specify domain or deployment number if single deployment scenario)
    • Short (one line) info
      • dwarfg_status
    • Systemctl process info
      • dwarfg_stats_proc
    • Extensive app status including process, version, license, device counters, variables, overrides and threads
      • dwarfg_stats_app
  • Using Dwarfguard daemon control script
    • /opt/dwarfg_<your_domain>/dwarfg_ctl.sh --status

Stopping Dwarfguard service

  • Using systemctl integration (unless you made custom manual install it's there)
    • systemctl stop dwarfg_<your_domain>
  • Using cmdline integration (no need to specify domain or deployment number if single deployment scenario)
    • dwarfg_stop
  • Using Dwarfguard daemon control script
    • /opt/dwarfg_<your_domain>/dwarfg_ctl.sh --stop

Starting Dwarfguard service

  • Using systemctl integration (unless you made custom manual install it's there)
    • systemctl start dwarfg_<your_domain>
  • Using cmdline integration (no need to specify domain or deployment number if single deployment scenario)
    • dwarfg_start
  • Using Dwarfguard daemon control script
    • /opt/dwarfg_<your_domain>/dwarfg_ctl.sh --start

Structure of Dwarfguard deployment

Every Dwarfguard deployment has two main directories:

  1. /opt/dwarfg_<your_domain>
    • Houses all installation files for the particular deployment including daemon binary, web pages, agents and configuration.
    • Some system configuration files (like Apache2 configuration) links to the inside of /opt/dwarfg_<domain> tree.
    • Notable elements:
      • apache_dwarfg_<your_domain>.conf - Apache2 configuration
      • base_defs - basic deaployment definitions file. Do not change this file directly.
      • cmdline - directory containing this deployment cmdline integration
      • download_fw_advantech.sh - Script to download Advantech cellular router Firmware files. Can be executed manually.
      • dwarfg_ctl.sh - daemon control script. Called by systemctl but could be used directly.
      • dwarfgd - Dwarfguard daemon
      • dwarfg.ini - admin-customizable configuration of the daemon
      • dwarfg_license.lic - actual license file (if updated, daemon restart is needed)
      • dwarflib_cfg.txt - admin-customizable configuration, mainly logging levels
      • exchange_server_ip.sh - agent repackaging script. Called automatically, can be used manually WITH CAUTION.
      • grab_logs.sh - script to gather logging files when you need to provide additional information for problem investigation
      • licman - license manager to tell status of license or exchange license from cmdline (if not possible via GUI)
      • secrets - directory containing passowrds and SSH keys when accessing system router using SSH to send SMS. Readable only to dwarfg user.
      • send_sms.py - Python script to send SMS via the system router. Needs information that can be read from DB, usually not feasible to run from cmdline. Best to use GUI, second best curl to call webservice.
      • ssl - contains HTTPS certificate if self-signed certificate was generated during deployment.
      • user.ini - contains web admin configuration. Do not update from command line.
  2. /srv/dwarfg_<your_domain> - Houses runtime files of the deployment - like logs, downloaded Firmware links and similar. Notable elements:
    • logs/log_dwarfg.txt - Dwarfguard daemon log file
    • var/log/* - Dwarfguard web interface logs

Locations outside of the Dwarfguard installation tree

There are a few notable system locations that are updated during Dwarfguard deployment residing out of the mentioned tree above. Most of these locations are simply links into the app tree but some are not. List of the locations and their purpose:

  1. Apache configuration
    • /etc/apache2/sites-enabled/dwarfg_*
    • /etc/apache2/sites-available/dwarfg_*
  2. Systemd configuration
    • /etc/systemd/system/dwarfg_*
  3. Crontab for daily/monthly traffic counters count and RSS re-generation
    • /etc/cron.d/dwarfg_*
  4. sudoers so that web and daemon (running under dwarfg user) can perform certain actions
    • /etc/sudoers.d/dwarfg_*

Locations shared between all Dwarfguard deployments on server

Next to the per-deployment directories, there is a directory shared between all the deployments on server:

  1. /opt/cache_dwarfg - server wide directory containing primarily
    1. dwarfg_deployments - directory containing one file per one Dwarfguard deployment.
    2. home - home directory of the dwarfg user. Important content:
      1. .dwarfg_site_config - contains system-wide configuration, namely the DB password for the dwarfg user. File is readable only for the dwarfg user.
      2. .ssh - contains file authorized_keys which lists all public keys for all prepared webtunnels
    3. cmdline - directory containing last installed version of the cmdline integration. /usr/bin/dwarfg_* links here
  2. /opt/backup_dwarfg/ - directory containing any backups made by cmdline dwarfg_backup or dwarfg_upgrade.sh script

CMDline integration overview

To make the tasks of the server admin easier, Dwarfguard brings a command-line tool facilitating some of the tasks.

There are several commands available. The list of commands together with a short description could be obtained by executing dwarfg_help command:

  • help [command] ... list cmds or provide longer help on one command

  • status ... show runtime status of Dwarfguard service (dwarfgd)

  • stop ... stops Dwarfguard service

  • start ... starts Dwarfguard service

  • restart ... restarts Dwarfguard service

  • reload ... reloads dwarflib configuration

  • stats_proc ... shows systemd-gathered information (if under systemd)

  • stats_app ... shows app stats like # of devices, license, last restart

  • logview ... opens daemon log in view

  • loggrep [level] ... displays logs messages. CRIT|ERR|*WARN*|NOTE

  • logexport [fname] ... create log archive. may add suffix: fname[.tgz]

  • backup ... run backup NOW (downtime!) via dwarfg_upgrade.sh

  • restore ... interactive mode: dwarfg_upgrade.sh (backup+restore)

  • deplist ... show all Dwarfguard deployments found on server

  • depstatus ... show service status for each deployment

Calling any command with "help" as first parameter shows the particular command help as well.

The commands are stored in the /opt/cache_dwarfg/cmdline directory but links are created from /usr/bin so that no updates to the PATH variable are needed.

Please note that in the case of multiple Dwarfguard deployments on the server, you will end up with the highest version of the CMDline commands among your deployments in the shared location. The deployment-original commands can alway be found at /opt/dwarfg_<deployment_domain>/cmdline

Handling license (from cmdline)

The license for Dwarfguard SW is stored in the file dwarfg_license.lic under the deployment directory.

Dwarfguard reads the license during startup and dumps the license information into the logfile.

The license is usually handled from the GUI, which actually executes the licman binary and restarts Dwarfguard daemon if needed.

You can obtain information about the license or perform any license management task from commandline by calling the licman binary directly. Do not forget that to do any actual management of the license, you need to be the root user.

Some of the usual usage cases for the licman follows:

  • licman -h ... display all commandline arguments and their description
  • licman -n -d ... dump actual license information without checking number of devices in DB
  • licman -n -d <licfile> ... dump information of license stored in the licfile file (without connecting to DB)

There are other options that are used when GUI checks the newly uploaded license prior to license update and to perform the actual license update but they are not really interesting to the server admin. The license is just a file. You can backup it by copying it and replace with new one by simply overwriting it. Just don't forget to restart the Dwarfguard service after you replace the file.

Fine-tuning application logging

Some of the application parameters can be set via GUI, some are accessible only to the server admin. The server-admin configuration is exposed in a few files, one of them being dwarflib_cfg.txt

This file contains configuration options for Dwarf library, that is used for a number of purposes.

There are three predefined configurations set provided in the deployed application tree:

  • dwarflib_cfg_prod.txt - contains setting appropriate for production deployment, cutting logging entries down to a limited amount. This is the default option for regular production deployment.
  • dwarflib_cfg_log.txt - more verbose production logging, specifially without log messages throttling and bigger message buffer (see below on more explanations). The size of the logging file will be bigger while still reasonable. Use it if you want to have a more detailed audit log or want to check for a potential problem details.
  • dwarflib_cfg_dbg.txt - contains debug setting for Dwarf library logging. NOTE that the size of the logging file when this settings is active is much bigger and if your storage is slow, it may impact application performance significantly. Do not use unless you are asked by support to do so. Also, if you need to debug, please inspect the configuration - you can decrease the general verbosity and debug only a particular area. See below on more explanation.

The configuration file itself is typically a symbolic link to one of the files above so that you can easily switch between the prepared configurations. The application does not require the file to be a link. You can simply copy one of the predefined files.

To apply any changes (e.g. linking another predefined configuration file or simply after the file edit) you need to send a reload signal the the daemon using one of the below ways:

  1. Using Dwarfguard cmdline integration:
    • dwarfg_reload
  2. Using control script directly from the deployed path:
    • /opt/dwarfg_<your_domain>/dwarfg_ctl.sh --reload-cfg

NOTE that you can also (naturally) restart the service for the Dwarflib configuration change to take effect - but that introduces service interruption while the reload operation does not.

As for customizing the configuration options itself, note that the file is reasonably commented, explaining each and every configuration option, listing its default values and having all the actual values located at the file end. The options are given in the format OPTION=VALUE. The most interesting configuration options are these:

  • DBG_GENERIC ... specifies logging level for all areas. Usually this is set to a lower level than the specific area you want to see more detailed information about. If you set this to e.g. MDL_INFO, all log messages on the MDL_INFO level and more important will be stored in the debug file, regardless of which area they belong to.
  • by using DBG_AREA_* variables you are able to set more detailed logging for a particular area than what is set via the DBG_GENERIC.
  • If you want to inspect what is happening on the system or want to do any troubleshooting, your first action should be to disable the debug message throttling - set DBG_THROTTLE=0
  • If you want to stop the debug messages being cut to a maximal length, you may set the DBG_NOLIMIT=1
  • If you want to review the data send by individual devices and responses sent by Dwarfguard back to the devices, look at DBG_DUMPDATAPUSH and DBG_DUMPDATARESP (note that this should not be used continually as it continues to dump the received and sent data on your storage inside /srv/dwarfg_<your_domain>/logs/datadums/)
  • Do not (ever) set DBG_MEMIN on production deployments. This is a purely development configuration.

Backup and restore

Looking at the structure of the Dwarfguard deployment above, it is evident there are multiple locations on the system that needs to be backed up and restored if needed.

There is another data source that needs to be backed up and restored if such need arise and that is the database where the data about your devices are retained.

While you can attempt to back up the whole deployment manually yourself, there are two more conveniet ways how to do that:

  1. Using the implemented backup&restore functionality.
  2. Backing up the whole system or VM (if you deploy on a VM)

As the second method depends on the system you use (typically a VM management) we are going to focus on the internal backup and restore.

The backup and restore system is exposed to the administrator via two interfaces:

  1. Directly in the deployment tree via the dwarfg_upgrade.sh script that facilitates backup and restore functionality itself.
  2. Cmdline integration command dwarfg_backup that makes it easy to use in case on multi-deployment. In the end, the integration command is calling the dwarfg_upgrade.sh script mentioned above.

Backup

Note that backup operation incurs some downtime. The length of the downtime depends on amount of data needed to be backed up and storage speed. Usually the backup is quite fast.

The backup operation first stops the Dwarfguard service, performs the backup and restarts the service immendiately afterwards.

The backup itself is stored in /opt/backup_dwarfg/<domain>%<version> directory. Please note that next to your data, the backup may contain sensitive information like passwords or SSH keys if you decided to store credentials to some of your devices in the Dwarfguard. While the information is not stored in plaintext format, a determined attacker will be able to get to the information sooner or later if you make the backup available to her. Keep the backup data private and safe!

You can start the backup simply by one of these ways:

  1. dwarfg_backup
  2. /opt/dwarfg_<domain>/dwarfg_upgrade.sh backup

Restore

When you want to restore the deployment, follow these steps:

  1. Make sure the deployment you want to restore is not still present in the system. If it is, you need to clean the running deployment first by executing /opt/dwarfg/cleanup.sh
  2. Locate the backup data directory.
  3. Run the dwarfg_upgrade.sh script from the backup:
    1. For interactive operation:
      • /opt/backup_dwarfg/dwarfg_<domain>%<version>/data/dwarfg_upgrade.sh
    2. For starting up in restore mode (still needs user confirmation):
      • /opt/backup_dwarfg/dwarfg_<domain>%<version>/data/dwarfg_upgrade.sh restore

If your restore operation fails, please read the "Manual recovery / restore" chapter in Upgrading to a new version article.