Logging Support
Introduction
The system produces a wealth of information in log files. Most of the log messages generated in the system are that of general events from the various system components. For some of the more important events that are logged, see Security Related Events.
The following methods for logging are available:
-
Logging of default log files: By default all logs are written to an internal RAM disk. Log files written to the internal RAM disk are not persistent across reboots.
-
Logging of default log files to USB: When enabled it will log the same things that are logged to the internal RAM disk, but it will be done to a connected external USB drive.
-
Logging to console port: It is possible to direct logging messages to the console port. Messages of severity level DEBUG or higher are shown on the console port.
-
Logging to custom logging sink: For more advanced logging, including logging to remote machines, the system supports creation of user defined logging sinks. Each individual sink describes a logging target that can be either a remote logging server, a file on external media (USB or SD Card), or a local file. In addition, for each sink the operator defines filters for what log messages to be written based on Facility (subsystem) and Severity (level). The filter provides a greater control in regards to the messages that should be logged.
For example an use-case see:
Overview
The system produces and presents a lot of information in the form of log messages that are stored in log files. Any message that is generated by the system will be associated with a so called Facility and a Severity, in accordance with the syslog standard specified in RFC 5424.
Facility
The Facility is supposed to signify what specific area any particular message belongs to. All facilities and how the system attempts to conform to them can be seen in the table below.
# | Facility | Description |
---|---|---|
0 | kern | Kernel log messages |
1 | user | User-level messages |
2 | Unused | |
3 | daemon | System daemons |
4 | auth | Security/Authorization messages |
5 | syslog | Unused |
6 | lpr | Unused |
7 | news | Unused |
8 | uucp | Unused |
9 | cron | Unused |
10 | authpriv | Unused |
11 | ftp | Unused |
12 | ntp | NTP |
13 | security | Log audit, for audit trail |
14 | console | Log alert |
16 | local0 | Alarm sub-system |
17 | local1 | Unused |
18 | local2 | PPP |
19 | local3 | Unused |
20 | local4 | OpenVPN, IPsec |
21 | local5 | Reserved, OEM customer specific |
22 | local6 | Unused |
23 | local7 | Unused |
Table 1: All of the standard syslog facilities and their uses in the system.
The facilities
security
andconsole
are displayed asLOGAUDIT
andLOGALERT
in Wireshark, which is the de facto tool to debug all network traffic.
Severity
The Severity is supposed to represent the level of criticality for each individual message. All severities and how the system attempts to conform to them can be seen in the table below.
# | Severity | Description |
---|---|---|
0 | emerg | System level service only |
1 | alert | System level service only |
2 | crit | System level service only |
3 | err | Severe error, daemon/service may restart |
4 | warning | Significant problem, e.g. no connection to Radius server |
5 | notice | General log message, e.g. login successful |
6 | info | Informational, less important |
7 | debug | Developer/low-level debug messages |
Table 2: All of the standard syslog severities and their general usage in the system.
Simple Example
As an example, a normal configuration setup for remote logging is to log all messages that fall within a specific severity and above. In the following example below, the device is configured to send all log messages of severity notice and above to a remote log server located at foobar.example.com:
example:/#> configure
example:/config/#> logging
example:/config/logging/#> sink
example:/config/logging/sink-1/#> target udp address foobar.example.com
example:/config/logging/sink-1/#> priority *.notice
example:/config/logging/sink-1/#> end
example:/config/logging/#> show sink
IDX TARGET PRIORITY FORMAT
1 udp address foobar.example.com *.notice rfc3164
example:/config/logging/#> leave
example:/#>
Default Log Files
The system have a number of default logging files that are generated. In order to see all the available log files on the system use the command:
example:/#> dir log
/log/ MOD. DATE/LINK SIZE MD5
auth.log 2019-11-28 17:35 264
messages 2019-11-28 18:02 7435
[...]
The messages log file is the main log file in the system. Use the
show log
command to show it, without any additional information it
displays the messages log:
example:/#> show log [...]
To show a specific log file:
example:/#> show log://auth.log [...]
Remember the short-keys to the show
command when listing long files:
R
: show the rest of the fileq
: Quit viewer
Also, the follow
command is very useful when monitoring log files. It
continuously lists new lines as they are appended to a file. Use
Ctrl-C
to stop and return to the command prompt.
Security Related Events
The system produces a lot of different log messages, but there are a few that are considered a bit more important in terms of security. As such they are listen below:
Event | Facility | Severity | Example |
---|---|---|---|
Login successful | auth | notice | Authentication successful for user ‘admin’ from 10.0.0.1 |
Login failed | auth | warning | Authentication failed for user ‘admin’ from 10.0.0.1 |
Audit trail | security | notice | User ‘admin’ changed the configuration |
System startup | console | notice | \\/ Westermo WeOS v5.7, entering runlevel 3 |
System shutdown | console | notice | \\/ Westermo WeOS v5.7, entering runlevel 6 |
System update | console | notice | \\/ Westermo WeOS v5.7, entering runlevel 9 |
Enter Maintenance mode | user | notice | Entering maintenance mode for upgrade to WeOS-5.7.x.pkg … |
Exit Maintenance mode | user | notice | Exiting maintenance mode for upgrade |
Service start | console | notice | Starting mdnsd:1, PID: 1203 |
Service stop | console | notice | Stopping mdnsd:1, PID: 1203, sending SIGTERM… |
Service restarting | console | notice | Restarting mdnsd:1, PID: 1203, sending SIGHUP… |
Service Died | console | warning | Service mdnsd:1 died, restarting (1/10) |
Service FAIL | console | warning | Service mdnsd:1 keeps crashing, not restarting |
Link up | console | notice | Link up on eth1 |
Link down | console | notice | Link down on eth1 |
DHCP Unknown host ID | console | warning | DHCPDISCOVER(11:22:33:44:55:66) vlan1 no address available |
NTP reply not synced | ntp | info | reply from 10.0.0.1: not synced (alarm), next query 64s |
NTP synced | ntp | info | clock is now synced |
NTP peer valid | ntp | info | peer 10.0.0.1 now valid |
NTP update FAIL | ntp | err | no reply received in time, skipping initial time setting |
Table 3: Security related events that are logged in the system.
Example sink
configuration to match the listed security related events listed
above, sent to two remote log servers located at 192.168.1.10 and
192.168.1.12:
example:/#> configure
example:/config/#> logging
example:/config/logging/#> sink
example:/config/logging/sink-1/#> target udp address 192.168.1.10
example:/config/logging/sink-1/#> priority auth;security;console;ntp
example:/config/logging/sink-1/#> end
example:/config/logging/#> sink target udp address 192.168.1.12 priority auth;
security;console;ntp
example:/config/logging/sink-2/#> end
example:/config/logging/#> show
Console syslog : Disabled
USB/SD logging : Disabled
Secure mode : Enabled
______________________________________________________________________________
Sinks
IDX TARGET PRIORITY FORMAT
1 udp address 192.168.1.10 auth.*;security.*;console.*;n... rfc3164
2 udp address 192.168.1.12 auth.*;security.*;console.*;n... rfc3164
Configuration
The logging in the system can be configured from the top-level configuration context in the CLI:
example:/#> configure example:/config/#> logging example:/config/logging/#>
[no] console
-
Use to enable or disable logging directly on the console.
Default: Disabled
no
- Disable logging to the console.
[no] usb [SIZE [COUNT]]
-
Enable or disable logging of default log files to USB (or SD-Card).
When enabled the same log files that are logged to on the local RAM disk will also be logged to on any attached USB or SD-Card.
A folder called /log/ will be created on the USB if it does not exist, this is where the log files will be created.
All log files are automatically compressed from generation .1
Example:
Enable USB logging and Keep 10 log files, each 10MiB (uncompressed):
example:/config/logging/#> usb 10M 10
Default: Enabled
Sinks can be configured to log to a USB file, regardless if this setting is enabled or not. This setting only handles logging of the systems default log files to the USB.
no
- Disable logging of default log files to usb.
SIZE
- Max size of each log file, in uncompressed form. Takes an optional k/M size modifier.
COUNT
- Number of log files to keep, in total.
[no] secure-mode
-
Enable or disable secure mode. This setting determines whether the unit reject or accept any logging messages from remote machines. By default secure mode is enabled, meaning the unit operates in a secure way and do not accept any messages from remote machines.
Default: Enabled
no
- Disable secure mode.
[no] template NAME priority [EXPR [;EXPR]]
-
Manage syslog sink priority templates. These templates can be applied to a sinks priority configuration for a quick filtering setup.
The system also provide a number of default template, that cannot be removed or changed.
Default: N/A
no
- Use
no template
to remove all template, except for default templates (default templates are not removable). Useno template NAME
to remove a specific template (e.g.no template my-template
). NAME
- A free form string.
EXPR
facility.[!|=]severity
facility
[auth | console | daemon | kern | ntp | security | user | local0 | local1 | local2 | local3 | local4 | local5 | local6 | local7]
severity
[debug | info | notice | warning | err | crit | alert | emerg]
[no] sink [INDEX]
-
Manage syslog sinks, for custom logging to file and remote.
This command is used to manage and create syslog sinks. Each sink defines a logging target and filter on what messages to send towards the target destination based on log message Facility and Severity.
Logging sinks can be configured to log to remote machines, local files, or files on a connected USB drive.
Enters a sub-configuration context for the sink.
no
- Use
no sink
to remove all configured logging sinks, and useno sink INDEX
to remove a specific sink definition (e.g.no sink 3
). INDEX
-
Numerical id of the sink, between
1-8
.When creating a new sink the
INDEX
argument is optional, when it is left out the next free index is automatically selected for you.
Sink configuration
Each sink also have its own configuration context:
example:/config/logging/#> sink example:/config/logging/sink-1/#>
[no] target [udp [address [IP|FQDN] | dhcp] | file [internal|external] PATH]
-
Specify the target destination of the sink. The target can specify the following types of destinations for the sink:
-
File on local RAM disk: Log to a local file specified by a file path, relative to the systems log file folder.
-
File on USB or SD-Card: Log to a file on a connected USB or SD-Card, specified by a file path relative to the target device.
-
Remote machine: Specify a remote machine to log to by providing an IP address or and FQDN.
-
Remote machine (Dynamic): Specify that the sink should log to a remote machine specified by an IP address received from a DHCP lease providing a log server (DHCP Option 7).
The target cannot be the same for multiple different sink instances.
Folders specified in the PATH when creating a file target, must exist before, they will not be created by the sink, only the file will.
Examples:
example:/config/logging/sink-1/#> target udp address 10.0.0.1
example:/config/logging/sink-1/#> target udp dhcp
example:/config/logging/sink-1/#> file internal my-file.log
no
- A target is required for every sink. Therefore, it is not possible to remove, it can only be changed by configuring another target.
IP
- IP address in standard quad-dotted notation, e.g. 192.168.1.1.
FQDN
- A fully qualified domain name.
PATH
- String definition of the file path and name.
-
[no] priority <EXPR [;EXPR] | template NAME>
-
Configure a sink priority expression, defines what type of messages are sent to the sink target.
The priority per sink consist of up to 16 different expressions. Each expression consists of a facility, a modifier, and a severity. These expressions are combined to define the syslog priority, which in turn dictates what type of messages are sent to the sink target.
Another thing to keep in mind, ordering of the expression have an effect on how the filtering is applied. Meaning that expressions are processed in the order that they appear, i.e. the order they are added.
As an example an expression
*.*
is added specifying that all messages are to be logged, of any facility and severity. If additional rules, sayauth.!*
andauthpriv.!*
, are added this means that we will now exclude all messages with the auth and authpriv facility based on preceding expression rules. If we would have added the exclude rules first, in this example, it would not do anything, since no base to exclude them from existed.Filter Expression Modifiers
Mod Usage Description .
facility.severity
Include messages of facility with severity equal or greater .!
facility.!severity
Exclude messages of facility with severity equal or greater .=
facility.=severity
Include messages equal to this facility and severity .!=
facility.!=severity
Exclude messages equal to this facility and severity *
*.severity
Include all messages of a severity *
facility.*
Include all messages of a facility *
*.*
Include all messages of any facility or severity EXPR
facility.[!|=]severity
facility
[auth | console | daemon | kern | ntp | security | user | local0 | local1 | local2 | local3 | local4 | local5 | local6 | local7]
severity
[debug | info | notice | warning | err | crit | alert | emerg]
[no] format bsd|rfc3164|rfc5424
-
Set the formatting to be used when sending the messages.
BSD
-
Legacy formatting for remote syslog servers;
<PRI> message
RFC3164
-
Standardized formatting;
<PRI>Nov 18 09:36:42 hostname proc[123] message
RFC5224
-
Modern syslog formatting;
<PRI>1 2019-11-18T09:36:42.000321+01:00 hostname proc 123 - - message
Example
This is a simple example how to configure a sink:
example:/#> configure
example:/config/#> logging
example:/config/logging/#> sink
example:/config/logging/sink-1/#> target udp address 10.0.0.1
example:/config/logging/sink-1/#> format rfc3164
example:/config/logging/sink-1/#> priority console.notice;security.warning
example:/config/logging/sink-1/#> show
Target : udp address 10.0.0.1
Format : rfc3164
______________________________________________________________________________
Priority
Expr Facility Modifier Severity
1 console . notice
2 security . warning
example:/config/logging/sink-1/#> leave
example:/#>
The same setup can also be configured with a one-liner:
example:/#> configure
example:/config/#> logging
example:/config/logging/#> sink target udp address 10.0.0.1 format rfc3164 pri
ority console.notice;security.warning
example:/config/logging/sink-1/#> show
Target : udp address 10.0.0.1
Format : rfc3164
______________________________________________________________________________
Priority
Expr Facility Modifier Severity
1 console . notice
2 security . warning
example:/config/logging/sink-1/#> leave
example:/#>
This sink now sends any message of facility console
and severity
notice
or higher, and facility security
with severity warning
or
higher, to a syslog server at 10.0.0.1, UDP port 514. All messages are
also formatted according to RFC 3164, see above for details.