Log Management with AMETHYST

Introduction

By default, the log files generated by Mapping are in a proprietary format so they can be read by the MAPPING user interface. The log files can be converted to text using the map_log_txt command but in this case you won't be able to change the formatting. With Amethyst, you can now generate log files with customizable formatting for use with various third-party solutions. Enabling Amethyst and disabling the old log system also improves system performance.

Limitations

Amethyst is still in its early stage of development. In release 9.1.0 you can use Amethyst for Workflows only (map_809 binary). Future enhancements will be made to use Amethyst with other binaries (mapcpysplf, map_lpr and more).

Supported output

The log files can be read with:

• Text file
• The console who called the process
• HTTP request

Requirements

• Onyx Server v9.1.0 or higher

Configuration

To enable Amethyst log management, the changes below need to be made:

mapping.conf

1. Go to LOGS
2. Add the following parameters

• LOG_TYPE
  • Default : Use the old log system
  • Amethyst : Use the new log system (Amethyst)
  • All : Use both log system (old and Amethyst)
• PATH_AMETHYST_LOG
  • Path to Amethyst configuration file

config.yaml

Amethyst needs a configuration file called config.yaml by default and saved in the Mapping conf or settings folder. Its name and path are specified in the file mapping.conf under LOGS with the parameter PATH_AMETHYST_LOG The configuration file is in yaml format and includes the configurations of the Loggers for the new log system.

Config.yaml file

Below is the description of the configuration file:

config

• config

• server

• port: log serveur port
• threads: number of threads used by the server (max. 250)

loggers

• loggers

• source: name of the log source retrieved by the programs (`workflow` is the name of the logger for map_809)
  • Workflow
• level: minimum log level
  • debug
  • information
  • warning
  • error
• channels: list of channels for each type

• type: channel type

FileChannel

• path: log file path (when Type = FileChannel)

• Log file path

• rotation: new log file creation frequency (one file a day, a month, a week...)

• never: no log rotation
• [day,][hh]:mm: the file is rotated on specified day/time day - day is specified as long or short day name (Monday|Mon, Tuesday|Tue, ... ); day can be omitted, in which case log is rotated every day hh - valid hour range is 00-23; hour can be omitted, in which case log is rotated every hour mm - valid minute range is 00-59; minute must be specified
• daily: the file is rotated daily
• weekly: the file is rotated every seven days
• monthly: the file is rotated every 30 days
• <n> minutes: the file is rotated every <n> minutes, where <n> is an integer greater than zero.
• <n> hours: the file is rotated every <n> hours, where <n> is an integer greater than zero.
• <n> days: the file is rotated every <n> days, where <n> is an integer greater than zero.
• <n> weeks: the file is rotated every <n> weeks, where <n> is an integer greater than zero.
• <n> months: the file is rotated every <n> months, where <n> is an integer greater than zero and a month has 30 days.
• <n>: the file is rotated when its size exceeds <n> bytes.
• <n> K: the file is rotated when its size exceeds <n> Kilobytes.
• <n> M: the file is rotated when its size exceeds <n> Megabytes.

• compress: specifies whether the old log files must be compressed in a compressed folder
  • false: the files are not compressed
  • true: the files are compressed
    
    
• purgeAge: deletes old log files automatically based on their create date
  • <n> seconds: deletes files older than <n> seconds.
  • <n> minutes: deletes files older than <n> minutes.
  • <n> hours: deletes files older than <n> hours.
  • <n> days: deletes files older than <n> days.
  • <n> weeks: deletes files older than <n> weeks.
  • <n> months: deletes files older than <n> months.
    
    
• purgeCount: deletes older log files automatically when the maximum number of files has been reached.
  • <n>: only the <n> more recent files are kept.
    
    
• format : format of each log message

• %s - message source
• %t - message text
• %l - message priority level (1 .. 7)
• %p - message priority (Fatal, Critical, Error, Warning, Notice, Information, Debug, Trace)
• %q - abbreviated message priority (F, C, E, W, N, I, D, T)
• %P - message process identifier
• %T - message thread name
• %I - message thread identifier (numeric)
• %N - node or host name
• %U - message source file path (empty string if not set)
• %u - message source line number (0 if not set)
• %w - message date/time abbreviated weekday (Mon, Tue, ...)
• %W - message date/time full weekday (Monday, Tuesday, ...)
• %b - message date/time abbreviated month (Jan, Feb, ...)
• %B - message date/time full month (January, February, ...)
• %d - message date/time zero-padded day of month (01 .. 31)
• %e - message date/time day of month (1 .. 31)
• %f - message date/time space-padded day of month ( 1 .. 31)
• %m - message date/time zero-padded month (01 .. 12)
• %n - message date/time month (1 .. 12)
• %o - message date/time space-padded month ( 1 .. 12)
• %y - message date/time year without century (70)
• %Y - message date/time year with century (1970)
• %H - message date/time hour (00 .. 23)
• %h - message date/time hour (00 .. 12)
• %a - message date/time am/pm
• %A - message date/time AM/PM
• %M - message date/time minute (00 .. 59)
• %S - message date/time second (00 .. 59)
• %i - message date/time millisecond (000 .. 999)
• %c - message date/time centisecond (0 .. 9)
• %F - message date/time fractional seconds/microseconds (000000 - 999999)
• %z - time zone differential in ISO 8601 format (Z or +NN.NN)
• %Z - time zone differential in RFC format (GMT or +NNNN)
• %L - convert time to local time (must be specified before any date/time specifier; does not itself output anything)
• %E - epoch time (UTC, seconds since midnight, January 1, 1970)
• %v[width] - the message source (%s) but text length is padded/cropped to 'width'
• %[name] - the value of the message parameter with the given name
• %% - percent sign

Example: '[%q]%Y-%m-%d %H:%M:%S:%F [PID:%P][THREAD %I] %t'

ConsoleChannel

• format : the format of each log message

• %s - message source
• %t - message text
• %l - message priority level (1 .. 7)
• %p - message priority (Fatal, Critical, Error, Warning, Notice, Information, Debug, Trace)
• %q - abbreviated message priority (F, C, E, W, N, I, D, T)
• %P - message process identifier
• %T - message thread name
• %I - message thread identifier (numeric)
• %N - node or host name
• %U - message source file path (empty string if not set)
• %u - message source line number (0 if not set)
• %w - message date/time abbreviated weekday (Mon, Tue, ...)
• %W - message date/time full weekday (Monday, Tuesday, ...)
• %b - message date/time abbreviated month (Jan, Feb, ...)
• %B - message date/time full month (January, February, ...)
• %d - message date/time zero-padded day of month (01 .. 31)
• %e - message date/time day of month (1 .. 31)
• %f - message date/time space-padded day of month ( 1 .. 31)
• %m - message date/time zero-padded month (01 .. 12)
• %n - message date/time month (1 .. 12)
• %o - message date/time space-padded month ( 1 .. 12)
• %y - message date/time year without century (70)
• %Y - message date/time year with century (1970)
• %H - message date/time hour (00 .. 23)
• %h - message date/time hour (00 .. 12)
• %a - message date/time am/pm
• %A - message date/time AM/PM
• %M - message date/time minute (00 .. 59)
• %S - message date/time second (00 .. 59)
• %i - message date/time millisecond (000 .. 999)
• %c - message date/time centisecond (0 .. 9)
• %F - message date/time fractional seconds/microseconds (000000 - 999999)
• %z - time zone differential in ISO 8601 format (Z or +NN.NN)
• %Z - time zone differential in RFC format (GMT or +NNNN)
• %L - convert time to local time (must be specified before any date/time specifier; does not itself output anything)
• %E - epoch time (UTC, seconds since midnight, January 1, 1970)
• %v[width] - the message source (%s) but text length is padded/cropped to 'width'
• %[name] - the value of the message parameter with the given name
• %% - percent sign

Exemple de valeur : [Console] %H:%M:%S [%p]%s: %t

ColorConsoleChannel

• path: path to the generated log file (when Type = FileChannel)

• Path to the generated log file

• format: format of each log message

• %s - message source
• %t - message text
• %l - message priority level (1 .. 7)
• %p - message priority (Fatal, Critical, Error, Warning, Notice, Information, Debug, Trace)
• %q - abbreviated message priority (F, C, E, W, N, I, D, T)
• %P - message process identifier
• %T - message thread name
• %I - message thread identifier (numeric)
• %N - node or host name
• %U - message source file path (empty string if not set)
• %u - message source line number (0 if not set)
• %w - message date/time abbreviated weekday (Mon, Tue, ...)
• %W - message date/time full weekday (Monday, Tuesday, ...)
• %b - message date/time abbreviated month (Jan, Feb, ...)
• %B - message date/time full month (January, February, ...)
• %d - message date/time zero-padded day of month (01 .. 31)
• %e - message date/time day of month (1 .. 31)
• %f - message date/time space-padded day of month ( 1 .. 31)
• %m - message date/time zero-padded month (01 .. 12)
• %n - message date/time month (1 .. 12)
• %o - message date/time space-padded month ( 1 .. 12)
• %y - message date/time year without century (70)
• %Y - message date/time year with century (1970)
• %H - message date/time hour (00 .. 23)
• %h - message date/time hour (00 .. 12)
• %a - message date/time am/pm
• %A - message date/time AM/PM
• %M - message date/time minute (00 .. 59)
• %S - message date/time second (00 .. 59)
• %i - message date/time millisecond (000 .. 999)
• %c - message date/time centisecond (0 .. 9)
• %F - message date/time fractional seconds/microseconds (000000 - 999999)
• %z - time zone differential in ISO 8601 format (Z or +NN.NN)
• %Z - time zone differential in RFC format (GMT or +NNNN)
• %L - convert time to local time (must be specified before any date/time specifier; does not itself output anything)
• %E - epoch time (UTC, seconds since midnight, January 1, 1970)
• %v[width] - the message source (%s) but text length is padded/cropped to 'width'
• %[name] - the value of the message parameter with the given name
• %% - percent sign

Example : '[%q]%Y-%m-%d %H:%M:%S:%F [PID:%P][THREAD %I] %t'

• traceColor: message colour for trace level (when Type = ColorConsoleChannel)
• informationColor : message colour for information level (when Type = ColorConsoleChannel)
• noticeColor : message colour for notice level (when Type = ColorConsoleChannel)
• warningColor : message colour for warning level (when Type = ColorConsoleChannel)
• criticalColor : message colour for critical level (when Type = ColorConsoleChannel)
• fatalColor : message colour for fatal level (when Type = ColorConsoleChannel)

Supported colours for ColocConsoleChannel:

• default
• black
• red
• green
• brown
• blue
• magenta
• cyan
• gray
• darkgray
• lightRed
• lightGreen
• yellow
• lightBlue
• lightMagenta
• lightCyan
• white

HttpChannel

• url

• URL executed for sending logs

• method

• POST
• PUT

• timeout: HTTP request timeout

• value in seconds (e.g. '2')

• headers : HTTP header sent with the data

• Authorization: HTTP security header (e.g. 'Bearer eyJhbGciOiJIUzUxMiJ9.eyJzdWIiOiJhZG1pbiIsImF1dGgiOiJST0xFX0FETUlOLFJPTEVfVVNFUiIsImV4cCI6MTU3MDA4OTIwMX0.xhmD8lCmnJVJs5b6VelcRnSJmoeEABMBf7-A3KdIM1uUbjOstSDAv0c7c8hnL1U54qYnmkiV46L13TSsByuMIQ'
• Content-Type: content header (e.g. 'application/json')

• format: log message format

• %s - message source
• %t - message text
• %l - message priority level (1 .. 7)
• %p - message priority (Fatal, Critical, Error, Warning, Notice, Information, Debug, Trace)
• %q - abbreviated message priority (F, C, E, W, N, I, D, T)
• %P - message process identifier
• %T - message thread name
• %I - message thread identifier (numeric)
• %N - node or host name
• %U - message source file path (empty string if not set)
• %u - message source line number (0 if not set)
• %w - message date/time abbreviated weekday (Mon, Tue, ...)
• %W - message date/time full weekday (Monday, Tuesday, ...)
• %b - message date/time abbreviated month (Jan, Feb, ...)
• %B - message date/time full month (January, February, ...)
• %d - message date/time zero-padded day of month (01 .. 31)
• %e - message date/time day of month (1 .. 31)
• %f - message date/time space-padded day of month ( 1 .. 31)
• %m - message date/time zero-padded month (01 .. 12)
• %n - message date/time month (1 .. 12)
• %o - message date/time space-padded month ( 1 .. 12)
• %y - message date/time year without century (70)
• %Y - message date/time year with century (1970)
• %H - message date/time hour (00 .. 23)
• %h - message date/time hour (00 .. 12)
• %a - message date/time am/pm
• %A - message date/time AM/PM
• %M - message date/time minute (00 .. 59)
• %S - message date/time second (00 .. 59)
• %i - message date/time millisecond (000 .. 999)
• %c - message date/time centisecond (0 .. 9)
• %F - message date/time fractional seconds/microseconds (000000 - 999999)
• %z - time zone differential in ISO 8601 format (Z or +NN.NN)
• %Z - time zone differential in RFC format (GMT or +NNNN)
• %L - convert time to local time (must be specified before any date/time specifier; does not itself output anything)
• %E - epoch time (UTC, seconds since midnight, January 1, 1970)
• %v[width] - the message source (%s) but text length is padded/cropped to 'width'
• %[name] - the value of the message parameter with the given name
• %% - percent sign

Value: '{"source":"%s", "date":"%Y-%m-%d %H:%M:%S:%F", "pid":%P, "message":"%t"}'

Mapping Variables

Mapping variables such as MAP_PRINTER or MAP_JOBNUM can be used in YAML.

For example, MAP_PRINTER can be used to give a log file the name of a queue:

path: '/apps/mapping/data/[%MAP_PRINTER%].log'

Amethyst also supports variables added using map_lp but need to comply with the following rules:

• The variable must be written between '[%' and '%]'
• The variable must not have spaces
• The only supported special characters are '-' and '_'.
• If a variable is not found, it is deleted. For example, if "[%MY_PARAM%]" does not exist, it is replaced with "".

Sample configuration file

Example 1

config:
   server:
       port: 7476
       threads: 11
   loggers:          
       -   
           source: 'Workflow'
           level: 'information'
           channels:
               -
                   type: 'FileChannel'
                   path: '/apps/mapping/data/MyLog.log'
                   rotation: 'daily'
                   format: '[%q]%Y-%m-%d %H:%M:%S:%F [PID:%P][THREAD %I]    %t'

Generated log file: Log file /apps/mapping/data/MyLog.log

# Log file created/rotated Thursday, 19 Sep 19 10:36:18 GMT
[I]2019-09-19 10:36:18:129349 [PID:6881][THREAD 0]    Run file /apps/mapping/data/conf/rules/workflow/rulesfile.rules.xml
[I]2019-09-19 10:36:18:131010 [PID:6881][THREAD 0]    Set PRINTER=MYPRINTERVALUE
[I]2019-09-19 10:36:18:131186 [PID:6881][THREAD 0]    Set PRINTERNAME=MYPRINTERNAME
[I]2019-09-19 10:36:18:131192 [PID:6881][THREAD 0]    Run SET
[I]2019-09-19 10:36:18:131247 [PID:6881][THREAD 0]    /apps/mapping/data/cmd.sh
[I]2019-09-19 10:36:18:152919 [PID:6881][THREAD 0]    End of the Workflow /apps/mapping/data/conf/rules/workflow/rulesfile.rules.xml.Time = 0.010 seconds.
[I]2019-09-19 10:36:18:546324 [PID:6901][THREAD 0]    Run file /apps/mapping/data/conf/rules/workflow/rulesfile.rules.xml
[I]2019-09-19 10:36:18:547191 [PID:6901][THREAD 0]    Set PRINTER=MYPRINTERVALUE
[I]2019-09-19 10:36:18:547350 [PID:6901][THREAD 0]    Set PRINTERNAME=MYPRINTERNAME
[I]2019-09-19 10:36:18:547356 [PID:6901][THREAD 0]    Run SET
[I]2019-09-19 10:36:18:547409 [PID:6901][THREAD 0]    /apps/mapping/data/cmd.sh
[I]2019-09-19 10:36:18:569399 [PID:6901][THREAD 0]    End of the Workflow /apps/mapping/data/conf/rules/workflow/rulesfile.rules.xml.Time = 0.010 seconds.
[I]2019-09-19 10:36:18:943406 [PID:6921][THREAD 0]    Run file /apps/mapping/data/conf/rules/workflow/rulesfile.rules.xml
[I]2019-09-19 10:36:18:944466 [PID:6921][THREAD 0]    Set PRINTER=MYPRINTERVALUE
[I]2019-09-19 10:36:18:944615 [PID:6921][THREAD 0]    Set PRINTERNAME=MYPRINTERNAME
[I]2019-09-19 10:36:18:944620 [PID:6921][THREAD 0]    Run SET
[I]2019-09-19 10:36:18:944703 [PID:6921][THREAD 0]    /apps/mapping/data/cmd.sh
[I]2019-09-19 10:36:18:973966 [PID:6921][THREAD 0]    End of the Workflow /apps/mapping/data/conf/rules/workflow/rulesfile.rules.xml.Time = 0.010 seconds.
[I]2019-09-19 10:36:19:263098 [PID:6941][THREAD 0]    Run file /apps/mapping/data/conf/rules/workflow/rulesfile.rules.xml
[I]2019-09-19 10:36:19:264103 [PID:6941][THREAD 0]    Set PRINTER=MYPRINTERVALUE
[I]2019-09-19 10:36:19:264284 [PID:6941][THREAD 0]    Set PRINTERNAME=MYPRINTERNAME
[I]2019-09-19 10:36:19:264290 [PID:6941][THREAD 0]    Run SET
[I]2019-09-19 10:36:19:264343 [PID:6941][THREAD 0]    /apps/mapping/data/cmd.sh
[I]2019-09-19 10:36:19:287644 [PID:6941][THREAD 0]    End of the Workflow /apps/mapping/data/conf/rules/workflow/rulesfile.rules.xml.Time = 0.000 seconds.

Example 2: Detailed configuration file

config:
   server:
       port: 7476
       threads: 11
   loggers:
       -
           source: 'Conversion'
           level: 'debug'
           channels:
               -
                   type: 'FileChannel'
                   path: 'C:\Users\jdobbels\Documents\6-M-Designer\KrakenLogs\conversion.log'
                   rotation: '2 K'
                   format: '---------[%q]%Y-%m-%d %H:%M:%S:%F [PID:%P][THREAD %I]    %t'
               -
                   type: 'ConsoleChannel'
                   format: '[Console] %H:%M:%S     [%p]        %s: %t'
       -   source: 'Workflow'
           level: 'information'
           channels:
               -
                   type: 'FileChannel'
                   path: '/apps/mapping/mapout/workflow1.log'
                   rotation: 'daily'
                   format: '[%q]%Y-%m-%d %H:%M:%S:%F [PID:%P][THREAD %I]    %t'
               -
                   type: 'FileChannel'
                   path: '/apps/mapping/mapout/[%MAP_PRINTER%].log'
                   rotation: '50 M'
                   compress: 'true'
                   purgeCount: '25'
                   format: '%Y-%m-%d %H:%M:%S:%F %t'
               -
                   type: 'ColorConsoleChannel'
                   format: '[ColorConsole] %H:%M:%S     [%p]        %s: %t'
                   traceColor: 'gray'
                   debugColor: 'lightcyan'
                   informationColor: 'lightgreen'
                   noticeColor: 'lightgreen'
                   warningColor: 'yellow'
                   errorColor: 'lightRed'
                   criticalColor: 'lightred'
                   fatalColor: 'darkgray'
               -
                   type: 'HttpChannel'
                   url: 'http://localhost:8080/api/log-mps'
                   method: 'POST'
                   timeout: '2 s'
                   headers:
                       Content-Type: 'application/json'    
                   format: '{"source":"%s", "date":"%Y-%m-%d %H:%M:%S:%F", "pid":%P, "message":"%t"}'

External links

Supported dynamic values: https://pocoproject.org/docs/Poco.PatternFormatter.html
Supported colours: https://pocoproject.org/docs/Poco.ColorConsoleChannel.html