Doc update, added log4net handlers, general cleanup

- Lot’s of previously missing info in the README.md file added.
- Added UDP server for task failure notifications
- Refactored node.js file for greater readability
- Changed name of REST server from “Qlik Sense Butler” to “Butler for
Qlik Sense”

README.md

@@ -4,16 +4,19 @@ Node.js based proxy app for providing features accessible from Qlik Sense load s
 
 The app started out as a way of posting to [Slack](https://slack.com/) from [Qlik Sense](http://www.qlik.com/products/qlik-sense) (or [QlikView](http://www.qlik.com/products/qlikview)) load scripts, but has since been generalized and now supports the following features/endpoints:
 
-- Post to Slack.
+- Post to Slack from Sense load scripts
       Endpoint: /slack
 - Create directories on the server where Butler is running
       Endpoint: /createDir
 - Check available disk space
       Endpoint: /getDiskSpace
 - Publish message to MQTT topic
       Endpoint: /mqttPublishMessage
+- Forward task failure notifications to Slack
 
-Other endpoints can be added if/when needed - ideas include linking to services like [Pushover](https://pushover.net/), sending tweets, controlling USB status lights like [Blink(1)](https://blink1.thingm.com/) etc. Given the large number of node.js modules available, it is quite easy to add new integrations.
+- Forward user audit events (session start/stop, connection open/close) to Slack and MQTT
+
+Other endpoints can be added if needed - ideas include linking to services like [Pushover](https://pushover.net/), sending tweets, controlling USB status lights like [Blink(1)](https://blink1.thingm.com/) etc. Given the large number of node.js modules available, it is quite easy to add new integrations.
 
 Current work in progress is focused on having Butler subscribing to MQTT topics, and start Sense tasks based on messages arriving in those topics.
 This will open up for upstream (MQTT enabled) data sources telling Sense that new data is available. Many cases where upstream sources are today polled can be avoided, with lower server load and more up-to-date data for end users as results.
@@ -26,12 +29,29 @@ The following configuration needs to be entered in the source code (all of them
 - Enter the webhook URL you got when configuring incoming webhooks in Slack
 - MQTT broker to be used
 - IP number of server where Butler is running
+- Slack channel to which task failure notifications should be posted
+- Slack channel to which session start/stop and connection open/close events should be posted
+
+Two additional configurations are optional (you don't have to configure them if you do not plan to use these features):
+
+**Task failure notifications**
+
+If forwarding of task failure events (i.e. failure of tasks started from the QMC) is to be used, the log4net XML config file (called LocalLogConfig.xml, in the log4net_task-failed directory) must be placed in the C:\ProgramData\Qlik\Sense\Scheduler directory on the Sense server where the scheduler service is running.
+
+The XML file included in the repo contains two parts:
 
-If forwarding of session start/stop and connection open/close messages (i.e. when users log in/out of Sense) is to be used, the log4net XML config file (called LocalLogConfig.xml) must also be placed in the correct directory on the Sense server.
+1. Sending emails when tasks fail. Fill in the fields enclosed by brackets, i.e. <>
+2. Sending a UDP message to Butler when tasks fail. Same thing here, update text within brackets as needed
 
-A sample LocalLogConfig.xml is included in the repo, update it with the IP of the server where Butler is running, then deploy it to the Sense server where the proxy service is running. If you have a Sense cluster with multiple virtual proxies, linked to different proxies, you need to deploy the XML file on all those proxy servers (assuming you want to monitor them all for session start/stop etc).
+Save the XML file to the above mentioned location. The scheduler service should pick up the XML file right away, no service restart needed (probably..).  
 
-The XML file should be copied to C:\ProgramData\Qlik\Sense\Proxy.
+**User audit event notifications**
+
+If forwarding of session start/stop and connection open/close messages (i.e. when users log in/out of Sense) is to be used, the log4net XML config file (this one is also named LocalLogConfig.xml, in the log4net_user-audit-event directory) must be placed in the correct directory (C:\ProgramData\Qlik\Sense\Proxy) on the Sense server where the proxy service(s) is(are) running.
+
+The LocalLogConfig.xml file included in the repo is almost complete, update it with the IP of the server where Butler is running, then deploy it to the Sense server where the proxy service is running. Note that these can be two different servers.
+
+If you have a Sense cluster with multiple virtual proxies, linked to different proxies, you need to deploy the XML file on all those proxy servers (assuming you want to monitor them all for session start/stop etc).
 
 
 Starting
@@ -41,6 +61,11 @@ node /path/to/app/butler.js
 
 The app can also be managed by a node.js process monitor, such as [PM2](https://github.com/Unitech/pm2) or [Forever](https://www.npmjs.com/package/forever). Both have successfully been used with Butler. The advantage of using a process monitor is that it will automatically restart Butler if it for some reason terminates.
 
+A rather serious, not yet resolved issue is that both PM2 and Forever seems to have trouble auto-starting node-js apps when a Windows server reboots.   
+They can be configured to start when a user logs into the newly rebooted Windows server, but that extra step (logging into the server) is of course not good. Way better would be if PM2 or Forever (or some other tool) could start on server boot.  
+There for sure are solutions for this - feel free to investigate, fork and send a pull request with a solution. 
+
+
 Usage
 -----
 General usage instructions follow, it is recommended to also take a look at the .qvs include files, to understand how parameters are passed etc.
@@ -51,7 +76,7 @@ General usage instructions follow, it is recommended to also take a look at the
   Some of Butler's features require certain mapping tables etc to be in place.
   Therefore, Butler's init function should be called before any other calls to Butler are made:
 
-      CALL ButlerInit;     // Only done once at beginnning of load script
+      CALL ButlerInit;     // Only done once at beginning of load script
 
 - **Send messages to Slack**
 
@@ -120,6 +145,32 @@ General usage instructions follow, it is recommended to also take a look at the
         CALL PostToMQTT('qliksense/my_app/last_reload', Timestamp(Now()));
 
 
+  - **Failed task notifications**
+
+    Note: This is not an endpoint in Butler's REST API, but rather a combination of Sense's log4net logging library, and features built into Butler.
+
+    When a task started by the Sense QMC fails, log4net will (if you have configured the XML file described above) send a UDP message to Butler, using port 9998.
+    Butler will then forward this message to Slack, using the Slack channel defined at the top of the butler.js file.
+
+    log4net will also send an email to the email address specified in the XML file. In this case Butler is not involved in any way.
+
+    More information on log4net and how it is used in Sense can be found [here](http://help.qlik.com/en-US/sense/2.2/Subsystems/PlanningQlikSenseDeployments/Content/Server/Server-Logging-Using-Appenders-QSRollingFileAppender-Built-in-Appenders.htm).
+
+
+  - **Session start/stop, connection open/close notifications**
+
+    Note: This is not an endpoint in Butler's REST API, but rather a combination of Sense's log4net logging library, and features built into Butler.
+
+    When a user start or stops a session, or when a user's connection opens or closes, the log4net library will (assuming it has been configured as described above) send a UDP message to Butler, using port 9997.
+    Butler will then forward this message to
+    1. the Slack channel defined in butler.js, and
+    2. the following MQTT topics
+      * qliksense/session/start
+      * qliksense/session/stop
+      * qliksense/connection/open
+      * qliksense/connection/close
+
+
 Warning
 -------
 - You should make sure to configure the firewall of the server where Buter is running, so it only accepts calls from the desired clients/IP addresses.

butler.js

@@ -8,7 +8,9 @@ var os = require('os');
 
 // Set up various objects and variables needed by the app
 var slackWebhookURL = '<fill in your web hook URL from Slack>';
-const SLACK_LOGIN_NOTIFICATION_CHANNEL = '<fill in name of Slack chanel where audit events (login/logoff etc) should be displayed>';
+const SLACK_LOGIN_NOTIFICATION_CHANNEL = '<fill in name of Slack chanel where audit events (login/logoff etc) should be posted>';
+const SLACK_TASK_FAILURE_CHANNEL = '<fill in name of Slack channel where task failure events should be posted>';
+
 
 var slack = new slack(slackWebhookURL);
 
@@ -21,14 +23,21 @@ var mqttClient  = mqtt.connect('mqtt://<IP of MQTT server>');
 
 
 var restServer = restify.createServer({
-  name: 'Qlik Sense Butler'
+  name: 'Butler for Qlik Sense'
 });
 
 
+const UDP_HOST = '<IP of server where Butler is running>';
+
 // Listen on port 9997 for incoming UDP connections regarding session starting/stoping, or connection opening/closing
-var udpServer = dgram.createSocket({type:"udp4", reuseAddr:true});
-const UDP_PORT = 9997;
-const UDP_HOST = '<IP of server where Butler is running>';  // Should work with localhost too... but it doesn't. Investigation needed
+var udpServerSessionConnection = dgram.createSocket({type:"udp4", reuseAddr:true});
+const UDP_PORT_SESSION_CONNECTION = 9997;
+
+// Listen on port 9998 for incoming UDP connections regarding failed tasks
+var udpServerTaskFailure = dgram.createSocket({type:"udp4", reuseAddr:true});
+const UDP_PORT_TASK_FAILURE = 9998;
+
+
 
 
 function respondSlack(req, res, next) {
@@ -126,21 +135,25 @@ mqttClient.on('error', function (topic, message) {
 restServer.use(restify.queryParser());    // Enable parsing of http parameters
 
 
+// -------------------------------------------------------------------------
 // Set up UDP server for sending acting on session events from Sense
-udpServer.on('listening', () => {
-  var address = udpServer.address();
+// -------------------------------------------------------------------------
+udpServerSessionConnection.on('listening', () => {
+  var address = udpServerSessionConnection.address();
   console.log('UDP server listening on %s:%s', address.address, address.port);
-  mqttClient.publish('qliksense/butler/session_server', 'start');   // Publish MQTT message that UDP server has started
+  // Publish MQTT message that UDP server has started
+  mqttClient.publish('qliksense/butler/session_server', 'start');
 });
 
-udpServer.on('error', () => {
-  var address = udpServer.address();
+udpServerSessionConnection.on('error', () => {
+  var address = udpServerSessionConnection.address();
   console.log('UDP server error on %s:%s', address.address, address.port);
-  mqttClient.publish('qliksense/butler/session_server', 'error');   // Publish MQTT message that UDP server has reported an error
+  // Publish MQTT message that UDP server has reported an error
+  mqttClient.publish('qliksense/butler/session_server', 'error');
 });
 
 // Main handler for UDP messages relating to session and connection events
-udpServer.on('message', function(message, remote) {
+udpServerSessionConnection.on('message', function(message, remote) {
   var msg = message.toString().split(';');
   console.log('%s: %s for user %s/%s', msg[0], msg[1], msg[2], msg[3])
 
@@ -173,6 +186,37 @@ udpServer.on('message', function(message, remote) {
 });
 
 
+// -------------------------------------------------------------------------
+// Set up UDP server for acting on failed task events
+// -------------------------------------------------------------------------
+udpServerTaskFailure.on('listening', () => {
+  var address = udpServerTaskFailure.address();
+  console.log('UDP server listening on %s:%s', address.address, address.port);
+  // Publish MQTT message that UDP server has started
+  mqttClient.publish('qliksense/butler/task_failure', 'start');
+});
+
+udpServerTaskFailure.on('error', () => {
+  var address = udpServerTaskFailure.address();
+  console.log('UDP server error on %s:%s', address.address, address.port);
+  // Publish MQTT message that UDP server has reported an error
+  mqttClient.publish('qliksense/butler/task_failure', 'error');
+});
+
+udpServerTaskFailure.on('message', function(message, remote) {
+  var msg = message.toString().split(';');
+  console.log('%s: Task "%s" failed, associated with app "%s', msg[0], msg[1], msg[2], msg[3])
+
+  slack.send({
+    text: 'Failed task: "' + msg[1] + '", linked to app "' + msg[2] + '".',
+    channel: SLACK_TASK_FAILURE_CHANNEL,
+    username: msg[0],
+    icon_emoji: ':ghost:'
+  });
+});
+
+
+
 
 // Set up endpoints for REST server
 restServer.get('/slack', respondSlack);
@@ -187,4 +231,6 @@ restServer.listen(8080, function() {
   console.log('REST server listening on %s', restServer.url);
 });
 
-udpServer.bind(UDP_PORT, UDP_HOST);
+// Set up UDP server for incoming messages from Sense log4net appenders
+udpServerSessionConnection.bind(UDP_PORT_SESSION_CONNECTION, UDP_HOST);
+udpServerTaskFailure.bind(UDP_PORT_TASK_FAILURE, UDP_HOST);

butler_init.qvs

@@ -1,5 +1,5 @@
 // ------------------------------------------------------------
-// ** Initialise Butler data structures **
+// ** Initialize Butler data structures **
 //
 // Create mapping table for conversion from utf8 to URL encoded
 // Could be a good idea to save this to a QVD rather than to read it from online, in order to

log4net_task-failed/LocalLogConfig.xml

@@ -0,0 +1,54 @@
+<?xml version="1.0"?>
+<configuration>
+    <!-- Mail appender-->
+    <appender name="MailAppender" type="log4net.Appender.SmtpAppender">
+        <filter type="log4net.Filter.LevelRangeFilter">
+            <param name="levelMin" value="ERROR" />
+			<!--Sets the level of logging, in this case any ERROR in the log will be sent as an email-->
+        </filter>
+        <filter type="log4net.Filter.DenyAllFilter" />
+        <evaluator type="log4net.Core.LevelEvaluator">
+            <param name="threshold" value="ERROR"/>
+			<!--Sets the level of logging, in this case any ERROR in the log will be sent as an email-->
+        </evaluator>
+        <param name="to" value="<email address to send failed task notification emails to>" />
+        <param name="from" value="<sender email address used in notification emails>" />
+        <param name="subject" value="Qlik Sense failed task (server <servername>)" />
+        <param name="smtpHost" value="smtp.gmail.com" />
+        <param name="port" value="587" />
+        <param name="EnableSsl" value="true" />
+        <param name="Authentication" value="Basic" />
+        <param name="username" value="<Gmail username>" />
+        <param name="password" value="<Gmail password>" />
+        <param name="bufferSize" value="0" /> <!-- Set this to 0 to make sure an email is sent on every error -->
+        <param name="lossy" value="true" />
+        <layout type="log4net.Layout.PatternLayout">
+            <param name="conversionPattern" value="%newline%date %-5level %newline%property{TaskName}%newline%property{AppName}%newline%message%newline%newline%newline" />
+			<!--Defined conversion pattern for the output. To be able to output custom properties in the log (example, Taskname), append %property{propertyname} to the output pattern-->
+        </layout>
+    </appender>
+
+    <appender name="NodeTaskFailureLogger" type="log4net.Appender.UdpAppender">
+        <filter type="log4net.Filter.LevelRangeFilter">
+            <param name="levelMin" value="ERROR" />
+			<!--Sets the level of logging, in this case any ERROR in the log will be sent as an email-->
+        </filter>
+        <param name="remoteAddress" value="<IP of server where Butler is running>" />
+        <param name="remotePort" value="9998" />
+        <layout type="log4net.Layout.PatternLayout">
+            <converter>
+                <param name="name" value="hostname" />
+                <param name="type" value="Qlik.Sense.Logging.log4net.Layout.Pattern.HostNamePatternConverter" />
+             </converter>
+            <param name="conversionpattern" value="%hostname;%property{TaskName};%property{AppName};%property{UserId}" />
+        </layout>
+    </appender>
+
+    <!--Send mail on task failure-->
+    <logger name="System.Scheduler.Scheduler.Slave.Tasks.ReloadTask">
+	<!--Logger name identifies the component to monitor. This can be found by investigating the actual log file-->
+        <appender-ref ref="MailAppender" />
+        <appender-ref ref="NodeTaskFailureLogger" />
+		<!--appender-ref should match the name identifying the appender. More than one appender can be configured in the same configuration file-->
+    </logger>
+</configuration>

package.json

@@ -1,6 +1,6 @@
 {
   "name": "qliksensebutler",
-  "version": "0.0.1",
+  "version": "1.1.0",
   "description": "Proxy for carrying out features that Qlik Sense or Qlikview cannot do out of the box.",
   "dependencies": {
     "diskusage": "^0.1.4",