Better logstash filter to analyze SystemOut.log and some more

2016-05-29_22-56-05_featured

Last week I wrote a post about Using Docker and ELK to Analyze WebSphere Application Server SystemOut.log, but i wasn’t happy with my date filter and how the websphere response code is analyzed. The main problem was, that the WAS response code is not always on the beginning of a log message, or do not end with “:” all the time.

I replaced the used filter (formerly 4 lines with match) with following code:

grok {
        # was_shortname need to be regex, because numbers and $ can be in the word
        match => ["message", "\[%{DATA:wastimestamp} %{WORD:tz}\] %{BASE16NUM:was_threadID} (?<was_shortname>\b[A-Za-z0-9\$]{2,}\b) %{SPACE}%{WORD:was_loglevel}%{SPACE} %{GREEDYDATA:message}"]
        overwrite => [ "message" ]
        #tag_on_failure => [ ]
    }
grok {
        # Extract the WebSphere Response Code
        match => ["message", "(?<was_responsecode>[A-Z0-9]{9,10})[:,\s\s]"]
        tag_on_failure => [ ]
    }

(more…)

Using Docker and ELK to Analyze WebSphere Application Server SystemOut.log

2016-05-28_16-47-58

I often get SystemOut.log files from customers or friends to help them analyzing a problem. Often it is complicated to find the right server and application which generates the real error, because most WebSphere Applications (like IBM Connections or Sametime) are installed on different Application Servers and Nodes. So you need to open multiple large files in your editor, scroll each to the needed timestamps and check the lines before for possible error messages.

Klaus Bild showed on several conferences and in his blog the functionality of ELK, so I thought about using ELK too. I started to build a virtual machine with ELK Stack (Elasticsearch, Logstash & Kibana) and imported my local logs and logs i got mailed. This way is cool to analyze your environment, but adding just some SystemOut.logs from outside is not the best way. It’s hard to remove this stuff after analyzing from a ELK instance.

Then I found a tutorial to install ELK with Docker, there is even a great Part 2 of this post, which helps us installing an ELK Cluster. Just follow this blog posts, install Docker and Docker-compose. It’s really fast deployed. In my case i do not use flocker, it’s enough to use a local data container for the elasticsearch data.

Why do I use Docker instead of a virtual machine? It’s super easy to just drop the data container and begin with an empty database.

My setup

I created a folder on my Mac to put all needed stuff for the Docker images to it:

mkdir elk
cd elk

(more…)

IBM Connections 5.5 install Ephox Editors

Several people told me that installing the Editors is not described very well in the IBM Connections documentation. So i decided to write down the steps I used to deploy the editors. Hope it helps.

Check Installation on ephox: http://docs.ephox.com/display/EphoxForIBMConnections/Installing+Textbox.io+Services

  • Extract EPHOX_EDITORS3.0.1_CONN5.5.zip
    • edit config/config.js

      • editor: 'textbox.io | EditLive | CKEditor | role-based'
        This sets the default editor. When you use role-based you can enable textbox.io and EditLive for different user groups.
        Be aware that editlive needs a Java plugin which is mostly deactivated or outdated in actual browsers.
      • Enable Spelling-Servie URL (you need to deploy tbioServices_c5.ear):
        spellingServiceUrl: "https://connections-host/ephox-spelling"
        You have to set to https, that spell-checking works with http and https access to Connections.
        If you have selected role-based, you must deploy the EphoxEditorsForConnections.ear
      • When you want to use the builtin spellchecking, you need to install services/tbioServices_c5.ear
    • Start ./install.sh or install.exe

      • ./install.sh root@webspherehost <customization-dir> <webressources-dir>
        You need to type the root password 3 times, because installation uses seperate ssh calls for the installation
  • Create /opt/ephox/application.conf (WINDOWS: WAS_INSTALLATION_DRIVE:\opt\ephox\application.conf)
        
ephox {
     allowed-origins {
       origins = [
          "https://connections-55.panastoeps.local",
          "http://connections-55.panastoeps.local"
       ],
       url = "https://connections-55.panastoeps.local/ephox-allowed-origins/cors"
     }
}
  • Map Applications to your webserver, update the Connections versionstamp and restart Common and Ephox Applications
  • Verify: http://<your server and port>/connections/resources/web/ephox.editors.connections/verify.html

Missing images in Wikis after migration to IBM Connections 5.5

Wikis in IBM Connections 5.5 have a little bug, because the link (/library instead of /wikis/form/api/library) for images are wrong and so they are not displayed.

There is a technote, which should solve this issue, but the used way with ProxyPass is not what i want to use. When you use ProxyPass and ProxyPassReverse you should add a ProxyRequest off to be more secure. ProxyPass to localhost can be a problem too, i would suggest to change localhost to the Connections IHS Hostname.

Why do I use a different approach?

Most of my deployments already use a RewriteRule to redirect the hostname to there Connections Homepage, so i don’t need an additional module (which needs ressources and can have security considerations), when i can solve the image issue through mod_rewrite.

RewriteRule "^/library/(.*)" "/wikis/form/api/library/$1" [R,L]

If you haven’t set <forceConfidentialCommunications enabled="true"/> in LotusConnections-config.xml you need to set the RewriteRule and the ProxyPass config within your http and your https configuration parts!

Example httpd.conf:

... 
# HTTP configuration
<VirtualHost *:80>
    ServerName connections.example.com

    RewriteEngine On

     # Redirect hostname to Homepage
    RewriteRule ^\/$ https://connections.example.com/homepage [noescape,L,R]

    # Fix wrong wiki image URL
    RewriteRule "^/library/(.*)" "/wikis/form/api/library/$1" [R,L]
</VirtualHost>
# HTTPS configuration
<VirtualHost *:443>
    ServerName connections.example.com

    RewriteEngine On

    # Redirect hostname to Homepage
    RewriteRule ^\/$ https://connections.example.com/homepage [noescape,L,R]

    # Fix wrong wiki image URL
    RewriteRule "^/library/(.*)" "/wikis/form/api/library/$1" [R,L]

    SSLEnable
    SSLProtocolDisable SSLv2 SSLv3
</VirtualHost>
...