:icons: font

After some deployments of IBM Connections pink and IBM Cloud private, I want to share some tools, links and hopefully helpful information around these products.

Sources to start with troubleshooting Orientme

#

Blogs:

#

• Nico Meisenzahl

• Socially Integrated

IBM Think:

#

• Two Wrongs Don’t Make a Right—Troubleshooting IBM Connections

IBM Technotes:

#

• If Redis is not reachable, Indexing and Analysis tries to reconnect for 8 times and then fails

• Mongo and Redis Pods in an Unknown State

Some shortcuts, commands and tools

#

During my reading and researches, I optimized some of the shown scripts in the above mentioned links. On the other side I hate to open logs in each single pod, so I searched for a better solution.

Getting the logs of multiple pods

#

On the command line you can just use kubetail. Just download this script to your master node of IBM Cloud private. Move it to the PATH and make executable:

wget https://raw.githubusercontent.com/johanhaleby/kubetail/master/kubetail
sudo mv kubetail /usr/local/bin
sudo chmod +x /usr/local/bin/kubetail

With kubetail you get log messages in different colors for different pods.

kubetail es-master -s 15s

This shows the logs of the last 15 seconds of all elasticsearch pods and until you hit Ctrl+c you will get all new messages too.

Set default namespace in CfC/ICP

#

With Orientme 6.0.0.4 the namespace of all pink pods is moved from default to connections. So you need to add the namespace to each kubectl command you run on the console. So I like to change the default namespace for kubectl:

kubectl config set-context $(kubectl config current-context) --namespace=connections

To switch this back to the IBM default use:

You need to switch the default namespace back before you update your environment! The update script for 6.0.0.5 will not run properly when you set a different namespace!

kubectl config set-context $(kubectl config current-context) --namespace=default

To check which settings are used:

kubectl config view

Speed up the technote commands

#

Mongo and Redis Pods in an Unknown State

#

Original from Mongo and Redis Pods in an Unknown State

kubectl get pods -n connections | grep redis-server
kubectl get pods -n connections | grep mongo

Or combine the two commands with:

kubectl get pods -n connections | grep -E "redis-server|mongo"

mongo-0                                2/2       Unknown   0          19h
mongo-1                                2/2       Running   0          19h
mongo-2                                2/2       Running   0          19h
redis-server-0                         1/1       Running   0          19h
redis-server-1                         1/1       Unknown   0          19h
redis-server-2                         1/1       Running   0          19h

None of the listed pods should have the state unknown. Resolving that issue:

kubectl delete pod <pod name=""> -n connections --grace-period=0 --force</pod>

For example:

kubectl delete pod mongo-1 -n connections --grace-period=0 --force

Or you automate it a little and delete all pods with state Unknown:

kubectl delete pod $(kubectl get pods -n connections | grep -E "redis-server|mongo" | grep Unknown | awk '{print $1}') -n connections --grace-period=0 --force

Kubernetes will recreate the pod after a short time.

If Redis is not reachable, Indexing and Analysis tries to reconnect

#

for 8 times and then fails

Original from If Redis is not reachable, Indexing and Analysis tries to reconnect for 8 times and then fails

kubectl get pods -n connections | grep -E "analysisservice|indexingservice"

This will return a list of running pods:

analysisservice-132048526-bh02z 1/1 Running 4 2d
analysisservice-132048526-dhm45 1/1 Running 3 2d
indexingservice-3077571376-9b541 1/1 Running 0 3d
indexingservice-3077571376-qvq7j 1/1 Running 3 3d
indexingservice-3077571376-t61wh 1/1 Running 0 3d

Check the logs (use kubetail) of these pods! If you see “Exception while connecting”, delete the pods and kubernetes will recreate them.

# Show logs of all pods in the default namespace, including last 5 lines of each log
kubetail --tail 5

# Show all logs on console and append output to logfile
kubetail --tail 5 | tee -a logfile

# This will show logs of indexingservice and analysisservice
kubetail '(indexingservice|analysisservice)' -e regex -s 30s

# Delete all indexingservice and analysisservice pods
kubectl delete pod $(kubectl get pods -n connections | grep -E "analysisservice|indexingservice" | awk '{print $1}') -n connections --grace-period=0 --force

So next time I try to get all that directly from the ICP Admin Interface or Elasticsearch.