Update Elasticsearch certificates in Componentpack

Elasticsearch in HCL Connections Componentpack is secured with Searchguard and needs certificates to work properly. These certificates are generated by bootstrap during the initial container deployment with helm.

These certificates are valid for 10 years (chain_ca.pem) or 2 years (elasticsearch*.pem) and stored in the Kubernetes secrets elasticsearch-secret, elasticsearch-7-secret. So when your HCL Connections deployment is running for 2 years, the certficates stop working.

The documentation on bootstrap is a little bit misleading and my suggested update does not make it into a technote or documentation update since nearly one year.

Set up bootstrap charts

#

Bootstrap is one-time job. It creates secrets and certificates for various components, including Redis, MongoDB and ElasticSearch.

Be aware that simply rerunning/upgrading bootstrap also means recreating all those secrets and certificates, and requires in some cases redoing the steps which were dependent on it (like setting up certificates again for Metrics).

Starting with Component Pack 7, bootstrap will also create secrets and certificates for ElasticSearch 7.

https://help.hcltechsw.com/connections/v7/admin/install/cp_install_services_tasks.html?hl=bootstrap#cp_install_services_tasks__section_iqb_24c_qmb

For Elasticsearch this is wrong. Bootstrap does only generate new certificates, if the variable env.force_regenerate is set to true. The default is false, so we have to set it during the helm command.

Update Elasticsearch certificates (Example commands for Elasticsearch 7)

#

If you are still using Elasticsearch 5, then just remove -7 from the commands below.

Get certificate from Kubernetes secret

#

kubectl get secret elasticsearch-7-secret -n connections \
   -o=jsonpath="{.data['elasticsearch-admin\.crt\.pem']}" | base64 -d > elasticsearch-admin.crt.pem

Check dates for certificate

#

openssl x509 -startdate -enddate -noout -in elasticsearch-admin.crt.pem
notBefore=Aug 16 08:10:54 2022 GMT
notAfter=Aug 15 08:10:54 2024 GMT

Delete bootstrap

#

helm delete bootstrap -n connections

Reinstall with helm

#

If we use the command from documentation, the certificate extracted from Kubernetes has still the same enddate.

helm install bootstrap \
   /opt/hcl-cnx-component-pack/microservices_connections/hybridcloud/helmbuilds/bootstrap-0.1.0-20210418-223218.tgz \
   -f /home/ansible/generated_charts/bootstrap.yml -n connections

We need to run

helm install bootstrap \
   /opt/hcl-cnx-component-pack/microservices_connections/hybridcloud/helmbuilds/bootstrap-0.1.0-20210418-223218.tgz \
   -f /home/ansible/generated_charts/bootstrap.yml -n connections --set env.force_regenerate=true

Check if the bootstrap pod is completed, then check the certificate again:

kubectl get pods -n connections | grep bootstrap
bootstrap-p4rj6                                             0/1     Completed   0          49m

kubectl get secret elasticsearch-7-secret -n connections \
   -o=jsonpath="{.data['elasticsearch-admin\.crt\.pem']}" | base64 -d > elasticsearch-admin_neu.crt.pem

openssl x509 -startdate -enddate -noout -in elasticsearch-admin_neu.crt.pem
notBefore=Sep  2 10:01:56 2022 GMT
notAfter=Sep  1 10:01:56 2024 GMT

Restart Elasticsearch

#

kubectl rollout restart sts es-master-7 es-data-7 -n connections
kubectl rollout restart deploy es-client-7 -n connections

Recreate elasticsearch-metrics.p12

#

kubectl get secret elasticsearch-7-secret -n connections \
   -o=jsonpath="{.data['chain-ca\.pem']}" | base64 -d > chain-ca.pem

kubectl get secret elasticsearch-7-secret -n connections \
   -o=jsonpath="{.data['elasticsearch-metrics\.p12']}" | base64 -d > elasticsearch-metrics.p12

cd /opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin
./wsadmin.sh -lang jython -user wasadmin_user -password wasadmin_password

execfile('esSecurityAdmin.py')
enableSslForMetrics('/opt/IBM/certs/es_certs/elasticsearch-metrics.p12', 'Elasticsearch_CA_password', '/opt/IBM/certs/es_certs/chain-ca.pem', '30098')

Copy the updated elasticsearch-metrics.p12 file from the Deployment Manager to the same location on the WebSphere Application Server nodes. Then restart your WebSphere environment (with Deployment Manager and Node Agents).