Liferay Unconference 2017: Docker and Kubernetes

This is a “little” summary of the Liferay Unconference open-space discussion with the topics of Docker and Kubernetes held in Amsterdam on Wednesday, October 4th 2017.
Thanks to Arne and Lex for sending me their notes!

We started off with a little introduction round where everyone said hi and mentioned if and what they are using Docker for plus what their main questions and interests are. This resulted in a bit of a list of discussion topics as such:

• Clustering, how to do LR clustering in a container environment
• Continuous Integration, how to integrate docker in your CI pipeline
• Deployment both of plugins and of your containers
• Orchestration, how to set up and manage all these new containers
• Development, how can we use Docker to ease and improve development
• Search services, add some Solr or Elastic search containers
• Statelessness, the holy grail of stateless container, is it possible with LR

There are quite a few people in the group who are using Docker in some way or are experimenting with it. A few who have no experience yet, but are interested and
thus joined the discussion group. Also, not many people who are using Docker + LR in production yet, but have used Docker in combination with other application (PHP
or an ELK monitoring stack which monitors a LR in a traditional VM). Also not too many people who are using a higher abstraction level orchestration like Kubernetes or Redhat’s OpenShift.

As service providers, there are a few people using Google facilities, but also Microsoft Azure (including their “Elasticsearch as a service” offering) and of course Amazon Web Services. For clustering, Liferay containers sound like a good match, but you still have to have handled a few topics. In general, a database cluster is not a big problem since many service providers and vendors have an offering for that.

For managing the document and media library it gets a bit more complex. If you are on AWS it’s simpler, because Liferay supports a doclib on AWS’ S3 service out of the box. That feature is however pretty hardcoded to work only with S3, not with compatible services from other providers/vendors. In that case, you generally have to go for a distributed or clustered filesystem. The consensus in the group is that while real distributed filesystems, life GlusterFS or Ceph, are possible they are too slow to use in production. Google doesn’t offer a disk service which can be written to concurrently by multiple nodes. Setting up an NFS mount is a traditional method for this and seems still to be a decent option.

An alternative is to set up your cluster that there is only one node in which edits can be done, which then update the documents in a CDN or Nginx and all other nodes are read nodes.

Licences are still a problem/grey area for dynamic clustering. With scaling up and doing rolling update the amount of LRs which are running at the same time is variable. Legally you should have a number of licenses for the maximum number of LRs you are running, but that is a waste if you are not always running with that amount of nodes.

There is a concept of an “Elastic license” but that would require having at least 2 platinum licenses before you can even think about that one.

Many people are having questions regarding this and while there is awareness at Liferay about this issue, a “perfect” solution is yet to be found. Everybody who needs such a feature or flexibility should talk to Liferay about this, so we can create a big enough demand for this and it becomes a higher priority for Liferay.

With regards to deployment of plugins, there are 2 main ways to go. Bake the plugins into your Docker image at build time or deploy them runtime while you make sure that you have configured your Liferay container to store your web apps and OSGI data in a data volume, so they are persistent through restarts and container redeploys. As always it is good advice to back up your data before deployments.

Regarding orchestration three solutions were mentioned, Docker Swarm, Kubernetes and OpenShift. In general, Kubernetes has the highest flexibility and should be a good choice for larger and longer projects. Docker + Docker Swarm is easier to pick up but is not really the industry standard. OpenShift is based atop of Kubernetes and tries to hide and ease a lot of the complexity.

For using containers in development docker could be nice. In principle, you could run the same image locally on your development workstation as you are using on production. There is some tooling for this in Eclipse, but this does not seem to be used much. Running Kubernetes on your workstation is possible with Minikube, but is more complex to set up and manage than just running Docker which works well on every platform.

Due to the fact that Elastic Search’ cluster features are well suited for clustering, it’s generally a good match for running in a container. You still need a recommended minimal of 3 ES nodes for a proper cluster. If you cluster over 2 datacenters and you want to guard against the risk of one datacenter going down you still should use a 3rd location. The risk of a hypervisor or machine going down in a single datacenter is still larger than the complete datacenter going down, so 3+ elastic search nodes running would still guard against that.

Clustering Solr for LR6.2 is more complex to set up. If you’re using DXP, then Elastic Search is by far preferable.

The statelessness of an application is the holy grail of large container clusters. Due to the somewhat dynamic nature of Liferay, it’s not easy to make it completely stateless without sacrificing a lot of flexibility. Hot deploying and installing plugins from the marketplace are some of the things you need to sacrifice.

Also discussed was security when using Docker. In general, big vendor docker image is considered mostly trusted when getting them from Docker Hub. But very often you would fork them in your own repositories and modifying or enriching them for your own specific use cases. Docker Hub, as well as Github, do support cryptographic signatures on uploaded images and respectively commits. But those features are not used 100%.

Liferay does have some docker images for LR7, but that’s not officially supported.

For monitoring of logs, there are also 2 (and probably more) ways to go. The default practice with Docker is to log everything to standard out and standard error and use tooling to monitor those via docker. Alternatively you could point your log files to a separate data volume specifically for your log files. You can then mount this log volume in a different container which has something like Filebeat to read and keep track of those files and send them to a (centralised) ELK stack.

Metric and trend monitoring can be done using Prometheus and Grafana. Being able to combine event logging and metric logging in one application would be very useful.

All in all a lot was discussed and it was a very lively and productive discussion. Thanks to all the participants!

I’m sure I forgot some details, please drop us a note or a comment and we’ll append this summary.