Applications FAQ

Apache NiFi poses some challenges to get it running on Istio. These challenges come from the clustering requirements it has. For example, there is a requirement that cluster components must be uniquely addressable using cluster-wide host names. This requirement conflicts with Istio’s requirement that workloads bind and listen on localhost / 127.0.0.1 within the pod.

There are different ways to work around these issues based on your configuration requirements for your NiFi deployment. NiFi has at least three ways to specify what hostname should be used for cluster networking:

• nifi.remote.input.host - the host name that will be given out to clients to connect to this NiFi instance for Site-to-Site communication. By default, it is the value from InetAddress.getLocalHost().getHostName(). On UNIX-like operating systems, this is typically the output from the hostname command.

• nifi.web.https.host - The HTTPS host. It is blank by default. The jetty server will run on this hostname and it needs to be addressable across the cluster for replication with other nodes.

• nifi.cluster.node.address - The fully qualified address of the node. It is blank by default. This is used for cluster coordination as well and needs to be uniquely addressable within the cluster.

Some considerations:

• Using a blank or localhost setting for nifi.web.https.host doesn’t work in this case because of the networking requirements for unique addressing mentioned above.
• Unless you’re okay with all of your users having all access roles in your NiFi deployment, HTTP is not a viable solution as NiFi does not perform user authentication over HTTP.
• Explicitly specifying the networking interfaces that NiFi should use can help work around the issues and allow NiFi to work: Modify nifi.properties where xxx is the network interface that corresponds with the worker IP (differs based on environment/cloud provider) and yyy was the loopback interface (I.e lo) for the container/pod:

nifi.web.https.network.interface.default=xxx
nifi.web.https.network.interface.lo=yyy

A real-world example (valid for IBM Cloud, maybe others) would look like this:

nifi.web.https.network.interface.default=eth0
nifi.web.https.network.interface.lo=lo

By default, Cassandra broadcasts the address it uses for binding (accepting connections) to other Cassandra nodes as its address. This is usually the pod IP address and works fine without a service mesh. However, with a service mesh this configuration does not work. Istio and other service meshes require localhost (127.0.0.1) to be the address for binding.

There are two configuration parameters to pay attention to: listen_address and broadcast_address. For running Cassandra in an Istio mesh, the listen_address parameter should be set to 127.0.0.1 and the broadcast_address parameter should be set to the pod IP address.

These configuration parameters are defined in cassandra.yaml in the Cassandra configuration directory (e.g. /etc/cassandra). There are various startup scripts (and yaml files) used for starting Cassandra and care should be given to how these parameters are set by these scripts. For example, some scripts used to configure and start Cassandra use the value of the environment variable CASSANDRA_LISTEN_ADDRESS for setting listen_address.

There are two Elasticsearch configuration parameters that need to be set appropriately to run Elasticsearch with Istio: network.bind_host and network.publish_host. By default, these parameters are set to the network.host parameter. If network.host is set to 0.0.0.0, Elasticsearch will most likely pick up the pod IP as the publishing address and no further configuration will be needed.

If the default configuration does not work, you can set the network.bind_host to 0.0.0.0 or localhost (127.0.0.1) and network.publish_host to the pod IP. For example:

...
containers:
- name: elasticsearch
  image: docker.elastic.co/elasticsearch/elasticsearch:7.2.0
  env:
    - name: network.bind_host
      value: 127.0.0.1
    - name: network.publish_host
      valueFrom:
        fieldRef:
          fieldPath: status.podIP
   ...

Refer to Network Settings for Elasticsearch for more information.

Similar to other services deployed in an Istio service mesh, Redis instances need to listen on localhost (127.0.0.1). However, each Redis slave instance should announce an address that can be used by master to reach it, which cannot also be localhost (127.0.0.1).

Use the Redis configuration parameter replica-announce-ip to announce the correct address. For example, set replica-announce-ip to the IP address of each Redis slave instance using these steps:

Pass the pod IP address through an environment variable in the env subsection of the slave StatefulSet definition:

- name: "POD_IP"
  valueFrom:
    fieldRef:
      fieldPath: status.podIP

Also, add the following under the command subsection:

echo "" >> /opt/bitnami/redis/etc/replica.conf
echo "replica-announce-ip $POD_IP" >> /opt/bitnami/redis/etc/replica.conf

By default, Zookeeper listens on the pod IP address for communication between servers. Istio and other service meshes require localhost (127.0.0.1) to be the address to listen on.

There is a configuration parameter that can be used to change this default behavior: quorumListenOnAllIPs. This option allows Zookeeper to listen on all addresses including the localhost. Set this parameter to true by using the following command where $ZK_CONFIG_FILE is your Zookeeper configuration file.

$ echo "quorumListenOnAllIPs=true" >> $ZK_CONFIG_FILE