While trying to setup Cassandra database in a local Kubernetes cluster on a Mac OS (via Minikube), I am getting connection issues. It seems like Node.js is not able to resolve DNS settings correctly, but resolving via command line DOES work.
The setup is as following (simplified): Cassandra Service
apiVersion: v1
kind: Service
metadata:
labels:
app: cassandra
name: cassandra
spec:
type: NodePort
ports:
- port: 9042
targetPort: 9042
protocol: TCP
name: http
selector:
app: cassandra
In addition, there's a PersistentVolume and a StatefulSet.
The application itself is very basic
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: app1
labels:
app: app1
spec:
replicas: 1
selector:
matchLabels:
app: app1
template:
metadata:
labels:
app: app1
spec:
containers:
- name: app1
image: xxxx.dkr.ecr.us-west-2.amazonaws.com/acme/app1
imagePullPolicy: "Always"
ports:
- containerPort: 3003
And a service
apiVersion: v1
kind: Service
metadata:
name: app1
namespace: default
spec:
selector:
app: app1
type: NodePort
ports:
- port: 3003
targetPort: 3003
protocol: TCP
name: http
there also a simple ingress setup
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress
annotations:
kubernetes.io/ingress.class: nginx
spec:
rules:
- host: dev.acme.com
http:
paths:
- path: /app1
backend:
serviceName: app1
servicePort: 3003
And added to /etc/hosts the minikube ip address
192.xxx.xx.xxx dev.acme.com
So far so good.
When trying to call dev.acme.com/app1 via Postman, the node.js app itself is being called correctly (can see in the logs), HOWEVER, the app can not connect to Cassandra and times out with the following error:
"All host(s) tried for query failed. First host tried, 92.242.140.2:9042: DriverError: Connection timeout. See innerErrors."
The IP 92.242.140.2 seems to be just a public IP that is related to my ISP, I believe since the app is not able to resolve the service name.
I created a simple node.js script to test dns:
var dns = require('dns') dns.resolve6('cassandra', (err, res) => console.log('ERR:', err, 'RES:', res))
and the response is
ERR: { Error: queryAaaa ENOTFOUND cassandra at QueryReqWrap.onresolve [as oncomplete] (dns.js:197:19) errno: 'ENOTFOUND', code: 'ENOTFOUND', syscall: 'queryAaaa', hostname: 'cassandra' } RES: undefined
However, and this is where it gets confusing - when I ssh into the pod (app1), I am able to connect to cassandra service using:
cqlsh cassandra 9042 --cqlversion=3.4.4
So it seems as the pod is "aware" of the service name, but node.js runtime is not.
Any idea what could cause the node.js to not being able to resolve the service name/dns settings?
UPDATE
After re-installing the whole cluster, including re-installing docker, kubectl and minikube I am getting the same issue.
While running ping cassandra from app1 container via ssh, I am getting the following
PING cassandra.default.svc.cluster.local (10.96.239.137) 56(84) bytes of data. 64 bytes from cassandra.default.svc.cluster.local (10.96.239.137): icmp_seq=1 ttl=61 time=27.0 ms
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
Which seems to be fine. However, when running from Node.js runtime I am still getting the same error -
"All host(s) tried for query failed. First host tried, 92.242.140.2:9042: DriverError: Connection timeout. See innerErrors."
from K8s Issue connecting to Cassandra on Mac OS (via Node.js)
0 komentar:
Posting Komentar