mesh service_在Kubernetes上使用istio Service Mesh运行Jupyterhub进行故障排除

最新推荐文章于 2024-07-05 17:55:50 发布

weixin_26711867

最新推荐文章于 2024-07-05 17:55:50 发布

阅读量439

点赞数

文章标签： python

原文链接：https://medium.com/swlh/running-jupyterhub-with-istio-service-mesh-on-kubernetes-a-troubleshooting-journey-707039f36a7b

版权

本文介绍了如何在Kubernetes上利用Istio Service Mesh解决Jupyterhub运行时的问题，详细记录了一次故障排查的历程。

摘要由CSDN通过智能技术生成

mesh service

JupyterHub is an open-source tool that offers the ability to spin up Jupyter notebook servers on demand. The notebooks can be used for data analysis or to create and execute Machine learning models. Istio is a service mesh that offers secure and observable communication mechanism between different services in a Kubernetes cluster.One of the benefits of running JupyterHub in an istio-enabled cluster is to gain support for mTLS(mutual TLS) capabilities between different JupyterHub components. mTLS ensures that all communication between the hub and the user-notebook servers is encrypted and is safe from eavesdropping. This capability has been requested by many users in the JupyterHub community.

JupyterHub是一种开源工具，可以按需启动Jupyter笔记本服务器。笔记本可用于数据分析或创建和执行机器学习模型。 Istio是一个服务网格，可在Kubernetes集群中的不同服务之间提供安全且可观察的通信机制。在启用istio的集群中运行JupyterHub的好处之一是获得对不同JupyterHub组件之间的mTLS(相互TLS)功能的支持。 mTLS确保集线器和用户笔记本服务器之间的所有通信都经过加密，并且不会被窃听。 JupyterHub社区中的许多用户都要求此功能。

To follow along this journey, it is important to know the basic component interactions in JupyterHub

要遵循此过程，重要的是要了解JupyterHub中的基本组件交互

The Hub configures the proxy by calling proxy-api
集线器通过调用proxy-api配置代理
The proxy forwards all requests to the Hub by default
代理默认将所有请求转发到集线器
The Hub handles login, and spawns single-user notebook servers on demand
集线器处理登录，并按需生成单用户笔记本服务器
The Hub configures the proxy to forward url prefixes to single-user notebook servers
集线器将代理配置为将url前缀转发到单用户笔记本服务器

建立(Setup)

— Install istio

—安装istio

$ istioctl install --set profile=demo

— Install JupyterHub

—安装JupyterHub

Create the jupyterhub namespace to install the JupyterHub. Set the istio-injection label to configure the automatic injection of the istio-proxy sidecar in the pods that start in the namespace. Set the mTLS mode for all services on the namespace.

创建jupyterhub命名空间以安装JupyterHub。设置istio-injection标签，以配置在命名空间中开始的pod中自动注入istio-proxy sidecar。为名称空间上的所有服务设置mTLS模式。

$ kubectl create ns jupyterhub
$ kubectl label namespace jupyterhub istio-injection=enabled$ kubectl apply -n jupyterhub -f - <<EOF 
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
spec:
  mtls:
    mode: STRICT
EOF

Next, setup the helm charts repository.

接下来，设置头盔图表存储库。

$ helm repo add jupyterhub https://jupyterhub.github.io/helm-chart/
$ helm repo update

Setup the config for the helm chart.

设置舵图的配置。

$ echo -n "proxy:\n  secretToken: '$(openssl rand -hex 32)'\n" > config.yaml

Install JupyterHub in the jupyterhub namespace

在jupyterhub命名空间中安装JupyterHub

$ helm template  jupyterhub/jupyterhub \
  --version=0.9.0 \
  --values config.yaml | kubectl -n jupyterhub apply -f -
$ # Not using `helm install` is a personal preference. I prefer qbec instead for day to day use. Using helm here as it is used to package JupyterHub for Kubernetes in the community.

Next, we’ll verify the deployment to see if the pods are running. Both the hub and the proxy pods running as expected.

接下来，我们将验证部署以查看Pod是否正在运行。集线器和代理容器都按预期运行。

$ kubectl -n jupyterhub get po
hub-fd88f65b6-6zqb9     2/2     Running     1    5m31s
proxy-98fdbb5fd-bv7nt   2/2     Running     0    5m31s

The 2/2 part shows that there are two containers in the pod - the main container and a sidecar istio-proxy container. kubectl -n jupyterhub describe po hub-fd88f65b6-6zqb9 shows that the hub pod has an istio-init container and an istio-proxy sidecar.

2/2部分显示了容器中有两个容器-主容器和一个sidecar istio-proxy容器。 kubectl -n jupyterhub describe po hub-fd88f65b6-6zqb9显示集线器荚具有istio-init容器和istio-proxy边车。

Network traffic is routed through the istio-proxy sidecar. To validate, look at the access logs from the sidecar.

网络流量通过istio-proxy边车路由。为了进行验证，请查看边车的访问日志。

$ [2020-09-15T03:50:42.650Z] "GET /api/routes HTTP/1.1" 200 - "-" "-" 0 87 2 1 "-" "Mozilla/5.0 (compatible; pycurl)" "389f6b9c-c966-96d7-8cc3-2a565f623ccd" "10.106.101.111:8001" "10.1.0.19:8001" outbound|8001||proxy-api.jupyterhub.svc.cluster.local 10.1.0.18:52182 10.106.101.111:8001 10.1.0.18:36004 - default

So far so good. Looks like we got everything we need. But, navigating to the proxy-public results in an unexpected error.

到现在为止还挺好。看起来我们已经拥有了所需的一切。但是，导航到代理公共会导致意外错误。

$ kubectl -n jupyterhub port-forward svc/proxy-public 8080:80

Image for post — 404 when accessing hub in the Jupyterhub on the istio service mesh

Port-forwarding directly to the proxy pod also fails with the same 404 error

直接转发到代理Pod的端口也失败，并显示404错误

$ kubectl -n jupyterhub port-forward proxy-98fdbb5fd-265xq 8080:8000

调查中 (Investigation)

The 404 turns out to be a little tricky to debug. Everything checks out, the proxy container is able to connect to the hub container via the hub service, the user request on the port-forward lands on the proxy container. Could this be an issue with istio? Disabling the sidecar injection makes everything work again magically.

404调试起来有点棘手。一切都检查完了，代理容器能够通过集线器服务连接到集线器容器，用户对端口转发的请求将到达代理容器。 istio可能有问题吗？禁用sidecar注入可使一切神奇地再次运行。

$ kubectl label namespace jupyterhub istio-injection=disabled --overwrite

The port-forward lands at the login page.

端口转发登陆在登录页面上。

Digging deeper requires more networking expertise. Enabling sidecar injection again and simultaneously tailing logs of the istio-proxy(which is based on envoy) for the proxy pod shows the event that records the failure.

深入挖掘需要更多的网络专业知识。再次启用sidecar注入，并同时尾随istio-proxy(基于envoy )的代理容器日志，将显示记录故障的事件。

$ kubectl -n jupyterhub logs proxy-98fdbb5fd-svnzl -c istio-proxy -f
 
[2020-09-15T04:37:03.884Z] "GET / HTTP/1.1" 404 NR "-" "-" 0 0 0 - "127.0.0.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36" "b0bd27fe-90c3-9bfd-8ae7-e716baa9eb6e" "localhost:8080" "-" - - 10.105.254.81:8081 127.0.0.1:0 - -

10.105.254.81:8081 is destination(k8s service) where hub is listening but there is no corresponding inbound event in the logs for the istio-proxy on the hub pod. The real hint here is the 404 NR which means that there is “No Route” or in other words the target service is not known to envoy. Hence envoy drops the outbound request resulting in a 404 NOT FOUND response. To understand what’s going on, let’s fire up a shell on the proxy and make some tailored requests and reproduce the issue.

10.105.254.81:8081是集线器正在侦听的目的地(k8s服务)，但是在日志中没有相应的入站事件以供集线器容器上的istio-proxy使用。真正的提示是404 NR ，这意味着不存在“无路线”，换句话说，目标服务未知。因此，特使删除了出站请求，从而产生404 NOT FOUND响应。要了解发生了什么，让我们在代理上启动一个shell并提出一些定制的请求并重现问题。

$ kubectl -n jupyterhub  exec -it deploy/proxy -c chp sh
# curl hub:8081

The proxy sidecar logs show the event with a successful redirect as expected(302 in this case).

代理端履历日志显示按预期成功重定向的事件(在这种情况下为302)。

[2020-09-15T04:51:18.816Z] "GET / HTTP/1.1" 302 - "-" "-" 0 0 2 1 "-" "curl/7.67.0" "923377f0-4cb1-9c35-8ba9-0c42f8e247da" "hub:8081" "10.1.0.37:8081" outbound|8081||hub.jupyterhub.svc.cluster.local 10.1.0.38:46732 10.105.254.81:8081 10.1.0.38:50312 - default

The hub sidecar log also show the corresponding inbound event. The X-REQUEST-ID field can be used to track the logs across services in an istio service mesh.

集线器边车日志还显示相应的入站事件。 X-REQUEST-ID字段可用于在istio服务网格中跨服务跟踪日志。

[2020-09-15T04:51:18.816Z] "GET / HTTP/1.1" 302 - "-" "-" 0 0 1 1 "-" "curl/7.67.0" "923377f0-4cb1-9c35-8ba9-0c42f8e247da" "hub:8081" "127.0.0.1:8081" inbound|8081||hub.jupyterhub.svc.cluster.local 127.0.0.1:34838 10.1.0.37:8081 10.1.0.38:46732 outbound_.8081_._.hub.jupyterhub.svc.cluster.local default

Now let’s route the request through the chp(configurable-http-proxy) — the nodejs server that proxies the call. This results in the 404 error — the same error as the calls from the browser.

现在，让我们通过chp(configurable-http-proxy)路由请求—代理该调用的nodejs服务器。这将导致404错误-与来自浏览器的调用相同的错误。

# curl localhost:8000 # the entry below is from proxy sidecar logs
[2020-09-15T04:54:19.216Z] "GET / HTTP/1.1" 404 NR "-" "-" 0 0 0 - "127.0.0.1" "curl/7.67.0" "0fcf1f52-a809-9855-bb58-50501a17f694" "localhost:8000" "-" - - 10.105.254.81:8081 127.0.0.1:0 - -

Setting the correct Host header on the request works as expected.

在请求上设置正确的Host标头可以按预期工作。

# curl localhost:8000 -H "Host: hub:8081" -H "X-REQUEST-ID:
test"[2020-09-15T05:01:35.988Z] "GET / HTTP/1.1" 302 - "-" "-" 0 0 2 2 "127.0.0.1" "curl/7.67.0" "test" "hub:8081" "10.1.0.37:8081" outbound|8081||hub.jupyterhub.svc.cluster.local 10.1.0.38:46732 10.105.254.81:8081 127.0.0.1:0 - default

So, the issue with chp is the mismatch of the Host header on the request. Envoy would drop the outbound request with an NR and the proxy cannot hit the hub service when an external request is routed through it. (Some other tools and nodejs snippets used to narrow down the exact issue have been omitted for brevity.)

因此，chp的问题是请求上的主机标头不匹配。 Envoy会使用NR丢弃出站请求，并且当外部请求通过它路由时，代理无法访问集线器服务。 (为简洁起见，省略了一些用于缩小确切问题范围的其他工具和nodejs代码段。)

jupyterhub-istio-proxy进行救援 (jupyterhub-istio-proxy to the rescue)

While there are ways to hack the current proxy implementation(s) and in some cases use a less secure variant, it is kinda redundant as istio(underlying envoy to be precise) offers a first-class support for network proxying. Moreover, the chp proxy implementation becomes a bottleneck as soon as the Jupyterhub traffic grows. It cannot be scaled beyond one pod due to its technical limitations. jupyterhub-istio-proxy can be used to configure istio to do the actual network routing based on user interactions with JupyterHub. It also offers a horizontally scalable solution needed to run production workloads at scale.

尽管有多种方法可以破解当前的代理实现，并且在某些情况下使用不太安全的变体，但它有点多余，因为istio(确切地说是基础特使)为网络代理提供了一流的支持。此外，一旦Jupyterhub流量增加， chp代理实现就会成为瓶颈。 由于其技术限制，它不能扩展到一个吊舱之外。 jupyterhub-istio-proxy可用于将istio配置为基于用户与JupyterHub的交互进行实际的网络路由。它还提供了横向扩展的解决方案，可以大规模运行生产工作负载。

Create an istio gateway to handle ingress to the K8s cluster. The gateway is the entry point for network.

创建一个istio网关来处理K8s集群的入口。网关是网络的入口点。

$ kubectl -n jupyterhub apply -f - https://gist.githubusercontent.com/harsimranmaan/4315477268fccea65accf8674f5c49ef/raw/0298f3f420365c7e56aedab7949ad39e00ffbcc3/jupyterhub-istio-proxy-gateway.yaml

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: jupyterhub-gateway
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "*"

Remove the proxy-public service as it no longer needed.

删除不再需要的代理公共服务。

$ kubectl -n jupyterhub delete svc proxy-public

Replace the proxy deployment with the jupyterhub-istio-proxy:

将代理部署替换为jupyterhub-istio-proxy：

$ kubectl -n jupyterhub apply -f https://gist.githubusercontent.com/harsimranmaan/2e77cf65019439052122b7b89f926686/raw/d800b8c60c2ac10226d549c1fbc6d8d75e8e6142/jupyterhub-istio-proxy.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: jupyterhub
    component: proxy
  name: proxy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jupyterhub
      component: proxy
      release: RELEASE-NAME
  strategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: jupyterhub
        component: proxy
        release: RELEASE-NAME
    spec:
      containers:
        - command:
            - /proxy/jupyterhub-istio-proxy
          env:
            - name: CONFIGPROXY_AUTH_TOKEN
              valueFrom:
                secretKeyRef:
                  key: proxy.token
                  name: hub-secret
            - name: ISTIO_GATEWAY
              value: jupyterhub-gateway
            - name: K8S_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: SUB_DOMAIN_HOST
              value: '*'
            - name: VIRTUAL_SERVICE_PREFIX
              value: jupyterhub
            - name: WAIT_FOR_WARMUP
              value: "true"
          image: splunk/jupyterhub-istio-proxy:0.0.2
          imagePullPolicy: IfNotPresent
          name: proxy
          ports:
            - containerPort: 8000
              name: proxy-api
              protocol: TCP
          resources:
            limits:
              cpu: "1"
              memory: 256M
            requests:
              cpu: 100m
              memory: 256M
          securityContext:
            allowPrivilegeEscalation: false
      securityContext:
        runAsNonRoot: true
      terminationGracePeriodSeconds: 60
---
apiVersion: v1
kind: Service
metadata:
  name: proxy-api
spec:
  ports:
    - name: http-proxy-api
      port: 8001
      protocol: TCP
      targetPort: 8000
  selector:
    component: proxy
  type: ClusterIP
---

Once the above config is applied, a new virtual service would appear.

一旦应用了上述配置，就会出现一个新的虚拟服务。

$ kubectl -n jupyterhub get vs
jupyterhub-8a5edab282632443219e051e4ade2d1d5bbc671c781051bf1437897cbdfea0f1   [jupyterhub-gateway]   [*]     37m

Everything should now work in theory(right?), but there are a few more issues to address. By default, hub is configured to tell the proxy-api(jupyterhub-istio-proxy) to route traffic to its IP instead of using its service name. This causes the istio virtual service to be configured with the IP

现在一切都应该在理论上起作用(对吗？)，但是还有更多问题需要解决。默认情况下，集线器配置为告诉proxy-api( jupyterhub-istio-proxy )将流量路由到其IP，而不使用其服务名称。这将导致使用IP配置istio虚拟服务

Hub log:
[I 2020-09-15 06:28:24.481 JupyterHub proxy:400] Adding default route for Hub: / => http://10.105.254.81:8081

resulting in invalid VS config

导致无效的VS配置

- destination:
        host: 10.105.254.81.jupyterhub.svc.cluster.local
        port:
          number: 8081

Patch the Jupyterhub config to set the Jupyterhub.hub_connect_ip property to the service name instead of the IP. The PROXY_PUBLIC_SERVICE_HOST and PROXY_PUBLIC_SERVICE_PORT are no longer in use and can be set to the external hostname and port (localhost:80 in this setup).

修补Jupyterhub配置，以将Jupyterhub.hub_connect_ip属性设置为服务名称而不是IP。 PROXY_PUBLIC_SERVICE_HOST和PROXY_PUBLIC_SERVICE_PORT不再使用，可以设置为外部主机名和端口(在此设置中为localhost：80)。

kubectl -n jupyterhub get cm/hub-config -o yaml | sed  "s/os\.environ\['HUB_SERVICE_HOST'\]/'hub'/g" | sed  "s/os\.environ\['PROXY_PUBLIC_SERVICE_HOST'\]/'localhost'/g" | sed  "s/os\.environ\['PROXY_PUBLIC_SERVICE_PORT'\]/'80'/g" | kubectl -n jupyterhub apply -f -

Restart the hub pod to pick up the new config. The service name is set for the default route.

重新启动集线器窗格以获取新配置。服务名称已设置为默认路由。

[I 2020-09-15 07:27:09.068 JupyterHub proxy:400] Adding default route for Hub: / => http://hub:8081- destination:
        host: hub.jupyterhub.svc.cluster.local
        port:
          number: 8081

Follow the official guide to determine the istio Gateway URL. Navigate to http://YOUR_GATEWAY_URL and you’ll see JupyterHub running. TLS termination for web requests is not covered here but is fairly straightforward to setup with istio gateway. It is left as an exercise to the readers.

按照官方指南确定istio网关URL。导航到http：// YOUR_GATEWAY_URL ，您将看到JupyterHub正在运行。 Web请求的TLS终止不在此处，但使用istio网关进行设置非常简单。它留给读者练习。

The last missing piece in the puzzle is to ensure that user-notebook servers can be spun up and users can run their favourite notebooks.This requires patching another JupyterHub component(kubespawner). The details can be found in this PR: https://github.com/jupyterhub/kubespawner/pull/425

难题中最后缺少的部分是确保可以启动用户笔记本服务器，并且用户可以运行自己喜欢的笔记本。这需要修补另一个JupyterHub组件(kubespawner)。可以在此PR中找到详细信息： https : //github.com/jupyterhub/kubespawner/pull/425

引擎盖下(Under the hood)

jupyterhub-istio-proxy creates an Istio virtual service for every route request from the hub. Hub forwards routing requests to jupyterhub-istio-proxy which sets up the desired destination route and waits for the route to warm up before sending the confirmation back to the hub. Once the route is created, hub redirects the user to their notebook server.

jupyterhub-istio-proxy为来自集线器的每个路由请求创建一个Istio虚拟服务。集线器将路由请求转发到jupyterhub-istio-proxy ，后者设置所需的目标路由并等待路由预热，然后再将确认发送回集线器。创建路由后，集线器会将用户重定向到其笔记本服务器。

If you have questions or would like to contribute to the development of jupyterhub-istio-proxy, drop a note or send in your contributions at https://github.com/splunk/jupyterhub-istio-proxy/issues

如果您有任何疑问或想为jupyterhub-istio-proxy的发展做出贡献，请在https://github.com/splunk/jupyterhub-istio-proxy/issues上发送注释或发送文稿