Configure secure serving for controller-manager & scheduler

Closed

luxas opened this issue Nov 28, 2018 · 8 comments

Closed

Configure secure serving for controller-manager & scheduler #1285

luxas opened this issue Nov 28, 2018 · 8 comments

Comments

Member

luxas commented Nov 28, 2018 •

xref: #1285 insecure serving is deprecated. We should switch to secure serving. controller-manager logs in v1.13.0-rc.1: I1128 12:40:04.680584 1 serving.go:318] Generated self-signed cert in-memory I1128 12:40:05.697603 1 controllermanager.go:151] Version: v1.13.0-rc.1 I1128 12:40:05.699084 1 secure_serving.go:116] Serving securely on [::]:10257 I1128 12:40:05.699649 1 deprecated_insecure_serving.go:51] Serving insecurely on 127.0.0.1:10252 scheduler logs in v1.13.0-rc.1: I1128 12:40:04.787327 1 serving.go:318] Generated self-signed cert in-memory W1128 12:40:05.278190 1 authentication.go:373] failed to read in-cluster kubeconfig for delegated authentication: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory W1128 12:40:05.278217 1 authentication.go:235] No authentication-kubeconfig provided in order to lookup client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication won't work. W1128 12:40:05.278227 1 authentication.go:238] No authentication-kubeconfig provided in order to lookup requestheader-client-ca-file in configmap/extension-apiserver-authentication in kube-system, so request-header client certificate authentication won't work. W1128 12:40:05.278252 1 authorization.go:177] failed to read in-cluster kubeconfig for delegated authorization: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory W1128 12:40:05.278411 1 authorization.go:146] No authorization-kubeconfig provided, so SubjectAccessReview of authorization tokens won't work. W1128 12:40:05.285538 1 authorization.go:47] Authorization is disabled W1128 12:40:05.285562 1 authentication.go:55] Authentication is disabled I1128 12:40:05.285579 1 deprecated_insecure_serving.go:49] Serving healthz insecurely on 127.0.0.1:10251 I1128 12:40:05.286312 1 secure_serving.go:116] Serving securely on [::]:10259 @sttts What do we need to do here?

luxas added area/security priority/important-longterm kind/feature labels Nov 28, 2018

luxas added this to the v1.14 milestone Nov 28, 2018

luxas changed the title Properly configure secure serving for the controller-manager Configure secure serving for controller-manager & scheduler Nov 28, 2018

Member Author

luxas commented Nov 28, 2018

cc @timothysc @liztio

neolit123 mentioned this issue Nov 28, 2018

Kubeadm use deprecated controller-manager and scheduller flags #1234

Closed

neolit123 mentioned this issue Nov 28, 2018

Use --bind-address for the controller-manager #1284

Closed

sttts commented Nov 29, 2018

You have to pass --authentication-kubeconfig and --authorization-kubeconfig and then configure RBAC to allow to read the kube-system/extension-apiserver-authentication configmap: https://github.com/kubernetes/kubernetes/blob/1588af4031e746e3e48855256054133e8707f018/test/integration/serving/serving_test.go#L133 allow to do delegated authorization via SubjectAccessReview: https://github.com/kubernetes/kubernetes/blob/1588af4031e746e3e48855256054133e8707f018/test/integration/serving/serving_test.go#L116

luxas mentioned this issue Dec 20, 2018

Use secure port for scheduler and controller-manager liveness probe #1327

Closed

joshrosso mentioned this issue Jan 8, 2019

cmd/kubeadm: use secure port for liveness probe on controller-manager and scheduler kubernetes/kubernetes#72159

Closed

Member

alexbrand commented Jan 30, 2019

To be able to scrape metrics on the secure port of the scheduler, we need to set the --authentication-kubeconfig and authorization-kubeconfig flags. We already do this on the controller manager. In the case of the scheduler, where the flags are not set, any request to /metrics on the secure port are rejected, because they are not authentication/authorized. By setting the flag, we tell the scheduler to perform tokenaccessreview and subjectaccessreview for requests coming in on the secure port.

timothysc assigned yagonobre Feb 13, 2019

lbogdan mentioned this issue Feb 26, 2019

Further exploration octetz/secure-port-k8s-cm-sched#1

Open

lbogdan commented Feb 27, 2019

All RBAC setup was already done in kubernetes/kubernetes#72491 , so it looks like it's only a matter of adding --authentication-kubeconfig and --authorization-kubeconfig to kube-scheduler. If no one is working on this, I'd like to take a shot at it.

Member

neolit123 commented Mar 7, 2019

@lbogdan sorry for the delayed reply. I'd like to take a shot at it. that would be appreciated, but mind that we are in code freeze soon, so a PR for this can be merged after 1.14 is out.

neolit123 removed this from the v1.14 milestone Mar 7, 2019

neolit123 added this to the v1.15 milestone Mar 7, 2019

daili mentioned this issue Apr 9, 2019

install kubernetes 1.14.0 using kube-aws won't work kubernetes-retired/kube-aws#1578

Closed

fl-max commented Apr 23, 2019

For those that stumble upon this and are using Kubeadm, the Kubeconfig is already generated and is mounted into the scheduler pod at /etc/kubernetes/scheduler.conf. You simply need to add authentication-kubeconfig & authorization-kubeconfig to your configuration like so: apiVersion: kubeadm.k8s.io/v1beta1 kind: ClusterConfiguration ... scheduler: extraArgs: authentication-kubeconfig: "/etc/kubernetes/scheduler.conf" authorization-kubeconfig: "/etc/kubernetes/scheduler.conf"

Member

fabriziopandini commented Apr 26, 2019

/assign

k8s-ci-robot assigned fabriziopandini Apr 26, 2019

dannymk commented May 21, 2019 •

Hmmm... I added those flags to: /etc/kubernetes/manifests/kube-scheduler.yaml ... - --authentication-kubeconfig=/etc/kubernetes/scheduler.conf - --authorization-kubeconfig=/etc/kubernetes/scheduler.conf ... I also added the kubernetes generated ca to the system: sudo cp /etc/kubernetes/pki/ca.crt /usr/local/share/ca-certificates/kubernetes-local-ca.crt sudo update-ca-certificates That did the trick for me.

neolit123 removed this from the v1.15 milestone Jun 3, 2019

neolit123 added this to the v1.16 milestone Jun 3, 2019

StanYago mentioned this issue Jul 4, 2019

Serviced accounts blocked in mirror pods brancz/kube-rbac-proxy#49

Closed

neolit123 unassigned yagonobre Jul 25, 2019

neolit123 mentioned this issue Aug 3, 2019

kubeadm: enable secure serving for the kube-scheduler kubernetes/kubernetes#80951

Merged

neolit123 self-assigned this Aug 3, 2019

neolit123 added the lifecycle/active label Aug 3, 2019

k8s-ci-robot closed this in #80951 Aug 5, 2019

转载至https://github.com/kubernetes/kubeadm/issues/1285