kube-inject: support XDS address directly

[howardjohn]

Currently, kube-inject has 3 modes:

1. Automagically read configmaps in cluster and apply
2. Same as (1), but users overrides with flags like --injectConfigMapName
3. User passes in files directly for the template

This leads to our multicluster install needlessly installing these configmaps in the remote cluster, which is complex, confusing for users, and likely to fall out of sync.

Additionally, none of these support revisions.

I propose we make these changes:

1. Deprecate (2)

2. Re-implement (1) to call the Istiod service, rather than reading the configmaps (ie implementation details)
   This will be done by reading the mutating webhook configurations. If its a Service, we will do a port-forward, and call /inject. If its a URL (external istiod), we will call it directly.
   An additional flag, like --injection-url will be added, to not look at webhook configs.
   If there are no webhooks and --injection-url is not set, we will continue to use configmaps and display a deprecation warning
   An additional flag, --revision will be added. When this is set, we will look at the revision webhook rather than

[howardjohn]

cc @esnible any thoughts? I will probably put this on UX agenda

[esnible]

@howardjohn Makes sense. Just to confirm, the plan is to read the MutatingWebhookConfig, and proxy a call as if Kube API did it, using the caBundle? Will the Istiod user always have the ability to read the webhook config and caBundle?

[howardjohn]

well, technically they can have whatever permissions they want - including lack of configmap permissions. But I don't see why MutatingWebhookConfig should be restricted in a average clusters, its not privileged at all to have read access. So it could break someone, but I suspect it would not. We can have it fall back to the configmaps with warnings, and if users complain maybe we keep around the configmaps forever

[howardjohn]

Maybe revision is not needed since we have the revision as part of the URL

[linsun]

Thanks @howardjohn, this makes sense and aligns with users need to specify XDS discovery address for certain istioctl cmds when interacting with remote clusters.

[howardjohn]

@carolynhu are you working on this? if not please let us know so we can find someone