auth: wire secure-storage cache into CLI

[simonfaltum]

Stack

Final PR in the opt-in secure token storage stack. Review and merge top-to-bottom:

1. auth: import FileTokenCache into CLI and wire DualWrite #5056 auth: import FileTokenCache into CLI and wire DualWrite
2. libs/auth/storage: add dormant secure-storage foundation #5008 libs/auth/storage: add dormant secure-storage foundation
3. auth: wire secure-storage cache into CLI #5013 auth: wire secure-storage cache into CLI (this PR)

Depends on #5008 for the StorageMode resolver and KeyringCache, and on #5056 for the CLI-owned FileTokenCache that ResolveCache now returns in legacy mode.

Why

Opt-in OS-native secure token storage for every CLI command that authenticates via the databricks-cli strategy. Users turn it on by setting DATABRICKS_AUTH_STORAGE=secure (per-shell) or [__settings__].auth_storage = secure in .databrickscfg (per-machine). Everyone else stays on the legacy file-backed cache. The default does not change.

Storage backend is a per-machine setting, not a per-invocation choice. This PR deliberately ships no --secure-storage flag so that login, token, logout, and every other command can never drift apart on the same machine.

Changes

Before: auth login, auth token, auth logout, and every workspace client built through CLICredentials always went through the SDK's default file-backed TokenCache.

Now: every path that talks to OAuth runs through a single resolver.

• New helper libs/auth/storage/cache.go:ResolveCache(ctx, override) resolves the mode via ResolveStorageMode and returns the corresponding cache.TokenCache. Split into a public form and an injectable core (resolveCacheWith) so unit tests exercise the secure path with a fake cache. Production always passes override = "" and relies on env -> config -> default. Legacy and plaintext modes return the CLI-owned storage.FileTokenCache from auth: import FileTokenCache into CLI and wire DualWrite #5056, not the SDK's file cache.
• auth login, auth token (via newTokenCommand + runInlineLogin), and auth logout call storage.ResolveCache(ctx, "") and plumb the resolved cache into u2m.WithTokenCache(...) at every NewPersistentAuth call site.
• libs/auth.CLICredentials.Configure also routes through ResolveCache so every workspace client built through the CLI credentials strategy reads from the same backend login writes to. This covers auth profiles, jobs list, clusters list, bundle deploy, and every other non-auth command. Without this, secure-storage users would hit "cache: token not found" on the first non-auth command.
• dualWriteLegacyHostKey helper centralises the post-Challenge mirror to the legacy host-based key. Runs only when mode is legacy so secure-mode users do not end up with duplicate keyring entries.
• discoveryLogin takes a discoveryLoginInputs struct after review feedback (8 positional args was over the threshold).
• CLICredentials.persistentAuth no longer opens its own file cache; it relies entirely on the caller to pass WithTokenCache. Previously it opened a file cache and prepended a WithTokenCache option, only to have it immediately overridden by the keyring one appended by Configure (last-one-wins). Single source of truth now.
• Acceptance tests under acceptance/cmd/auth/storage-modes/ cover the two CLI-visible behaviors that do not require an OS keyring: invalid env var surfaces a clear error, and explicit legacy mode behaves identically to the default path.

Out of scope:

• Dedicated plaintext storage implementation. The resolver accepts plaintext; a follow-up will route it to a non-dual-write file backend.
• Write path for [__settings__].auth_storage. Users hand-edit the config for now.
• Automatic migration of existing token-cache.json entries. Users re-login after upgrading.
• Flipping the default to secure.

Known limitation: duplicate keyring entries until SDK bump

The CLI currently pins databricks-sdk-go v0.128.0, which still contains PersistentAuth.dualWrite and calls it from both Challenge() and refresh(). dualWrite stores the token under GetCacheKey() (profile name) AND hostCacheKey() (the host URL). In legacy mode this mirrors into the file cache, which is the desired backward-compat behavior. In secure mode it mirrors into the keyring, producing two entries per login:

• databricks-cli / <profile> -> token
• databricks-cli / https://<host> -> same token

databricks/databricks-sdk-go#1646 removes the SDK-side dualWrite and the host-key fallback in loadToken. It is already merged to SDK main but blocked on #5056 shipping in a CLI release before it can be released itself. Once the SDK ships a new version and the CLI bumps to it, secure mode will produce a single keyring entry per login and the dualWriteLegacyHostKey helper in this PR remains the only source of the legacy host-key mirror (file cache only, by design).

No code change needed in this PR; the resolver, helper gating, and CLICredentials wiring are already correct for the post-bump world.

Test plan

• Unit tests for ResolveCache covering default legacy, explicit override, env-var selection, plaintext fallback, invalid override, invalid env, and file-factory error propagation. Secure path uses a fake cache.TokenCache so CI never touches the OS keyring.
• Unit tests for CLICredentials.Configure confirming ResolveCache is invoked and the resulting cache is passed to NewPersistentAuth via WithTokenCache.
• Acceptance tests under acceptance/cmd/auth/storage-modes/: invalid-env (bogus DATABRICKS_AUTH_STORAGE surfaces a clear error via auth token) and legacy-env-default (explicit legacy mode clears the file cache via auth logout, matching default behavior).
• make checks, make test, make lint pass.
• Existing cmd/auth/login, cmd/auth/token, cmd/auth/logout, cmd/auth/profiles acceptance test suites still pass (regression).
• Manual smoke: with DATABRICKS_AUTH_STORAGE=secure, run auth login, then auth profiles, auth token, jobs list, auth logout; confirm the token goes to the keyring and reads from there.