Should ClusterQueue manage all workloads or coexist with ResourceQuota?

[kimminw00]

I currently have a question regarding the best practice for managing resources in a kubernetes cluster.

I’ve read through Issue #696, which provided some context, but I am still debating the best approach.

My question is: Should we aim to manage all workloads via ClusterQueue (Kueue), or is a hybrid approach preferred?

• Option A:
  Manage everything through ClusterQueue (Kueue).
• Option B:
  Use native ResourceQuota for long-running services which do not use expensive accelerators (e.g., deployment, statefulset).
  Use ClusterQueue specifically for batch jobs, model training jobs, and inference services.

I am curious if ClusterQueue is intended to replace standard ResourceQuota entirely, or if it is better suited to coexist with them.

Any advice or real-world examples would be appreciated!

[kannon92]

I'm not sure it's intended to replace but we usually aim to have Kueue quota used instead of ResourcrQuota.

The main use case we find is that people want Kueue managing quota for GPUs. If some workloads don't need to go through Kueue the quota handling gets confusing. You can have some workloads that "skip" the line while others have to wait for their GPUs.

I could see a use for RQ as a hard limit as they are a built in admission controller.

[kannon92]

For our largest users they want to replicate the Slurm experience. All jobs are managed by scheduler and it should be impossible to request workloads without Kueue checking.

This is surprising difficult to enforce with all the different workloads that exist in K8s but that has been our goal.