Skip to content

[core][flink] supports count star pushdown for all-global-indexed filters.#7680

Open
steFaiz wants to merge 4 commits intoapache:masterfrom
steFaiz:flink_index_count_pushdown
Open

[core][flink] supports count star pushdown for all-global-indexed filters.#7680
steFaiz wants to merge 4 commits intoapache:masterfrom
steFaiz:flink_index_count_pushdown

Conversation

@steFaiz
Copy link
Copy Markdown
Contributor

@steFaiz steFaiz commented Apr 21, 2026

Purpose

This introduce a PredicateVisitor to decide whether a Predicate is all covered by scalar index (e.g. btree now).
Then we can accelerate count(*) queries on indexed columns without touching data files. This is useful in some AI situations. For example, before filtering high quality data, we might want to check the portion to decide the threshold.

Tests

org.apache.paimon.globalindex.ScalarIndexedFieldsVisitorTest for UnitTest
org.apache.paimon.flink.BatchFileStoreITCase for ITCase

@steFaiz steFaiz changed the title [WIP][core][flink] supports count star pushdown for all-global-indexed filters. [core][flink] supports count star pushdown for all-global-indexed filters. Apr 22, 2026
@steFaiz
Copy link
Copy Markdown
Contributor Author

steFaiz commented Apr 22, 2026

I'm not sure whether it's safe. The result may be inaccurate. This PR assumes that: If the whole data predicates can be covered by global index, those predicates can be fully consumed by paimon. But actually this is not right, because the global index might be not covering all data. Moreover, current Index is not exact, for example for unsupported queries e.g. LIKE, NOT_LIKE, the index reader will return ALL_NON_NULL_ROWS.

@steFaiz
Copy link
Copy Markdown
Contributor Author

steFaiz commented Apr 22, 2026

I think an alternative approach is to introduce a new procedure or a special option.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant