4.0.5 Release Notes

[yiguolei]

New Features & Improvements

Index & Search

• Support MATCH projection as virtual column for inverted index ([feature](search) support MATCH projection as virtual column for inverted index evaluation #61092)
• Add PrefixQuery, PhrasePrefixQuery, and UnionPostings support ([feat](query_v2) Add PrefixQuery, PhrasePrefixQuery and UnionPostings support  #60701)
• Support BM25 scoring in inverted index queries ([feature](score) support BM25 scoring in inverted index query_v2 #59847)
• Implement phrase-level BM25 scoring with phrase frequency ([fix](inverted index) implement phrase-level BM25 scoring with phrase frequency #60331)
• Change default search mode from standard to lucene ([improvement](search) change default mode of search function from standard to lucene #61055)

Security & Authentication

• Add LDAPS support via configuration ([enhance](auth) introduction of ldaps support via configuration property #60275)
• Enhance Azure object storage client certificate troubleshooting ([feat](be) Enhance CA cert troubleshooting in Azure object storage client #61102)

Bugfix

Query & Execution

• Fix query cache inconsistency with variant subcolumns ([fix](query-cache) include variant subcolumn path in query cache digest #61709)
• Fix MATCH crash on alias slots ([fix](fe) Fix MATCH crash on alias slots and push down as virtual column #61584)
• Fix set operation crash when data size exceeds 4GB ([fix](set) fix coredump of set op if total data size exceeds 4G #61471)
• Fix TopN filter issues with nullable slots across outer joins ([fix](topnFilter)Fix TopN filter probe expressions to wrap nullable slots when pushed through outer joins. #59074)
• Fix incorrect MV functional dependency calculation ([fix](mtmv) Fix query err when calc mv functional dependency which has variant and log more detailed info for troubleshoot a problem #59933)
• Fix type coercion issues in SUM and AVG ([fix](typeCoercion) remove checkLegalityBeforeTypeCoercion in sum and avg #59602)

Functions & Types

• Add support for NDV on decimalv2 ([Bug](function) add ndv decimalv2 support #61546)
• Fix incorrect ABS result for decimalv2 ([fix](decimalv2) fix wrong result of abs(decimalv2) #61397)
• Fix LARGEINT handling in array_apply ([fix](function)Fix array_apply being unable to handle LARGEINT. #61081)

Storage & Compaction

• Fix stale rowset commit issues ([fix](ms) Add job check for stale commit rowset #61427)
• Fix compaction timeout calculation errors ([Fix](warmup) Fix wrong rowset data size when calculating timeout of waiting delay compaction #61188)
• Fix segment corruption not triggering cache retry ([fix](cloud) fix segment footer CORRUPTION not triggering file cache retry #61386)
• Fix move-memtable related stream write errors ([fix](move-memtable) fix StreamWrite EINVAL error when report tablet load info #60688)

Index & Search

• Fix deterministic reader selection for multi-index columns ([fix](inverted index) Make select_best_reader deterministic for multi-index columns #61596)
• Fix search query parsing issues with special characters ([fix](search) Fix slash character in search query_string terms #61599)
• Fix MUST_NOT handling with NULL rows ([fix](search) Replace ExcludeScorer with AndNotScorer for MUST_NOT to handle NULL rows #61200)
• Fix inverted index overflow issue ([fix](inverted index) fix StringRef to std::string reinterpret_cast overflow #61120)

Load & Streaming

• Fix load memory regression ([fix](load) fix load memory usage is more than branch-3.1 (#61173) #61327)
• Fix streaming job stuck due to ignored S3 auth errors ([fix](job) fix streaming job stuck when S3 auth error is silently ignored in fetchRemoteMeta #61284)
• Fix streaming job scheduling issues ([fix](job) fix streaming job fails with "No new files found" on second scheduling  #61249)
• Fix streaming insert replay NPE ([fix](streaming) Fix NPE in StreamingInsertJob when MetricRepo is not initialized during replay #61253)
• Fix routine load job incorrectly cancelled after FE restart ([fix](job) fix routine load job incorrectly cancelled on FE restart after swap table #61046)
• Fix streaming job config not refreshed after ALTER ([fix](streamingjob) fix streaming insert job not refreshing TVF props after ALTER SQL #61451)

Cloud & External Systems

• Fix empty rowset handling in batch delete ([fix](cloud) Skip empty rowsets before accessor lookup in batch delete_rowset_data #60919)
• Reject invalid admin operations in cloud mode ([fix](cloud) reject ADMIN SET REPLICA VERSION in cloud #60875)
• Fix RPC metric naming issues ([fix](cloud) change cloud fe getVersion rpc metric to getTableVersion and getPartitionVersion #60745)
• Fix Iceberg table read and rewrite failures ([fix](iceberg)fix core when read a iceberg table that have schema change and equality delete. #59984, [fix](iceberg)fix iceberg rewrite_data_file fail when table had been updated. #61112)
• Fix ES query issues with array keyword fields ([fix](es-catalog) Fix query error when ES keyword field contains array data #61236)

File Cache & IO

• Fix file cache concurrency crash ([fix](filecache) fix crash on reset_range when clearing cache concurrently #60271)
• Fix empty block file handling ([fix](filecache) add check and exception handle for empty block file #59646)

Metadata & Job System

• Fix job scheduling and execution issues ([fix](coordinator) Fix NPE in updateLoadCounters during reportExecStatus for non-Load queries #61467, [fix](job) fix streaming job stuck when S3 auth error is silently ignored in fetchRemoteMeta #61284)
• Fix backup snapshot upload on invalid storage paths ([fix](backup) reject upload snapshots on broken storage path #61251)

Security & Auth

• Fix LDAP injection vulnerability by proper escaping ([fix](ldap) Replace custom LDAP filter escaping with LdapEncoder.filterEncode to prevent injection vulnerabilities and add related documentation. #61662)
• Fix password lock failure after invalid login ([fix](password) password lock failed after invalid login #61592)
• Restrict catalog privilege checks to valid operations ( [fix](auth) restrict skip_catalog_priv_check to SHOW and SELECT on catalog privilege checks #61147)

Stability & Misc

• Fix FlushToken counter inconsistency ([fix](be) keep FlushToken running count symmetric on cancel #61684)
• Fix profile metrics inaccuracies ([Bug](profile) fix wrong count of scan wait worker timer #61064, [fix](fe) Fix bugs in SummaryProfile and StmtExecutor metric reporting #61601)
• Fix timezone-related issues ([Fix](tz) Fix some issues about timezone #61151)
• Fix ORC heap-use-after-free crash ([fix](orc)fix coredump because rewriteLeaves function heap-use-after-free. #61138)
• Fix frontend TVF metadata inconsistency ([fix] Preserve CurrentConnected in frontends() TVF #61087)
• Fix insert overwrite behavior ([fix](insert-overwrite) should use un strict replace #61082)
• Fix local shuffle strategy ([fix](local shuffle) Fix local shuffle strategy #61057)
• Fix scan node missing column size metrics ([fix](fe) add column_data_sizes to BackendPartitionedSchemaScanNode #61086)