Skip to content

feat: normalize Arrow JS numeric inputs in column builders - #81

Closed
rustyconover wants to merge 1 commit into
uwdata:mainfrom
Query-farm:upstream/arrow-type-compat
Closed

rustyconover wants to merge 1 commit into
uwdata:mainfrom
Query-farm:upstream/arrow-type-compat

Conversation

@rustyconover

Copy link
Copy Markdown
Contributor

Passing an Apache Arrow JS numeric descriptor, such as new Int32(), to a column builder can allocate a Uint8Array and encode the wrong signedness. Values can be corrupted without an error.

Normalize integer and floating-point inputs at the column-building boundary using Flechette's existing type constructors. This derives storage and signedness from the logical type and gives the encoder a canonical Flechette type. Native descriptors retain identity; foreign descriptors are not mutated. columnFromArray, columnFromValues, and the types option of tableFromArrays accept the new structural ColumnBuilderType.

Compatibility is deliberately limited to integer and floating-point descriptors. Other Arrow JS types can use different physical layouts, so this does not add a generic ArrayType fallback. Arrow JS declares specialized numeric type IDs while returning the standard Int/Float IDs at runtime; the structural input type accommodates those declarations without adding a runtime or declaration dependency on apache-arrow.

Validation:

  • 226 tests pass on Node 24 and 26; lint and bundle/declaration builds pass.
  • Tests use all 11 actual Arrow JS numeric classes, signed/unsigned boundaries, nulls, multiple batches, frozen inputs, arrays/iterables, and IPC stream/file round trips through both readers.
  • 32 of the 34 new runtime cases fail before the fix.
  • A strict consumer type-check against emitted declarations verifies real Arrow JS inputs; CI runs it on both Node versions and checks rejection of unsupported inputs.

@jheer

jheer commented Oct 22, 2026

Copy link
Copy Markdown
Member

Changes look good, but I'm curious about the motivation here, as it goes beyond Flechette's current design scope. Are there common use cases requiring Arrow-JS type instances for Flechette encoding? An alternative might be to perform data type instance conversion outside of Flechette.

@rustyconover

Copy link
Copy Markdown
Contributor Author

Hi @jheer,

I have code called vgi-rpc which is an RPC framework based on Arrow IPC. It turns out using Arrow JS was too large for Cloudflare Workers, so I added support for using Flechette as the implementation. You're kind of right, I could handle this outside of Flechette but if I was just trying to have pluggable arrow libraries these conversions are convenient.

Rusty

@jheer

jheer commented Oct 23, 2026

Copy link
Copy Markdown
Member

After some more thought, my lean is that this conversion probably doesn't belong in Flechette. It's easy to do outside of the core lib and I'm hesitant to bake in duck-typing assumptions (even if tested) about other libraries here.

Thanks for the set of PRs! I'll get a release with the other bug fixes going soon. And I'm glad Flechette fits a need in your work!

@jheer jheer closed this Oct 23, 2026
@rustyconover

Copy link
Copy Markdown
Contributor Author

You're welcome, thanks for taking the fixes!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants