Repository navigation
feat: normalize Arrow JS numeric inputs in column builders - #81
rustyconover wants to merge 1 commit into
Conversation
|
Changes look good, but I'm curious about the motivation here, as it goes beyond Flechette's current design scope. Are there common use cases requiring Arrow-JS type instances for Flechette encoding? An alternative might be to perform data type instance conversion outside of Flechette. |
|
Hi @jheer, I have code called vgi-rpc which is an RPC framework based on Arrow IPC. It turns out using Arrow JS was too large for Cloudflare Workers, so I added support for using Flechette as the implementation. You're kind of right, I could handle this outside of Flechette but if I was just trying to have pluggable arrow libraries these conversions are convenient. Rusty |
|
After some more thought, my lean is that this conversion probably doesn't belong in Flechette. It's easy to do outside of the core lib and I'm hesitant to bake in duck-typing assumptions (even if tested) about other libraries here. Thanks for the set of PRs! I'll get a release with the other bug fixes going soon. And I'm glad Flechette fits a need in your work! |
|
You're welcome, thanks for taking the fixes! |
Passing an Apache Arrow JS numeric descriptor, such as
new Int32(), to a column builder can allocate a Uint8Array and encode the wrong signedness. Values can be corrupted without an error.Normalize integer and floating-point inputs at the column-building boundary using Flechette's existing type constructors. This derives storage and signedness from the logical type and gives the encoder a canonical Flechette type. Native descriptors retain identity; foreign descriptors are not mutated.
columnFromArray,columnFromValues, and thetypesoption oftableFromArraysaccept the new structuralColumnBuilderType.Compatibility is deliberately limited to integer and floating-point descriptors. Other Arrow JS types can use different physical layouts, so this does not add a generic
ArrayTypefallback. Arrow JS declares specialized numeric type IDs while returning the standard Int/Float IDs at runtime; the structural input type accommodates those declarations without adding a runtime or declaration dependency on apache-arrow.Validation: