bug(node): Apache Arrow Typescript error - "Could not find a declaration file for module 'apache-arrow'"

[j-norwood-young]

LanceDB version

v0.4.12

What happened?

Hi there!

I'm getting a Typescript warning for the Apache Arrow version that LanceDB requires:

import * as lancedb from "vectordb";
import { Schema, Field, Float32, FixedSizeList, Int32, Float16, Utf8 } from "apache-arrow";

Error:

Could not find a declaration file for module 'apache-arrow'. 'xxx/node_modules/apache-arrow/Arrow.node.mjs' implicitly has an 'any' type.
There are types at 'xxx/node_modules/apache-arrow/Arrow.node.d.ts', but this result could not be resolved when respecting package.json "exports". The 'apache-arrow' library may need to update its package.json or typings.ts(7016)

Installing v15 of Apache Arrow solves the Typescript error, but introduces an NPM error:

npm ERR! code ERESOLVE
npm ERR! ERESOLVE could not resolve
npm ERR! 
npm ERR! While resolving: vectordb@0.4.12
npm ERR! Found: apache-arrow@15.0.1
npm ERR! node_modules/apache-arrow
npm ERR!   apache-arrow@"^15.0.1" from the root project
npm ERR! 
npm ERR! Could not resolve dependency:
npm ERR! peer apache-arrow@"^14.0.2" from vectordb@0.4.12
npm ERR! node_modules/vectordb
npm ERR!   vectordb@"^0.4.12" from the root project
npm ERR! 
npm ERR! Conflicting peer dependency: apache-arrow@14.0.2
npm ERR! node_modules/apache-arrow
npm ERR!   peer apache-arrow@"^14.0.2" from vectordb@0.4.12
npm ERR!   node_modules/vectordb
npm ERR!     vectordb@"^0.4.12" from the root project
npm ERR! 
npm ERR! Fix the upstream dependency conflict, or retry
npm ERR! this command with --force or --legacy-peer-deps
npm ERR! to accept an incorrect (and potentially broken) dependency resolution.

Are there known steps to reproduce?

"dependencies": {
"typescript": "^5.3.3",
"vectordb": "^0.4.12",
...
}

[westonpace]

Yes, it looks like this was a bug in apache-arrow and they fixed it in apache/arrow@17b946c which was released in version 15.

vectordb is currently using version 14. We had made arrow a peer dependency because of the way apache-arrow works. This forces you to use the same version that vectordb is using. This is why you get the error when you upgrade to version 15.

The reason arrow needs to be the same version is that, if it is not, then instanceof checks will fail, and it is impossible to do things like "create an empty table based on an arrow schema". We have recently loosened this restriction in #1058 by adding a workaround. If something comes in that looks like an arrow-14-schema (e.g. an arrow-15-schema) then we will rebuild the arrow-14-schema based on that.

As a result, I think the simplest thing to do is probably for us to remove the peer dependency. This will let you use version 15 even if we are still using version 14. It also lets us upgrade to version 15 without that being a breaking change. I'll try and get a release out in the next few days with this change. In the meantime, you should be ok ignoring the above warning by using --force.

[awatson1978]

+1. Running into a similar Appache Arrow bug, but during npm install. Didn't manage to grab a screenshot, but same basic thing.

[westonpace]

@awatson1978 thanks for letting us know. Are you using arrow version 14 or version 15 in your project?