support connecting to lance files in an S3 bucket

[Analect]

Thanks for a great tool. Could you envisage this working against an S3 URI as well as a filesystem path, as currently implemented?

[gordonmurray]

Thanks for posting this, Sorry for such a delay in responding.

Its very doable, I have other projects using Lance on S3/other object stores.

We currently wrap DATA_PATH in Path() and run local filesystem checks against it. This mangles the URI into s3:/bucket before LanceDB sees it, and the startup script then refuses to launch because the “directory” does not exist.

Most of the work involves removing those local filesystem assumptions and documenting the required credential environment variables, including AWS_ENDPOINT for MinIO users.

One important difference with object storage is that you cannot mount a bucket as read-only in the same way as a local folder. The read-only guarantee therefore needs to come from read-only credentials. The application does not currently write any data, but the documentation should make this explicit.

There is also an endpoint that reads an entire table into memory before slicing it. That may be acceptable on local disk, but it would perform poorly over S3, so I would address that as part of this work.