next.js / duckdb-wasm parquet demo

Example Next.js app using duckdb-wasm to read/fetch Parquet files, in Node and the browser.

Demo: runsascoded.com/next-duckdb-parquet-demo

Relevant files:

• pages/index.tsx: shows 3 ways of loading Parquet data (on a mix of client and server):
  1. useParquetBuf: load parquet file as byte array on server, send to client, decode in browser (minimize data sent to client while loading a static page)
  2. loadparquet: load parquet file on server, send to client
  3. useParquet: fetch parquet file by URL on client
• src/parquet.ts: utilities for instantiating DuckDB and fetching/loading Parquet data
  • Copied from runsascoded/next-utils/parquet.ts, adapted from ilyabo/graphnavi/lib/useDuckConn.ts
  • See also duckdb-wasm#1148
• public/people.parquet: demo Parquet file, generated by init-data.js

Live demo

runsascoded.com/next-duckdb-parquet-demo:

Parquet data loaded on server (in getStaticProps, with loadParquet):

[{"id":1,"name":"Mark"},{"id":2,"name":"Hannes"}]

Parquet data buffer loaded on server, parsed on client (with useParquetBuf):

[{"id":1,"name":"Mark"},{"id":2,"name":"Hannes"}]

Parquet data fetched on client, from https://runsascoded.com/next-duckdb-parquet-demo/people.parquet (with useParquet):

[{"id":1,"name":"Mark"},{"id":2,"name":"Hannes"}]

Run locally

Dev mode:

next dev  # serve dev-mode site at http://127.0.0.1:3000

Build + Export:

next build
next export
http-server out  # serve static site from out/ dir

Generate sample data

node init-data.js

Runs init-data.js:

const duckdb = require('duckdb')
const db = new duckdb.Database(':memory:')
db.run(`
CREATE TABLE people(id INTEGER, name VARCHAR);
INSERT INTO people VALUES (1, 'Mark'), (2, 'Hannes');
COPY (SELECT * FROM people) TO 'public/people.parquet' (FORMAT 'parquet');
`)

⚠️ Broken under "App Router" ⚠️

For some reason, the server-side loadParquet path doesn't seem to work under Next.js's new "App router" (also reported at next.js#57819).

Here's a trimmed down example (from the @app-broken branch):

• pages/pages.tsx ✅ works
• app/app/page.tsx ❌ broken

Attempting to load the latter emits this on the server side:

duckdb-wasm fetch
bestBundle: {
  mainModule: '/Users/ryan/c/next-duckdb-parquet-demo/node_modules/@duckdb/duckdb-wasm/dist/duckdb-eh.wasm',
  mainWorker: '/Users/ryan/c/next-duckdb-parquet-demo/node_modules/@duckdb/duckdb-wasm/dist/duckdb-node-eh.worker.cjs',
  pthreadWorker: null
}
made logger
made db
worker terminated with 1 pending requests

and the client hangs.

Similarly, next build times out, meaning I can't deploy a demo. Here's a GitHub Actions run that failed due to this issue.

diff pages/pages.tsx app/app/page.tsx:

-export async function getStaticProps() {
-    const data = await loadParquet<Person>(parquetPath)  // ✅ works fine under pages/
-    return { props: { data } }
-}
-
-export default function Home({ data }: { data: Person[] }) {
+export default async function Home() {
+    const data = await loadParquet<Person>(parquetPath)  // ❌ broken under app/

This line:

await db.instantiate(bundle.mainModule, bundle.pthreadWorker);

emits this error:

worker terminated with 1 pending requests

I haven't found any further leads about what might be going on.