# Query JSONL, Parquet and Arrow shards by partition Local workflow · Windows and macOS Find which shard contains a record without merging the whole folder first. Inspect useful fields locally and keep the source file, row and partition with the result. Updated Nov 1, 2026 · [RowCove](https://rowcove.com/) **How do I query a folder of JSONL, Parquet or Arrow shards?** Choose the dataset folder, review discovered shards and partition keys, select a useful partition scope and project the required fields. Apply a supported field filter and follow continuation until complete when you need all matches. Collection browsing is read-only and does not imply a global sort or aggregate. **In this guide** [1. Choose the dataset root and review discovery](https://rowcove.com/guides/query-dataset-shards#discover) [2. Prune partitions and project fields](https://rowcove.com/guides/query-dataset-shards#scope) [3. Follow continuation and preserve provenance](https://rowcove.com/guides/query-dataset-shards#continue) [4. Use the same scope with an AI client](https://rowcove.com/guides/query-dataset-shards#mcp) [5. Complete four-shard mock traversal](https://rowcove.com/guides/query-dataset-shards#mock) [6. Choose a workflow worth repeating](https://rowcove.com/guides/query-dataset-shards#value) [All practical guides ↗](https://rowcove.com/guides) ## 1. Choose the dataset root and review discovery Choose folder in Open data and select the narrow dataset root in the native directory picker. RowCove detects supported shards automatically. Review discovery before trusting the scope: a root containing train and validation is different from the train folder alone. Keep unrelated files outside the selected scope. All examples and measurements below use generated mock data. No customer files or trading records are used. ``` mock-dataset/ split=train/shard-000.parquet split=train/shard-001.parquet split=validation/shard-000.parquet ``` JSONL and Arrow shards follow the same folder-selection approach. Discovered partition values and actual source formats determine what can be selected. ## 2. Prune partitions and project fields In collection setup, select the relevant split or supported partition range. Project only the fields needed for the investigation, such as id and group. Apply group equals mock-42 in the mock example. Partition predicates can exclude irrelevant shard files before scanning; a record filter alone may still require reading all selected shards. For the question “which file contains this record?”, begin with its stable identifier and a partition you can justify. Return the identifier and fields needed to check the match, then retain the source file and original row. For the different question “are there any more matches?”, continue through the selected scope until the scan is complete. Skipping a partition and projecting columns are distinct optimizations. JSONL still needs sequential parsing of selected text; columnar projection can avoid decoding omitted fields. Do not promise the same physical disk IO across formats. ## 3. Follow continuation and preserve provenance Review the bounded result page, source file, original source row and partition context. Continue with the next page until the scan reports completion if you need all matches. A short or empty intermediate page does not prove that the remaining collection has no matches. Keep the collection unchanged during traversal. Restart if the source snapshot changes rather than combining cursors from different source versions. Schema observations from a bounded page are not a certified global schema or a total row count. For a correction, locate the exact source file, open that individual file and use its supported save-copy workflow. Collection scans do not stage a corpus-wide edit or a global sort. See [JSONL corrections](https://rowcove.com/guides/query-edit-jsonl) and [bounded columnar editing](https://rowcove.com/guides/edit-parquet-arrow) . ## 4. Use the same scope with an AI client In the [MCP workspace](https://rowcove.com/docs-mcp) , authorize only the intended local root and inspect the installed collection tool schemas. Request projected fields, explicit partition predicates and bounded pages; preserve the returned cursor and source provenance. Test the call and inspect its wire log/history before building a longer automation. Local processing keeps source files on the device. Rows returned to an external AI client can still be transmitted by that client: send only the necessary fields and use mock data in public demonstrations. Use the [agent documentation and skills](https://rowcove.com/agent/index.md) for supported tools and safety boundaries. ## 5. Complete four-shard mock traversal All examples and measurements below use generated mock data. No customer files or trading records are used. 1,048,576 mock records across four train shards; three-trial medians | Full selected-source filter traversal | JSONL | Parquet | Arrow | | --- | --- | --- | --- | | Return id/group for all 10,810 matches | 1.139 s | 1.138 s | 1.289 s | | Return all fields for all 10,810 matches | 1.260 s | 4.438 s | 2.201 s | Measured on Windows with a Release native engine build on 27 October 2026: three fresh-process trials per task, the same 1,048,576 logical mock records, and a 1,024-byte synthetic payload. Median elapsed times include opening, index/schema preparation and the requested work. UI rendering, IPC and the separate correctness pass are excluded. OS caches were not cleared; these are not cold-disk guarantees or a comparison with another application. These engineering measurements cover the updated engine included in 0.1.1; results depend on the device, source and requested operation. The timer includes repeated catalog discovery and snapshot checks for every page. All selected source records were scanned; every returned match, exact identifier, source-file/row and partition was verified. These are full filtered traversals, not first-page timings, global sorts or total-row-count metadata reads. Projected native peak working set was at most 26.3 MiB across the three formats; all-field traversal reached 40.0 MiB. No retained temporary scratch files were observed at result time. Output copies, desktop renderer memory and transient disk peaks are separate costs. The four-shard fixture does not establish performance for millions of shard files or a TB corpus. ## 6. Choose a workflow worth repeating Pro is useful when locating and correcting records is repeated work: fewer one-off conversion scripts, less manual file splitting and a reviewed independent output. Free Preview opens up to 100 records read-only. Try representative files first; occasional small-file inspection may already be covered by free tools. See [introductory prices and renewal terms](https://rowcove.com/pricing) : first month US$5, then US$10/month; first year US$39, then US$100/year, for eligible new subscribers. Time saved depends on the tasks you repeat and your existing tools. These timings do not establish purchase intent, paid conversion or a universal speed ranking. The practical benefit is a shorter path from “which shard contains this record?” to an inspectable source location. Evaluate a training-data repair, a market-history partition check or a repeated research investigation before committing to a subscription. ## Continue your workflow - [Edit an individual JSONL source](https://rowcove.com/guides/query-edit-jsonl) - [Columnar edit limits and copies](https://rowcove.com/guides/edit-parquet-arrow) - [Bounded collection queries through MCP](https://rowcove.com/docs-mcp) ## Try the first page on your own data. Start on Windows or macOS: source information and up to 100 records, read-only. Review reader capabilities before planning a full-file Pro workflow. [Download for Windows ↓](https://rowcove.com/download/windows) [Download for macOS ↓](https://rowcove.com/download/macos) Version 0.1.1 · Windows x64 · macOS Apple Silicon · [Install guide →](https://rowcove.com/download) [Reader capabilities →](https://rowcove.com/formats) ## Find a record and decide the next step Questions, answered [Contact support →](https://rowcove.com/support) ### How do I open JSONL, Parquet or Arrow data split across many files? Use the folder picker or drop a dataset folder into Open data. For supported collections, RowCove discovers shards and available split or partition information. Choose the relevant source scope, project the fields you need and browse bounded result pages with source context. Keep the scope explicit when comparing results: one split, a date partition and the entire collection are different populations. Follow the continuation state through pages when you need every matching record. A bounded collection scan should not be treated as an automatic global sort, aggregate or complete schema reconciliation across every shard. [Browse a pretraining shard collection](https://rowcove.com/guides/pretraining-shards) [Work with columnar datasets](https://rowcove.com/guides/parquet-datasets) ### Can I correct the record after finding its source shard? Collection browsing is read-only. Keep the matching record’s source file, original row and partition, then open that individual file. If its reader supports editing, review and stage the correction and save to a new destination. Editing one source file does not rewrite the collection or replace its other shards. Check the source’s editing limits first. Parquet and Arrow edit-copy currently allow up to 128 MiB, 200,000 rows and 512 columns per source. A query can inspect a larger source without making that file eligible for editing. For an ineligible source, retain the provenance and use a suitable downstream transformation workflow. [Trace a match to its source](https://rowcove.com/guides/query-dataset-shards#continue) [Review columnar edit limits](https://rowcove.com/guides/edit-parquet-arrow#edit) ### Can an assistant draw conclusions from the first result page? A first page establishes what was observed in that page, not a global count, complete search, corpus-wide quality result or research conclusion. Ask the assistant to state the source, split or partitions, projection, filters and whether the operation has finished. Use continuation or result pagination when the task needs every matching record, and distinguish a completed full-file operation from a bounded scan. Before relying on a finding, inspect representative source records in RowCove and keep their provenance. Use the supplied agent skills for repeatable task steps. [Use the RowCove skills](https://rowcove.com/agent/skills.md) [Read result-scope rules](https://rowcove.com/agent/workflows.md) --- Canonical page: https://rowcove.com/guides/query-dataset-shards Updated: 2026-11-01 [Related Agent reference](https://rowcove.com/agent/workflows.md)