IterableData

Iterable Data

Iterable Data is a Python library for reading and writing data files row by row in a consistent, iterator-based interface. It provides a unified API for working with various data formats (CSV, JSON, Parquet, XML, etc.) similar to csv.DictReader but supporting many more formats.

This library simplifies data processing and conversion between formats while preserving complex nested data structures (unlike pandas DataFrames which require flattening).

Features

• Unified API: Single interface for reading/writing multiple data formats
• Automatic Format Detection: Detects file type and compression from filename or content (magic numbers and heuristics)
• Format Capability Reporting: Programmatically query format capabilities (read/write/bulk/totals/streaming/tables)
• Support for Compression: Works seamlessly with compressed files
• Preserves Nested Data: Handles complex nested structures as Python dictionaries
• DuckDB Integration: Optional DuckDB engine for high-performance queries with pushdown optimizations
• Pipeline Processing: Built-in pipeline support for data transformation
• Encoding Detection: Automatic encoding and delimiter detection for text files
• Bulk Operations: Efficient batch reading and writing
• Native Batch Conversion: Opt-in columnar-to-columnar transfers with projection, row-range, and batch-size selection
• Bounded Columnar I/O: Shared row/bulk cursors and configurable Parquet row groups keep large reads and writes bounded
• Codec Performance Profiles: Choose fast, balanced, or max compression settings with effective-setting diagnostics
• Table Listing: Discover available tables, sheets, and datasets in multi-table formats
• Context Manager Support: Use with statements for automatic resource cleanup
• DataFrame Bridges: Convert iterable data to Pandas, Polars, and Dask DataFrames with one-liner methods
• Cloud Storage Support: Direct access to S3, GCS, and Azure Blob Storage via URI schemes
• Database Engine Support: Read-only access to SQL and NoSQL databases (PostgreSQL, ClickHouse, MySQL, MongoDB, Elasticsearch, etc.) as iterable data sources
• Atomic Writes: Production-safe file writing with temporary files and atomic renames
• Bulk File Conversion: Convert multiple files at once using glob patterns or directories
• Progress Tracking and Metrics: Built-in progress bars, callbacks, and structured metrics objects
• Error Handling Controls: Configurable error policies and structured error logging; malformed input raises typed errors by default instead of reading as empty datasets
• Security Hardening: XXE-safe XML parsing, AST-whitelisted filter expressions, and explicit pickle trust acknowledgement
• Performance Regression Gate: CI-enforced baselines for representative read/convert workloads
• Container Formats: Stream records from TAR archives without extracting members to disk
• Type Hints and Type Safety: Complete type annotations with typed helper functions for dataclasses and Pydantic models
• Lakehouse Tables: Read and write Delta Lake, Iceberg, and DuckLake; read Hudi; experimental Apache Paimon tables plus Row/Mosaic file formats

Supported File Types

Core Formats

• JSON - Standard JSON files
• JSONL/NDJSON - JSON Lines format (one JSON object per line)
• JSON-LD - JSON for Linking Data (RDF format)
• CSV/TSV - Comma and tab-separated values
• Annotated CSV - CSV with type annotations and metadata
• CSVW - CSV on the Web (with metadata)
• PSV/SSV - Pipe and semicolon-separated values
• LTSV - Labeled Tab-Separated Values
• FWF - Fixed Width Format
• XML - XML files with configurable tag parsing
• ZIP XML - XML files within ZIP archives
• HTML - HTML files with table extraction

Binary Formats

• BSON - Binary JSON format
• MessagePack - Efficient binary serialization
• CBOR - Concise Binary Object Representation
• UBJSON - Universal Binary JSON
• SMILE - Binary JSON variant
• Bencode - BitTorrent encoding format
• Avro - Apache Avro binary format (read & write)
• Pickle - Python pickle format (untrusted input is unsafe; pass trust=True to acknowledge)

Columnar & Analytics Formats

• Parquet - Apache Parquet columnar format
• ORC - Optimized Row Columnar format
• Arrow/Feather - Apache Arrow columnar format
• GeoParquet - GeoParquet metadata-aware Parquet profile with geometry/CRS preservation
• Lance - Modern columnar format optimized for ML and vector search
• Vortex - Modern columnar format with fast random access
• Paimon Row - Apache Paimon row format for O(1) row-number access
• Paimon Mosaic - Apache Paimon columnar-bucket format for wide tables
• Paimon - Apache Paimon warehouse/catalog tables
• Delta Lake - Delta Lake format (read & write)
• Iceberg - Apache Iceberg format (read & write)
• DuckLake - DuckLake lakehouse tables (read & write)
• Hudi - Apache Hudi format (read; writes deferred)

Database Formats

• SQLite - SQLite database files
• DBF - dBase/FoxPro database files
• MySQL Dump - MySQL dump files
• PostgreSQL Copy - PostgreSQL COPY format
• DuckDB - DuckDB database files

Statistical Formats

• SAS - SAS data files
• Stata - Stata data files
• SPSS - SPSS data files
• R Data - R RDS and RData files
• fst - R fst columnar on-disk frames (fst extra; experimental)
• PX - PC-Axis format
• ARFF - Attribute-Relation File Format (Weka format)
• LIBSVM - Sparse labeled feature vectors (read & write)
• NumPy - .npy / .npz array rows (read & write; npy extra)

Scientific Formats

• NetCDF - Network Common Data Form for scientific data
• CDF - NASA Common Data Format (space science)
• HDF5 - Hierarchical Data Format
• Zarr - Chunked array stores (zarr extra; experimental)
• XYZ - Molecular/point coordinate tables
• CIF - Crystallographic Information File (atom_site loops; experimental)
• PDB - Protein Data Bank ATOM/HETATM records
• MATLAB MAT - MATLAB .mat variables (mat extra; experimental)
• SEG-Y - Seismic traces (geophysical extra; experimental)
• GRIB2 - Meteorological messages (geophysical extra; experimental)
• miniSEED - Seismological waveform windows (geophysical extra; experimental)

Geospatial Formats

• GeoJSON - Geographic JSON format
• GeoJSON Text Sequence - RFC 8142 line-delimited GeoJSON Features (.geojsonl, .geojsons); streaming-friendly
• GeoPackage - OGC GeoPackage format
• GML - Geography Markup Language
• KML - Keyhole Markup Language
• KMZ - KML Zipped (ZIP archive containing KML)
• GPX - GPS Exchange Format (waypoints, routes, tracks)
• Shapefile - ESRI Shapefile format
• File Geodatabase - ESRI FileGDB layers via Fiona (geospatial extra; experimental)
• MapInfo MIF - MapInfo Interchange Format (geospatial extra; experimental)
• Esri ASCII Grid - Raster grids as cell or row records (.asc)
• ArcInfo E00 - Interchange exports (experimental subset)
• LAS - LiDAR point clouds (lidar extra; experimental)
• BAG - Bathymetric Attributed Grid (hdf5 extra; experimental)
• CZML - Cesium CZML document packets
• FlatGeobuf - Streaming geospatial features with optional spatial-index filtering
• MVT/PBF - Mapbox Vector Tiles
• TopoJSON - Topology-preserving GeoJSON extension

RDF & Semantic Formats

• JSON-LD - JSON for Linking Data
• RDF/XML - RDF in XML format
• Turtle - Terse RDF Triple Language
• N-Triples - Line-based RDF format
• N-Quads - N-Triples with context
• TriG - RDF Triple Graph format
• N3 - Notation3 RDF format
• TriX - XML Triple RDF format
• HDT - Header-Dictionary-Triples compact RDF (rdf extra; experimental)

Feed Formats

• Atom - Atom Syndication Format
• RSS - Rich Site Summary feed format

Network Formats

• PCAP - Packet Capture format
• PCAPNG - PCAP Next Generation format

Log & Event Formats

• Apache Log - Apache access/error logs
• CEF - Common Event Format
• GELF - Graylog Extended Log Format
• WARC - Web ARChive format
• CDX - Web archive index format
• ILP - InfluxDB Line Protocol
• HTML - HTML files with table extraction

Email Formats

• EML - Email message format
• MBOX - Mailbox format
• MHTML - MIME HTML format

Configuration Formats

• INI - INI configuration files
• TOML - Tom's Obvious Minimal Language
• YAML - YAML Ain't Markup Language
• HOCON - Human-Optimized Config Object Notation
• EDN - Extensible Data Notation

Office Formats

• XLS/XLSX - Microsoft Excel files
• XLSB - Excel Binary format
• ODS - OpenDocument Spreadsheet
• Microsoft Access - Access .mdb / .accdb tables (access extra; experimental)
• Lotus 1-2-3 - Legacy WK1 / .123 spreadsheets (experimental)

Business & Exchange Formats

• EDI - X12 / EDIFACT segment streams (experimental)
• IATI - Aid-transparency activity XML (xml extra; experimental)

CAD Formats

• DXF - AutoCAD Drawing Exchange Format

Graph Formats

• GraphML - Graph Markup Language
• GEXF - Graph Exchange XML Format
• DOT - GraphViz DOT format

Sequence & Alignment Formats

• FASTA - Sequence format (protein/nucleotide)
• FASTQ - Sequence with quality format
• SAM - Sequence Alignment/Map (text)
• BAM - Binary SAM format
• Genomic VCF/BCF - Variant Call Format for genomic data (distinct from vCard .vcf; requires bio extra)
• CRAM - Reference-compressed sequence alignments (requires alignment extra and an explicit reference when needed)
• BED - BED3–BED12 genomic intervals
• GFF3/GTF - Genomic feature annotations with coordinate and attribute preservation

Streaming & Big Data Formats

• Kafka - Apache Kafka format
• Pulsar - Apache Pulsar format
• Flink - Apache Flink format
• Beam - Apache Beam format
• RecordIO - RecordIO format
• SequenceFile - Hadoop SequenceFile
• TFRecord - TensorFlow Record format
• WebDataset - TAR shards grouped into ML sample dicts (format="webdataset"; experimental)

Protocol & Serialization Formats

• Protocol Buffers - Google Protocol Buffers
• Cap'n Proto - Cap'n Proto serialization
• FlatBuffers - FlatBuffers serialization
• FlexBuffers - FlexBuffers format
• Thrift - Apache Thrift format
• ASN.1 - ASN.1 encoding format
• Ion - Amazon Ion format
• OTLP JSON/Protobuf - OpenTelemetry traces, logs, and metrics export profiles (requires otlp extra)

Other Formats

• TAR - Multi-file archive container (read-only; streams members without extracting to disk)
• vCard (VCF) - Electronic business cards (RFC 6350); not genomic Variant Call Format
• iCal - iCalendar format
• LDIF - LDAP Data Interchange Format
• TXT - Plain text files

See the formats documentation (or docs/docs/formats/ in this repo) for per-format parameters, record shapes, and extras.

Supported Compression Codecs

• GZip (.gz)
• BZip2 (.bz2)
• LZMA (.xz, .lzma)
• LZ4 (.lz4)
• ZIP (.zip)
• Brotli (.br)
• ZStandard (.zst, .zstd)
• Snappy (.snappy, .sz) — streaming decompression for framed files
• LZO (.lzo, .lzop) — streaming decompression with legacy blob fallback
• 7z (.7z; requires py7zr via iterabledata[compression])

Requirements

Python 3.10+

Installation

pip install iterabledata

The PyPI package is iterabledata. Import iterable:

from i