Module arrow2::io::parquet::read

APIs to read from Parquet format.

Re-exports

pub use parquet2::fallible_streaming_iterator;

pub use schema::infer_schema;

Modules

schema

APIs to handle Parquet <-> Arrow schemas.

statistics

APIs exposing parquet2’s statistics as arrow’s statistics.

Structs

BasicDecompressor

A FallibleStreamingIterator that decompresses CompressedDataPage into DataPage.

ColumnChunkMetaData

Metadata for a column chunk.

ColumnDescriptor

A descriptor for leaf-level primitive columns. This encapsulates information such as definition and repetition levels and is used to re-assemble nested data.

CompressedDataPage

A CompressedDataPage is compressed, encoded representation of a Parquet data page. It holds actual data and thus cloning it is expensive.

DataPage

A DataPage is an uncompressed, encoded representation of a Parquet data page. It holds actual data and thus cloning it is expensive.

Decompressor

Decompressor that allows re-using the page buffer of PageIterator.

FileMetaData

Metadata for a Parquet file.

FileReader

An iterator of Chunks coming from row groups of a parquet file.

PageIterator

A page iterator iterates over row group’s pages. In parquet, pages are guaranteed to be contiguously arranged in memory and therefore must be read in sequence.

ReadColumnIterator

A MutStreamingIterator of pre-read column chunks

RowGroupDeserializer

An Iterator of Chunk that (dynamically) adapts a vector of iterators of Array into an iterator of Chunk.

RowGroupMetaData

Metadata for a row group.

RowGroupReader

An [Iterator<Item=RowGroupDeserializer>] from row groups of a parquet file.

TimestampType

Timestamp logical type annotation

Enums

DataPageHeader

LogicalType

ParquetError

Errors generated by this crate

ParquetTimeUnit

ParquetType

Representation of a Parquet type. Used to describe primitive leaf fields and structs, including top-level schema. Note that the top-level schema type is represented using GroupType whose repetition is None.

PhysicalType

PrimitiveConvertedType

State

State of MutStreamingIterator.

Traits

ColumnChunkIter

Trait describing a MutStreamingIterator of column chunks.

DataPages

Trait describing a FallibleStreamingIterator of DataPage

FallibleStreamingIterator

A fallible, streaming iterator.

MutStreamingIterator

Functions

_get_page_iterator

Returns a new PageIterator by seeking reader to the begining of column_chunk.

_get_page_stream

Returns a stream of compressed data pages

_read_metadata

Reads a file’s metadata.

_read_metadata_async

column_iter_to_arrays

An iterator adapter that maps multiple iterators of DataPages into an iterator of Arrays.

decompress

Decompresses the page, using buffer for decompression. If page.buffer.len() == 0, there was no decompression and the buffer was moved. Else, decompression took place.

get_column_iterator

Returns a ColumnIterator of column chunks corresponding to field. Contrarily to get_page_iterator that returns a single iterator of pages, this iterator returns multiple iterators, one per physical column of the field. For primitive fields (e.g. i64), ColumnIterator yields exactly one column. For complex fields, it yields multiple columns.

get_page_iterator

Creates a new iterator of compressed pages.

int96_to_i64_ns

read_columns

Reads all columns that are part of the parquet field field_name

read_columns_async

Reads all columns that are part of the parquet field field_name

read_columns_many

Returns a vector of iterators of Array (ArrayIter) corresponding to the top level parquet fields whose name matches fields’s names.

read_columns_many_async

Returns a vector of iterators of Array corresponding to the top level parquet fields whose name matches fields’s names.

read_metadata

Reads parquets’ metadata syncronously.

read_metadata_async

Reads parquets’ metadata asynchronously.

to_deserializer

Converts a vector of columns associated with the parquet field whose name is Field to an iterator of Array, ArrayIter of chunk size chunk_size.

Type Definitions

ArrayIter

Type def for a sharable, boxed dyn Iterator of arrays

PageFilter

Type declaration for a page filter