Table of Contents

Apache Arrow

Apache Arrow is a columnar in-memory format for analytical data. DuckDB.NET can hand you a query result directly as Arrow record batches, so you can move data into Arrow-aware libraries without reading it row by row.

DuckDB.NET builds the batches with DuckDB's Arrow C Data Interface. Each DuckDB data chunk becomes an Arrow RecordBatch with no per-value marshaling. The Arrow types come from the Apache.Arrow package, which ships as a dependency of DuckDB.NET.Data, so you do not need to add it yourself.

Stream Arrow record batches

Call ExecuteArrowStream to get an IArrowArrayStream. The stream exposes the result Schema and reads one RecordBatch per DuckDB data chunk. You own the returned stream and must dispose it.

using var duckDBConnection = new DuckDBConnection("Data Source=file.db");
duckDBConnection.Open();

using var command = duckDBConnection.CreateCommand();
command.CommandText = "SELECT city, temp_lo, temp_hi FROM weather ORDER BY city;";

// The caller owns the stream and must dispose it.
using IArrowArrayStream stream = command.ExecuteArrowStream();

Console.WriteLine("Columns: {0}", string.Join(", ", stream.Schema.FieldsList.Select(field => field.Name)));

while (await stream.ReadNextRecordBatchAsync() is { } batch)
{
    using (batch)
    {
        var city = (StringArray)batch.Column("city");
        var tempLo = (Int32Array)batch.Column("temp_lo");
        var tempHi = (Int32Array)batch.Column("temp_hi");

        for (var row = 0; row < batch.Length; row++)
        {
            Console.WriteLine("{0}: {1} to {2}", city.GetString(row), tempLo.GetValue(row), tempHi.GetValue(row));
        }
    }
}

Each column in a batch is an Arrow array. Cast it to the matching Arrow type, such as StringArray or Int32Array, to read values.

Stream batches asynchronously

ExecuteArrowBatchesAsync returns an IAsyncEnumerable<RecordBatch> that you consume with await foreach. It produces the same batches as ExecuteArrowStream and accepts a CancellationToken.

using var duckDBConnection = new DuckDBConnection("Data Source=file.db");
duckDBConnection.Open();

using var command = duckDBConnection.CreateCommand();
command.CommandText = "SELECT city, temp_lo, temp_hi FROM weather ORDER BY city;";

var rowCount = 0;

await foreach (RecordBatch batch in command.ExecuteArrowBatchesAsync())
{
    using (batch)
    {
        rowCount += batch.Length;
    }
}

Console.WriteLine("Read {0} rows from the result set.", rowCount);

Materialized and streaming mode

The Arrow methods honor the same execution modes as the rest of DuckDB.NET. By default DuckDB materializes the whole result in memory before producing batches. Set the UseStreamingMode property to true to fetch the result chunk by chunk with bounded memory. See Basic Usage for more on materialized and streaming mode.

Multiple statements

The Arrow methods return only the first result set in the command. They execute statements up to and including that first result set, then stop. Any statement after it does not run, so its side effects (for example a CREATE TABLE or INSERT after a SELECT) are skipped.

If your command has more than one result set, use ExecuteReader and walk the results with NextResult instead. See Batching. For Arrow, prefer a single-statement command.

Edit this page