Enum ScalarValue

pub enum ScalarValue {
Show 43 variants    Null,
    Boolean(Option<bool>),
    Float16(Option<f16>),
    Float32(Option<f32>),
    Float64(Option<f64>),
    Decimal128(Option<i128>, u8, i8),
    Decimal256(Option<i256>, u8, i8),
    Int8(Option<i8>),
    Int16(Option<i16>),
    Int32(Option<i32>),
    Int64(Option<i64>),
    UInt8(Option<u8>),
    UInt16(Option<u16>),
    UInt32(Option<u32>),
    UInt64(Option<u64>),
    Utf8(Option<String>),
    LargeUtf8(Option<String>),
    Binary(Option<Vec<u8>>),
    FixedSizeBinary(i32, Option<Vec<u8>>),
    LargeBinary(Option<Vec<u8>>),
    FixedSizeList(Arc<FixedSizeListArray>),
    List(Arc<GenericListArray<i32>>),
    LargeList(Arc<GenericListArray<i64>>),
    Struct(Arc<StructArray>),
    Date32(Option<i32>),
    Date64(Option<i64>),
    Time32Second(Option<i32>),
    Time32Millisecond(Option<i32>),
    Time64Microsecond(Option<i64>),
    Time64Nanosecond(Option<i64>),
    TimestampSecond(Option<i64>, Option<Arc<str>>),
    TimestampMillisecond(Option<i64>, Option<Arc<str>>),
    TimestampMicrosecond(Option<i64>, Option<Arc<str>>),
    TimestampNanosecond(Option<i64>, Option<Arc<str>>),
    IntervalYearMonth(Option<i32>),
    IntervalDayTime(Option<i64>),
    IntervalMonthDayNano(Option<i128>),
    DurationSecond(Option<i64>),
    DurationMillisecond(Option<i64>),
    DurationMicrosecond(Option<i64>),
    DurationNanosecond(Option<i64>),
    Union(Option<(i8, Box<ScalarValue>)>, UnionFields, UnionMode),
    Dictionary(Box<DataType>, Box<ScalarValue>),
}

Expand description

A dynamically typed, nullable single value.

While an arrow [Array]) stores one or more values of the same type, in a single column, a ScalarValue stores a single value of a single type, the equivalent of 1 row and one column.

 ┌────────┐
 │ value1 │
 │ value2 │                  ┌────────┐
 │ value3 │                  │ value2 │
 │  ...   │                  └────────┘
 │ valueN │
 └────────┘

   Array                     ScalarValue

stores multiple,             stores a single,
possibly null, values of     possible null, value
the same type

§Performance

In general, performance will be better using arrow [Array]s rather than ScalarValue, as it is far more efficient to process multiple values at once (vectorized processing).

§Example

// Create single scalar value for an Int32 value
let s1 = ScalarValue::Int32(Some(10));

// You can also create values using the From impl:
let s2 = ScalarValue::from(10i32);
assert_eq!(s1, s2);

§Null Handling

ScalarValue represents null values in the same way as Arrow. Nulls are “typed” in the sense that a null value in an [Int32Array] is different than a null value in a [Float64Array], and is different than the values in a [NullArray].

// You can create a 'null' Int32 value directly:
let s1 = ScalarValue::Int32(None);

// You can also create a null value for a given datatype:
let s2 = ScalarValue::try_from(&DataType::Int32)?;
assert_eq!(s1, s2);

// Note that this is DIFFERENT than a `ScalarValue::Null`
let s3 = ScalarValue::Null;
assert_ne!(s1, s3);

§Nested Types

List / LargeList / FixedSizeList / Struct are represented as a single element array of the corresponding type.

§Example: Creating `ScalarValue::Struct` using [`ScalarStructBuilder`]

// Build a struct like: {a: 1, b: "foo"}
let field_a = Field::new("a", DataType::Int32, false);
let field_b = Field::new("b", DataType::Utf8, false);

let s1 = ScalarStructBuilder::new()
   .with_scalar(field_a, ScalarValue::from(1i32))
   .with_scalar(field_b, ScalarValue::from("foo"))
   .build();

§Example: Creating a null `ScalarValue::Struct` using [`ScalarStructBuilder`]

// Build a struct representing a NULL value
let fields = vec![
    Field::new("a", DataType::Int32, false),
    Field::new("b", DataType::Utf8, false),
];

let s1 = ScalarStructBuilder::new_null(fields);

§Example: Creating `ScalarValue::Struct` directly

// Build a struct like: {a: 1, b: "foo"}
// Field description
let fields = Fields::from(vec![
  Field::new("a", DataType::Int32, false),
  Field::new("b", DataType::Utf8, false),
]);
// one row arrays for each field
let arrays: Vec<ArrayRef> = vec![
  Arc::new(Int32Array::from(vec![1])),
  Arc::new(StringArray::from(vec!["foo"])),
];
// no nulls for this array
let nulls = None;
let arr = StructArray::new(fields, arrays, nulls);

// Create a ScalarValue::Struct directly
let s1 = ScalarValue::Struct(Arc::new(arr));

§Further Reading

See datatypes for details on datatypes and the format for the definitive reference.

Variants§

§

Null

represents DataType::Null (castable to/from any other type)

§

Boolean(Option<bool>)

true or false value

§

Float16(Option<f16>)

16bit float

§

Float32(Option<f32>)

32bit float

§

Float64(Option<f64>)

64bit float

§

Decimal128(Option<i128>, u8, i8)

128bit decimal, using the i128 to represent the decimal, precision scale

§

Decimal256(Option<i256>, u8, i8)

256bit decimal, using the i256 to represent the decimal, precision scale

§

Int8(Option<i8>)

signed 8bit int

§

Int16(Option<i16>)

signed 16bit int

§

Int32(Option<i32>)

signed 32bit int

§

Int64(Option<i64>)

signed 64bit int

§

UInt8(Option<u8>)

unsigned 8bit int

§

UInt16(Option<u16>)

unsigned 16bit int

§

UInt32(Option<u32>)

unsigned 32bit int

§

UInt64(Option<u64>)

unsigned 64bit int

§

Utf8(Option<String>)

utf-8 encoded string.

§

LargeUtf8(Option<String>)

utf-8 encoded string representing a LargeString’s arrow type.

§

Binary(Option<Vec<u8>>)

binary

§

FixedSizeBinary(i32, Option<Vec<u8>>)

fixed size binary

§

LargeBinary(Option<Vec<u8>>)

large binary

§

FixedSizeList(Arc<FixedSizeListArray>)

Fixed size list scalar.

The array must be a FixedSizeListArray with length 1.

§

List(Arc<GenericListArray<i32>>)

Represents a single element of a [ListArray] as an [ArrayRef]

The array must be a ListArray with length 1.

§

LargeList(Arc<GenericListArray<i64>>)

The array must be a LargeListArray with length 1.

§

Struct(Arc<StructArray>)

Represents a single element [StructArray] as an [ArrayRef]. See ScalarValue for examples of how to create instances of this type.

§

Date32(Option<i32>)

Date stored as a signed 32bit int days since UNIX epoch 1970-01-01

§

Date64(Option<i64>)

Date stored as a signed 64bit int milliseconds since UNIX epoch 1970-01-01

§

Time32Second(Option<i32>)

Time stored as a signed 32bit int as seconds since midnight

§

Time32Millisecond(Option<i32>)

Time stored as a signed 32bit int as milliseconds since midnight

§

Time64Microsecond(Option<i64>)

Time stored as a signed 64bit int as microseconds since midnight

§

Time64Nanosecond(Option<i64>)

Time stored as a signed 64bit int as nanoseconds since midnight

§

TimestampSecond(Option<i64>, Option<Arc<str>>)

Timestamp Second

§

TimestampMillisecond(Option<i64>, Option<Arc<str>>)

Timestamp Milliseconds

§

TimestampMicrosecond(Option<i64>, Option<Arc<str>>)

Timestamp Microseconds

§

TimestampNanosecond(Option<i64>, Option<Arc<str>>)

Timestamp Nanoseconds

§

IntervalYearMonth(Option<i32>)

Number of elapsed whole months

§

IntervalDayTime(Option<i64>)

Number of elapsed days and milliseconds (no leap seconds) stored as 2 contiguous 32-bit signed integers

§

IntervalMonthDayNano(Option<i128>)

A triple of the number of elapsed months, days, and nanoseconds. Months and days are encoded as 32-bit signed integers. Nanoseconds is encoded as a 64-bit signed integer (no leap seconds).

§

DurationSecond(Option<i64>)

Duration in seconds

§

DurationMillisecond(Option<i64>)

Duration in milliseconds

§

DurationMicrosecond(Option<i64>)

Duration in microseconds

§

DurationNanosecond(Option<i64>)

Duration in nanoseconds

§

Union(Option<(i8, Box<ScalarValue>)>, UnionFields, UnionMode)

A nested datatype that can represent slots of differing types. Components: .0: a tuple of union type_id and the single value held by this Scalar .1: the list of fields, zero-to-one of which will by set in .0 .2: the physical storage of the source/destination UnionArray from which this Scalar came

§

Dictionary(Box<DataType>, Box<ScalarValue>)

Dictionary type: index type and value

Implementations§

§

impl ScalarValue

pub fn new_primitive<T>( a: Option<<T as ArrowPrimitiveType>::Native>, d: &DataType, ) -> Result<ScalarValue, DataFusionError>
where T: ArrowPrimitiveType,

Create a [Result<ScalarValue>] with the provided value and datatype

§Panics

Panics if d is not compatible with T

pub fn try_new_decimal128( value: i128, precision: u8, scale: i8, ) -> Result<ScalarValue, DataFusionError>

Create a decimal Scalar from value/precision and scale.

pub fn new_utf8(val: impl Into<String>) -> ScalarValue

Returns a ScalarValue::Utf8 representing val

pub fn new_interval_ym(years: i32, months: i32) -> ScalarValue

Returns a ScalarValue::IntervalYearMonth representing years years and months months

pub fn new_interval_dt(days: i32, millis: i32) -> ScalarValue

Returns a ScalarValue::IntervalDayTime representing days days and millis milliseconds

pub fn new_interval_mdn(months: i32, days: i32, nanos: i64) -> ScalarValue

Returns a ScalarValue::IntervalMonthDayNano representing months months and days days, and nanos nanoseconds

pub fn new_timestamp<T>( value: Option<i64>, tz_opt: Option<Arc<str>>, ) -> ScalarValue
where T: ArrowTimestampType,

Returns a ScalarValue representing value and tz_opt timezone

pub fn new_zero(datatype: &DataType) -> Result<ScalarValue, DataFusionError>

Create a zero value in the given type.

pub fn new_one(datatype: &DataType) -> Result<ScalarValue, DataFusionError>

Create an one value in the given type.

pub fn new_negative_one( datatype: &DataType, ) -> Result<ScalarValue, DataFusionError>

Create a negative one value in the given type.

pub fn new_ten(datatype: &DataType) -> Result<ScalarValue, DataFusionError>

pub fn data_type(&self) -> DataType

return the [DataType] of this ScalarValue

pub fn get_datatype(&self) -> DataType

👎Deprecated since 31.0.0: use data_type instead

Getter for the DataType of the value.

Suggest using Self::data_type as a more standard API

pub fn arithmetic_negate(&self) -> Result<ScalarValue, DataFusionError>

Calculate arithmetic negation for a scalar value

pub fn add<T>(&self, other: T) -> Result<ScalarValue, DataFusionError>
where T: Borrow<ScalarValue>,

Wrapping addition of ScalarValue

NB: operating on ScalarValue directly is not efficient, performance sensitive code should operate on Arrays directly, using vectorized array kernels

pub fn add_checked<T>(&self, other: T) -> Result<ScalarValue, DataFusionError>
where T: Borrow<ScalarValue>,

Checked addition of ScalarValue

NB: operating on ScalarValue directly is not efficient, performance sensitive code should operate on Arrays directly, using vectorized array kernels

pub fn sub<T>(&self, other: T) -> Result<ScalarValue, DataFusionError>
where T: Borrow<ScalarValue>,

Wrapping subtraction of ScalarValue

NB: operating on ScalarValue directly is not efficient, performance sensitive code should operate on Arrays directly, using vectorized array kernels

pub fn sub_checked<T>(&self, other: T) -> Result<ScalarValue, DataFusionError>
where T: Borrow<ScalarValue>,

Checked subtraction of ScalarValue

NB: operating on ScalarValue directly is not efficient, performance sensitive code should operate on Arrays directly, using vectorized array kernels

pub fn mul<T>(&self, other: T) -> Result<ScalarValue, DataFusionError>
where T: Borrow<ScalarValue>,

Wrapping multiplication of ScalarValue

NB: operating on ScalarValue directly is not efficient, performance sensitive code should operate on Arrays directly, using vectorized array kernels.

pub fn mul_checked<T>(&self, other: T) -> Result<ScalarValue, DataFusionError>
where T: Borrow<ScalarValue>,

Checked multiplication of ScalarValue

NB: operating on ScalarValue directly is not efficient, performance sensitive code should operate on Arrays directly, using vectorized array kernels.

pub fn div<T>(&self, other: T) -> Result<ScalarValue, DataFusionError>
where T: Borrow<ScalarValue>,

Performs lhs / rhs

Overflow or division by zero will result in an error, with exception to floating point numbers, which instead follow the IEEE 754 rules.

NB: operating on ScalarValue directly is not efficient, performance sensitive code should operate on Arrays directly, using vectorized array kernels.

pub fn rem<T>(&self, other: T) -> Result<ScalarValue, DataFusionError>
where T: Borrow<ScalarValue>,

Performs lhs % rhs

Overflow or division by zero will result in an error, with exception to floating point numbers, which instead follow the IEEE 754 rules.

NB: operating on ScalarValue directly is not efficient, performance sensitive code should operate on Arrays directly, using vectorized array kernels.

pub fn is_unsigned(&self) -> bool

pub fn is_null(&self) -> bool

whether this value is null or not.

pub fn distance(&self, other: &ScalarValue) -> Option<usize>

Absolute distance between two numeric values (of the same type). This method will return None if either one of the arguments are null. It might also return None if the resulting distance is greater than usize::MAX. If the type is a float, then the distance will be rounded to the nearest integer.

Note: the datatype itself must support subtraction.

pub fn to_array(&self) -> Result<Arc<dyn Array>, DataFusionError>

Converts a scalar value into an 1-row array.

§Errors

Errors if the ScalarValue cannot be converted into a 1-row array

pub fn to_scalar(&self) -> Result<Scalar<Arc<dyn Array>>, DataFusionError>

Converts a scalar into an arrow [Scalar] (which implements the Datum interface).

This can be used to call arrow compute kernels such as lt

§Errors

Errors if the ScalarValue cannot be converted into a 1-row array

§Example

use datafusion_common::ScalarValue;
use arrow::array::{BooleanArray, Int32Array};

let arr = Int32Array::from(vec![Some(1), None, Some(10)]);
let five = ScalarValue::Int32(Some(5));

let result = arrow::compute::kernels::cmp::lt(
  &arr,
  &five.to_scalar().unwrap(),
).unwrap();

let expected = BooleanArray::from(vec![
    Some(true),
    None,
    Some(false)
  ]
);

assert_eq!(&result, &expected);

pub fn iter_to_array( scalars: impl IntoIterator<Item = ScalarValue>, ) -> Result<Arc<dyn Array>, DataFusionError>

Converts an iterator of references ScalarValue into an [ArrayRef] corresponding to those values. For example, an iterator of ScalarValue::Int32 would be converted to an [Int32Array].

Returns an error if the iterator is empty or if the ScalarValues are not all the same type

§Panics

Panics if self is a dictionary with invalid key type

§Example

use datafusion_common::ScalarValue;
use arrow::array::{ArrayRef, BooleanArray};

let scalars = vec![
  ScalarValue::Boolean(Some(true)),
  ScalarValue::Boolean(None),
  ScalarValue::Boolean(Some(false)),
];

// Build an Array from the list of ScalarValues
let array = ScalarValue::iter_to_array(scalars.into_iter())
  .unwrap();

let expected: ArrayRef = std::sync::Arc::new(
  BooleanArray::from(vec![
    Some(true),
    None,
    Some(false)
  ]
));

assert_eq!(&array, &expected);

pub fn new_list( values: &[ScalarValue], data_type: &DataType, ) -> Arc<GenericListArray<i32>>

Converts Vec<ScalarValue> where each element has type corresponding to data_type, to a single element [ListArray].

Example

use datafusion_common::ScalarValue;
use arrow::array::{ListArray, Int32Array};
use arrow::datatypes::{DataType, Int32Type};
use datafusion_common::cast::as_list_array;

let scalars = vec![
   ScalarValue::Int32(Some(1)),
   ScalarValue::Int32(None),
   ScalarValue::Int32(Some(2))
];

let result = ScalarValue::new_list(&scalars, &DataType::Int32);

let expected = ListArray::from_iter_primitive::<Int32Type, _, _>(
    vec![
       Some(vec![Some(1), None, Some(2)])
    ]);

assert_eq!(*result, expected);

pub fn new_list_from_iter( values: impl IntoIterator<Item = ScalarValue> + ExactSizeIterator, data_type: &DataType, ) -> Arc<GenericListArray<i32>>

Converts IntoIterator<Item = ScalarValue> where each element has type corresponding to data_type, to a [ListArray].

Example

use datafusion_common::ScalarValue;
use arrow::array::{ListArray, Int32Array};
use arrow::datatypes::{DataType, Int32Type};
use datafusion_common::cast::as_list_array;

let scalars = vec![
   ScalarValue::Int32(Some(1)),
   ScalarValue::Int32(None),
   ScalarValue::Int32(Some(2))
];

let result = ScalarValue::new_list_from_iter(scalars.into_iter(), &DataType::Int32);

let expected = ListArray::from_iter_primitive::<Int32Type, _, _>(
    vec![
       Some(vec![Some(1), None, Some(2)])
    ]);

assert_eq!(*result, expected);

pub fn new_large_list( values: &[ScalarValue], data_type: &DataType, ) -> Arc<GenericListArray<i64>>

Converts Vec<ScalarValue> where each element has type corresponding to data_type, to a [LargeListArray].

Example

use datafusion_common::ScalarValue;
use arrow::array::{LargeListArray, Int32Array};
use arrow::datatypes::{DataType, Int32Type};
use datafusion_common::cast::as_large_list_array;

let scalars = vec![
   ScalarValue::Int32(Some(1)),
   ScalarValue::Int32(None),
   ScalarValue::Int32(Some(2))
];

let result = ScalarValue::new_large_list(&scalars, &DataType::Int32);

let expected = LargeListArray::from_iter_primitive::<Int32Type, _, _>(
    vec![
       Some(vec![Some(1), None, Some(2)])
    ]);

assert_eq!(*result, expected);

pub fn to_array_of_size( &self, size: usize, ) -> Result<Arc<dyn Array>, DataFusionError>

Converts a scalar value into an array of size rows.

§Errors

Errors if self is

a decimal that fails be converted to a decimal array of size
a Fixedsizelist that fails to be concatenated into an array of size
a List that fails to be concatenated into an array of size
a Dictionary that fails be converted to a dictionary array of size

pub fn convert_array_to_scalar_vec( array: &dyn Array, ) -> Result<Vec<Vec<ScalarValue>>, DataFusionError>

Retrieve ScalarValue for each row in array

Example 1: Array (ScalarValue::Int32)

use datafusion_common::ScalarValue;
use arrow::array::ListArray;
use arrow::datatypes::{DataType, Int32Type};

// Equivalent to [[1,2,3], [4,5]]
let list_arr = ListArray::from_iter_primitive::<Int32Type, _, _>(vec![
   Some(vec![Some(1), Some(2), Some(3)]),
   Some(vec![Some(4), Some(5)])
]);

// Convert the array into Scalar Values for each row
let scalar_vec = ScalarValue::convert_array_to_scalar_vec(&list_arr).unwrap();

let expected = vec![
vec![
    ScalarValue::Int32(Some(1)),
    ScalarValue::Int32(Some(2)),
    ScalarValue::Int32(Some(3)),
],
vec![
   ScalarValue::Int32(Some(4)),
   ScalarValue::Int32(Some(5)),
],
];

assert_eq!(scalar_vec, expected);

Example 2: Nested array (ScalarValue::List)

use datafusion_common::ScalarValue;
use arrow::array::ListArray;
use arrow::datatypes::{DataType, Int32Type};
use datafusion_common::utils::array_into_list_array;
use std::sync::Arc;

let list_arr = ListArray::from_iter_primitive::<Int32Type, _, _>(vec![
   Some(vec![Some(1), Some(2), Some(3)]),
   Some(vec![Some(4), Some(5)])
]);

// Wrap into another layer of list, we got nested array as [ [[1,2,3], [4,5]] ]
let list_arr = array_into_list_array(Arc::new(list_arr));

// Convert the array into Scalar Values for each row, we got 1D arrays in this example
let scalar_vec = ScalarValue::convert_array_to_scalar_vec(&list_arr).unwrap();

let l1 = ListArray::from_iter_primitive::<Int32Type, _, _>(vec![
    Some(vec![Some(1), Some(2), Some(3)]),
]);
let l2 = ListArray::from_iter_primitive::<Int32Type, _, _>(vec![
    Some(vec![Some(4), Some(5)]),
]);

let expected = vec![
  vec![
    ScalarValue::List(Arc::new(l1)),
    ScalarValue::List(Arc::new(l2)),
  ],
];

assert_eq!(scalar_vec, expected);

pub fn raw_data(&self) -> Result<Arc<dyn Array>, DataFusionError>

Get raw data (inner array) inside ScalarValue

pub fn try_from_array( array: &dyn Array, index: usize, ) -> Result<ScalarValue, DataFusionError>

Converts a value in array at index into a ScalarValue

pub fn try_from_string( value: String, target_type: &DataType, ) -> Result<ScalarValue, DataFusionError>

Try to parse value into a ScalarValue of type target_type

pub fn cast_to( &self, data_type: &DataType, ) -> Result<ScalarValue, DataFusionError>

Try to cast this value to a ScalarValue of type data_type

pub fn eq_array( &self, array: &Arc<dyn Array>, index: usize, ) -> Result<bool, DataFusionError>

Compares a single row of array @ index for equality with self, in an optimized fashion.

This method implements an optimized version of:

    let arr_scalar = Self::try_from_array(array, index).unwrap();
    arr_scalar.eq(self)

Performance note: the arrow compute kernels should be preferred over this function if at all possible as they can be vectorized and are generally much faster.

This function has a few narrow usescases such as hash table key comparisons where comparing a single row at a time is necessary.

§Errors

Errors if

it fails to downcast array to the data type of self
self is a Struct

§Panics

Panics if self is a dictionary with invalid key type

pub fn size(&self) -> usize

Estimate size if bytes including Self. For values with internal containers such as String includes the allocated size (capacity) rather than the current length (len)

pub fn size_of_vec(vec: &Vec<ScalarValue>) -> usize

Estimates size of Vec in bytes.

Includes the size of the Vec container itself.

pub fn size_of_vec_deque(vec_deque: &VecDeque<ScalarValue>) -> usize

Estimates size of VecDeque in bytes.

Includes the size of the VecDeque container itself.

pub fn size_of_hashset<S>(set: &HashSet<ScalarValue, S>) -> usize

Estimates size of HashSet in bytes.

Includes the size of the HashSet container itself.

Trait Implementations§

§

impl Clone for ScalarValue

§

fn clone(&self) -> ScalarValue

Returns a copy of the value. Read more

1.0.0 · source§

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source. Read more

§

impl Debug for ScalarValue

§

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter. Read more

§

impl Display for ScalarValue

§

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter. Read more

§