Packages

trait Encoder[T] extends Serializable

Used to convert a JVM object of type T to and from the internal Spark SQL representation.

Scala

Encoders are generally created automatically through implicits from a SparkSession, or can be explicitly created by calling static methods on Encoders.

import spark.implicits._

val ds = Seq(1, 2, 3).toDS() // implicitly provided (spark.implicits.newIntEncoder)

Java

Encoders are specified by calling static methods on Encoders.

List<String> data = Arrays.asList("abc", "abc", "xyz");
Dataset<String> ds = context.createDataset(data, Encoders.STRING());

Encoders can be composed into tuples:

Encoder<Tuple2<Integer, String>> encoder2 = Encoders.tuple(Encoders.INT(), Encoders.STRING());
List<Tuple2<Integer, String>> data2 = Arrays.asList(new scala.Tuple2(1, "a");
Dataset<Tuple2<Integer, String>> ds2 = context.createDataset(data2, encoder2);

Or constructed from Java Beans:

Encoders.bean(MyClass.class);

Implementation

  • Encoders should be thread-safe.
Annotations
@implicitNotFound()
Source
Encoder.scala
Since

1.6.0

Linear Supertypes
Serializable, AnyRef, Any

Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Encoder
  2. Serializable
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Abstract Value Members

  1.  abstract def clsTag: ClassTag[T]

    A ClassTag that can be used to construct an Array to contain a collection of T.

  2.  abstract def schema: StructType

    Returns the schema of encoding this type of object as a Row.

Concrete Value Members

  1.  final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2.  final def ##: Int
    Definition Classes
    AnyRef → Any
  3.  final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4.  final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5.  def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  6.  final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7.  def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  8.  final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  9.  def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  10.  final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  11.  final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  12.  final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  13.  final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  14.  final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  15.  def toString(): String
    Definition Classes
    AnyRef → Any
  16.  final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  17.  final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  18.  final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])

Deprecated Value Members

  1.  def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped