pyspark.sql.functions.last#

pyspark.sql.functions.last(col, ignorenulls=False)[source]#

Aggregate function: returns the last value in a group.

The function by default returns the last values it sees. It will return the last non-null value it sees when ignoreNulls is set to true. If all values are null, then null is returned.

New in version 1.3.0.

Changed in version 3.4.0: Supports Spark Connect.

Parameters

colColumn or column name: column to fetch last value for.
ignorenullsbool: if last value is null then look for non-null value. False` by default.

Returns

Column: last value of the group.

Notes

The function is non-deterministic because its results depends on the order of the rows which may be non-deterministic after a shuffle.

Examples

>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame([("Alice", 2), ("Bob", 5), ("Alice", None)], ("name", "age"))
>>> df = df.orderBy(df.age.desc())
>>> df.groupby("name").agg(sf.last("age")).orderBy("name").show()
+-----+---------+
| name|last(age)|
+-----+---------+
|Alice|     NULL|
|  Bob|        5|
+-----+---------+

To ignore any null values, set ignorenulls to True

>>> df.groupby("name").agg(sf.last("age", ignorenulls=True)).orderBy("name").show()
+-----+---------+
| name|last(age)|
+-----+---------+
|Alice|        2|
|  Bob|        5|
+-----+---------+