pyspark.RDDBarrier.mapPartitionsWithIndex#

RDDBarrier.mapPartitionsWithIndex(f, preservesPartitioning=False)[source]#

Returns a new RDD by applying a function to each partition of the wrapped RDD, while tracking the index of the original partition. And all tasks are launched together in a barrier stage. The interface is the same as RDD.mapPartitionsWithIndex(). Please see the API doc there.

New in version 3.0.0.

Parameters

ffunction: a function to run on each partition of the RDD
preservesPartitioningbool, optional, default False: indicates whether the input function preserves the partitioner, which should be False unless this is a pair RDD and the input function doesn’t modify the keys

Returns

RDD: a new RDD by applying a function to each partition

See also

RDD.mapPartitionsWithIndex()

Notes

This API is experimental

Examples

>>> rdd = sc.parallelize([1, 2, 3, 4], 4)
>>> def f(splitIndex, iterator): yield splitIndex
...
>>> barrier = rdd.barrier()
>>> barrier
<pyspark.core.rdd.RDDBarrier ...>
>>> barrier.mapPartitionsWithIndex(f).sum()
6