pyspark.sql.DataFrameWriter.options#

DataFrameWriter.options(**options)[source]#

Adds output options for the underlying data source.

New in version 1.4.0.

Changed in version 3.4.0: Supports Spark Connect.

Parameters

**optionsdict: The dictionary of string keys and primitive-type values.

Examples

>>> spark.range(1).write.options(key="value")
<...readwriter.DataFrameWriter object ...>

Specify options in a dictionary.

>>> spark.range(1).write.options(**{"k1": "v1", "k2": "v2"})
<...readwriter.DataFrameWriter object ...>

Specify the option ‘nullValue’ and ‘header’ with writing a CSV file.

>>> from pyspark.sql.types import StructType,StructField, StringType, IntegerType
>>> schema = StructType([
...     StructField("age",IntegerType(),True),
...     StructField("name",StringType(),True),
... ])
>>> import tempfile
>>> with tempfile.TemporaryDirectory(prefix="options") as d:
...     # Write a DataFrame into a CSV file with 'nullValue' option set to 'Hyukjin Kwon',
...     # and 'header' option set to `True`.
...     df = spark.createDataFrame([(100, None)], schema=schema)
...     df.write.options(nullValue="Hyukjin Kwon", header=True).mode(
...         "overwrite").format("csv").save(d)
...
...     # Read the CSV file as a DataFrame.
...     spark.read.option("header", True).format('csv').load(d).show()
+---+------------+
|age|        name|
+---+------------+
|100|Hyukjin Kwon|
+---+------------+