https://pysparkisrad.com/misc/ Recent content in Miscs on PySpark Is Rad Hugo -- gohugo.io en-us https://pysparkisrad.com/misc/dataframe/ Mon, 01 Jan 0001 00:00:00 +0000 https://pysparkisrad.com/misc/dataframe/ dataframe # creating a simple dataframe # Often times you want to create a simple dataframe to try out PySpark functions. Here is my favorite way to create a simple dataframe. data = [{"a": "hi"},{"a": "bye"},{"a": "fly"}] df = spark.createDataFrame(data) a hi bye fly For me, using a dictionary is the easiest way. You can use this technique to create arrays (lists). You can create nulls by not including a key. https://pysparkisrad.com/misc/uuid/ Mon, 01 Jan 0001 00:00:00 +0000 https://pysparkisrad.com/misc/uuid/ uuid # uuids in pyspark # I often need uuids in Databricks. I use both deterministic (repeatable) ids and random ids. Random UUIDs # For random ids I used to use a UDF. import uuid @udf def create_random_id(): return str(uuid.uuid4()) But as of Spark 3.0.0 there is a Spark SQL for random uuids. So now I use this: from pyspark.sql import functions as F df.withColumn("uuid", F.expr("uuid()")) This is nicer and should be faster since it uses native Spark SQL instead of a UDF which then runs python.