array_contains #

pyspark.sql.functions.array_contains(col, value) #

version: since 1.5.0

Collection function: returns null if the array is null, true if the array contains the given value, and false otherwise.

value: value or column to check for in an array

array_contains

Runnable Code:

from pyspark.sql import functions as F
# Set up dataframe
data = [{"a": [],"b": 1},{"a": [1,2,2],"b": 1},
        {"a": [4,5,5],"b": 1}]
df = spark.createDataFrame(data)
# Use function
df = (df
     .withColumn("array_contains_4",
       F.array_contains(F.col("a"),4))
     .withColumn("array_contains_b",
       F.array_contains(F.col("a"),
                       F.col("b")))
     )
df.show()

a	b	array_contains_4	array_contains_b
[]	1	false	false
[1, 2, 2]	1	false	true
[4, 5, 5]	1	true	false

Usage:

Notice that you can find either a value or a column’s value in an array. Pretty rad!

returns: Column(sc.\_jvm.functions.array_contains(\_to_java_column(col), value))

PySpark manual

tags: search array, search list, list, find in list, column in list