RayDP¶
Description:
- Run Apache Spark on a Ray cluster, so Spark jobs and Ray tasks can share the same cluster and data.
Resources:
- [GitHub] https://github.com/oap-project/raydp
- [Ray docs] https://docs.ray.io/en/latest/ray-more-libs/raydp.html
Requirements:
pip install raydp(it installsrayand a PySpark version that RayDP supports).- Java, because Spark runs on the JVM: install a JDK and set
JAVA_HOMEbefore starting Jupyter. ray.init()starts a local Ray instance. To use an existing Ray cluster instead, setRAY_ADDRESS(for exampleRAY_ADDRESS=autoon a cluster node) before starting Jupyter.
Example 1¶
Start Spark on Ray, count words in a small Spark dataframe, then stop Spark and Ray
In [ ]:
Copied!
import ray
import raydp
# Start a local Ray instance, or connect to the cluster named by the RAY_ADDRESS environment variable
ray.init()
# create a Spark cluster with specified resource requirements
# (small enough for a laptop; raise these on a real cluster)
spark = raydp.init_spark(app_name='RayDP Example',
num_executors=1,
executor_cores=1,
executor_memory='1GB')
# normal data processesing with Spark
df = spark.createDataFrame([('look',), ('spark',), ('tutorial',), ('spark',), ('look', ), ('python', )], ['word'])
df.show()
word_count = df.groupBy('word').count()
word_count.show()
# stop the spark cluster
raydp.stop_spark()
ray.shutdown()
import ray
import raydp
# Start a local Ray instance, or connect to the cluster named by the RAY_ADDRESS environment variable
ray.init()
# create a Spark cluster with specified resource requirements
# (small enough for a laptop; raise these on a real cluster)
spark = raydp.init_spark(app_name='RayDP Example',
num_executors=1,
executor_cores=1,
executor_memory='1GB')
# normal data processesing with Spark
df = spark.createDataFrame([('look',), ('spark',), ('tutorial',), ('spark',), ('look', ), ('python', )], ['word'])
df.show()
word_count = df.groupBy('word').count()
word_count.show()
# stop the spark cluster
raydp.stop_spark()
ray.shutdown()