spark

How to set up a local Pyspark Environment with Jupyter on your Mac

Whether it’s for social science, marketing, business intelligence or something else, the number of times data analysis benefits from heavy duty parallelization is growing all the time. Apache Spark is an awesome platform for big data analysis, so getting to know how it works and how to use it is probably a good idea. Setting up your own cluster, administering it etc. etc. is a bit of a hassle to just learn the basics though (although Amazon EMR or Databricks make that quite easy, and you can even build your own Raspberry Pi cluster if you want…), so getting Spark and Pyspark running on your local machine seems like a better idea.