> For the complete documentation index, see [llms.txt](https://unstructured-playgroud.gitbook.io/unstructuredplayground/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://unstructured-playgroud.gitbook.io/unstructuredplayground/release-1.0/apache-spark-ecosystem.md).

# Apache Spark Ecosystem

Introduction to Apache Spark

**Apache Spark™** is a unified analytics engine for large-scale data processing developed at UC Berkeley in 2009.  It has received rapid acceptance from a wide range of industries, especially those that process at massive scale.  Apache Spark can process multiple petabytes of data residing on over 8,000 nodes.  It is  an open source project supported by over 1000 contributors from over 250 organizations.

![Image from DataBricks](https://3266175528-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LTtjxZz0mQ2dLXJi93S%2F-LUre14keHsjQNvJewJM%2F-LUrgbY3nzIl3Plue9z9%2FScreen%20Shot%202018-12-28%20at%207.00.04%20PM.png?alt=media\&token=4d1973ad-9c0f-4e2c-89e7-9dd604de632a)

## Slides

[slides - pdf](https://github.com/marilynwaldman/course/blob/master/spark/00-Introduction/00-Introduction.pdf)

[slides - pptx](https://github.com/marilynwaldman/course/blob/master/spark/00-Introduction/00-Introduction.pptx)

### Apache Spark is known for:

* **Speed**

  Run workloads 100x faster.  Apache Spark achieves high performance for both batch and streaming data, using a state-of-the-art DAG scheduler, a query optimizer, and a physical execution engine.
* **Ease of Use**

  Write applications quickly in Java, Scala, Python, R, and SQL.
* **Generality**

  Combines SQL, streaming, and complex analytics.  Spark powers a stack of libraries including [SQL and DataFrames](https://spark.apache.org/sql/), [MLlib](https://spark.apache.org/mllib/) for machine learning, [GraphX](https://spark.apache.org/graphx/), and [Spark Streaming](https://spark.apache.org/streaming/). You can combine these libraries seamlessly in the same application.
* **Runs Everywhere**

  Spark runs on Hadoop, Apache Mesos, Kubernetes, standalone, or in the cloud. It can access diverse data sources.

  You can run Spark using its [standalone cluster mode](https://spark.apache.org/docs/latest/spark-standalone.html), on [EC2](https://github.com/amplab/spark-ec2), on [Hadoop YARN](https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html), on [Mesos](https://mesos.apache.org/), or on [Kubernetes](https://kubernetes.io/). Access data in [HDFS](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html), [Alluxio](https://alluxio.org/), [Apache Cassandra](https://cassandra.apache.org/), [Apache HBase](https://hbase.apache.org/), [Apache Hive](https://hive.apache.org/), and hundreds of other data sources.

[Credit : Apache Spark](https://spark.apache.org/)

![](https://3266175528-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LTtjxZz0mQ2dLXJi93S%2F-LUrhVBEbltdWw0xSbJY%2F-LUrilqIg255effVQJWG%2FScreen%20Shot%202018-12-28%20at%207.11.15%20PM.png?alt=media\&token=6af3d204-01ea-44c3-ba87-3226b156da8f)

#### <br>

\
&#x20;