# Kafka Introduction

## Introduction

* Pub/Sub
* Used for near realtime processing - events
* Kafka began at LinkedIn - Jay Krep - I Love Logs - also Amazon Kinesis
* Solves 3 problems - unreliable receivers, scaling, messy communications

![Dataflow for realtime event analytics](https://153111891-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LTtjxZz0mQ2dLXJi93S%2F-LTxUBhttCbuzGxdEhiJ%2F-LTxXEz8OtQzgeQOyrVU%2FScreen%20Shot%202018-12-17%20at%2011.58.15%20AM.png?alt=media\&token=8a8eace6-7a84-4452-ab94-c00fe8663ddb)

### Why event driven

* compare to batch
* fraud detection
* IOT
* show architecture of stuff
* infinite sequence ....

### What is Pub/Sub

* Producer/Consumer with a buffer
* Events are numbered and processed in order

![](https://153111891-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LTtjxZz0mQ2dLXJi93S%2F-LTxUBhttCbuzGxdEhiJ%2F-LTxYQSC3o6L2RlzRd00%2FScreen%20Shot%202018-12-17%20at%2012.02.40%20PM.png?alt=media\&token=4eab47e6-c25e-4925-8399-ebf4a7a1c2e2)

![](https://153111891-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LTtjxZz0mQ2dLXJi93S%2F-LTxbCSbMx5Z3A_EHzmW%2F-LTxbiqbuUKa6bkMS0ll%2FScreen%20Shot%202018-12-17%20at%2012.22.10%20PM.png?alt=media\&token=49702602-dce8-498c-b1de-eae6dedacff6)

### Architecture

![Kafka - put everything on the stream](https://153111891-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LTtjxZz0mQ2dLXJi93S%2F-LTxYWMqEDUpC73_ibSm%2F-LTxZagKnSmsKiFT3Ae2%2FScreen%20Shot%202018-12-17%20at%2012.08.43%20PM.png?alt=media\&token=461e4450-de62-480b-86bd-4bd853aad16a)

### Solves the Messy Communications Problem

![https://www.slideshare.net/KaiWaehner/apache-kafka-as-eventdriven-open-source-streaming-platform-prague-meetup?qid=64dec3ab-f2a2-4ee5-86df-23feef24840d\&v=\&b=\&from\_search=2](https://153111891-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LTtjxZz0mQ2dLXJi93S%2F-LTxYWMqEDUpC73_ibSm%2F-LTx_95hsHc9BthhzhFf%2FScreen%20Shot%202018-12-17%20at%2012.11.04%20PM.png?alt=media\&token=d24f4b48-b367-432c-a768-76d508a28d36)

### Scaling

![Open up another buffer or broker](https://153111891-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LTtjxZz0mQ2dLXJi93S%2F-LTxbCSbMx5Z3A_EHzmW%2F-LTxc7KiuqPwihnIrUIK%2FScreen%20Shot%202018-12-17%20at%2012.24.01%20PM.png?alt=media\&token=9b65b030-9533-4140-835e-af7439b8dfb3)

![Start more consumers](https://153111891-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LTtjxZz0mQ2dLXJi93S%2F-LTxbCSbMx5Z3A_EHzmW%2F-LTxcT0g7ObQ4ebh1KtW%2FScreen%20Shot%202018-12-17%20at%2012.25.31%20PM.png?alt=media\&token=68c3e7fa-ec1e-4a92-9201-ca884e8de8d6)

### Replication for reliability

![https://www.saowen.com/a/e25edaf7cf39676b6cb617bbb1e988a0ecf9fe3cc0275e85a2ff5d76bc59f14b](https://153111891-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LTtjxZz0mQ2dLXJi93S%2F-LTxckpMxP8P_Stg5tEc%2F-LTxe7V8kXJPgfsUYF3Z%2FScreen%20Shot%202018-12-17%20at%2012.32.59%20PM.png?alt=media\&token=3b5939e1-788c-42e6-a431-c229b63959f9)

### Brokers, Partitions, Topics

* Broker - URL
* Partition
* Topic&#x20;
* <http://codeflex.co/what-is-apache-kafka/>
