Μηχανικός Apache Kafka

Διάρκεια: 1 μήνας 25 ημέρες

Κόστος: 450 ευρώ

Φόρτος εργασίας: 80 ώρες

ECVET: 4

A program for software engineers who develop applications that use Kafka’s APIs and production engineers (also called SREs, devops, or sysadmins) who install, configure, tune, and monitor Kafka in production. Furthermore this training can be useful for data architects and data engineers in mind, those responsible for designing and building an organization’s entire data infrastructure.

Εκδήλωσε Ενδιαφέρον

Αντικείμενο και Σκοπός

This training provides the needed information in order to use Apache Kafka in production and build robust and performant applications on top of it. We highlighted the popular use cases: message bus for event-driven microservices, stream-processing applications, and large-scale data pipelines. We also focused on making the training general and comprehensive enough so it will be useful to anyone using Kafka, no matter the use case or architecture. Topics such as how to install and configure Kafka and how to use the Kafka APIs are covered. Finally, Kafka’s design principles are presented and discussed.

The main audience is software engineers who develop applications that use Kafka’s APIs and production engineers (also called SREs, devops, or sysadmins) who install, configure, tune, and monitor Kafka in production. Furthermore this training can be useful for data architects and data engineers in mind, those responsible for designing and building an organization’s entire data infrastructure.

Μαθησιακοί Στόχοι

Με την ολοκλήρωση του προγράμματος, ο/η επιμορφούμενος/η θα είναι σε θέση να:

Χρήση Apache Kafka στην παραγωγή και στην υποστήριξη αποδοτικών εφαρμογών.
Εγκαθιστά και παραμετροποιεί Apache Kafka.
Αναπτύσσει επαγγελματικό λογισμικό.
Σχεδιάζει και υλοποιεί ανεξάρτητα της περίπτωσης ή της αρχιτεκτονικής.

Μεθοδολογία Υλοποίησης

Διαδικασία Υλοποίησης της Επιμόρφωσης
Online εξ αποστάσεως

Μέθοδοι Υλοποίησης της Επιμόρφωσης
Το εκπαιδευτικό υλικό θα αποτελείται από βιντεοσκοπημένες ενότητες και παραδείγματα προγραμματισμού. Θα χρησιμοποιηθεί η μέθοδος της ασύγχρονης τηλεκπαίδευσης. Το υλικό θα διατίθεται σταδιακά. Κάθε εβδομάδα και μία ενότητα.

Μέθοδοι Αξιολόγησης & Κριτήρια Απόδοσης Πιστοποιητικού Επιμόρφωσης
Η μέθοδος αξιολόγησης θα αποτελείται από εργασίες με την μορφή προγραμματισμού σε εβδομαδιαία βάση.

Από το αποτέλεσμα των εργασιών θα προκύψει και ο τελικός βαθμός των εκπαιδευομένων.

Το πρόγραμμα δεν θα τηρεί απουσιολόγιο.

Κόστος προγράμματος: €300

Εκπτωτικές κατηγορίες:

- 10% έκπτωση σε άνεργους
- 10% έκπτωση σε φοιτητές
- 10% έκπτωση σε ένστολους
- 10% έκπτωση σε ιδιωτικούς υπαλλήλους
- 10% έκπτωση σε υπαλλήλους Δημοσίου και ΟΤΑ
- 10% έκπτωση σε συμμετέχοντες άνω των 40 ετών
- 10% έκπτωση σε προκαταβολή του συνολικού ποσού

1. Introduction to Kafka & Installing Kafka

Σύνολο φόρτου εργασίας: 10 ώρες

In this session, we provide the important information for trainees to understand the concept of publish/subscribe messaging and why it is important. Publish/subscribe messaging is a pattern that is characterized by the sender (publisher) of a piece of data (message) not specifically directing it to a receiver. Instead, the sender (publisher) classifies the message somehow, and the receiver (subscriber) subscribes to receive certain classes of messages. Furthermore, this session describes how to get started with the Apache Kafka broker, including how to set up Apache Zookeeper, which is used by Kafka for storing metadata for the brokers. It will also cover the basic configuration options for a Kafka deployment, as well as criteria for selecting the correct hardware to run the brokers on. Finally, it will cover how to install multiple Kafka brokers as part of a single cluster and some specific concerns when using Kafka in a production environment.

Duration
● Video Sessions: 60 min
● Programming Assignment: max 180 min
Video Sessions
1. Publish/Subscribe Messaging
2. Enter Kafka
3. Why Kafka
4. Data Ecosystem
5. Prerequisites
6. Installing a Kafka Broker
7. Broker Configuration
8. Hardware Selection
9. Kafka Clusters
10. Production Concerns
Programming Assignments
No programming assignment.

Εβδομαδιαία συνεδρία: Introduction to Kafka & Installing Kafka

2. Kafka Producers & Consumers

Σύνολο φόρτου εργασίας: 10 ώρες

This session provides information on how to use the Kafka producers and consumers, starting with an overview of its design and components. It will show how to create KafkaProducer and ProducerRecord objects, how to send records to Kafka, and how to handle the errors that Kafka may return. We’ll then review the most important configuration options used to control the producer behavior. Finally, a deeper look at how to use different partitioning methods and serializers, and how to write your own serializers and partitioners. Furthermore, reading data from Kafka is a bit different than reading data from other messaging systems, and there are few unique concepts and ideas involved. We’ll start by explaining some of the important concepts, and then we’ll go through some examples that show the different ways consumer APIs can be used to implement applications with varying requirements.

Duration
● Video Sessions: 90 min
● Programming Assignment: max 180 min
Video Sessions
1. Overview
2. Constructing a Message to Kafka
3. Sending a Message with Kafka
4. Configuring producers
5. Serializers – Duration
6. Partitions – Duration
7. Creating Kafka Consumer
8. Commits and Offsets
9. Deserializers – Duration
10. Consuming Records with Specific Offsets
11. Rebalance Listeners – Duration
Programming Assignments
One programming assignment solving problems covering the presented topics.

Εβδομαδιαία συνεδρία: Kafka Producers & Consumers

3. Kafka Internals

Σύνολο φόρτου εργασίας: 10 ώρες

In this session, we focus on three topics that are especially relevant to Kafka practitioners: (1) How Kafka replication works; (2) How Kafka handles requests from producers and consumers; (3) How Kafka handles storage such as file format and indexes.Understanding these topics in-depth will be especially useful when tuning Kafka and understanding the mechanisms that the tuning knobs control goes a long way toward using them with precise intent rather than fiddling with them randomly.

Duration
● Video Sessions: 60 min
● Programming Assignment: max 180 min
Video Sessions
1. Cluster Membership
2. Replication
3. Request Processing
4. Physical Storage
Programming Assignments
One programming assignment solving problems covering the presented topics.

Εβδομαδιαία συνεδρία: Kafka Internals

4. Reliable Data Delivery

Σύνολο φόρτου εργασίας: 10 ώρες

In this session, we will start by talking about different kinds of reliability and what they mean in the context of Apache Kafka. Then we will talk about Kafka’s replication mechanism and how it contributes to the reliability of the system. Furthermore, we will discuss Kafka’s brokers and topics and how they should be configured for different use cases. Then we will discuss the clients, producer, and consumer, and how they should be used in different reliability scenarios. Last, we will discuss the topic of validating the system’s reliability, because it is not enough to believe a system is reliable.

Duration
● Video Sessions: 60 min
● Programming Assignment: max 180 min
Video Sessions
1. Reliability Guarantees
2. Broker Configuration
3. Using Producers in a Reliable System
4. Using Consumers in a Reliable System
5. Validating System Reliability
Programming Assignments
One programming assignment solving problems covering the presented topics.

Εβδομαδιαία συνεδρία: Reliable Data Delivery

5. Building Data Pipelines

Σύνολο φόρτου εργασίας: 10 ώρες

In this session, we’ll discuss some of the common issues that you need to take into account when building data pipelines. Those challenges are not specific to Kafka, but rather general data integration problems. Nonetheless, we will show why Kafka is a good fit for data integration use cases and how it addresses many of those challenges. We will discuss how the Kafka Connect APIs are different from the normal producer and consumer clients, and when each client type should be used. Then we’ll jump into some details of Kafka Connect. While a full discussion of Kafka Connect is out‐side the scope of this session, we will show examples of basic usage to get you started and give you pointers on where to learn more. Finally, we’ll discuss other data integration systems and how they integrate with Kafka.

Duration
● Video Sessions: 60 min
● Programming Assignment: max 180 min
Video Sessions
1. Building Data Pipelines
2. Kafka Connect
3. Alternatives approaches
Programming Assignments
One programming assignment solving problems covering the presented topics.

Εβδομαδιαία συνεδρία: Building Data Pipelines

6. Cross Clustering Mirroring

Σύνολο φόρτου εργασίας: 10 ώρες

In this session, we will present cross-cluster mirroring concepts of all or part of the data, including some use cases and architectures that are used to implement these use cases and provide the pros and cons of each architecture pattern. Furthermore, we will discuss MirrorMaker, and how to use it, including operational tips, including deployment and performance tuning. Finally, we will also present some other alternatives.

Duration
● Video Sessions: 60 min
● Programming Assignment: max 180 min
Video Sessions
1. Multi Cluster Architectures
2. Apache Kafka’s Mirror Maker
3. Alternative Solutions
Programming Assignments
One programming assignment solving problems covering the presented topics.

Εβδομαδιαία συνεδρία: Cross Clustering Mirroring

7. Administering & Monitoring Kafka

Σύνολο φόρτου εργασίας: 10 ώρες

In this session, we will present how to use several command-line interface (CLI) utilities that are useful for making administrative changes to the Kafka clusters. The tools are implemented in Java classes, and a set of scripts are provided to call those classes properly. These tools provide basic functions, but you may find they are lacking for more complex operations. We will also present the most critical metrics to monitor Apache Kafka all the time, and how to respond to them. We’ll also describe some of the more important metrics to have on hand when debugging problems. This is not an exhaustive list of the metrics that are available, however, because the list changes frequently, and many will only be informative to a hardcore Kafka developer.

Duration
● Video Sessions: 90 min
● Programming Assignment: max 180 min
Video Sessions
1. Topics Operations
2. Consumer Groups
3. Dynamic Configuration Parameters
4. Partition Management
5. Consuming and Producing
6. Client ACLs
7. Metrics Basics
8. Kafka Broker Metrics
9. Client Monitoring
10. End-to-End Monitoring
11. Delay Monitoring
Programming Assignments
One programming assignment solving problems covering the presented topics.

Εβδομαδιαία συνεδρία: Administering & Monitoring Kafka

8. Stream Processing

Σύνολο φόρτου εργασίας: 10 ώρες

This session is intended to be a brief introduction to stream processing and will cover basic Kafka Streams features. We will explain what we mean by stream processing (since this term is frequently misunderstood), then discuss some of the basic concepts of stream processing and the design patterns that are common to all stream-processing systems and provide information on how to use Kafka steam library, including some examples and use cases.

Duration
● Video Sessions: 60 min
● Programming Assignment: max 180 min
Video Sessions
1. Overview
2. Stream Processing Concepts
3. Stream Processing Design Patterns
4. Kafka Streams by Example
5. Architecture overview of Kafka Streams
6. Stream Processing Use Cases
Programming Assignments
One programming assignment solving problems covering the presented topics.