Data Engineering

Data are often considered the “new oil”, the raw material from which value is created. To harness the power of data, the data need to be stored and processed on a technical level. This course introduces the four “Vs” of data, as well as typical data sources and types.

The course discusses the most common data storage formats encountered in modern systems, focusing both on text-based as well as binary data formats.

Handling large amounts of data poses significant challenges for the underlying infrastructure. The course discusses the most important distributed and streaming data handling frameworks which are used in leading edge applications.

Kursinhalte

• Data Types and Data Sources
  • The 4Vs of data: volume, velocity, variety, veracity
  • Data sources
  • Data types
• Working with Common Data Formats
  • Text-Based Formats (CSV, XML, JSON)
  • Binary Formats (HDF5, Parquet, Arrow)
• NoSQL data stores
  • Introduction and motivation
  • Approaches and technical concepts
• Distributed Systems
  • Hadoop & MapReduce
  • Hadoop file system (HDFS)
  • Spark
  • DASK
• Streaming Frameworks
  • Spark streaming
  • Kafka

Fakten zum Modul: Niveau: Bachelor · Unterrichtssprache: English · 5 ECTS-Punkte · Modulprüfung: Examen, 90 Minuten

Many of the recent advances in data science, particularly machine learning and artificial intelligence, rely on comprehensive data storage and computing power. Cloud computing is one way of providing that power in a scalable way, without considerable upfront investment in hardware and software resources. This course introduces the area of cloud computing together with its enabling technologies. Moreover, the most cutting-edge advances like serverless computing and storage are illustrated. Finally, a thorough overview on popular cloud offerings, especially in regard to analytics capabilities, is given.

Kursinhalte

• Introduction to Cloud Computing
  • Fundamentals of Cloud computing
  • Cloud Service Models
  • Benefits and Risks
• Enabling Technology
  • Virtualization and Containerization
  • Storage Technology
  • Networks and RESTful Services
• Serverless Computing
  • Introduction to Serverless Computing
  • Benefits
  • Limitations
• Established Cloud Platforms
  • General Overview
  • Google Cloud Platform
  • Amazon Web Services
  • Microsoft Azure
  • Platform Comparison
• Data Science in the Cloud
  • Provider-independent services and tools
  • Google Data Science and Machine Learning Services
  • Amazon Web Services Data Science and Machine Learning Services
  • Microsoft Azure Data Science and Machine Learning Services

Fakten zum Modul: Niveau: Bachelor · Unterrichtssprache: English · 5 ECTS-Punkte · Modulprüfung: Examen, 90 Minuten

This course explores concepts of data engineering. Data engineering is concerned with the infrastructure aspects of data science such as data storage and provision, as well as the provisioning of suitable operational environments.

After laying out foundational notions and concepts of the discipline, this course addresses important developments in storage technology; aspects of systems architecture for processing data at scale; containerization as a modern take on virtualization; and the logic of data pipelines and associated operational aspects. Important issues pertaining to data security and protection are also given appropriate attention.

Kursinhalte

• Foundations of Data Engineering
  • Reliability
  • Scalability
  • Maintainability
• NoSQL In Depth
  • Fundamentals of NoSQL
  • Established NoSQL solutions
• Architectures for Data Processing at Scale
  • Batch processing architectures
  • Architectures for stream and complex event processing
  • Lambda architecture
• Containerization In Depth
  • Docker containers
  • Container management
• Governance & Security
  • Data protection
  • Data security
  • Data governance
• Operational Aspects
  • Defining principles of DataOps
  • Building and maintaining data pipelines
  • Metrics and monitoring

Fakten zum Modul: Niveau: Bachelor · Unterrichtssprache: English · 5 ECTS-Punkte · Modulprüfung: Examen, 90 Minuten

The focus of this course is the implementation of a real-world data engineering use case in the form of a student portfolio.

To this end, students choose a project subject from the various sub-domains of data engineering. Examples include setting up a Docker container environment or dockerized service; implementing a data pipeline according to DataOps principles; and setting up an NoSQL data store.

The goal is for students to demonstrate they can transfer theoretical knowledge to an implementation scenario that closely mimics practical work in a professional data engineering setting.

Kursinhalte

This course covers the practical implementation of approaches and techniques covered in the preceding methodological course in a project-oriented setting. Each participant must produce a portfolio detailing and documenting the work. Porfolio themes are chosen from a list, or suggested by the students in accord with the tutor.

Fakten zum Modul: Niveau: Bachelor · Unterrichtssprache: English · 5 ECTS-Punkte · Modulprüfung: Portfolio