Databricks Spark 3.0 ceritification

Prep for the databricks spark 3 certification using Learning Spark v2 book resource
Sample datasets have been taken from the databricks community edition storage.

The repo covers the practical aspects of dataframe api including the following:

  • Subsetting DataFrames (select, filter, etc.)
  • Column manipulation (casting, creating columns, manipulating existing columns, complex column types)
  • String manipulation (Splitting strings, regex)
  • Reading/writing DataFrames (schemas, formats- parquet, avro, json etc)
  • Rows, Columns and Expressions
  • Common Dataframe operations (filter, select, where, distinct, sort, limit)
  • Wide and Narrow Transformations
  • Working with dates (extraction, formatting, etc)
  • Aggregations (groupBy, orderBy, count)
  • Statistical methods (avg, sum, max, min, describe, correlation, sampleBy)
  • UDFs
  • Combining datasets (joins, unions, broadcasting)
  • Optimising and tuning (caching and persistence, repartitioning, shuffle, catalyst optimiser – logical, optimised plans)

In addition, there is also an example of mlflow pipeline although not part of the certification


View Github