Adopt Atum Agent and Reader to work for Spark 4

[lsulak]

Background

Some parts of our ecosystem are already using, or soon gonna be using, Spark 4. Therefore it is important to keep our tech up to date. Currently Atum Agent+Reader were not tested against Spark 4 and the goal of this ticket is to implement necessary changes and tests to achieve this.

Feature

Atum Agent and Reader can work in Spark 4 environment.

Business Value

Some of our applications might not work if we do not update.

Proposed Solution

Not much of a solution really, but here I want to enumerate list of items what are known to do.

So, today the agent is cross-built only against Spark 3.5.5, Scala 2.12.18 + 2.13.13, targeting Java 8. What Spark 4 forces:

• Java 17 minimum (so we'll need to update this)
• Scala 2.13 only (so will need to drop Scala 2.12)
• and underlying Spark Commons also has to be supported (this has been done recently: https://github.com/AbsaOSS/spark-commons/releases/tag/v1.0.0)

Further to do or to check:

• Spark internals or anything from Spark library we use in Agent or Reader - does it still work and is compatible? E.g. we use internal.Logging on some places
• Build matrix / system - this needs to be updated, dependencies, CICD, build artifact path and so on. Ideally we should continue to support Spark 3.5.x with Scala 2.12 and 2.13 and also support Spark 4 with Scala 2.13.