Introduction In Part 2 we walked through an example modern data architecture (shown below). In this third part we will discuss the people required to build it and discuss the types of skills required. This discussion will focus on three key areas of the platform: Ingestion of data into the lake Transformation of the data so it is ready for consumers The infrastructure to support these two processes This means that with the exception of analysts, roles that are focused on data sources (e.g. DBAs) or data consumers (e.g. Data Scientists) will not be discussed in this part. Required Skills The skills required for the example architecture can be broken down roughly into the following: Infrastructure - Terraform, IAM, Networking & Security, ECS, S3, Glue Data Ingestion - Python, S3, Glue, Lambda, ECS Data Transformation - Redshift, SQL, DBT, Python, ECS Some areas will overlap and the level of skill required will vary on the team. For example, the Python knowledge to run DBT is signifi...
Occasional thoughts about data