By Sandeep Mehta, Engineering Lead, Data Platforms
In this digital age, data is the new oil - it's valuable, powerful, and everyone wants a piece of it. From healthcare to education to finance to marketing, every business is digging for data gold to fuel their growth and beat the competition.
Mining data is not easy - it requires a special set of skills and tools, and you need a sturdy data platform that can handle scalability, robustness and security. Here at Dojo, we also need a data platform to revolutionise the experience economy by empowering our customers with data products.
That's where our Data Platform Engineering team comes in. A combination of Data Engineering and Platform Engineering, it's a unique group of specialists with the power to build a robust, scalable, and fault-tolerant data infrastructure that can deal with any data challenge.
What is a Data Platform?
The official definition is, “An end to end data management system which provides solutions for ingesting, transforming, aggregating and analysing data at scale.”
A data platform is like a super-charged brain for your organisation - it's the ultimate multitasker that stores, processes, and manages all your data. It's a superhero sidekick that never steals the spotlight, but always saves the day.
This isn’t an ordinary sidekick - a data platform has skills like extensive software automation, building network and security infrastructure, and knowledge of data governance policies. It’s a master of user experience and developer experience. It takes care of all the dirty work without you even noticing, making even the most complex data processes feel like a walk in the park.
That’s why if you want to give your organisation the super-brain it deserves, a data platform is the way to go. It's the superhero sidekick of data, the wizard of automation, and the multitasker of UX and DX all rolled into one.
The role of a Data Platform Engineer at Dojo
Data Platform Engineers at Dojo work in a centralised team or group of teams that collaborate closely with domain software engineers, analytics engineers, insight or business analysts, data scientists, and other stakeholders.
Data Mesh is an architectural concept that decentralises data ownership, treating data as a product managed by cross-functional teams within a data platform, enabling distributed governance and overcoming the limitations of centralised data lakes or warehouses. The Data Platform team is one of the most important pillars when it comes to the Data mesh, as they aim to understand each team’s data requirements and the design systems that meet their needs. They are responsible for implementing automation and providing self-service features to enable all domains to efficiently process, manage and own their own data.
Data Platform Engineers also play a critical role in ensuring the security and privacy of sensitive data. They design and implement security protocols, such as access controls and encryption, and ensure compliance with data privacy regulations, such as GDPR and PCI.
Skills Required for Data Platform Engineering
Data Platform Engineering is often misunderstood as just a job for those who know a bit about coding and big data technologies. It's actually much more than that. Data Platform Engineers need to be proficient in a variety of skills, including data technologies, cloud-native features, programming languages, and data governance policies. They need to have strong problem-solving and communication skills, software engineering experience, and, more importantly, platform engineering expertise.
At Dojo there are 4 types of skill sets required to be a successful Data Platform Engineer.
1. Data Skills:
- Proficiency in database systems like SQL and NoSQL.
- Experience with big data technologies such as PubSub, Kafka, Flink, Airflow etc.
- Knowledge of data warehousing concepts and tools such as Snowflake, Redshift, and BigQuery.
- Familiarity with data governance policies and compliance requirements.
2. Software Engineering Skills:
- Proficiency in programming languages such as Python, Golang, Java, Scala etc.
- Experience with microservices architecture and serverless computing.
- Understanding of software development methodologies such as Agile and DevOps.
- Familiarity with containerization and container orchestration technologies such as Docker and Kubernetes.
3. Platform Engineering Skills:
- Knowledge of cloud platforms such as AWS, Azure, and Google Cloud Platform, including cloud storage, serverless and managed data services.
- Familiarity with platform engineering tools such as Kubernetes, Grafana, Prometheus, ArgoCD
- Knowledge and experience in writing modular Infrastructure as code in Terraform or Pulumi.
4. Product Skills:
- Product management: Collaborate with stakeholders to define product vision, strategy, and roadmaps.
- Be the developer advocate of the products and tooling developed to enhance the user adoption within the company.
- User experience (UX) design: Design intuitive and easy-to-use data platforms that meet the needs of end-users.
- Agile development methodologies: Manage projects, collaborate with cross-functional teams, and deliver products iteratively.
Keeping pace with the constantly evolving data and platform technology landscape poses the greatest challenge in this job. Crucially, staying current with emerging technologies and making informed architectural decisions in line with requirements is a key skill to master.
Data Platform Engineering is a critical role that enables organisations to efficiently process, manage, and analyse data to make informed decisions and achieve business objectives. Data Platform Engineers possess a diverse skill set that includes data technologies, software engineering, and platform engineering expertise. They work collaboratively with stakeholders to understand their data requirements and design data platforms that meet their needs. By implementing automation, self-service features, and data governance policies, they ensure that organisations can leverage their data assets to gain a competitive advantage in the data-driven economy. With a skilled team of Data Platform Engineers and a robust data platform, organisations can unlock the full potential of their data and ride the wave of AI-driven innovation to success.