Data Engineer focused on distributed pipelines, IoT/sensor ingestion, and modern lakehouse architectures.
Currently working at the AI Model Development Lab (Sensors & Data) @ CNT.
I design and build resilient, production-ready data pipelines and scalable ingestion architectures. Currently, at the AI Model Development Laboratory (Sensors & Data Sector) at CNT (Confederação Nacional do Transporte), I focus on handling sensor telemetries, structuring raw physical-world metrics, and preparing enterprise datasets for AI/ML workloads.
Core Areas of Expertise:
- Sensor Data & AI Readiness: Designing ingestion layers for IoT and sensor streams, transforming noisy edge metrics into structured datasets for analytical consumption and predictive modeling.
- Distributed Computing & Lakehouses: Implementing Medallion Architectures (Bronze/Silver/Gold) with ACID compliance via Delta Lake, utilizing PySpark for high-throughput batch and stream transformations.
- Data Governance & Privacy-by-Design: Embedding dynamic data masking and cryptographic salted hashing (SHA-256) into ingestion pipelines to guarantee compliance with privacy regulations (LGPD/GDPR).
- Enterprise Data Warehousing: Dimensional modeling (Star Schema, SCD Type 2), SQL-driven data reconciliation, and automated orchestration (Prefect).
| Domain | Technologies |
|---|---|
| Languages & Core | Python (3.10+) SQL (ANSI/PostgreSQL) Java Bash/Linux |
| Distributed & Big Data | Apache Spark (PySpark) Delta Lake Databricks (Lakeflow) |
| Data Warehousing & DBs | PostgreSQL SQLite Dimensional Modeling (Star Schema) SAP Data Services |
| Orchestration & DataOps | Prefect Docker Git / GitHub Actions PyTest |
| Cloud Infrastructure | Oracle Cloud Infrastructure (OCI Certified) AWS Fundamentals |
|
PySpark • Delta Lake • Prefect • Data Governance • LGPD End-to-end modern Lakehouse implementing the Medallion pattern. Enforces privacy-by-design through automated dynamic data masking and salted SHA-256 cryptographic hashing to isolate Personally Identifiable Information (PII) before training Machine Learning models. |
PySpark • Parquet • Star Schema • PostgreSQL • Fault Tolerance Scalable pipeline processing 20+ years (2000–2020) of Brazilian flight records. Implements distributed cleaning in PySpark, partitioned columnar storage (Parquet), and a Graceful Degradation fallback routing mechanism to ensure high availability. |
|
PDI (Pentaho) • PostgreSQL • REST APIs • Dimensional Modeling Enterprise data integration pipeline merging legacy CSV transactional streams with live external REST APIs. Features Slowly Changing Dimensions (SCD Type 2), deterministic hashing for keyless datasets, and watermark-driven incremental ingestion. |
React.js • Context API • UI/UX • Systems Integration Single-Page Application (SPA) serving as an interactive engineering hub. Demonstrates full-stack capability, client-side routing, and responsive application architecture to bridge core infrastructure with user interfaces. |