Telecom Customer Journey & Churn Root Cause Analysis Platform

End-to-end data engineering platform,Detects churn signals, reconstructs customer journeys, and classifies root causes as behavioral, network-driven, or mixed — at production scale using — Spark · Iceberg · Airflow · DuckDB

The Problem

Every telecom knows customers churn. The expensive mistake is treating all churn the same way. A customer who left because of poor network coverage needs a different response than one who simply stopped recharging. This platform classifies every churned customer into a root cause — behavioral, network-driven, or mixed — so retention teams can act on evidence, not guesses.

Objective

Build an end-to-end data engineering platform that simulates telecom operator's ingestion pipeline, enforces data quality at every layer, models a star schema warehouse, and computes churn root causes across behavioral, network-driven, and mixed categories as in production.

Data Sources

Usage Events — service activity per customer: customer_id, event_date, event_type, data_mb, duration_min Billing Records — monthly billing: customer_id, billing_month, amount_sdg, status Recharge Records — top-up history: customer_id, recharge_date, amount_sdg Network Tickets — support tickets: ticket_id, customer_id, ticket_date, issue_type, status, resolution_days Customers — subscriber master: customer_id, msisdn, region, plan_type, age, gender, activation_date, monthly_arpu_sdg

🏗️ Data Architecture

The pipeline follows a layered architecture — raw data lands as Parquet after SFTP ingestion, passes through a quality gate, gets enriched and classified by Spark, persisted as a real Iceberg table, modeled into a star schema warehouse, and served through a DuckDB analytics mart. An Airflow DAG orchestrates every step.

📐 Warehouse — Star Schema

fact_customer_churn_snapshot sits at the center with foreign keys to five dimension tables. The root_cause_key is the core analytical signal — joining to dim_root_cause tells the business whether a churned customer was lost to behavioral disengagement, network failure, or both.

Key Design Decisions

• Iceberg over plain Parquet — snapshot isolation, time travel, and idempotent overwrites. A plain Parquet overwrite destroys history; an Airflow retry silently duplicates rows. Iceberg prevents both.
• Two fact tables, not one — churn snapshots and network tickets have different grains. Merging them silently inflates every aggregate.
• SCD Type 2 on dim_customer — a customer who changed plans in June should not retroactively change the attribution of a March churn event.
• Rule engine over ML — every classification comes with a human-readable reason a retention agent can act on. A probability score cannot do that.
• Cross-phase monitoring — the monitoring script reads manifests from all phases simultaneously and catches issues no single phase can detect on its own, like a row count mismatch between classification and the mart.

Full rationale in docs/architecture_decisions.md.

What It Found

Running on a simulated cohort of 5,000 Sudanese subscribers:

• 14.36% overall churn rate (718 customers)
• 81.75% of churned customers are behavioral — silence and non-payment, no dominant network signal
• 11.28% are network-driven — 3+ tickets raised before going silent
• Atbara has the highest churn rate at 18.18%, but it is behavioral, not network — the right intervention is a recharge incentive, not a tower upgrade

Repository Structure

telecom-churn-platform/
├── requirements.txt
├── phase 1- data generation/       → run_phase1.py
├── phase 2- raw storage layer/     → run_phase2.py
├── phase 3- airflowDAG/            → local_orchestrator.py
│                                     telecom_churn_pipeline.py
├── phase 4/                        → process_layer.py
│                                     root_cause_engine.py
│                                     iceberg_writer.py
├── phase 5/                        → phase5_star_schema_builder.py
├── phase 6/                        → build_mart.py
├── Notebooks/                      → 06_analytics_mart_exploration.ipynb
├── phase 7- monitoring/            → pipeline_monitor.py
├── dashboard/                      → churn_dashboard.html
└── docs/                           → architecture_decisions.md

🚀 Quick Start

# Install dependencies
pip install -r requirements.txt   # Python 3.12+ and Java 11+ required

# Run the full pipeline in one command
cd "phase 3- airflowDAG"
python local_orchestrator.py

# Or open the dashboard directly (no server needed)
open dashboard/churn_dashboard.html

📋 Requirements

Python 3.12+ Java 11+ (required for PySpark) — download from adoptium.net See requirements.txt for full package list

📚 Documentation

Architecture & Design Decisions