[2025-12] task_id=mofid running
Data Engineer @ Mofid Securities
Dec 2025 – Present
- Built the company's first unified text-data platform on Elasticsearch and Kibana, consolidating four sources (call transcripts and chat logs from MongoDB, customer SMS messages and stock-market search events) into one denormalised schema enriched with customer and trading data.
- Designed Airflow ETL pipelines that join session metadata with transcripts, rebuild each conversation as a chronological agent–customer dialogue and load flat, search-ready documents; backfilled a full year of history.
- Replaced N+1 per-record ClickHouse lookups with a single bulk fetch and in-memory mapping, cutting a batch enrichment job from 70 minutes to under 5 (>92% faster).
- Secured Elasticsearch and Kibana with Active Directory (LDAP) sign-in, role-based access control and field-level security masking PII such as national IDs, plus Filebeat-based audit logging.
- Deployed a 3-node Elasticsearch cluster with Ansible and Docker, then right-sized shard and index settings after a production capacity review to cut memory overhead and inter-node traffic.
- Refactored the shared ETL library used across Airflow DAGs, fixing database connection leaks and redundant engine creation, and extended the customer cash-flow pipeline with new transaction types.