1. Integrating Bureau and Account Aggregator Data
- Technical Problem Statement :
NBFCs typically rely on fragmented datasets like Bureau data (credit scores, payment history) and proprietary transactional data from banking sources. These silos hinder the ability to gain a unified customer view, which is crucial for generating relevant, personalized credit offers. Without real-time data integration, credit products remain generic, missing opportunities for personalization, leading to higher rejection rates and customer churn. - Data Engineering Solution:
To resolve this, we developed a real-time data integration pipeline using AWS DMS for streaming data and DRIFT, an in-house Python and SQL-based ETL tool, to merge Bureau data with AA data. Bureau data provided historical credit behaviour, while AA data offered real-time transactional insights. The pipeline architecture included: - Data Source Layer:
We ingested structured and semi-structured data from APIs (Bureau data) and transactional records from AA. - Transformation Layer:
Data was transformed using AWS Glue with Python and SQL scripts for data cleansing, normalization and feature enrichment. This allowed us to derive critical features like the debt-to-income ratio, spending patterns and savings rate. - Data Storage:
Processed data was stored in AWS Redshift for analytics and Amazon DocumentDB was used as a data lake and for storing partner responses. - Technical Impact:
Data latency was reduced by 50%, enabling near real-time credit offers.
Unified customer profiles allowed advanced modeling, increasing offer acceptance by 30%.
2. Custom Machine Learning Models for Credit Offer Precision
Problem Statement:
NBFCs must ensure credit offers are relevant and personalized. Precise customer segmentation, affinity scoring and risk assessment are crucial, based on each customer’s financial behaviour. Existing rule-based systems were inadequate for scaling personalization, particularly in a high-volume environment.
ML Solution Architecture:
We implemented four key ML models using MLFlow for model development and management:
- Customer Segmentation Model :
- Approach: Using unsupervised learning (K-means clustering), we segmented customers based on repayment history, account balance and loan frequency.
- Technical Stack: Python, AWS DMS for real-time data replication, MLFlow for MLOps.
- Feature Engineering: Key features like default risk, liquidity and monthly cash flow were used to classify customers into high-risk, low-risk and high-value categories.
Impact: Improved marketing ROI by 25% through more precise, targeted offers.
- User Affinity Index :
- Approach: A supervised learning model (LightGBM) calculated an affinity score for each customer based on transactional patterns, prior engagements and demographic information.
- Technical Stack: Python, AWS Redshift, MLFlow and Airflow for orchestration.
- Feature Engineering: Key features include historical clickstream metrics to capture users' digital and product preferences along with bureau data on existing credit product inquiries to determine the overall user affinity index.
Impact: Increased conversions by 15%, focusing on customers most likely to engage.
- Income Imputation Model :
- Approach: Using XGBoost, we imputed missing income data based on transactional patterns and derived financial indicators.
- Technical Stack: Python, AWS Redshift, MLFlow and SQL for feature engineering.
- Feature Engineering: Key features include demographic, geographic and employment related data points, company characteristics along with bureau transactions history, current credit obligations are also considered.
Impact: Reduced rejection rates by 18% due to accurate income estimation.
- Propensity Modeling :
- Approach: U A logistic regression model predicted a customer’s propensity to accept specific offers based on historical engagement and financial stability.
- Technical Stack: Python, AWS DMS, Airflow and AWS Redshift for seamless orchestration and data management.
- Feature Engineering: Key features include bureau data on minimum, maximum and median credit balances, combined with campaign response data and clickstream events, to assess the overall likelihood of customer engagement in campaigns.
Impact: Increased offer conversion rates by 25% through personalized targeting.
3. Martech Integration: Real-Time Delivery of Offers
- Problem Statement :
Once credit offers are generated, they need to be delivered in real-time across channels like SMS, email and WhatsApp using Martech automation. - Solution :
We integrated the personalized credit offers with the client’s Martech platform through REST APIs. Key components included: - Offer Engine API :
Developed using Python (FastAPI) and deployed on AWS EC2 behind Nginx, the API served offers to the Martech platform. - Real-Time Messaging Pipeline :
Credit offers were delivered via Apache Kafka, ensuring near-instantaneous cross-channel messaging. - Campaign Monitoring :
We used AWS Redshift and Tableau for tracking response metrics, engagement rates and conversions in real-time.
Technical Impact:
Reduced time-to-market for credit offers by 15%.
Improved customer engagement with cross-channel touchpoints, increasing engagement by 20%.
4. Ensuring Compliance and Risk Mitigation
- Problem Statement :
NBFCs must comply with strict regulatory frameworks, particularly regarding data privacy, consent, fraud detection and Do Not Disturb (DND) regulations. - Data Engineering Solution :
We built a compliance management framework as a middleware service using Python and SQL. It ensured that all credit offers adhered to; - User Consent :
Validated via APIs integrated with consent framework. - Blacklist and DND Verification :
SQL queries checked offers against preloaded blacklists and DND data sources. - Fraud Detection :
A rule-based engine, leveraging AWS Redshift, flagged suspicious activity to prevent credit offers from being sent to high-risk profiles.
Impact :
Zero compliance violations, improving customer trust.
Reduced legal risk by 10%.
5. Campaign Optimization Using Martech Response Data
- Problem Statement :
Campaign performance required constant optimization to maximize offer acceptance and minimize marketing spend. - Data Analysis Solution :
We built an automated campaign analysis pipeline using AWS Lambda and Redshift. Python and SQL scripts analyzed customer engagement data to recalibrate ML models based on real-time feedback. - Key Metrics :
Conversion Rate : A/B testing identified the most effective offer types.
Cost Efficiency : Insights led to a 10% reduction in marketing costs by focusing on high-conversion segments.
Conclusion:
This project highlights the power of data integration, ML-driven analytics and Martech automation in revolutionizing credit offer generation. By leveraging AWS DMS, Redshift, MLFlow and in-house ETL tools like DRIFT, we improved offer acceptance by 30%, reduced rejection rates and optimized marketing spend, all while ensuring compliance.
Key Outcomes:
30% increase in offer acceptance.
18% reduction in rejections.
25% improvement in overall campaign ROI.
This AWS-powered architecture showcases how financial institutions can harness data engineering, machine learning and scalable cloud infrastructure to achieve strategic growth in personalized credit offerings.
