Back
Year
2025
Tech & Technique
Next.js, React, Typescript, Python, FastAPI, GitHub Actions (CI/CD), Docker, AWS, PostgreSQL, Numpy, Pandas, Scikit-learn, DoWhy
Description
A Drag-and-Drop system for building ETL Pipelines and Causal Inference Models.
Key Features:
Key Features:
- ✨ Visual Workflow Design: Intuitively build and manage complex ETL pipelines and Causal Inference models using an interactive drag-and-drop interface.
- 🧩 Modular Component Library: Easily connect a variety of pre-built and customizable blocks to create flexible and tailored data processing and modeling workflows.
- ⚙️ End-to-End ETL Automation: Streamline the entire data lifecycle, from extraction and transformation to loading, with robust automation and scheduling capabilities for your pipelines.
- 🔎 Integrated Causal Inference: Seamlessly design, train, and evaluate causal inference models to uncover cause-and-effect relationships directly within your data workflows.
- 🚀 Rapid Development & Deployment: Accelerate the creation and iteration of data pipelines and causal studies, enabling quicker insights and deployment of data-driven solutions.
My Role
Full Stack Developer
- 🎨 Drag-and-Drop Interface Development: Designed and implemented an intuitive and responsive drag-and-drop user interface.
- 🏗️ ETL Pipeline Engine Development: Developed the robust backend engine capable of interpreting the user-designed pipelines using Graph Data Structure and Depth First SearchDFS.
- 📊 Causal Inference Model Integration: Integrated and implemented functionalities for defining and running causal inference models within the platform.
- 🔌 API Design and Development: Created and maintained APIs to facilitate communication between the frontend drag-and-drop interface and the backend processing engine.
- 💾 Data Management and Storage: Designed and implemented solutions for managing the metadata of pipelines and models.
- ☁️ Scalability and Performance: Focused on building a scalable architecture capable of handling varying data volumes and computational demands for both ETL processes and causal model execution.
- 🔄 CI/CD and Deployment: Established and maintained CI/CD pipelines to automate the build, testing, and deployment of the full-stack application.
- 🐞 Monitoring and Troubleshooting: Implemented monitoring solutions and was responsible for identifying and resolving issues across the entire stack to ensure the platform's stability and performance in production.