Implement Phenix Data Platform

Implement a Data platform as a product, that fulfils the following criteria:

• Establish a Central Data Hub: Develop a robust Data Platform serving as the singular, indisputable source of truth for all organisational data, ensuring accuracy and consistency across the board.
• Empower Data Science and Research Teams: Provide state-of-the-art tools and resources to our Data Science and Research analyst teams, enabling them to delve deep into data, extract meaningful patterns, and drive innovative research initiatives.
• Revolutionise Machine Learning Lifecycles: Provide tools and technics to Implement an end-to-end framework for Machine Learning models, encompassing exploration, training, evaluation, deployment, and seamless serving, with a focus on advanced LLM models, ensuring cutting-edge AI capabilities.
• Optimised Workload Management: Engineer the platform to seamlessly handle both batch and real-time data processing with minimal latency, guaranteeing swift and efficient data processing regardless of the workload type, enhancing overall operational efficiency.
• Global Deployment, Local Compliance: Design the platform to be deployable in diverse international settings while rigorously adhering to specific local regulations, such as GDPR in the European Union, ensuring compliance and data integrity on a global scale.

Achievements:

• Successfully onboarded Data Science & Research teams to the Platform
• Completed development of the first prototypes for the risk assessment model & conversation simulator for our Volunteer Counselors
• Deployed the Platform to 2 countries (comming more)

Technical accomplishments:

• Installed all Data platform components using Infrastructure as Code (IaC), enabling us to rapidly deploy the platform to new countries within few days
• Developed robust ingest & transformation data pipelines, including data scrubbing/anonymization to ensure the removal of Personally Identifiable Information (PII), with built-in monitoring & data quality features
• Automated Continuous Integration/Continuous Deployment (CI/CD) pipelines
• Data processing adhered to local regulations

The following technologies have been used:

• AWS (infrastructure, AWS DMS, S3, etc)
• Databricks
• Github
• Terraform & Terragrunt for IaC

• Initially, we decided to use Change Data Capture (CDC) to gather data from source systems. However, we encountered troubleshooting issues with AWS DMS and realised the effort required for implementing incremental ingestion and transformations. As a result, we opted for batch ingestion initially, pulling data multiple times per day. This approach was considered sufficient from a business perspective.
• The initial infrastructure was deployed without Infrastructure as Code (IaC), which caused issues when adjustments were needed and when multi-country support was expected. Consequently, we implemented IaC to address these concerns.
• Our CI/CD process is implemented using various technologies. Some parts are automated using GitHub Actions, while others rely on executing Terraform/Terragrunt scripts. In the future, we aim to unify and simplify our process.

At the core of our company's mission lie empathy and innovation. Our primary objective is to enhance mental well-being for individuals worldwide, transcending geographical barriers.

To realize this mission, we recognize the need to harness modern technologies, including LLM models. The central focus of our Data Platform is transitioning from our existing legacy system to a more robust, dynamic, and industry-grade infrastructure. This transition will empower us with cutting-edge capabilities, enabling us to deliver unparalleled mental health support. On top of the Data Platform, we will implement the following key enhancements:

• Developing realistic conversation simulations for volunteers using LLM models, built based on our historical data.
• Improving message classification (risk level) to ensure the prompt assignment of Volunteer Crisis Сounselors to critical cases.
• Real-time monitoring of volunteer demand spikes, facilitating rapid outreach to Volunteer Crisis Counselors to address increased needs.

With these initiatives, we aim to make a significant positive impact on mental well-being, regardless of individuals' locations.