Data Platform for text-based mental health support
- Customer
- Crisis Text Line
- Project manager on the customer side
-
- Project timeline
- December, 2022 - September, 2023
- Project scope
- 1050 man-hours
- Goals
-
Implement Data platform, which can support/offer:
- Single point of truth
- Batch processing
- Real-time data processing
- Machine Learning pipelines/life cycle
- LLM
- ML models serving
- Project Results
-
- Installed the data platform components
- Established ETL pipelines (ingest & transformation)
- Established SDLC (software development life cycle)
- Automated CI/CD
- Established monitoring (ETL, data freshness, notifications)
- The whole infrastructure as a code
- Onboarded DS & R&I teams
The uniqueness of the project
The Data platform should meet the following criteria:
- Support batch and stream processing.
- Support ML pipelines.
- Be deployed in different countries (US, EU, Canada).
- Support multi-language for data anonymization.
- Monitor that data from one country cannot be uploaded or stored in an instance from another (e.g. EU data in US instance).
- Has deployment flexibility (independent feature deployment to different instances/countries).
- IaC (infrastructure as a code)
- Used software
-
The following technologies have been used:
- AWS (infrastructure, AWS DMS, S3, etc)
- Apache Spark (Data processing framework)
- Github
- Terraform & Terragrunt for IaC
- Difficulty of implementation
-
- From the beginning, we decided to use CDC (change data capture). However, we encountered troubleshooting issues with AWS DMS and also assessed the effort required to implement incremental ingestions and transformations. As a result, we decided to initially go with batch ingestion, pulling data several times per day. This approach was deemed sufficient from a business perspective.
- The initial infrastructure was deployed without IaC (Infrastructure as Code), which led to issues when adjustments needed to be made and when multi-country support was anticipated. Consequently, we implemented IaC to address these concerns.
- Our CI/CD process is implemented using different technologies. Some parts are automated using GitHub actions, while others rely on running terraform/terragrunt scripts. In the future, we aim to unify and simplify our process.
- Project Description
-
#fff; --tw-ring-color: rgba(69,89,164,0.5); --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-shadow: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-shadow-colored: 0 0 transparent; --tw-blur: ; --tw-brightness: ; --tw-contrast: ; --tw-grayscale: ; --tw-hue-rotate: ; --tw-invert: ; --tw-saturate: ; --tw-sepia: ; --tw-drop-shadow: ; --tw-backdrop-blur: ; --tw-backdrop-brightness: ; --tw-backdrop-contrast: ; --tw-backdrop-grayscale: ; --tw-backdrop-hue-rotate: ; --tw-backdrop-invert: ; --tw-backdrop-opacity: ; --tw-backdrop-saturate: ; --tw-backdrop-sepia: ; caret-color: #374151; color: #374151; font-family: Söhne, ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, Ubuntu, Cantarell, "Noto Sans", sans-serif, "Helvetica Neue", Arial, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Noto Color Emoji"; white-space: pre-wrap;">At the core of our company's mission lie empathy and innovation. Our primary objective is to enhance mental well-being for individuals worldwide, transcending geographical barriers.
#fff; --tw-ring-color: rgba(69,89,164,0.5); --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-shadow: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-shadow-colored: 0 0 transparent; --tw-blur: ; --tw-brightness: ; --tw-contrast: ; --tw-grayscale: ; --tw-hue-rotate: ; --tw-invert: ; --tw-saturate: ; --tw-sepia: ; --tw-drop-shadow: ; --tw-backdrop-blur: ; --tw-backdrop-brightness: ; --tw-backdrop-contrast: ; --tw-backdrop-grayscale: ; --tw-backdrop-hue-rotate: ; --tw-backdrop-invert: ; --tw-backdrop-opacity: ; --tw-backdrop-saturate: ; --tw-backdrop-sepia: ; caret-color: #374151; color: #374151; font-family: Söhne, ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, Ubuntu, Cantarell, "Noto Sans", sans-serif, "Helvetica Neue", Arial, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Noto Color Emoji"; white-space: pre-wrap;">To realize this mission, we recognize the need to harness modern technologies, including LLM models. Our Data Platform's central focus is transitioning from our existing legacy system to a more robust, dynamic, and industry-grade infrastructure. This transition will empower us with cutting-edge capabilities, enabling us to deliver unparalleled mental health support. Some key enhancements include:
- Improving message classification (risk level) to ensure prompt assignment of Volunteer Crisis Counselors to critical cases.
- Real-time monitoring of demand spikes, facilitating rapid outreach to Volunteer Crisis Counselors to address increased needs.
- Developing realistic simulations through the utilization of LLM models
-
#fff; --tw-ring-color: rgba(69,89,164,0.5); --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-shadow: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-shadow-colored: 0 0 transparent; --tw-blur: ; --tw-brightness: ; --tw-contrast: ; --tw-grayscale: ; --tw-hue-rotate: ; --tw-invert: ; --tw-saturate: ; --tw-sepia: ; --tw-drop-shadow: ; --tw-backdrop-blur: ; --tw-backdrop-brightness: ; --tw-backdrop-contrast: ; --tw-backdrop-grayscale: ; --tw-backdrop-hue-rotate: ; --tw-backdrop-invert: ; --tw-backdrop-opacity: ; --tw-backdrop-saturate: ; --tw-backdrop-sepia: ; list-style: none; margin: 1.25em 0px; padding: 0px; counter-reset: list-number 0; display: flex; flex-direction: column; caret-color: #374151; color: #374151; font-family: Söhne, ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, Ubuntu, Cantarell, "Noto Sans", sans-serif, "Helvetica Neue", Arial, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Noto Color Emoji"; white-space: pre-wrap;">
#fff; --tw-ring-color: rgba(69,89,164,0.5); --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-shadow: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-shadow-colored: 0 0 transparent; --tw-blur: ; --tw-brightness: ; --tw-contrast: ; --tw-grayscale: ; --tw-hue-rotate: ; --tw-invert: ; --tw-saturate: ; --tw-sepia: ; --tw-drop-shadow: ; --tw-backdrop-blur: ; --tw-backdrop-brightness: ; --tw-backdrop-contrast: ; --tw-backdrop-grayscale: ; --tw-backdrop-hue-rotate: ; --tw-backdrop-invert: ; --tw-backdrop-opacity: ; --tw-backdrop-saturate: ; --tw-backdrop-sepia: ; caret-color: #374151; color: #374151; font-family: Söhne, ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, Ubuntu, Cantarell, "Noto Sans", sans-serif, "Helvetica Neue", Arial, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Noto Color Emoji"; white-space: pre-wrap;">With these initiatives, we aim to make a significant positive impact on mental well-being, regardless of where individuals are located.
- Project geography
- US, EU
On November 30, the professional IT community GlobalCIO hosted a large-scaled international conference "Global CIO Insights: Digital Transformation with AI". During the event, leading experts shared their practical experience in launching projects utilizing artificial intelligence (AI) and highlighted approaches that helped elevate their companies to new heights.
Voting for projects participating in the "Project of the Year" contest is open. The voting began on December 1st and will continue until January 15th inclusive. The winners will be announced on February 7th, 2024.
According to the statistics agency, the number of telecommunication and IT companies has increased 1.8 times in the last five years. At the beginning of 2023, there are more than 12 thousand ICT companies operating in the country. More than 100 thousand people are employed in the industry. At the end of 2022, the volume of ICT services grew by 125.5% and amounted to 22.9 trillion soums, of which 4.2 trillion soums were programming services provided by Uzbek companies and specialists. The total revenue of Uzbekistan's IT sector for the first quarter of 2023 reached 2.38 trillion soums - almost four times more than in the same period of 2022. Exports of digital services increased to $57.2 million. According to IT Park, net income amounted to more than 90% of revenue or 2.158 trillion soums.
According to the statistics agency, the number of telecommunication and IT companies has increased 1.8 times in the last five years. At the beginning of 2023, there are more than 12 thousand ICT companies operating in the country. More than 100 thousand people are employed in the industry. At the end of 2022, the volume of ICT services grew by 125.5% and amounted to 22.9 trillion soums, of which 4.2 trillion soums were programming services provided by Uzbek companies and specialists. The total revenue of Uzbekistan's IT sector for the first quarter of 2023 reached 2.38 trillion soums - almost four times more than in the same period of 2022. Exports of digital services increased to $57.2 million. According to IT Park, net income amounted to more than 90% of revenue or 2.158 trillion soums.
The IT strategy of a large number of companies includes the automation (digitization) of not only accounting processes but also production and logistics processes.