Our Client
One of the biggest independent power producers in the country is an OdyxAI client. Delivering power goods and services to commercial and industrial customers is the exclusive focus of their retail electricity business. In addition to Washington, DC, they currently serve commercial clients in Delaware, Maryland, New Jersey, Ohio, and Pennsylvania.
Challenge
They used manual data management approaches and had much real-time data, files stored on external drives and data from other parties. Additionally, they must ingest real-time data from the energy sector into data lakes for simple access and speedy analysis.
Solution
For managing real-time data, file drops, data obtained through APIs, and third-party data gathered from numerous data sources and ingested into the data lake; OdyxAI suggested using the AWS Cloud Solution.
This data was transferred from the S3 buckets to the processing layer using some lambdas, where it underwent ETL processes for batch and real-time processing. In the processing layer, we developed data warehouses and data marts. Most data was kept in AWS Redshift, while we saved time-series data in AWS DynamoDB.
Tableau and AWS Athena provided real-time data directly delivered from the processing layer for ad-hoc analysis. We use AWS CloudWatch to provide trustworthy log recordings for monitoring. OdyxAI created, set up, and implemented this entire architecture.
Result
Our client saw some significant improvements in their operations after replacing the manual data management techniques, including:
- Auto-upscale/downscale the consumption depending on the available capacity.
- Reliable logs and AWS CloudWatch-based operation monitoring.
- Effective use of API gateways and Lambda triggers.
- Automated data cleansing, loading, and exporting as necessary
.
Technology Used
- AWS Kinesis Data Streams
- AWS SQS/SNS
- AWS API Gateway
- AWS Lambda
- S3 Bucket
- Amazon Redshift
- AWS DynamoDB
- AWS Cloudwatch