Overriding Spring Boot properties in Amazon Lambda Tech by Sunny Srinidhi - February 11, 2020February 11, 20200 I want to start this post by bluntly saying that using a Spring Boot project as your Amazon Lambda function is a bad, bad idea, for so many reasons. I don’t want to get into that in this post. But sometimes, you can’t stop this from happening because you’re not calling the shots. Read more... “Overriding Spring Boot properties in Amazon Lambda”
Getting started with Chalice to create AWS Lambdas in Python – Step by Step Tutorial Tech by Sunny Srinidhi - November 14, 2019November 14, 20190 If you’re into serverless stuff, you already know what is AWS Lambda. But if you don’t know, AWS Lambda is a serverless service provided by Amazon where you can create ‘functions’ and deploy them in AWS, which you can run without having any server instances (such as EC2). Read more... “Getting started with Chalice to create AWS Lambdas in Python – Step by Step Tutorial”
Invoke an AWS Lambda Function from another Lambda Function Data Science Tech by Sunny Srinidhi - November 4, 2019November 4, 20190 I recently discovered that you can't invoke more than one Lambda function in AWS for an S3 event, with the same prefix and suffix (or just with the same suffix, which was the issue in my case). So I wanted a way to invoke one Lambda function from another Lambda function. If you're feeling kind of lost, check out the problem statement in my Github project. That could possibly add some context to the problem. If you don't want to go there, I'll try to explain it here again. The Problem and the Requirement In one of our projects, we have a Lambda function which is invoked whenever a text file is uploaded to a particular S3 bucket. The Lambda function takes
How to automatically trigger AWS Lambda functions using CloudWatch Tech by Sunny Srinidhi - November 2, 20190 If you have AWS Lambda functions which need to be triggered periodically, like CRON jobs, there are many ways to achieve this. But I recently discovered a very easy and AWS-way of doing this, which makes life a lot easier. So, there are a lot of ways you can trigger Lambda functions periodically. One of the most common ways I've see people doing this is adding an API Gateway to the Lambda function, and then calling that API periodically as a CRON job from one of the machines in the setup. I actually thought this is how you're supposed do to that. Okay, let me make this clear. I'm not a DevOps guy. I just learn these things as and when
Cleaning and Normalizing Data Using AWS Glue DataBrew Data Science by Sunny Srinidhi - January 17, 2022January 17, 20221 A major part of any data pipeline is the cleaning of data. Depending on the project, cleaning data could mean a lot of things. But in most cases, it means normalizing data and bringing data into a format that is accepted within the project. Read more... “Cleaning and Normalizing Data Using AWS Glue DataBrew”
Kinesis Data Streams vs. Kinesis Firehose Delivery Streams Data Science by Sunny Srinidhi - May 25, 2020May 25, 20200 I have talked about Kinesis before, and I'm sure you've been using Kinesis for longer than me. But according to what I've seen, not all teams or companies use all parts of Kinesis. And, there are four parts in Kinesis: Ingest and process streaming data with Kinesis streams - Kinesis Data StreamsDeliver streaming data with Kinesis Firehose delivery streams - Kinesis Firehose Delivery StreamsAnalyse streaming data with Kinesis analytics applications - Kinesis AnalyticsIngest and process media streams with Kinesis video streams - Kinesis Video Streams All these four parts offer something different. Well, the last two are definitely different than the first two. But it's the first two that I see a lot of people getting confused with. So I thought I'll
How to build a simple data lake using Amazon Kinesis Data Firehose and Amazon S3 Data Science by Sunny Srinidhi - March 3, 2020March 3, 20203 As the data generated from IoT devices, mobile devices, applications, etc. increases at an hourly rate, creating a data lake to store all that data is getting crucial for almost any application at scale. There are many tools and services that you could use to create a data lake. Read more... “How to build a simple data lake using Amazon Kinesis Data Firehose and Amazon S3”
Proof of Concepts (POCs) I write a lot of POC projects, especially when I'm learning something new or I need to quickly test if a data pipeline works, or maybe I'm just testing a new integration. I make all these POCs public as Github repositories. I wanted to consolidate the list of POCs in an easy to search fashion. And that's why I have this page here. Below is a list of all the POCs that I've written so far. If a particular POC has an accompanying blog post which explains the code in the POC, I have linked that blog post as well in the list below. Let me know if any of these POCs have helped you in any way.