Overriding Spring Boot properties in Amazon LambdaTech by Sunny Srinidhi - February 11, 2020February 11, 20200 In this post, we’ll see how we can maintain Spring Boot properties in an Amazon Lambda function without making code changes.
Getting started with Chalice to create AWS Lambdas in Python – Step by Step TutorialTech by Sunny Srinidhi - November 14, 2019November 14, 20190 Using Chalice, you can write a Lambda function, test it locally, and even deploy the Lambda function to your development, test, or production environments. In this post, we’ll see how we can install Chalice on our local machines, write a simple REST API to return the famous “Hello, world!” response, and deploy it to a dev stage on AWS Lambda.
Invoke an AWS Lambda Function from another Lambda FunctionData ScienceTech by Sunny Srinidhi - November 4, 2019November 4, 20190 I recently discovered that you can't invoke more than one Lambda function in AWS for an S3 event, with the same prefix and suffix (or just with the same suffix, which was the issue in my case). So I wanted a way to invoke one Lambda function from another Lambda function. If you're feeling kind of lost, check out the problem statement in my Github project. That could possibly add some context to the problem. If you don't want to go there, I'll try to explain it here again. The Problem and the Requirement In one of our projects, we have a Lambda function which is invoked whenever a text file is uploaded to a particular S3 bucket. The Lambda function takes
How to automatically trigger AWS Lambda functions using CloudWatchTech by Sunny Srinidhi - November 2, 20190 If you have AWS Lambda functions which need to be triggered periodically, like CRON jobs, there are many ways to achieve this. But I recently discovered a very easy and AWS-way of doing this, which makes life a lot easier. So, there are a lot of ways you can trigger Lambda functions periodically. One of the most common ways I've see people doing this is adding an API Gateway to the Lambda function, and then calling that API periodically as a CRON job from one of the machines in the setup. I actually thought this is how you're supposed do to that. Okay, let me make this clear. I'm not a DevOps guy. I just learn these things as and when
Cleaning and Normalizing Data Using AWS Glue DataBrewData Science by Sunny Srinidhi - January 17, 2022January 17, 20221 In this post, we’ll see what is AWS Glue DataBrew and how to use it to clean and transform our data in a data pipeline.
Kinesis Data Streams vs. Kinesis Firehose Delivery StreamsData Science by Sunny Srinidhi - May 25, 2020August 27, 20240 I have talked about Kinesis before, and I'm sure you've been using Kinesis for longer than me. But according to what I've seen, not all teams or companies use all parts of Kinesis. And, there are four parts in Kinesis: Ingest and process streaming data with Kinesis streams - Kinesis Data Streams Deliver streaming data with Kinesis Firehose delivery streams - Kinesis Firehose Delivery Streams Analyse streaming data with Kinesis analytics applications - Kinesis Analytics Ingest and process media streams with Kinesis video streams - Kinesis Video Streams All these four parts offer something different. Well, the last two are definitely different than the first two. But it's the first two that I see a lot of people getting confused with. So I thought I'll
How to build a simple data lake using Amazon Kinesis Data Firehose and Amazon S3Data Science by Sunny Srinidhi - March 3, 2020March 3, 20203 In this post, we’ll see how we can create a very simple, yet highly scalable data lake using Amazon’s Kinesis Data Firehose and Amazon’s S3.
Proof of Concepts (POCs) I write a lot of POC projects, especially when I'm learning something new or I need to quickly test if a data pipeline works, or maybe I'm just testing a new integration. I make all these POCs public as Github repositories. I wanted to consolidate the list of POCs in an easy to search fashion. And that's why I have this page here. Below is a list of all the POCs that I've written so far. If a particular POC has an accompanying blog post which explains the code in the POC, I have linked that blog post as well in the list below. Let me know if any of these POCs have helped you in any way.