Real-Time Data Processing: Understanding the What, Why, Where, Who, and HowData ScienceTech by Sunny Srinidhi - October 22, 20240 In today’s data-driven world, businesses and organizations are continuously generating massive amounts of data. While processing data in batch mode remains useful, the need for instant decision-making has led to an increasing focus on real-time data processing. This article delves into what real-time data processing is, why it's essential, its various applications, the tools used to achieve it, trends shaping its evolution, and real-world use cases. What is Real-Time Data Processing? Real-time data processing refers to the capability to continuously ingest, process, and output data as soon as it is generated, with minimal latency. Unlike batch processing, which collects and processes data in large groups at set intervals (e.g., daily or hourly), real-time processing works with data immediately as it becomes available,
Understanding Data Governance: A Comprehensive GuideData ScienceTech by Sunny Srinidhi - October 18, 2024October 18, 20240 Data governance is a set of practices, policies, and standards that ensure data is managed as an asset in a consistent and reliable manner across an organization. It involves defining who owns the data, who has the right to make decisions about it, and how it can be used. This comprehensive guide aims to shed light on what data governance entails, its importance, how it can be achieved, best practices, and who should be involved in the process. What is Data Governance? Data governance refers to the collection of policies, roles, responsibilities, and procedures that oversee the management of data assets within an organization. It ensures that data is accurate, consistent, accessible, and protected from misuse. The main goal of data governance
Streamline Data Transfer with AWS DataSync: A Comprehensive GuideData Science by Sunny Srinidhi - March 9, 2024March 9, 20240 Discover the power of AWS DataSync for seamless, secure, and accelerated data transfers. Learn how to optimise workflows with ease!
Understanding the Battle of Database Storage: Row-Oriented vs. ColumnarData Science by Sunny Srinidhi - March 8, 2024March 8, 20241 In the realm of database storage, row-wise and columnar approaches stand as stalwarts with distinct advantages. Row-wise storage excels in transactional operations, ensuring data integrity with simplicity. Conversely, columnar storage revolutionizes analytical querying, leveraging vertical organization for rapid attribute retrieval. Understanding their nuances is pivotal in crafting efficient, tailored database solutions for diverse data-driven needs.
Enhancing Data Security and Privacy in the Cloud with AWS Clean RoomsData Science by Sunny Srinidhi - May 26, 2023January 17, 20240 Data security and privacy in the cloud is becoming crucial as more organisations are embracing cloud computing and cloud storage. In this post, we’ll see how AWS Clean Rooms can help maintain data security and privacy.
Use Amazon CloudSearch to quickly search through dataTech by Sunny Srinidhi - March 29, 2023January 17, 20240 Amazon CloudSearch provides a number of powerful search capabilities, including full-text search, faceted search, and customizable relevance ranking. In this post, we’ll see what CloudSearch is
Cleaning and Normalizing Data Using AWS Glue DataBrewData Science by Sunny Srinidhi - January 17, 2022January 17, 20221 In this post, we’ll see what is AWS Glue DataBrew and how to use it to clean and transform our data in a data pipeline.
Kinesis Data Streams vs. Kinesis Firehose Delivery StreamsData Science by Sunny Srinidhi - May 25, 2020August 27, 20240 I have talked about Kinesis before, and I'm sure you've been using Kinesis for longer than me. But according to what I've seen, not all teams or companies use all parts of Kinesis. And, there are four parts in Kinesis: Ingest and process streaming data with Kinesis streams - Kinesis Data Streams Deliver streaming data with Kinesis Firehose delivery streams - Kinesis Firehose Delivery Streams Analyse streaming data with Kinesis analytics applications - Kinesis Analytics Ingest and process media streams with Kinesis video streams - Kinesis Video Streams All these four parts offer something different. Well, the last two are definitely different than the first two. But it's the first two that I see a lot of people getting confused with. So I thought I'll
Explore your Amazon S3 data online using FilestashTech by Sunny Srinidhi - April 29, 2020April 29, 20200 Filestash is a very handy tool in your browser which helps you nativage your S3 buckets and folders easily, and even edit files online.
How to build a simple data lake using Amazon Kinesis Data Firehose and Amazon S3Data Science by Sunny Srinidhi - March 3, 2020March 3, 20203 In this post, we’ll see how we can create a very simple, yet highly scalable data lake using Amazon’s Kinesis Data Firehose and Amazon’s S3.