{"id":28014,"date":"2024-12-23T12:34:43","date_gmt":"2024-12-23T07:04:43","guid":{"rendered":"https:\/\/blogrevamp.cashfree.com\/?p=28014"},"modified":"2024-12-23T12:35:12","modified_gmt":"2024-12-23T07:05:12","slug":"smart-log-management-system","status":"publish","type":"post","link":"https:\/\/blogrevamp.cashfree.com\/smart-log-management-system\/","title":{"rendered":"From Log Overload to Cost-Efficient Storage: Cashfree Payments\u2019 Journey to Smarter Log Management"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_81 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #005c31;color:#005c31\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #005c31;color:#005c31\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/blogrevamp.cashfree.com\/smart-log-management-system\/#Challenges_for_Storing_Logs\" >Challenges for Storing Logs<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/blogrevamp.cashfree.com\/smart-log-management-system\/#How_to_Solve_a_Log_Explosion\" >How to Solve a Log Explosion?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/blogrevamp.cashfree.com\/smart-log-management-system\/#How_Did_We_Optimise_the_Storage\" >How Did We Optimise the Storage?<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/blogrevamp.cashfree.com\/smart-log-management-system\/#Comparison_of_ELK_Kibana_and_Data_Lake_Table\" >Comparison of ELK (Kibana) and Data Lake Table\u00a0<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/blogrevamp.cashfree.com\/smart-log-management-system\/#ELK_Kibana_and_Data_Lake_for_Application_Logs\" >ELK (Kibana) and Data Lake for Application Logs<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/blogrevamp.cashfree.com\/smart-log-management-system\/#New_Log_Retention_Flow\" >New Log Retention Flow<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/blogrevamp.cashfree.com\/smart-log-management-system\/#Components_Used_in_Running_the_Log_Processing_in_Data_Lake\" >Components Used in Running the Log Processing in Data Lake<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/blogrevamp.cashfree.com\/smart-log-management-system\/#Table_Structure\" >Table Structure<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/blogrevamp.cashfree.com\/smart-log-management-system\/#Number_of_Logs_Processed_and_the_Run_Time_The_batch_run_happens_every_20_mins\" >Number of Logs Processed and the Run Time (The batch run happens every 20 mins)<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-10\" href=\"https:\/\/blogrevamp.cashfree.com\/smart-log-management-system\/#Monolithic_Service_Logs_Optimisation\" >Monolithic Service Logs Optimisation<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-11\" href=\"https:\/\/blogrevamp.cashfree.com\/smart-log-management-system\/#Log_Optimisation\" >Log Optimisation<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-12\" href=\"https:\/\/blogrevamp.cashfree.com\/smart-log-management-system\/#Migrate_Logs_to_Data_Lake\" >Migrate Logs to Data Lake<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-13\" href=\"https:\/\/blogrevamp.cashfree.com\/smart-log-management-system\/#Monolith_Stack_Flow\" >Monolith Stack Flow<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-14\" href=\"https:\/\/blogrevamp.cashfree.com\/smart-log-management-system\/#Monolith_Stack_Table_Structure\" >Monolith Stack Table Structure<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-15\" href=\"https:\/\/blogrevamp.cashfree.com\/smart-log-management-system\/#Conclusion\" >Conclusion<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-16\" href=\"https:\/\/blogrevamp.cashfree.com\/smart-log-management-system\/#The_Road_Ahead\" >The Road Ahead&nbsp;<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n\n<p class=\"wp-block-paragraph\">At Cashfree Payments, we have multiple microservices running on Kubernetes, alongside a few monolithic services on EC2, both contributing to a significant volume of logs. These logs are essential for debugging and are also retained for extended periods to meet compliance requirements.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">However, this leads to escalating storage costs. In this blog, we share our insights on building efficient log storage systems to address these challenges.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Challenges_for_Storing_Logs\"><\/span><strong>Challenges for Storing Logs<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Log explosion<\/strong>:<br>As the business grows, the existing service produces more logs, and new services are added. This leads to an increase in storage space and incurs more cost.<br><\/li>\n\n\n\n<li><strong>Identifying optimum storage<\/strong>:\n<ul class=\"wp-block-list\">\n<li>ELK (Kibana): The increasing volume of logs and the need for a longer retention policy significantly add enormous costs to ELK-based logging systems and the cost of our infrastructure.<\/li>\n\n\n\n<li>Cloudwatch: Monolithic services operating in EC2 as legacy solutions were costly as we used Cloudwatch for log storage.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"How_to_Solve_a_Log_Explosion\"><\/span><strong>How to Solve a Log Explosion?<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">We had an initial round of log reduction exercises for our microservices to solve this. Here is how we did it:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Changed the log format from text to JSON<\/strong>: Switching to JSON enabled easier parsing of logs, simplified field searches using JSON keys, and reduced the index size.<\/li>\n\n\n\n<li><strong>Removed redundant logs<\/strong>: We identified and eliminated logs that were no longer needed.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">As a result, we successfully reduced the daily log volume by 30%.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"How_Did_We_Optimise_the_Storage\"><\/span><strong>How Did We Optimise the Storage?<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Despite efforts to reduce log volume, we continued to incur significant ELK (Kibana) costs, with a retention period limited to just 12 days. To address this, we explored using a data lake, which allowed us to process data at a much lower cost and with fewer resources. By leveraging S3 storage, which is more cost-effective than other data storage options, we were able to store more logs at a reduced cost.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Comparison_of_ELK_Kibana_and_Data_Lake_Table\"><\/span><strong>Comparison of ELK (Kibana) and Data Lake Table\u00a0<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXdWZ7pbx6ODIsfWGjax6KC-Sn2JGyY2NmFPuIZ9bwJNsDg_Gv4CmSWHb0oucKdSjBEYqjv00x8DLCVxFA9h0CJLh3-RQOcRQstKDS3KAJCiiAsILvr8MiJHPqtEa5ZYN9rcjJcp?key=CATd1eHazNafjrSrI5XqHw\" alt=\"\"\/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Since ELK (Kibana) had lower latency but was considerably more expensive, and the Data Lake solution offered significant cost benefits at the expense of higher latency, we developed a hybrid solution for our log storage and retention. As part of this initiative, we also addressed issues related to our log archival procedures.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We resolved almost 98% of our debugging use cases by reviewing logs from the previous four to five days. However, we discovered that the debugging process occasionally required logs older than five to seven days.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Initially, we indexed all keys in the ELK cluster present in the logs, but most were not utilized in our search queries. To optimise this, we limited the keys by using a curated list of the most frequently used indexed keys in ELK.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"ELK_Kibana_and_Data_Lake_for_Application_Logs\"><\/span><strong>ELK (Kibana) and Data Lake for Application Logs<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Reduced the log retention in ELK to 5 days from the previous 12 days. This reduced our cluster size by nearly 50%, thus decreasing our ELK expenditure.<\/li>\n<\/ol>\n\n\n\n<ol start=\"2\" class=\"wp-block-list\">\n<li>Reduced the number of indexed keys. Since all the keys were indexed initially, indexing required much more memory than usual. By reducing the number of keys to be indexed we redacted the storage and compute required to build and store indexes. Our storage has shrunk due to this change.<br><\/li>\n\n\n\n<li>Added a review process to add indexes on new keys. This removed the addition of redundant keys to the index list.<br><\/li>\n\n\n\n<li>Logs retention in the Data Lake table was set to 30 days. This allows us to troubleshoot issues whose logs are up to 30 days old.<br><\/li>\n\n\n\n<li>Deprecated the existing archival process and moved to the new archival flow for long retention.<br>\n<ul class=\"wp-block-list\">\n<li>We used the parquet files from the Data Lake table instead of the raw txt files for the archival process.<\/li>\n\n\n\n<li>File transfer cost reduced by 100 times.<\/li>\n\n\n\n<li>The metadata overhead became negligible as the average file size increased.<\/li>\n\n\n\n<li>Since we moved the processed files to S3 Instant Retrieval, we can store the logs as Data Lake tables with zero retrieval time.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">These changes have helped us decrease the cost of the logging system by more than 50%. As a result of the process, we had a new, efficient, and more powerful and efficient log storage process in place.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"New_Log_Retention_Flow\"><\/span><strong>New Log Retention Flow<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXdvkNRNzBiIFb5lH-aplZmdrDTale-FqRYnmEQzGV8RhCuw_tMrZhrAMb0wIfkjJc8YMcJJxx0oc6iHppBsk7y_5C1ThcAqNFghRRmLqPSa5eBY57PgW92hh87VmyFbhfhnunENsA?key=CATd1eHazNafjrSrI5XqHw\" alt=\"\"\/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Components_Used_in_Running_the_Log_Processing_in_Data_Lake\"><\/span><strong>Components Used in Running the Log Processing in Data Lake<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXcixtKTJWJtF5a1xy6wn-fZ-Jjd_LkcYlpxgPUMYJzDUAZZsYt16w5YHH56JaUSuTf0pnMPKbPplyXCZb0O-JXDp8NkffmsL8GLnwn2_rPCw9HSo57uLJO8IxhcirJreYAxrm4A?key=CATd1eHazNafjrSrI5XqHw\" alt=\"\"\/><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Apache Spark<\/strong> framework to process the logs<\/li>\n\n\n\n<li><strong>EMR on EKS<\/strong> is used to deploy Spark applications.\n<ul class=\"wp-block-list\">\n<li>Using the <strong>spot node<\/strong> in our EKS cluster<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Parquet<\/strong> file format is used to store the logs for faster retrieval and query.\n<ul class=\"wp-block-list\">\n<li>\u00a0Using snappy compression along with parquet to store the data.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>AWS GLUE Catalog<\/strong> to store the metadata of the tables<\/li>\n\n\n\n<li><strong>Athena<\/strong> to access the logs<\/li>\n\n\n\n<li><strong>Airflow<\/strong> for scheduling<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Table_Structure\"><\/span><strong>Table Structure<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXfyVglmgw7KEhst7bzBGltEarwr-E01nT1PM2w88GZW7LGUTwtillpGos-wjsyT5KYzLI5x-4-vLEP1AQtXP8BKt4cvmPsI-zWxRcbxNt42jENDGw35w8aAZRiYfRuzqCRqfcmqvQ?key=CATd1eHazNafjrSrI5XqHw\" alt=\"\"\/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Number_of_Logs_Processed_and_the_Run_Time_The_batch_run_happens_every_20_mins\"><\/span><strong>Number of Logs Processed and the Run Time (The batch run happens every 20 mins)<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXcvLNLHC8van3UkygDehFJCSIX48SZu2SogIXaa1Ph-xQILa3QJnc-wZvaiZ3MGnkNDJ6T_KEFFtfaTT1V8wf7H-o9vKXQACy2gtxKgm_cuZ-AGaTmmTMekTCuz4T2IzhxV8F1e?key=CATd1eHazNafjrSrI5XqHw\" alt=\"\"\/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Monolithic_Service_Logs_Optimisation\"><\/span><strong>Monolithic Service Logs Optimisation<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Some monolithic applications running on EC2 machines could not push logs to ELK (Kibana) due to architectural constraints. Initially, we used the AWS log agent to collect and push logs to CloudWatch, but the cost of storing, querying, and alerting with CloudWatch was prohibitive.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We initiated two exercises for these logs. First, log optimisation, and second, migrate the logs to the data lake.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Log_Optimisation\"><\/span><strong>Log Optimisation<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">We decreased the log retention on the cloudwatch from a lifetime to 30 days. This made a huge difference in the cost but there was a limit to cost reduction here.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Migrate_Logs_to_Data_Lake\"><\/span><strong>Migrate Logs to Data Lake<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Since we were directly getting the logs from EC2 machines, we used the <strong>log rotation<\/strong> to create a new log file every hour and then push the created file to S3. Once the logs are in S3, we process them and store them in the data lake table using the Spark data pipeline.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This approach eliminated our reliance on CloudWatch for monolithic services, significantly reducing costs. The expenses incurred through this method are substantially lower than those of AWS CloudWatch.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Monolith_Stack_Flow\"><\/span><strong>Monolith Stack Flow<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXfjhjf2MeM6gL0jUQ-G9keGDqbT9r_o95piuihgrrVfn1rnjLewDVX_FU6QspdDvAm4y9HYYmAtNz4li6MW7VrQp0d4bPO-0ogMJP38hjbEhOo1e0O46Jl0VDEXLC2bfWduJzsI?key=CATd1eHazNafjrSrI5XqHw\" alt=\"\"\/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">For this, we had to set up a different data pipeline as the log structure differed from Kibana. With this, we can store the logs for a longer period without increasing costs exponentially.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Monolith_Stack_Table_Structure\"><\/span><strong>Monolith Stack Table Structure<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXe5eAa_qGe1jOIbXvZe_VWkgsCuTlDI15wS-kRfEmeCPzDIAT_h-Ls38ojyRBrSBu2vOxqd5ESHf08c4KeOZ55UbKMJ_1eppZRs4BpNjcGdMfxjC3xK-LU-GV2ymWVbB4_xDA3jHQ?key=CATd1eHazNafjrSrI5XqHw\" alt=\"\"\/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Conclusion\"><\/span><strong>Conclusion<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Cutting log storage costs is both feasible and critical for businesses handling increasing data volumes. Strategies like tiered storage, log compression, and retention policies enabled significant cost savings, with reductions of <strong>80%<\/strong> in application log storage and <strong>70%<\/strong> for monolithic logs. By adopting similar measures, organisations can achieve scalable, efficient, and budget-friendly log management.&nbsp;<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"The_Road_Ahead\"><\/span><strong>The Road Ahead&nbsp;<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Here are our focus points for the near future:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Log Archival: Moving the logs from normal S3 to Glacier.<\/li>\n\n\n\n<li>Audit logs: Moving audit logs to the longer retention period for internal audits.<\/li>\n\n\n\n<li>Canonical Logs: Building customer debugging logs for move visibility.<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>At Cashfree Payments, we have multiple microservices running on Kubernetes, alongside a few monolithic services on EC2, both contributing to a significant volume of logs. These logs are essential for debugging and are also retained for extended periods to meet compliance requirements.&nbsp; However, this leads to escalating storage costs. In this blog, we share our<\/p>\n","protected":false},"author":125,"featured_media":28015,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"_themeisle_gutenberg_block_has_review":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_wpcom_ai_launchpad_first_post":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[1497],"tags":[],"class_list":["post-28014","post","type-post","status-publish","format-standard","has-post-thumbnail","category-engineering"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Cashfree Payments\u2019 Journey to Smarter Log Management<\/title>\n<meta name=\"description\" content=\"Learn how Cashfree Payments optimised log storage for Kubernetes and EC2 services, reducing costs while ensuring compliance and efficient debugging.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.cashfree.com\/blog\/smart-log-management-system\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Cashfree Payments\u2019 Journey to Smarter Log Management\" \/>\n<meta property=\"og:description\" content=\"Learn how Cashfree Payments optimised log storage for Kubernetes and EC2 services, reducing costs while ensuring compliance and efficient debugging.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.cashfree.com\/blog\/smart-log-management-system\/\" \/>\n<meta property=\"og:site_name\" content=\"Cashfree Payments Blog\" \/>\n<meta property=\"article:published_time\" content=\"2024-12-23T07:04:43+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-12-23T07:05:12+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2024\/12\/FLO_Tn%404x.png?fit=5002%2C3335&ssl=1\" \/>\n\t<meta property=\"og:image:width\" content=\"5002\" \/>\n\t<meta property=\"og:image:height\" content=\"3335\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Anant Singh\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Anant Singh\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"7 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Cashfree Payments\u2019 Journey to Smarter Log Management","description":"Learn how Cashfree Payments optimised log storage for Kubernetes and EC2 services, reducing costs while ensuring compliance and efficient debugging.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.cashfree.com\/blog\/smart-log-management-system\/","og_locale":"en_US","og_type":"article","og_title":"Cashfree Payments\u2019 Journey to Smarter Log Management","og_description":"Learn how Cashfree Payments optimised log storage for Kubernetes and EC2 services, reducing costs while ensuring compliance and efficient debugging.","og_url":"https:\/\/www.cashfree.com\/blog\/smart-log-management-system\/","og_site_name":"Cashfree Payments Blog","article_published_time":"2024-12-23T07:04:43+00:00","article_modified_time":"2024-12-23T07:05:12+00:00","og_image":[{"width":5002,"height":3335,"url":"https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2024\/12\/FLO_Tn%404x.png?fit=5002%2C3335&ssl=1","type":"image\/png"}],"author":"Anant Singh","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Anant Singh","Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.cashfree.com\/blog\/smart-log-management-system\/#article","isPartOf":{"@id":"https:\/\/blogrevamp.cashfree.com\/smart-log-management-system\/"},"author":{"name":"Anant Singh","@id":"https:\/\/blogrevamp.cashfree.com\/#\/schema\/person\/0cd69fa01cbfa7a371778ddae38ecf15"},"headline":"From Log Overload to Cost-Efficient Storage: Cashfree Payments\u2019 Journey to Smarter Log Management","datePublished":"2024-12-23T07:04:43+00:00","dateModified":"2024-12-23T07:05:12+00:00","mainEntityOfPage":{"@id":"https:\/\/blogrevamp.cashfree.com\/smart-log-management-system\/"},"wordCount":1111,"commentCount":0,"image":{"@id":"https:\/\/www.cashfree.com\/blog\/smart-log-management-system\/#primaryimage"},"thumbnailUrl":"https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2024\/12\/FLO_Tn%404x.png?fit=5002%2C3335&ssl=1","articleSection":["Engineering"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.cashfree.com\/blog\/smart-log-management-system\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/blogrevamp.cashfree.com\/smart-log-management-system\/","url":"https:\/\/www.cashfree.com\/blog\/smart-log-management-system\/","name":"Cashfree Payments\u2019 Journey to Smarter Log Management","isPartOf":{"@id":"https:\/\/blogrevamp.cashfree.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.cashfree.com\/blog\/smart-log-management-system\/#primaryimage"},"image":{"@id":"https:\/\/www.cashfree.com\/blog\/smart-log-management-system\/#primaryimage"},"thumbnailUrl":"https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2024\/12\/FLO_Tn%404x.png?fit=5002%2C3335&ssl=1","datePublished":"2024-12-23T07:04:43+00:00","dateModified":"2024-12-23T07:05:12+00:00","author":{"@id":"https:\/\/blogrevamp.cashfree.com\/#\/schema\/person\/0cd69fa01cbfa7a371778ddae38ecf15"},"description":"Learn how Cashfree Payments optimised log storage for Kubernetes and EC2 services, reducing costs while ensuring compliance and efficient debugging.","breadcrumb":{"@id":"https:\/\/www.cashfree.com\/blog\/smart-log-management-system\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.cashfree.com\/blog\/smart-log-management-system\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.cashfree.com\/blog\/smart-log-management-system\/#primaryimage","url":"https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2024\/12\/FLO_Tn%404x.png?fit=5002%2C3335&ssl=1","contentUrl":"https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2024\/12\/FLO_Tn%404x.png?fit=5002%2C3335&ssl=1","width":5002,"height":3335},{"@type":"BreadcrumbList","@id":"https:\/\/www.cashfree.com\/blog\/smart-log-management-system\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/blogrevamp.cashfree.com\/"},{"@type":"ListItem","position":2,"name":"Engineering","item":"https:\/\/blogrevamp.cashfree.com\/category\/engineering\/"},{"@type":"ListItem","position":3,"name":"From Log Overload to Cost-Efficient Storage: Cashfree Payments\u2019 Journey to Smarter Log Management"}]},{"@type":"WebSite","@id":"https:\/\/blogrevamp.cashfree.com\/#website","url":"https:\/\/blogrevamp.cashfree.com\/","name":"Cashfree Payments Blog","description":"Cashfree Payments- Payment Gateway for India","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/blogrevamp.cashfree.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/blogrevamp.cashfree.com\/#\/schema\/person\/0cd69fa01cbfa7a371778ddae38ecf15","name":"Anant Singh","url":"https:\/\/blogrevamp.cashfree.com\/author\/anant-singh\/"}]}},"jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p9MjXo-7hQ","jetpack_likes_enabled":false,"jetpack_featured_media_url":"https:\/\/i0.wp.com\/blogrevamp.cashfree.com\/wp-content\/uploads\/2024\/12\/FLO_Tn%404x.png?fit=5002%2C3335&ssl=1","_links":{"self":[{"href":"https:\/\/blogrevamp.cashfree.com\/wp-json\/wp\/v2\/posts\/28014","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogrevamp.cashfree.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogrevamp.cashfree.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogrevamp.cashfree.com\/wp-json\/wp\/v2\/users\/125"}],"replies":[{"embeddable":true,"href":"https:\/\/blogrevamp.cashfree.com\/wp-json\/wp\/v2\/comments?post=28014"}],"version-history":[{"count":2,"href":"https:\/\/blogrevamp.cashfree.com\/wp-json\/wp\/v2\/posts\/28014\/revisions"}],"predecessor-version":[{"id":28019,"href":"https:\/\/blogrevamp.cashfree.com\/wp-json\/wp\/v2\/posts\/28014\/revisions\/28019"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blogrevamp.cashfree.com\/wp-json\/wp\/v2\/media\/28015"}],"wp:attachment":[{"href":"https:\/\/blogrevamp.cashfree.com\/wp-json\/wp\/v2\/media?parent=28014"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogrevamp.cashfree.com\/wp-json\/wp\/v2\/categories?post=28014"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogrevamp.cashfree.com\/wp-json\/wp\/v2\/tags?post=28014"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}