This is the first of many (I hope) articles in which I will set aside the more technical and conceptual approach and speak a little more directly about FinOps. Often, we look at the FinOps content that is produced, and it seems that everything is always very conceptual, filled with keywords for SEO, and in some cases, even avoiding the risk of producing something that goes beyond the already established concept.
Without a doubt, my biggest challenge in executing FinOps has been, and still is, the automation of tasks. To put it better… the lack of it! With the aim of boosting the level of FinOps maturity, as well as considering the complexity of the cloud environment I work in, task automation has been a significant challenge with great returns.
This is because in recent months I have had the opportunity to work directly on improving various domains and capabilities in the company I work for, especially in topics related to understanding cloud usage and costs. The initial scenario was facing an Azure cloud environment with low cost allocation, inefficient metadata, no standardization, and no analysis of consumption and usage metrics. In a few words, the company was crawling in most domains and capabilities, but mainly in understanding cloud usage and costs.
I understood the urgency and priority of having accurate data through which the company could understand its cloud usage, and I established a routine, initially monthly, for collecting data from Cost Management + Billing, Azure Reservations consumption, Tag collection, and metrics of computing resource consumption. The first sprints were efficient, and the company gained approximately 90% visibility of allocated costs. Currently, Tags are being redefined, and rightsizing actions are being taken based on the collected consumption metrics. Important decisions, such as the cloud budget for the upcoming accounting periods, are being based on collected and accurate data.
Task automation has become my biggest challenge in refining this monitoring and reporting environment. This is because the current goal is to reduce the duration of sprints so that the phases of Inform, Optimize, and Operate are completed in increasingly shorter cycles. This becomes an effective and real necessity in environments where it is necessary to analyze metrics from hundreds of computing resources, for example, and the manual process of checking memory and CPU usage to make a rightsizing recommendation based on that assessment can take about 40 hours to complete.
To tackle this problem, I created an automation using the Azure Python SDK and the Pandas library to extract metrics via API, evaluate consumption data, and based on that, generate rightsizing recommendations. As a result, a task estimated at 40 hours is now completed in a maximum of 8 hours. This is because the FinOps Analyst remains an important piece in analyzing the data, assessing whether the recommendations generated by the automation are coherent and effective. The data collection itself takes no more than 20 minutes!
Satisfied with the result, I have been investing time in new automations, and I am currently working on a complete ETL structure, using the Pandas library from Python to gather necessary information for KPIs and real-time decision-making based on data, using the cost CSV generated by Azure. Among my main Dataframes are: Resources created in the period, resources deleted in the period, TOP 10 highest costs, instance reservation utilization, and period comparisons.
My goal is to automate everything from Data Ingestion to Reporting & Analytics, integrating various data sources, processing them, and using them to feed a Business Intelligence structure through the PowerBI tool.
In practice, automating tasks has allowed me to gain time to develop more precise strategies, improve professionally, and dedicate more time to meet client demands. If you have reliable automations, you can indeed have the freedom to study more and focus on developing new projects. When I automated part of my tasks, I began to dedicate more time to enhancing the delivery processes of FinOps for my clients, generating and standardizing methodologies, and sharing knowledge with colleagues.
Currently, I am dedicating myself to training other professionals, using part of the time that I would previously have spent on manual checks. My goal is to automate more! My goal is to do more! As a company matures in FinOps, automations become a measure of success.
Trust, while being cautious
The only tip I leave for you is to be coherent and precise in your analyses. You have already saved time consolidating information, so you don’t need to save time analyzing the results. Task automation is not about gaining time for idleness, a gastronomic outing at the company, or anything else.
Task automation is about gaining time to analyze data accurately, with shorter delivery times. Make no mistake… Shorter delivery times! The evaluation MUST BE PRECISE. I said in another post, and I repeat it here: you do not make mistakes with other people's money. Check the data once, twice, three times! Take advantage of the fact that by automating tasks you have gained more time to do this!
Check the original article in Portuguese here
