Welcome to the Learning FOCUS blog series. If this is your first post, I recommend you start with Introducing an open billing data format to get a high-level picture of what the FinOps Open Cost and Usage Specification (FOCUS) is and what it covers. This week, I’ll cover the columns that help you identify and describe the cloud resources that are incurring charges. Resource usage makes up the vast majority of most costs, so these columns will be used for many scenarios. Buckle up! There’s a lot to cover this week…

⮜ Previous post (Dates)   ·   Next post (Services) ⮞

Identifiers, names, and types in FOCUS

As we cover new columns, I like to take a second to explain common conventions you’ll find across the schema. One example is how FOCUS handles identifiers and names; another is how FOCUS handles “types”.

In FOCUS, when there’s an entity that could have an ID and a name, the columns are always named *Id and *Name. This is pretty straightforward, so there’s not much to explain really. Today, we’ll discuss ResourceId, ResourceName, RegionId, and RegionName. You may have noticed a few “Name” columns that don’t have corresponding IDs in the FOCUS schema. Those were named as such to allow for the addition of ID columns in a future release. We’ll touch on those later.

We discussed “type” columns when I covered charge types and pricing models. FOCUS has “category” columns and “type” columns. Columns that use provider-specific terms are named “Type” and columns that use a predefined set of provider-agnostic terms are named “Category”. When it comes to resources, we only have ResourceType today, which is a provider-specific value, but there have been discussions about adding a ResourceCategory for provider-agnostic terms for the different types of resources.

One other thing to call out about type columns – and this applies to many columns – you’ll notice that most columns have friendly display names or labels rather than programmer-friendly enumerations (e.g., “MyLongValue” without spaces) or codes (e.g., “my-long-value”). I’ll touch on that later.

Resource ID and name

In general, you’ll find that most of what you’re charged for are tangible things. With a strong foundation in cloud computing, FOCUS has adopted the term “resource” to represent those tangible things that you deploy and use. Given most are familiar with cloud resources, I won’t belabor the point here but let me know if anyone would like a more detailed explanation and I can expand on that in a future blog post.

ResourceId is a unique string that represents a tangible thing you’re charged for. In the cloud, this is a deployed resource, like a virtual machine or database. If you’re generating FOCUS data for on-premises costs, that could be a server or even a building, depending on what you’re tracking. (That may sound too high-level, but let’s save that for a future blog post if anyone’s interested in how to generate FOCUS data for non-cloud costs.) In Azure, the ResourceId is a globally unique identifier that can be referenced programmatically, which makes it very useful when joining with other datasets. Microsoft’s FOCUS dataset lowercases all ResourceId values to ensure consistent grouping and filtering.

ResourceName is a custom name for the resource. You can think of the ResourceName as a friendly name, but it isn’t precisely a display name, given they are usually part of the ResourceId (at least for Microsoft cloud resources).

Anatomy of an Azure resource ID

Before we talk about resource types, I want to take a minute to explain what an Azure resource ID is. This is very useful information as it helps you understand how resources relate to each other, how access is managed, and even how inheritance is applied in some cases like policies and Cost Management tag inheritance.

As I mentioned, Azure resource IDs are globally unique identifiers. Resource IDs usually include a subscription ID, resource group name, resource type, and resource name. However, some resources are deployed outside of resource groups or even subscriptions, and may not have one or both of those constructs. The typical structure for an Azure resource ID is:

/subscriptions/{sub-id}/resourceGroups/{rg-name}/providers/{provider-id}/{type}/{resource-name}

Azure resource types may also be nested. For instance, let’s say we create a resource group named Foo and deploy a SQL database named Baz into a SQL server named Bar. The ResourceId for the database would be:

/subscriptions/###/resourceGroups/Foo/providers/Microsoft.SQL/servers/Bar/databases/Baz

Azure resource IDs consist of list-instance pairs. In this example, they are:

Understanding this hierarchy is helpful because it informs you what objects a resource “belongs” to and may inherit settings from. As an example, if I have Reader access to the Bar server, then I also have access to all nested databases, but I don’t have access to everything in the Foo resource group. You’ll see similar behaviors across other cross-cutting services, like Azure Monitor, Azure Policy, and Microsoft Cost Management.

And one thing that’s interesting about this example is the nested resource – the SQL database is a child to the SQL server. And since there can be other SQL databases with the same name, you’ll notice that nested resources are treated slightly differently when it comes to their name. ResourceName for nested resources includes the parent resource name. For instance, our database would be “Bar/Baz” to indicate the server/database parent-child relationship.

This is also represented in Azure resource types. An Azure resource type is a code that indicates the resource provider and the kind of resource it is. For example, a virtual machine would be “Microsoft.Compute/virtualMachines” while our nested SQL database is “Microsoft.SQL/servers/databases”. You can technically extract this from the resource ID by trimming everything before and including the last “/providers/” reference and removing all the type name values.

For those curious, the default resource provider when there isn’t a “/providers/” value is “Microsoft.Resources”. For instance, “Microsoft.Resources/subscriptions” for subscriptions.

Understanding how resource IDs work can be helpful. And these fully qualified resource IDs are used throughout the FOCUS dataset, so expect to hear more about them as we cover other object identifiers.

Resource types

So far, we covered ResourceId and ResourceName, we discussed how the name is part of the resource ID for Azure resources, and we also introduced the concept of a resource type that’s also part of the resource ID. Now, let’s look at how resource types differ between Azure and FOCUS.

In FOCUS, ResourceType is a user-friendly display label for the type of resource. You’ll see values like “Virtual machine” or “SQL database”. The string is a capitalized, singular display name and very explicitly not a code. This is another common convention you’ll see in FOCUS. FOCUS generally avoids “codes” or enumeration-style strings that don’t use spaces. FOCUS prefers human-readable display strings to make data more approachable to people of all backgrounds.

Given this requirement for ResourceType to be a human-readable display string and how important the Azure resource type is, Microsoft also includes an x_ResourceType column with the Azure resource type code, like “Microsoft.Compute/virtualMachines” or “Microsoft.SQL/servers/databases”.

This is a core principle with the Microsoft implementation of FOCUS. We believe your FOCUS dataset should have no compromises. You should have everything you need from the native formatted dataset directly in your FOCUS dataset.

Resource type metadata outside of FOCUS

To share a bit of the internals, Microsoft Cost Management implemented a transformation layer that joins and converts the cost and usage data from an internal schema to align with FOCUS. And since FOCUS introduces a few new concepts that aren’t part of the underlying systems, there was a need to create additional resources to facilitate the FOCUS transform. And since this transform is useful in many scenarios, such as the FinOps Foundation FOCUS converter project, we published an open data file that contains resource type metadata to facilitate mapping resource types to display names, GA state, descriptions, icons, and useful links.

While the Microsoft FOCUS dataset does this mapping for you, you may also find this mapping file useful for custom tools and reports that need a more polished touch. To learn more, see Resource types open data in the FinOps toolkit. Resource types are updated with each release to account for new resources. If you feel any are missing are should be updated, please create an issue in GitHub and we’ll strive to update those for the next release. We also welcome and encourage direct pull requests to update these values.

Other resource properties

Beyond the ID, name, and type, there are a few other common properties of resources that I’d like to touch on briefly, but I won’t spend much time on since I’ll cover each of them in more detail in future blog posts.

RegionName is the human-readable display label and RegionId is the internal identifier for the Azure region the resource was deployed to. Similar to ResourceType, RegionName contains user-friendly strings with spaces (e.g., “Brazil South”, “Japan East”). On the other hand, RegionId for Azure is the lowercase RegionName without spaces (e.g., “brazilsouth”, “japaneast”). Note that RegionId values may be different for other providers. FOCUS does not require any specific formats for identifiers.

Tags is a JSON column with the key-value pairs for user- and system-defined tags for each resource. This behaves the same as the existing Tags column from native datasets, except that the FOCUS Tags column is consistently formatted with curly braces (“{…}”) to facilitate parsing.

SubAccountId and SubAccountName are the subscription ID and name that the resource was deployed to. SubAccountId uses the fully-qualified Azure resource ID format we discussed earlier rather than a simple GUID.

Lastly, x_ResourceGroupName is the name of the resource group the resource was deployed to, if applicable. Note that some resources may not be deployed to resource groups, so this value may be null.

I’ll discuss each of these in more detail in a future blog post.

Charge types and pricing models

Now that we’ve covered resources, which make up the majority of cloud costs, it’s probably worth reviewing how resource columns relate to the other columns we’ve covered.

In most cases, you’ll find ChargeCategory is “Usage” when ResourceId is not null. While it is possible for purchases to have a resource ID, this information is only available in a few cases, like with commitment discount purchases. Generally, when you are looking at a list of resources, you may want to filter down to ChargeCategory == “Usage” or possibly where ResourceId is not null, depending on the question you’re trying to answer.

PricingCategory is also useful for filtering down to the “Committed” costs from commitment discounts or analyzing Spot VM usage using the “Dynamic” value. We’ll discuss commitment discounts in a future blog post, but it’s important to keep this relationship in mind as it will be

Transitioning to FOCUS

Whether you’re updating reports, transforming data, validating FOCUS, or simply curious about how FOCUS compares to the historical actual and amortized datasets, you’re probably looking for a more direct mapping of columns. We have separate articles covering each of these scenarios in more detail, but here’s a summary regarding the date columns I covered above.

For more details, refer to the following articles:

• Update reports to use FOCUS columns
• Convert cost and usage data to FOCUS 
• Validate FOCUS data 

Reviewing cost in Power BI

Resources are the core functional unit in the cloud, so you can imagine there’s a lot to dig into now that we’ve covered resource columns. We’ll use the FinOps toolkit Cost summary report again.

Let’s begin with the most obvious starting point… The Resources page shows the resources that are incurring the most cost within your accounts. You’ll see many of the concepts that we covered above, like the name, type, region, resource group, and subaccount (subscription). This report also summarizes savings and the Effective Savings Rate (ESR), but we’ll discuss those more in a future blog post.

Next, I’d like to pivot into a slightly different perspective of our cost and look at the Resource inventory page. The Resource inventory page summarizes cost by resource type but also provides details about how many resources you might have and what the average cost per resource is over the period. You may notice the resource types in the table are grouped, but we won’t cover that today.

I could also cover the Resource groups, Subscriptions, or Regions pages since we quickly touched on each, but I’ll hold off and cover them in more detail in future blog posts.

While these examples are fairly simple, I hope they give you some ideas of ways you can use resource columns to track your costs back to the resources that incurred the costs. This will get even more interesting when we cover SKUs, since those are the true source of incurred costs, but we’ll discuss that more in the future.

To learn more about these and other reports, see FinOps toolkit Power BI reports.

Querying cost in FinOps hubs

Now let’s look at a few queries you can run using FinOps hubs with Data Explorer. Investigating resource costs is one of the most common first steps as people begin to analyze their costs and unexpected changes.

We’ll start with a simple list of top resources with the most cost this billing period.

Costs
| where BillingPeriodStart == startofmonth(now())
| summarize
    ResourceName = take_any(ResourceName),
    ResourceType = take_any(ResourceType),
    x_ResourceType = take_any(x_ResourceType),
    RegionName = arraystring(make_set_if(RegionName, RegionName != 'Global')),
    x_ResourceGroupName = take_any(x_ResourceGroupName),
    SubAccountName = take_any(SubAccountName),
    EffectiveCost = round(sum(EffectiveCost), 2)
    by
    ResourceId
| order by EffectiveCost desc
| limit 10
| project-away ResourceId

I should probably explain a few things from this query:

• I used take_any() to select some of the values that don’t change for each resource ID, like name and type. This is more efficient and reduces computation compared to adding additional columns after the “by” keyword, which speeds up the query.
• I used make_set_if() to filter out global bandwidth charges and build a collection of the remaining regions since some resources can be moved between regions.
• The arraystring() function is a custom function that converts an array to a comma-delimited string for display purposes.

Next, let’s try something a little more interesting and get a list of the top resources incurring the most cost that were created within the last 3 months.

Costs
| summarize
    ResourceName = take_any(ResourceName),
    ResourceType = take_any(ResourceType),
    x_ResourceGroupName = take_any(x_ResourceGroupName),
    SubAccountName = take_any(SubAccountName),
    ChargePeriodStart = min(ChargePeriodStart),
    EffectiveCost = round(sumif(EffectiveCost, BillingPeriodStart == startofmonth(ago(1d))), 2)
    by
    ResourceId
| where ChargePeriodStart >= startofmonth(now(), -3)
| order by EffectiveCost desc
| limit 10
| project-away ResourceId

A few new things in this query are:

• I used min() to get the earliest ChargePeriodStart date for each resource ID.
• I changed the sum(EffectiveCost) to use sumif() so we could get the cost of only the current billing period when sorting by cost.
• I filtered ChargePeriodStart after summarizing to make sure we get the earliest date. Just keep in mind you can only go as far back as your exported data.

We can also chart this for an interesting view of how many resources are created over time:

Costs
| where isnotempty(ResourceId)
    and ChargeCategory == 'Usage'
    and CommitmentDiscountStatus != 'Unused'
| summarize
    ChargePeriodStart = min(ChargePeriodStart)
    by ResourceId
| summarize
    x_ResourceCount = dcount(ResourceId)
    by ChargePeriodStart
| where ChargePeriodStart > toscalar((Costs | summarize min(ChargePeriodStart)))
| render columnchart

And a few notes:

• I filtered out non-usage and non-resource rows. (I’ll explain CommitmentDiscountStatus in a future blog post.)
• I calculated start dates per resource ID first, then counted the unique values by start date. This double-summarize pattern can be quite useful, albeit sometimes confusing to work out.
• I filtered out the oldest date since everything remaining would have that value.

There are a lot of things we can do with resources. We’re just scratching the surface. But hopefully this will help spur some ideas or show you how you can get an answer to a question you have.

As always, we’ll continue to build on resource columns in future blog posts to demonstrate more interesting scenarios as we go. Leave a comment if you’d like to see any specific queries and I’ll make sure to cover it.

What next?

At this point, we have a high-level understanding of the types of charges we’re incurring, how much we’re being charged, when we incurred those charges, and what resources we deployed that incurred the charges. Next, we’ll dig into the services these resources roll up to.

If you need a refresher or have any questions about previous topics, this is a good time to review them. We’ll touch on a little of everything given the overlapping concepts.

• Introduction
• Costs
• Charge types and pricing models
• Dates

For a more directed walkthrough, the FinOps Foundation offers a free Introduction to FOCUS course. When you’re ready to dig into your own FOCUS data, check out the Power BI reports in the FinOps toolkit. These reports offer a great starting point that you can customize to meet your needs. And if you’re looking for more advanced analytics that can handle data at scale, check out FinOps hubs, which offer additional benefits, like pre-calculated savings for EA and MCA accounts.

 

⮜ Previous post (Dates)   ·   Next post (Services) ⮞