Back in the way back when, when killing Prince Malchezaar was a personal accomplishment, or even before, when all I wanted in the world could be found in Quagmirran’s Eye, I was content in my ignorance about performance. Fun and output were loosely correlated at best. But as I progressed into the world of competitive raiding, and then further still into a world where teams feel like failures if they fall out of the US top 20, I started to recognize the edge that comes from continual performance review. And when, about 2 years ago, I started with a firm that prides itself on the ability to evaluate projects based solely on key data points, the value of metrics finally solidified itself in my mind.
Over the years, I’ve read, heard and participated in a number of arguments about the pros and cons of meters. And nothing is more sure to get a response from a complete WoW stranger than linking a meter in chat and commenting on some perceived shortcoming. Even Ghostcrawler himself has weighed in on the topic saying:
Trying to evaluate how awesome a healer you are by looking at healing meters is extremely dangerous. Heck, it’s even dangerous to compare dps if you don’t look at what’s really going on. (Source)
And this oldie but goodie (wonderfully enough, from a Shaman thread about Ulduar):
The moral of the story is meters are very useful, but like any tool, their ability to measure what happens in reality has limitations. In my experience, players put too much emphasis on them, especially for healing. (Source)
Now I’ll credit GC with being an incredibly intelligent individual and certainly one of sufficient patience to put up with the attacks that frustrated and angry players level at him day in and day out. And I’d bet that at one point or another he’s had to delve into some class performance data to evaluate how much “nerfed to the ground” actually is. But on this point—on the topic of if meters can truly give you an accurate picture of a player’s performance—I have to respectfully disagree with him, and likely, many of you. Meters, and the metrics they present, absolutely matter. They’re not perfect at present, but I have some ideas of how they can get there.
(Do note: when I say “meter” in this post, I am using it as a general term encompassing every combat log parser out there—from real-time addons like Recount and Skada, to detailed online tools like WoL and WMO.)
Why do we need meters?
I am, quite unashamedly, an obsessive trinkerer. I am constantly looking for new ways to improve how I do things—from repetitive data handling (I love you, Visual Basic!) to casually tracking the fastest way to get home in rush hour traffic (bah, traffic!) In WoW, this means I generally spend a good deal of time and energy thinking and exploring ways to improve my performance. (Hell, I even come up with “optimal” farming routes, for those times when I want to zone out and still be productive.)
But I understand that this perspective isn’t one that’s shared by everyone. And in the 9th month of running ICC, even I abandoned the idea of process improvement long ago, content to simply enjoy the ride until Cata’s release. And yet I will still argue that meters are valuable to everyone, regardless of whether they are casual or uber-hardcore. Why?
The large majority of people think they’re above average.
Of course, you’re above average, I’m just talking about those other fools who don’t read my blog. Just like I’m talking about those other baddies who are the root of all evil in WoW and do things like behave badly, afk during trash, or ninja log during a bad LFD group. You and I would never do that … *cough* My point here is, you and I are both guilty of largely overestimating our performance and our own skill. It’s why each and every one of us can recount a tale of absolute baddies but casually dismiss our own bad behavior and choices. It’s also the same reason players point to meters and say, “that’s not representative of my contributions to the team”, because how we feel about our performance and how the numbers actually play out, are two separate things entirely.
This effect (Illusory Superiority) provides an explanation for something we all have trouble acknowledging—meters are important because we stink at gauging how good, or bad, we actually are. (For more interesting reading, check out: Why we overestimate our competence and Inaccurate Beliefs About Learning and Memory)
(Now let me stop you for one second before you race to the comments section to advise me just how wrong I am—this is not a post about the use or misuse of meters. This post is not about how capable we are of objectivity. This is about why and how we can use meters to help us become better players. Do read on, I’ve got some ideas in that regard.)
So what do meters tell us?
Aside from the obvious benefits to epeen enlargement, meters actually provide a good window into player performance, more than they’re really given credit for. Even if you’re not willing to delve too deep in your analysis and break out the comparison spreadsheets and data charts, most meters out there can still give you a good selection of information to look at. Aside from the “Damage Done” and “Healing Done” options, meters also generally display the following (in some way or another):
- Damage taken and healing taken (by raid member, by spell)
- Enemy damage done (to raid members, by spell, by percent)
- Dispels (by raid member)
- Interrupts (ohai, did we all forget about this one? Even you former RoS rogues?)
- Activity (by raid member)
- Overhealing done (by raid member)
- Targets healed (by raid member)
Now, I absolutely concede that these meters, as they currently exist, make it hard for players to see the full picture of a fight or instance, because it’s easy to get lost within all of the twists, turns and nuances of the average parse. But to say that the above metrics are useless because “they don’t tell the whole story” is simply a load of hogwash.
Meters, and combat logs, are the raw data of an encounter, without any of our personal bias thrown in. “Vixsin totally made the heal that saved the tank and prevented a wipe” isn’t appended on any of the lines, nor will my epic dodge of a Shadow Trap be recorded as a +1 Skill anywhere on the Skill Meter. But instead of seeing that as a bad thing—as players often argue—I instead see that as something positive. Yes, winning as a team is about group performance, and there is no I in a healing team, but if that were the long and short of the story, then there’d be no reason to track sports statistics, right?
So when GC or anyone else says that meters are dangerous, I instead take it to mean that he and they are talking about times where someone links a dps meter (with himself at the top, naturally) and concludes that he wins at the internet. One meter, one metric for that matter, does not tell an entire picture, this much is true. But why can’t we create a series of indicators that do?
A better perspective
As I see it, the problem with current combat log parsers centers around a couple key flaws:
- Problem: Meters contain little hierarchy of information. At their most detailed, they are like full-blown, detailed, cross-referenced, and avidly footnoted thesis, with absolutely no abstract or summary, no chapters, and no conclusion. In a business environment, this would be like lacking an Executive Summary, so named because company executives rarely have the time to read your 120 page report on the forensic analysis of costs on the new private performing arts center. They need the facts, your findings and your conclusions, and they need it in as few words as possible.
- Problem: The established gauges of excellence “DPS” and “HPS” are insufficient standards. DPS and HPS are not latched on to as “good” metrics because players understand the values they represent (and they represent a limited world view at best). They are latched onto because they are easy to understand (more dps kills bosses faster) and easy to use as a point of comparison (HPS = healing version of dps). But they are exactly that, one point.
- Problem: There is little way to cross-compare personal or guild performance. Oftentimes, looking at your own performance from week-to-week or looking at several guilds’ performance on a specific encounter leaves you struggling to identify similarities and differences within a multi-faceted parse. This means that tracking your own performance gains is boiled down, once again, to the easy and fast metrics of HPS and DPS, disregarding the supplemental factors that influenced that single number.
So what’s the solution? New metrics, and more of them.
Ultimately, the goal of any metric is to provide a system which allows raid leaders and raiders themselves to evaluate performance relative to their teammates, data groups, class, etc. But this data lacks context unless it can be compared to other data sets in an attempt to quantify improvements or slippages. So, in order to do this, we need to establish common, yet influential, performance elements, with apply in just about every boss fight, like the following:
- % activity / 100%
- Type: player-to-player comparison
- Benefit: An easy metric to understand how good a player is at staying alive (or how loved he/she is by healers). This is already something shown on WoL and WMO, but currently can’t be compared on a parse-to-parse basis.
- Best Score: 100 (for surviving the whole encounter)
- Total resources expended / 12 seconds
- Type: class-to-class
- Benefit: Although I’m not sure this would be a point of comparison that would be valid for death knights (maybe on a limited basis for runic power), this would allow most players to evaluate their resource management against other players of their class. Was one rougue able to eek out more dps because of his energy use? Was one healers’ regen far surpassing another healer of the same class?This metric would ultimately identify which players are doing more for less (a vital find at least for mana users in Cata).
- Best Score: (no limit)
- [(your damage to target1 / total damage to target1) + … (your damage to targetn / total damage to targetn)] / total number of targets to which damage was done (for healers, replace damage with healing)
- Type: player-to-player
- Benefit: It’s an oftentimes forgotten aspect of dps and healing performance, but the number of times you switch targets directly correlates to a reduction in throughput for both dps and hps. So, a focus metric not only allows you to easily determine whether or not you’re comparing a raid-healing pally to a tank-healing pally, but also evaluate on a basic level whether or not your healers and dps are doing the jobs that they should be doing.
- Best Score: (depends on assignments)
- (your # of interrupts, CC, cleanses) / (total # of interrupts, CC, cleanses)
- Type: player-to-player and player-to-team
- Benefit: It goes without saying, but sometimes performance of a simple task makes all the difference in the world (interrupts on Vezax, for example). And when these types of mechanics crop up in Cata, and they will, it will behoove you to know which members of the team are only working to further their own numbers and which are managing to multitask.
- Best Score: 100 (representing that you did all of the interrupts in an encounter)
- your damage taken / (total damage taken – outliers)
- Type: player-to-player
- Benefit: In an environment where mana matters, the damage your partners and team takes will start to have a pretty sizable impact on the success of the team. So, while this metric might not be valuable when evaluated for a single raid night, when looking at weeks of raids, you’ll be able to quickly see who has a knack for standing in fire and who is really worthy of the healers’ adoration.
- Max Score: 0 (you took no incoming damage during an encounter)
Personal and Team Values (at least for Healing)
- Personal value: activity / effective healing (Max of 100)
- Team value : (effective healing) x (% of total healing done) x (activity) x (normalization factor) so that ∑ team values = 1
- Type: player-to-player
- Benefit: Playing for a team is a delicate balance, and oftentimes when you start looking at “top parses” it’s easy to forget that personal accomplishment comes at the price of team accomplishment. The DPS that was Power Infusion’ed to the top of WoL got there because he also stood in fire, and ignored adds, not because he was particularly adept at his rotation. Likewise, the healer who deviated from a tank-healing assignment to raid-snipe (and thus caused other healers to have to supplement tank heals), was working more for himself than the team. By creating complimentary metrics for personal and team “worth” you can not only see performance gains within your team, but between players and between parses from week-to-week.
Ultimately, I think establishment of the above metrics could contribute to a better analysis of performance, and provide a clearer picture of where teams are succeeding and where they’re falling short. Inclusion of each of the above, on a summary page, would allow a player to quickly and easily assess themselves and others. True, you’ll never be able to see the value of that “one ZOMG EPIC clutch heal” but then, who’d want to reduce their healing contributions down to one twitch moment anyways?
So, I guess after all that schpiel, the only thing left to say is … so who knows lua?