SWTOR Combat Log Enhancements
Changelog
March 28, 2021 - Added new section for the bare minimum route.
April 28, 2020 - Added section about overheal and overkill.
April 27, 2020 - Added section about shields and absorbs.
April 21, 2020 - Added section about not exiting combat properly in Operations.
April 17, 2020 - Added raid target markers to the list of resources dumped with source and target units.
April 6, 2020 - Added a new high priority issue. In-order event logging. Heals are not logging correctly relative to the damage they are healing.
March 27, 2020 - Added a new section describing issues with Spend and Restore, and also added a section discussing implications of anonymity.
March 22, 2020 - Added new sections describing issues with AbilityActivate and AbilityInterrupt, and also added a section discussing dispels.
March 17, 2020 - Added suggestion for dumping the 9 Utility abilities in the PlayerInfo event section. Mentioned % absorbs in the absorbs section.
Introduction
The purpose of this document is to outline what features need to be added to the SWTOR combat log in order to support all of the features on swtorlogs.com. These tasks are broken up by priority. The high priority tasks are very important, and the lower priority tasks are all optional. The two most important requests are that the log include the actions of all players and all NPCs/enemies, and that the log be enhanced with health and position data for all units.
Update (March 2021) - The Bare Minimum
I wanted to talk about what would be the bare minimum needed to make a functional site, since this task list is way smaller than what would be required to make a feature complete site. If we just drop lots of capabilities and stuff and accept that the logs could be easily hackable, then we can end up with something much smaller in terms of requirements.
If we just assume people have to log for their contribution to be shown, and that I handle merging the logs somehow, then all that's really left is being able to identify the fight and whether or not it was a kill or a wipe.
You would need make only the following changes:
- Log the deaths of all NPCs. Do not only log the deaths of NPCs that the logging player killed. This doesn't leak any info and is safe to do. Doing this guarantees I see boss deaths so that I can confirm kills.
- Include the flashpoint/operation difficulty and size in EnterCombat so I can identify what I'm looking at.
That's it. I would then have enough info to put up damage parses etc. for specs even if nobody else in the run ever uploads.
The Original Document for a Full-Featured Site
High Priority Tasks
The tasks listed here should be considered very important in order to make swtorlogs.com a reality.
Add a BeginLog Event
Add a BeginLog event whenever a log is started. The BeginLog event should be written out when the log first starts and should include the following information:
- Log version. By having a log version number that can be revved, you can make changes to the log format and then rev the version, and I can know what version of the log I am looking at. This makes it easier to make breaking changes in the future.
- The logging player AND their server, e.g., Kihra, SateleShan
- The full date and time that the log started, .e.g, 3/6 2020 11:43. Right now the only way to know when a log started is to study the name of the file. If someone renames the file, you break the ability to know when the log was made. Adding a fully qualified date at the start of the log will allow the log file to be renamed safely. Alternatively you could just make every log line a fully qualified date.
- [Optional] The game build (either your internal build number or the public version ,e.g., 6.1).
- [Optional] The language used in the log file: en, fr, de.
Log All Players And NPCs
The combat log should include events for all players and NPCs. Players that opt out (this should be the default) are anonymous and should be dumped with a special name that clearly indicates they are anonymous (and not just a player with the name Anonymous).
Anonymous players need to be distinguishable from one another. The way ESO did this was by using a session-unique ID for the lifetime of the log file. This would mean you'd dump something like @@Anonymous32451301, or @@Anonymous { 32451301 }.
[@@Anonymous { 324451301 }]
Implications of Logging All Players and NPCS
You currently expose other players' info in healing events. If you decide to go with this Anonymous approach, you will actually need to tighten this up and make the healing sources/targets anonymous when appropriate. In other words, it's important that you don't sometimes dump a player as anonymous and sometimes dump their real name. You'll have to be 100% consistent.
Starparse will undoubtedly be updated to show everyone's DPS from the log. Once you put all players' info into the log, even with anonymity, it will sometimes be possible to infer which player is which (e.g., if there's just one of a class/spec in the raid). You'll have to decide (as ESO did) that you're ok with this. If this is a cause for concern, you could enable this logging mode only for Operations (and not for Flashpoints), although I would provide metrics and ranks for Flashpoints also if you did allow it. Ultimately this is your decision to decide how you want to handle anonymity.
The way other games handle logs once they dump info for all players is they dump them in "near-real-time" and not actual real-time, i.e., they allow some buffer to fill up before flushing. This prevents things like boss timers from being made by a third party app. They then flush immediately once combat ends and the raid has wiped or been victorious.
Include Hit Points and Positions of all Units
Include a new resources section for units in every log line. At a bare minimum this should include current/max HP, current absorb strength (e.g., sorcerer bubble), the current target marker on the unit (e.g., gear, saber, gun), and x/y/z position and facing (in degrees or radians). Dump a comma-separated list in parentheses after unit name and id that includes all of these fields.
[@Kihra (85201/85611, 4324, 0, 5.245, 3.623, 2.124, 0.541)]
Here is a second example for an NPC:
[Zakuul Probe {3866097631625216}:25300011083961 (85201/85611, 4324, 0, 5.245, 3.623, 2.124, 0.541)]
Note: HP and absorb values for a target unit should be the updated values after the target unit has taken/absorbed the damage or received the healing (for damage/healing events).
Deliver Events in the Correct Order
This is hugely important and needs to be addressed at the very least for healing and damage events.
You need to ensure correct delivery of healing and damage events relative to one another and relative to deaths and buff/debuffs applications and fades. In the example screenshot to the left, the Kolto Shell is clearly healing damage taken from the second Flurry that was logged AFTER the Kolto Shell!
You can tell this by the amount of threat generated by the Kolto Shell ability. My current algorithm for computing overheal is just relying on health deficits, which would work if the events were delivered in order. Even if I started using threat instead, the out of order event delivery is going to make it impossible for you to attach the correct hit points and position to each event.
In general it looks like the issue is with reactive abilities, e.g., reactive healing like companion Kolto Shell, and the reactive heal that juggernaut tanks can get when healing under 70% health. Saber Reflect is another reactive ability that seems to dump the reflected damage before the taken damage. It just seems like there's some simple bug with reactive abilities that is causing an out of order event issue.
Overheal and Overkill
Overheal and Overkill need to be included for all events. For example if a player is at 149,500 out of 150,000 health, and gets healed for 5000, 4500 of that is overheal. A similar principle applies for overkill in damage events. If an enemy has 100 health left and is hit for 1000, 900 of that is overkill. You need to enhance your events to show overheal and overkill values.
Add overheal and overkill when present by placing them after the real damage separated by a colon.
14:15:56.301] [@Dahri] [@Dahri:Risha {404542969610240}] [Kolto Cloud {1147159994957824}] [ApplyEffect {836045448945477}: Heal {836045448945500}] (288:288)The above example has 288 overheal, so that value is included after the raw heal with a : in between them.
These values only need to be dumped if present, so you can just omit them if there is no overheal or overkill.
Support for Absorbs
I need to be able to credit healer absorbs as healing.
The Easy Approach
Add the absorb strength to the ApplyEffect of the absorb buff, e.g., if a bubble is going to absorb 24,000 damage, then in the ApplyEffect of that buff, put the number 24,000 somewhere. Put the amount remaining on an absorb shield in the RemoveEffect. This allows me to know the total amount absorbed by subtracting the number in the RemoveEffect from the number in the ApplyEffect. I[11:43:58.551] [@Kihra] [@Kihra] [MyAbsorbAbility {3569530139836416}] [ApplyEffect {836045448945477}: MyAbsorbAbility {3569530139836416}] (24000 -absorb)
Put the remaining absorb strength in the parenthetical section of a RemoveEffect.
[11:43:58.551] [@Kihra] [@Kihra] [MyAbsorbAbility {3569530139836416}] [RemoveEffect {836045448945477}: MyAbsorbAbility {3569530139836416}] (6243 -absorb)
The Harder Approach
Create new events that fire concurrently with a damage event that dump the source (healer), the target (the person who took damage), the ability that did the absorbing (e.g., sorcerer bubble), and the amount it absorbed. With this approach, you would still want to dump the info outlined in "The Easy Approach" so that overheal could be computed for absorbs. These additional events would just provide more detailed information about which absorbs were working to absorb which damage abilities. They enable a feature on my site called "connected damage and healing" where I am able to show exactly what damage someone was absorbing or healing.Note I am fine with not having overheal for absorbs if you decided to just do the harder approach. It is superior in the sense that you know exactly what enemy ability got absorbed and which ability did the absorbing.
Fix Shield and Absorb Numbers in Damage Taken Events
Events can be shielded by tanks, but often the absorbed value isn't the amount shielded. Sometimes it's an actual absorb ability listed when a shield occurs in conjunction with an absorb (like Sonic Barrier). Other times it's a gigantic value that far exceeds the size of the original hit pre-shielding. This is possibly because you calculate the shielding value before damage reduction from armor, etc.
Ideally you'd fix this to stop conflating absorbing and shielding and express those values uniquely as separate fields. In other games like WoW, shielding (i.e., blocking) is just its own field that is distinct from absorbs. It's obvious looking at the log that you can both shield and absorb a hit at the same time, and ideally you would show both the shielded amount as well as the absorbed amount. Don't overload them into a single field.
From an analysis perspective, it would also be nice if you fixed the shielding value to account for damage reduction, so that it isn't this gigantic number.
Buff and Debuff Stacks
Information about buff and debuff stacks is missing from the logs. My suggestion for this is to add a new event called ModifyEffect that will fire when stacks change. For example, Kinetic Ward is 15 stacks initially, so this could be included in the ApplyEffect event or with a separate ModifyEffect that fires concurrently to establish the initial stacks.
[11:45:12.403] [@Kihra] [@Kihra] [Kinetic Ward {980828796485632}] [ModifyEffect {836045448945477}: Kinetic Ward {980828796485632}] (15 stacks)
Whenever the stack count changes, a ModifyEffect would fire, e.g.,
[11:45:13.102] [@Kihra] [@Kihra] [Kinetic Ward {980828796485632}] [ModifyEffect {836045448945477}: Kinetic Ward {980828796485632}] (13 stacks)
AbilityActivate
The AbilityActivate event needs to fire for NPCs. I asssume it's just being filtered out right now because of combat logs being limited to your player only, so this will be something to check once you enable the log for all players and NPCs.
In addition, the AbilityActivate event has a few issues that should be corrected.
The target of an activation event is always the source. Ideally this would be changed to be the actual target of the ability when a target is required. For example if I am casting Project on a specific target, that NPC should be the target of the event and not me. When a cast is targetless, e.g., a Sniper's Suppressive Fire, just dump [] as the target, since the ability doesn't require one.
When an ability has a cast time, AbilityActivate fires at the beginning of the cast, but nothing fires when the cast completes. AbilityActivate should be enhanced to indicate when a cast time exists and dump that cast time, so for example with a Sniper's Snipe ability, it could dump the expected cast duration in seconds, e.g., 1.5.
- When an ability completes casting (e.g., like a Sniper's Snipe, another event should fire, e.g., AbilityActivateCompleted, that indicates that the cast has really finished.
[Optional] Handle channels (e.g., Suppressive Fire, Series of Shots) as well as actual casts (e.g., Snipe). To handle a channel, you'd just need to have a flag on the initial AbilityActivate that indicates that the spell is a channel and then dump the duration just like you do for casts. I don't really handle channels well on my site, so I consider this fully optional. Just make sure if you choose to ignore channels that you don't dump a duration for them and confuse me into thinking they are casts.
Include Raid Difficulty and Size
Enhance the EnterCombat event to include the raid difficulty and raid size if the player is in a raid. This will make it much easier for me to support every single boss fight without having to have a huge table of health pool values for bosses just to figure out difficulty.
Optional Tasks
The following are all fully optional and if you don't do any of them, I could still build a decent site. I've listed these in priority order.
No Exit Combat On Operation Wipes
When a player finally releases and respawns after an Operation wipe on a boss, the logging player should exit combat. Right now no exit combat event fires, so it looks like the player is still in combat. I can work around these events being broken if I have to, but it effectively means I have to ignore enter and exit combat completely and special case fight by fight.
Interrupts
Interrupts are really bizarre in the log. The player is the source of the event when his spells are interrupted and also when interrupting enemy spells. This makes it really hard to tell what's going on. In addition the interrupt never has any target, so you know what spell was interrupted, but you don't really know whether the player interrupted the spell or the player's spell got interrupted. You have no idea which NPC was involved, since it's just not there.
Interrupts should be changed so that the source is the player or NPC that interrupted the spell, the target is the player or NPC that got interrupted, and the ability is the spell that was interrupted. Then have another field (e.g., like the spot where buffs/debuffs go in ApplyEffect) that indicates what spell (e.g., Mind Snap) did the interrupting.
This is what my interrupt of an enemy Med Scan looks like in the log right now.
[14:06:23.122] [@Kihra] [] [Med Scan {855630499807232}] [Event {836045448945472}: AbilityInterrupt {836045448945482}] ()
Note that who I interrupted is missing, and the ability I used to interrupt is missing. Here is one possible way to fix the event's syntax:
[14:06:23.122] [@Kihra] [Brekken Gang Lieutenant {3842131714113536}:42705003309711] [Med Scan {855630499807232}] [AbilityInterrupt {836045448945482}: Mind Snap {875086701658112}] ()
This is what an enemy's interrupt of my Cascading Debris channel looks like in the log right now.
[11:47:02.696] [@Kihra] [] [Cascading Debris {3414228417380352}] [Event {836045448945472}: AbilityInterrupt {836045448945482}] ()
Note that who interrupted me is missing, and the ability the enemy used to interrupt is missing. I am listed as the source, which makes it seem like I interrupted myself!
Here is one possible way to fix the event's syntax:
[14:06:23.122] [Knight of Zakuul {3839103762169856}:144772122829] [@Kihra] [Cascading Debris {3414228417380352}] [AbilityInterrupt {836045448945482}: Knockdown {1099842340258055}] ()
Note in this example, the enemy used crowd control to stop the spell rather than casting an actual interrupt ability. It would be totally fine (if it's hard to connect this up) to omit the Knockdown ability in the example above as long as you just get the source, target and interrupted ability correct.
This task is listed as optional mainly because I can look for the ModifyThreat event (with a tiny threat value) from real interrupts to track when the player uses a real interrupt ability, but this is obviously gross and I don't really want to rely on it.
Spend and Restore Events
Spend and Restore events are missing the ability responsible for the Spend/Restore. If there is an activated ability directly responsible for the Spend/Restore, it would be nice to include it in the event.
Obviously Energy can just regen, and you don't need to include the ability for that, but if an actual ability is responsible for the Spend/Restore, including it would be nice.
I designated this as optional because I could probably link Spends/Restores to the last seen AbilityActivate with a pretty high degree of reliability, but I'd get it wrong sometimes.
Dispels and Purges
No events seem to fire when a player dispels buffs and debuffs. For example when I pop Resilience as a Kinetic Combat Shadow, the harmful effects are removed, but nothing indicates that Resilience was responsible.
You have two options here. Either enhance RemoveEffect to include the dispel ability when one was used to remove a buff/debuff, or you can create a new AbilityDispel event that looks very similar to how I suggested the AbilityInterrupt event should look in the previous section (i.e., source, target, harmful effect, dispel ability).
Add Character IDs
Dump IDs for characters so that renames and transfers work. If characters in your system have global IDs, consider including them for non-anonymous players in the log file. ESO did this by hashing the global ID (just to not expose the real underlying global ID of the player). This allows players to be identified by an ID rather than a name, and it means they can rename or server transfer safely and still retain all their logs.
You could put this in the same place I suggested the anonymous session-ID. This is what ESO does.
then i go and rename to Dahri
[@Dahri { 3424858382 }] <-- Global id is the same, SWTORLogs can tell it's the same player and merges Kihra and Dahri automatically!
Player Info Events
Dump PlayerInfo events for every player in the party/raid when the logging player enters combat. This would fire right after an EnterCombat event.
(a) In WoW and ESO, this event dumps auras in effect on the players when combat starts. This allows all existing buffs to be known since they may have occurred before logging even started (or when the logger wasn't present). ESO dumped this as a bracketed list of ability ids and then a second bracketed list of stack counts. Without this information, long-term buffs like consumables or stat buffs won't be visible (which is pretty annoying to players). You could limit this to long-term and permanent buffs only (this is what ESO did), as any short-term buffs will fade during combat, and then I can show that they were up at the start. Having this feature is important for analysis, since it would let you see stat buffs, effects like Guard, consumable buffs, etc.
(b) You could also dump item level and spec (e.g., Kinetic Combat Shadow) here.
(c) Dump the 9 Utility abilities of the player. If you dumped these, I could give you statistical data on which ones are being chosen and how they are balanced, and players would be able to see what builds people are using on various boses.
(d) ESO and WoW also dumped gear for each player in this event. This can be as simple as just dumping the ids of the item in each gear slot, or going further, you can also dump info like what mods/augments/etc. are used. Neither ESO nor WoW used any names in this event, they just dumped IDs and relied on me to build up a database of names. For gear, since items are just shells that contain mods/amplifiers, etc. you would need to dump all the components inside each piece of gear for completeness.
Map and Zone Info
Map and zone info. You could fire an event when the logging player changes zones or maps and include the localized map/zone name and id. I noticed that you dump what appears to be a zone or map name when the player enters combat. It might be interesting to add the ID to that for localization if that's a zone/map.
Cross-Server Grouping
If you ever allow cross-server grouping, then the server for all players needs to be included. This could be done with the server name following the player name. WoW drops spaces and pushes multiple words together, so you could do the same, e.g., "Star Forge"" becomes "StarForge". For now, since you don't seem to allow this, you can just skip this suggestion. It's something to keep in mind if you ever do allow players to group across servers though.
[@Dari@StarForge]