Serrated Blades and Deadliness.

In 1.12 the Subtlety tree gets considerably buffed. It's not restricted to talents that buff your stealth, openers, finishers and flexibility but it also contains two talents that increase your (sustained) damage output.

How do the new talents Serrated Blades and Deadliness compare to the old damage talents like Malice, Lethality or Opportunity etc?

Serrated Blades (3/3)

This talent scales perfectly with gear and level. All our damage is physical and armor mitigation is percentage based so the only variable to gauge the power of Serrated Blades is the opponent's armor.

Armor mitigation calculates like this: armor / (armor + 400 + 85 * level)

With 3 points invested and for a level 60 rogue it ignores 300 of the target's armor. How does it affect the effective damage mitigation of a level 60 target?

1000 armor: Mitigation 15,4% -> 11,3%. Physical damage taken +4,84%.
2000 armor: Mitigation 26,7% -> 23,6%. Physical damage taken +4,17%.
3000 armor: Mitigation 35,3% -> 32,9%. Physical damage taken +3,66%.
5000 armor: Mitigation 47,6% -> 46,1%. Physical damage taken +2,94%.
10000 armor: Mitigation 64,5% -> 63,8%. Physical damage taken +1,97%.

So all in all, Serrated Blades is quite efficient against low armor targets. This makes it especially useful against clothies, sundered mobs or in combination with Expose Armor. Note that it also buffs your Rupture considerably.

Deadliness (5/5):

For most rogues the dps through attackpower is slightly higher than the weapon dps. These rogues would see their white damage increase by 5-6% through Deadliness.
(0.1 * attackpower_dps / (weapon_dps + attackpower_dps) * 100).
If Deadliness affects not only attackpower from gear but also attackpower added through buffs, the increase would be like 7-8% in groups.
Specials attacks like Sinister Strike, Backstab and post patch even Eviscerate and Rupture are also affected by attackpower. However a large part of the special damage is always a flat bonus which is not affected by Deadliness.
How much a special attack's damage will increase depends on how much of its damage is based on attackpower.

Hemo:
Your Hemo damage is 100% weapon damage and thus will increase by the same percentage as your white damage. (Usually a bit more than 5%).
Sinister Strikes:
Lets say you have a Vis'kag and with it your Sinister Strikes hit for an average of 380 damage on 0 armor targets. 68 damage of that is the flat bonus each SS gets. (100 + 187)/2 = 144 damage come from your weapon and the remaining 168 damage will come from your attackpower (~980AP). Deadliness would increase your Sinister Strike damage by 0.1 * 168 / 380 = 4,4%.
Backstabs:
Let's say you have a Perdition's Blade and your Rank 9 Backstabs hit for an average of 550 damage on 0 armor targets.
225 damage are due to the flat bonus. 1.5 * (73 + 137)/2 = 157 damage come from your weapon and the remaining 168 damage will come from attackpower (~922AP). Deadliness would increase your Backstabs by 0.1 * 168 / 550 = 3.1%

If you got exceptionally good gear then the flat bonuses of your specials aren't as significant and you should expect a proportionally higher dps increase. If your gear is worse than what I used in the examples, then you can expect a proportionally smaller dps increase through using specials.

To compare these new talents with old damage talents such as Lethality, Opportunity and Malice you can use the following values as reference points:


Malice (5/5):

White damage increases by ~5%. Sinister Strikes by ~4%. Backstabs by ~3% (3.8% with Lethality) for an epic equipped rogue. The overall effect becomes slightly smaller the better your gear (+hit and +crit).

Lethality (5/5):

With a basic crit rate of 25% 55 of 100 Backstabs will do 15% more damage than without Lethality. So your average Backstabs get a 8.25% damage increase. If you do 40% of your damage with Backstab, its a 3.3% dps increase.
For sword rogues with the same basic crit rate only 25% of your Sinister Strikes get a 15% damage boost so your average Sinister Strike gets a 3.75% dps boost. If you do 30% of your damage with Sinister Strikes, thats a lame 1.125% damage increase.

Opportunity (5/5):

Backstabs get a 20% dps increase, so if you do 40% of your total damage with Backstabs, you get a 8% overall dps increase. Pretty straight forward and very very strong compared to other talents. A must-have for dagger builds.

Precision (5/5):

If you have less than ~6% hit chance on your gear, points in Precision are kinda powerful. It will up your damage by ~4-5% depending on the spec and gear. As soon as your specials don't miss anyway +hit will only improve your white damage. If you are combat specced that is still a 3-4% dps boost, a Sealfate build will get only like 2-3% total dps increase out of speccing Precision.

Dual Wield (5/5):

This one is straight forward again. Your damage through normal melee attacks is upped by 1.75/1.50 = 16.67%. If you do 40% of your damage with white hits that's a 6.67% damage boost. Most rogues that are deep in the combat tree (especially raiding ones) do 60% or more of their damage with white hits. So Dual Wield would be a 10% damage boost for them.

Conclusion:

Blizzard added two very cool damage talents in the Sub tree. But as those that optimize their specs for raid damage won't want to waste lots of points into the lower tier utility talents, you might wonder what these talents are good for. Well, actually they make builds that go deep into the Subtlety tree more viable than they were pre patch. For example 21/8/22 dagger builds not only get better utility talents than before but they can pick up Serrated Blades and increase their dps in groups and against low armor targets by a couple of percent. The true winners however are Hemo rogues in my opinion as they get a number of interesting new choices. For example with patch 1.12 you can combine Hemo with 2/2 Dirty Deeds and Improved Kidney Shot. Or you could use Hemo together with Seal Fate and Cold Blood. But with Deadliness promising you 5% or more of total dps increase with both Hemo and white attacks it seems smart as well to spend 30 points in Sub and the remaining points into the Assa tree up to Cold Blood. And it's not less viable to spend full 31 points in the Sub tree and get the buffed Premediation and Gauge instead of Cold Blood.
So while Blizzard isn't buffing high-end raiding rogues there, they offer us a large range of new builds that are all tailored around PvP and at the same time stronger in PvE than current PvP builds and thus they make the Subtlety tree much more competative than before compared with the other trees.

Finisher Efficiency: Eviscerate

Eviscerate is the only direct damage finisher rogues have. The damage it does depends on the amount of combo points used. It is mitigated by armor and it can crit, but it's independant from your weapons and attackpower. In other words: it doesn't really scale with gear. Evi might have been a no-brainer when you were leveling, but with Backstabs and Sinister Strikes yielding more and more damage per energy point, and white melee hits making up an increasing share of your total damage, the good old Eviscerate seems to drag behind.
To decide right when to spend combo points how, let's take closer look on Eviscerate's efficiency.

The efficency of Eviscerate with different amounts of combopoints

Without talents this is what an Eviscerate does on a target with 0 armor:

1 CP Evi does 216-312 dmg = 264 average dmg = 28%
2 CP Evi does 384-480 dmg = 432 average dmg = 46%
3 CP Evi does 552-648 dmg = 600 average dmg = 64%
4 CP Evi does 720-816 dmg = 768 average dmg = 82%
5 CP Evi does 888-984 dmg = 936 average dmg = 100%

The damage increases proportionally to the amount of combo points invested. You could say an Eviscerate does 94 base damage and 170 damage per combo point.

An Eviscerate will cost a certain amount of energy which could be spent to generate CP and damage, and with Ruthlessness specced the Eviscerate will be cheaper the more CP you have to spend. So obviously the 5cp Eviscerate is the most efficient one.

For lvl 60 rogues with decent gear, Ruthlessness and Relentness Strikes, Eviscerate will generally start to pay off with 3 or more combopoints. If you do Eviscerates with only one or two combopoints, you lose so much energy that it negates all damage Eviscerate may yield.
While it starts to pay off with 3cp the efficiency gain from using more combopoints is quite large. For example for a Sealfate Rogue in epic gear Eviscerates with 4 CP will mean a 60% higher dps buff than Eviscerates with 3 CP. Full 5 CP Eviscerates are twice as much extra dps as 3 CP Eviscerates.

Eviscerates and Sealfate

So the best way to spend your CP on Eviscerate is with 5 points. However with a Sealfate build you might waste CP if you go for 5 points at all costs: If the last SS or Backstab you do crits, then you lose the extra CP Sealfate is awarding for crits. So what should you do if you have Sealfate and got 4 CP on a target?
For a Sealfate rogue using daggers doing 4 CP Eviscerates or risk wasting an extra CP to do a full Eviscerate doesn't make a difference. If you use Swords/Maces, you are better off going for 5cp as the risk to waste a combopoint is a lot lower with Sinister Strikes.

Short fights

In short fights where you are fighting only one (or the last) mob it's not always possible to wait until you can do a 5 point Eviscerate. If the mob dies too fast you might lose all your CP. Even if you managed to get the finisher off, you might not have time to spend the 25 Energy from Ruthlessness. In these situations 3-4cp Eviscerates before doing a last Backstab or Sinister Strike can be a good option.

Example:
Let's say you are SF dagger specced and have 110 energy. Your raid engages an Anubisath Eradicator. These fights take about 10 seconds, so the best way to start is with two Backstabs and if they both crit, you do an Eviscerate. That way you get a 80% chance to get 25 energy back which allows you to do a 3rd Backstab right after you get the 4th energy tick. If you had tried to do the 3 Backstabs first and then finish with an Eviscerate, you'd have needed 2 more energy ticks (or 4 seconds) which you won't get.

Conclusion

Generally go for 5 CP Eviscerates if possible.

4 CP and 3 CP Eviscerates should be only used if you don't have time to get more cp on a target. They are noticeably less efficient than 5 CP Eviscerates but still better than wasting the combo points.

Eviscerates with less than 3 CP shouldn't be used at all except if you are sure that the Eviscerate will kill the mob and you have time to regen to full energy before you engage the next one. This happens sometimes when grinding.

In situations where you will regen your full energy until you engage the next target, it might be better to use Eviscerate (and other finishers) not as the last spell. That way you get a chance to spend the 25 energy Ruthlessness might give you back.