Serrated Blades and Deadliness.

In 1.12 the Subtlety tree gets considerably buffed. It's not restricted to talents that buff your stealth, openers, finishers and flexibility but it also contains two talents that increase your (sustained) damage output.

How do the new talents Serrated Blades and Deadliness compare to the old damage talents like Malice, Lethality or Opportunity etc?

Serrated Blades (3/3)

This talent scales perfectly with gear and level. All our damage is physical and armor mitigation is percentage based so the only variable to gauge the power of Serrated Blades is the opponent's armor.

Armor mitigation calculates like this: armor / (armor + 400 + 85 * level)

With 3 points invested and for a level 60 rogue it ignores 300 of the target's armor. How does it affect the effective damage mitigation of a level 60 target?

1000 armor: Mitigation 15,4% -> 11,3%. Physical damage taken +4,84%.
2000 armor: Mitigation 26,7% -> 23,6%. Physical damage taken +4,17%.
3000 armor: Mitigation 35,3% -> 32,9%. Physical damage taken +3,66%.
5000 armor: Mitigation 47,6% -> 46,1%. Physical damage taken +2,94%.
10000 armor: Mitigation 64,5% -> 63,8%. Physical damage taken +1,97%.

So all in all, Serrated Blades is quite efficient against low armor targets. This makes it especially useful against clothies, sundered mobs or in combination with Expose Armor. Note that it also buffs your Rupture considerably.

Deadliness (5/5):

For most rogues the dps through attackpower is slightly higher than the weapon dps. These rogues would see their white damage increase by 5-6% through Deadliness.
(0.1 * attackpower_dps / (weapon_dps + attackpower_dps) * 100).
If Deadliness affects not only attackpower from gear but also attackpower added through buffs, the increase would be like 7-8% in groups.
Specials attacks like Sinister Strike, Backstab and post patch even Eviscerate and Rupture are also affected by attackpower. However a large part of the special damage is always a flat bonus which is not affected by Deadliness.
How much a special attack's damage will increase depends on how much of its damage is based on attackpower.

Hemo:
Your Hemo damage is 100% weapon damage and thus will increase by the same percentage as your white damage. (Usually a bit more than 5%).
Sinister Strikes:
Lets say you have a Vis'kag and with it your Sinister Strikes hit for an average of 380 damage on 0 armor targets. 68 damage of that is the flat bonus each SS gets. (100 + 187)/2 = 144 damage come from your weapon and the remaining 168 damage will come from your attackpower (~980AP). Deadliness would increase your Sinister Strike damage by 0.1 * 168 / 380 = 4,4%.
Backstabs:
Let's say you have a Perdition's Blade and your Rank 9 Backstabs hit for an average of 550 damage on 0 armor targets.
225 damage are due to the flat bonus. 1.5 * (73 + 137)/2 = 157 damage come from your weapon and the remaining 168 damage will come from attackpower (~922AP). Deadliness would increase your Backstabs by 0.1 * 168 / 550 = 3.1%

If you got exceptionally good gear then the flat bonuses of your specials aren't as significant and you should expect a proportionally higher dps increase. If your gear is worse than what I used in the examples, then you can expect a proportionally smaller dps increase through using specials.

To compare these new talents with old damage talents such as Lethality, Opportunity and Malice you can use the following values as reference points:


Malice (5/5):

White damage increases by ~5%. Sinister Strikes by ~4%. Backstabs by ~3% (3.8% with Lethality) for an epic equipped rogue. The overall effect becomes slightly smaller the better your gear (+hit and +crit).

Lethality (5/5):

With a basic crit rate of 25% 55 of 100 Backstabs will do 15% more damage than without Lethality. So your average Backstabs get a 8.25% damage increase. If you do 40% of your damage with Backstab, its a 3.3% dps increase.
For sword rogues with the same basic crit rate only 25% of your Sinister Strikes get a 15% damage boost so your average Sinister Strike gets a 3.75% dps boost. If you do 30% of your damage with Sinister Strikes, thats a lame 1.125% damage increase.

Opportunity (5/5):

Backstabs get a 20% dps increase, so if you do 40% of your total damage with Backstabs, you get a 8% overall dps increase. Pretty straight forward and very very strong compared to other talents. A must-have for dagger builds.

Precision (5/5):

If you have less than ~6% hit chance on your gear, points in Precision are kinda powerful. It will up your damage by ~4-5% depending on the spec and gear. As soon as your specials don't miss anyway +hit will only improve your white damage. If you are combat specced that is still a 3-4% dps boost, a Sealfate build will get only like 2-3% total dps increase out of speccing Precision.

Dual Wield (5/5):

This one is straight forward again. Your damage through normal melee attacks is upped by 1.75/1.50 = 16.67%. If you do 40% of your damage with white hits that's a 6.67% damage boost. Most rogues that are deep in the combat tree (especially raiding ones) do 60% or more of their damage with white hits. So Dual Wield would be a 10% damage boost for them.

Conclusion:

Blizzard added two very cool damage talents in the Sub tree. But as those that optimize their specs for raid damage won't want to waste lots of points into the lower tier utility talents, you might wonder what these talents are good for. Well, actually they make builds that go deep into the Subtlety tree more viable than they were pre patch. For example 21/8/22 dagger builds not only get better utility talents than before but they can pick up Serrated Blades and increase their dps in groups and against low armor targets by a couple of percent. The true winners however are Hemo rogues in my opinion as they get a number of interesting new choices. For example with patch 1.12 you can combine Hemo with 2/2 Dirty Deeds and Improved Kidney Shot. Or you could use Hemo together with Seal Fate and Cold Blood. But with Deadliness promising you 5% or more of total dps increase with both Hemo and white attacks it seems smart as well to spend 30 points in Sub and the remaining points into the Assa tree up to Cold Blood. And it's not less viable to spend full 31 points in the Sub tree and get the buffed Premediation and Gauge instead of Cold Blood.
So while Blizzard isn't buffing high-end raiding rogues there, they offer us a large range of new builds that are all tailored around PvP and at the same time stronger in PvE than current PvP builds and thus they make the Subtlety tree much more competative than before compared with the other trees.