RTMPE handshake type 10 reverse engineering

[ntoskrnl]

Long story short: I need to reverse engineer the final piece of a handshake algorithm so that it can be reimplemented in open source software. Unfortunately, it's obfuscated pretty heavily, and I'm not a very skilled reverse engineer.

This is what I've got so far (the first step of the algorithm):

for (i = 0; i < 16; i++) {    // The byte_* arrays can easily be dumped from memory.    in[i] = byte_536F0C40[byte_53701040[in[i]]];}

I can't get any further; it just gets too complicated. Maybe someone here is a more skilled reverse engineer? This is a really tough nut to crack. I've asked in many places but nobody has managed to do it so far.

Here's a C source file that you can compile and then debug: https://mega.co.nz/#!d8A22RqD!dR-sBxtKEQ84QCcQVnL8GEO3YtsV0k1nJPXkGwWmcdU

Alternatively, here's an EXE that can be directly debugged: https://mega.co.nz/#!w5wATZaT!ZWHmHnyWmfOqIRYK2N6figtL4PZpLvKYhJo8W4jT08g

The question is: What does the rtmpe10_sig function do when given 16-byte input and output buffers, and a keyid between 0 and 15 (inclusive)?

---

Background info: This algorithm is from Flash Player. Adobe occasionally releases new versions of the handshake. All previous versions have been reimplemented in open source software like rtmpdump, but this algorithm in type 10 has not been reverse engineered yet. As you can see in the source code, the relevant sections of the DLL have been identified and dumped already. It's just that the code is such a mess due to some sort of obfuscation.

Part of the algorithm is probably a standard cryptographic function. Type 8 uses XTEA and type 9 uses Blowfish.

Edited December 20, 2013 by ntoskrnl

[ExXxtreme]

Hello,

but this algorithm in type 10 has not been reverse engineered yet.

Xeebo already did handshake 10...

greetZ

ExXxtreme

Edited December 20, 2013 by ExXxtreme

[ntoskrnl]

Actually, Xeebo only identified and dumped the relevant code sections from the Flash Player DLL. My example application uses that data. Unfortunately the algorithm itself remains undiscovered. The algorithm has to be reverse engineered so that it can be reimplemented cleanly.

[ntoskrnl]

Interesting. Mega has taken down the files. Let me know if you'd like to take a shot at this and I'll reupload them.

[Lite]

We are not a hacking forum. There are lots of those on the internet.

That being said. Topic closed.