byuu's message board

Could anyone kind of explain in maybe a paragraph or two in 'layman's terms' how you guys go about reverse engineering something that has NO documentation whatsoever? Like the Super FX2 chip or something. I mean, from what I understand, someone was going to loan byuu a Life Fitness Extertainment bike if he wanted to spend the rest of his life emulating this crazy stuff. Ok, you get this giant box in the mail...Where do you even start? Do you literally just dive on in, unscrewing screws and physically looking at the motherboard/various chips with your own two eyes? Does that even help? This is so sad, but I am so far in the beginning of learning programming, I open up my own PC, see a giant mess of wires, fans, dust, and green 'chips' and say "yep, it's a computer." Ok, maybe I can at least tell some of the major components, but you know what I mean. Do you ever see some goofy looking IC or chip and go "uhhh, what the hell is this thing, and what's it's purpose?"

Or for instance even the Super Nintendo itself. I'm sure there is errata in the leaked 'Official Programming Manual' just as there are errors in the official ZiLOG Z80 documentation (from what I know). When an error is found, is it just assumed that EVERYTHING could be wrong, therefore, every single component must be tested 1,000,000 times to ensure accuracy? Or when you guys figure out a clock cycle to 500 decimal places...how do you do that? The original Game Boy (DMG-01) boot ROM was dumped by someone literally looking at bits with a microscope.

I was just curious to know a general idea behind it if anyone is willing to briefly explain a typical reverse engineering process for an undocumented piece of hardware to a slow dull-witted beginner like myself.

Thanks,
Nick

*EDIT* Good god, after researching the Game Boy boot ROM a little more, I found all of these pictures. I need to just stop and focus on my crappy little 2 dimensional arrays of objects in Java. This stuff is so inspiring, but depressing at the same time. LOL.

you start with trial and error and eventually work your way to electron microscopes.

You can take a look at the type numbers on the chips and search for technical documentation on the internet...

Chip product numbers on the PCBs can help get you the proper instruction set documentation, or at least give you a general idea of what you're working with. Discovering an opcode set with no documentation is absolutely brutal. Takes people several months to do that stuff.

But most actual emulation is just doing something, and observing the results. Now try and repeat them exactly under emulation, and move on. It's a lot easier when games are breaking. Once you have 100% compatibility, you have to figure out what to improve by blind guessing and experimentation. You basically make your own strenuous tests that do all the crazy shit you can imagine.

If you think of the whole forest, you'll never get anywhere. You have to be content to focus on one tree at a time, even when nothing at all runs, and you're staring at a black window for the first three months. If you can do that, the rest falls into place.

That is why you are awesome byuu. Infinite patience.

Hey thanks byuu. I was always curious to know how the "Pan Docs" were written for the Game Boy as I don't think there was ever any 'official' documentation leaked for it or anything. I just picture these guys with actual labs with fume hoods, crazy acids, and electron microscopes doing stuff like this guy. "I can read out the ROM. Anything goes at this point in time, versus glitching with a glitcher."

Meanwhile, I'm doing my best Marty McFly impression..."Whoa, what the hell is a glitcher!?"

Can you take a chip like this and install it in your own little custom PCB from Radio Shack, hook it up to your PC and just run a seemingly infinite amount of tests to try and kind of 'brute force' the opcodes? I guess what I'm asking is say for instance a Super FX chip (I have no idea how it was done, I guess I could research it more), but did someone buy a game like Star Fox or something, literally desolder the Super FX chip off of the cartridge PCB and install it into their own PCB and just bombard it with 'questions?'

Thanks again for your guys' replies.

Rarely is anyone rigging up hardware setups to reverse engineer ICs in gaming consoles. Clock glitching is more of an exception than the norm. RE almost never achieves perfection and doesn't really have to for things to be good enough. Usually information is refined as it is found to be incorrect more than it's thoroughly tested from the ground up.

A lot of baseline information comes from existing sources. For these old consoles there's always a lot of official documentation, and even if you never personally end up seeing it it doesn't mean it wasn't leaked - the odds of something for an old platform like Gameboy not having its documentation leaked is close to nothing when it must have passed thousands of eyes and the risk due to exposing it now is low.

Even if there isn't official documentation from the game console manufacturer, they will still be usually using a lot of off the shelf chips that have documentation publicly available or otherwise attainable. In the case where custom chips are used they'll often still be largely derived from existing designs, or at least will follow common methodologies or at least common sense. The more complex a chip is the more likely it is to not be a totally original design, for obvious reasons. If the vendor documentation didn't get leaked then at least they'll probably have given some architectural hints in their advertisement of the console.

People don't usually have to desolder chips and make custom interfaces to them because the interfaces can largely be controlled by software running on the CPU(s) in the console. Said CPUs are directly programmable via removable media like cartridges. So so long as you have a way of making a custom cartridge or CD or whatever then you're good to run code on the platform, and if you know even what architectural type the CPU is then you'll be ready to start executing code on it almost immediately after that. I doubt this is what anyone did for SuperFX. Even it is derived from a chip that has available documentation.

If you don't know the entire instruction set before hand then you really just need to know a subset of it, a very small subset in fact, to begin reverse engineering the rest of it. If you don't have any of the instruction set then things are substantially more difficult, but this isn't usually the case. Again for SuperFX, analyzing the non-FX game code would lend a lot towards understanding what the FX game code does.

The other side of RE is examining existing code, such as BIOS and game ROMs. From this you can derive a logistic overview of what the code expects to happen vs how it's interfacing with peripheral hardware. This can also give you some help in deciphering an instruction set if you're starting from nothing. ROMs in cartridges can be dumped with support electronics if you understand the cartridge interface, which itself can be derived (enough) from the component ROM datasheets and enough visible traces.

Ahhh, ok, thank you Exophase for that. That was very insightful. I appreciate it.

I think whether you are reverse engineering hardware or software, the process is basically the same.

First you dive in and try to recognize whatever you can. In software this could be blocks of code, with hardware it could be chips and boards. Then you look into what is connected to what and start asking why. Once you figure out the flow and how everything is connected, you can start making substitutions to test whether your assumptions thus far or correct. After that, you go from there.

At least with software, usually a combination of tracing and reading the input/output from a working system will get you as far as you need. For a good example of starting from nothing, look at the preliminary Shiren server I put together with byuu. We had a game and a couple packets logged years ago when there used to be a live server. A day later we had our own server capable of collecting 95% of the gameplay data, doing password recover, creating accounts and validating the username/password.

I keep hoping one day I'll get to take a crack at figuring out who to pass data to and from the BS ... currently no way to look at what it's doing. Would be pretty cook to decipher the request and have the emulator hit a server online, pull a response, then translate that back into something to return to the BS.