Dev Log: Star Ocean Canoe Patch

Monday, April 02, 2018

When I started the development of the Star Ocean canoe patch, for some reason I thought to try and keep a reasonable log of my point of view at that time as I thought that some people may find it interesting what goes into developing something like it. I planned to post it in a more broken up multi part post series, posting each part shortly after that part of development took place. But I ended up deciding against that as, I didn't know when the final patch would be released, and I didn't want to generate any extra hype for the patch that would feed any impatience. Doing so felt like it would just be an unfair teaser to people. So I held it off, thinking that I might post them shortly before I released the patch. But even then, I just wanted the patch out and elected to post it after, if at all.

Due to some recent unjust slander against me, causing me to have to defend myself against bogus claims, I'm now feeling more need to post my log. Originally this log would have made up 3 long posts, but I'm going to try and edit it and condense it down into this 1.

Preface:

Before this project, my ROM hacking skills and accomplishments consisted of mostly just the basics as far as I know, as taught to me by an old friend. Finding data in a ROM, manually or via relative searching, finding pointers that reference that data, and relocating that data if needed. Basic that see use in translations.

I used the skills rarely. Mostly just poking around ROMs to little effect. Probably the most that I did with them was some extra hacking in some Super Mario Kart course hacks, changing a few tiles and strings of text.

Then once the SNES Classic was released, I used the skills a bit more to help create the early SFA2 canoe patches.

Learning from SFA2:

Getting SFA2 working in canoe was never my goal. My goal was Star Ocean from day one. But SFA2 was a nessacery first step towards that.

SFA2 was ported to the WiiU/3DS Virtual Consoles, so I was quickly able to make it work on canoe after some early days discoveries. From SFA2 I observed and learned that NERD used a technique to get SFA2 working, similar to what Dejap did for SFA2 and SO years ago with ZSNES. They pre-decrypted the graphics data as another file/set of data, but implemented a unique ROM hack over all the encrypted graphics data still left inside the ROM. I figured from the start that the hacks were just 8 byte pointers. 4 bytes for a plain text "SDD1" tag, and 4 bytes for a address value that's likely an offset of that specific data in the un-encrypted graphics data. But I never bothered confirming that suspicion until later. I realized that if Star Ocean didn't boot at all with bad graphics like SFA2 did when un-patched, then there's no point in fixing its graphics. So this is where the project sat on the shelf for me for a bit.

The Breakthrough:

After needing a break from the then early development of SFROM Tool, I decided that since it has been a couple months, that I'd take another crack at Star Ocean. After all, I had learned a lot since the last time I tried. Particularly, I unlocked  the mysteries of Preset ID's and created the extensive list. My hope was to use what I had learned to find a ID that was compatible with SO enough to boot. I had obviously tried the 3 IDs belonging to the various region ports of SFA2, to no luck. But nothing else was striking me as the next best contenders.

So I decided to try a different approach. I wrote some tools to make compiling and loading new SFROMs with a range of different Preset IDs, quick and easy. It was very brute force. But in the end it paid off! I found a single Preset ID that booted Star Ocean, with predictably bad graphics, just like SFA2! Through those bad graphics, I saw no other issues at that time accept for maybe glitchy audio. It was enough that I now needed to start working on replicating what NERD did for SFA2, with Star Ocean.

Unlocking the SDA Pointer Format for Star Ocean:

I knew what my goal was, but at first I was unsure where I should start towards that goal. At this time there was no reference material that I could find on line that could help me locate the SDD1 compressed data in a ROM. So I had to come at it from a different angle. Because this SDA setup reminded me of the old Dejap GFX patches, I figured I could start there to learn more. With SFA2 I had both a Dejap pack and a NERD SDA hack to compare to each other, figuring that in doing so I could reverse engineer an understanding of what was going on, and use that understanding to unlock Star Ocean. This idea was proving successful at teaching me. Further, I decided that the best way to learn how do this in SO was to just use the Dejap pack. I figured there was even a chance that using the Dejap pack would shorten the load off what I'd have to do on the project if it was all done for me already.

So I wrote some routines to convert the Dejap gfx pack data to IPS/SDA Tile data. Not expecting much at the time, I applied a newly generated IPS patch to a ROM and inserted the SDA data. And to my surprise.... it worked! Sometimes you don't believe what you are doing is valid until you see results like that. But there it was, Star Ocean running on canoe with normal graphics!

The Drop:

While it working as it did surprised me due to keeping my expectations low, what did not surprise me was that it wasn't all going to be that easy, and that there was going to be bugs. Sure enough, I ran into a multiple crash points and soft-locks right away. So now it was time to figure out why those crashes/soft-locks were happening.

For soft locks, the short of it was that Dejap pack was far from perfect and erroneously saw non-SDD1 data areas of the ROM as SDD1 data areas. Like overwriting code that manages transitions between scenes for example.

For the crashes, I quickly learned that the SDA pointer technique was NOT designed with Star Ocean in mind. The short of that is that the SDA pointer format is 8 bytes long, and while that's fine for SFA2, some SDD1 data blocks in SO are less than 8 bytes. I think the lowest I've noticed is 5 bytes. So I dubbed this issue "8 byte conflicts". I decided to just write a routine to check for all such 8 byte conflicts and omit 1 of the 2 conflicting SDA pointers for now. And the resulting patch I labeled the 90% patch (based off how many of the total known SDA pointers at this time got disabled.

Working Towards 100% from the 90% Patch:

The 90% patch proved stable. But obviously wasn't release worthy. It was now a couple weeks into December at this time, and I saw myself wanting to get something closer to 100% before I'd announce this project on Christmas. That was my hope. I felt that I needed to tackle this 8 byte conflict in order to feel confident that I could complete this project. I wanted at least that to be able to announce it!

What was obvious to me at the time was that I needed to do some relocating. After mulling it over a bit, I came to the realization that the solution to this issue was easy. I didn't need to relocate entire large blocks of SDD1 data. I just needed a new location for the SDA pointer. And for that I needed to locate the reference pointers that point to the original SDD1 data (now point to the SDA pointers). Beyond that, for space to work with, the SDA pointer format renders the encrypted SDD1 data beyond the first 8 bytes unused. So I had lots of space to work with!

The 100% Dejap Pack Based Patch:

My solution to the 8 byte conflict worked, and I fixed the remaining 10%. And with a couple days to spare before Christmas! I sang my progress to a few in the loop people, and acquired Melthris' help to make a video to both wet the appetite of people who would be interested, and attempt to silence anyone who may otherwise think the project was fake.

Now at this point, the question was "what next?". Something I left myself to ponder over the Christmas holiday. I realized a bit ago that even when I generate a 100% Dejap pack based patch, that there were so many issues with it that I wouldn't be very proud to release such a patch. While it may be 100% of what the Dejap pack covered, I seriously doubted that it was 100% of the games true complete data. I saw little tidbits suggesting as much.

The Post Christmas Reboot:

I came to the conclusion that while the Dejap pack was great to learn with, I needed to scrap almost all of it and dump the address of all the SDD1 data manually and properly. The only exception was a certain type of large SDD1 data blocks, non-sprite data. Those did not have standard pointers to that data like most sprites do. But it was the rest of the SDD1 sprite data that I found evidence of that the Dejap pack was lacking. So while not 100% confident that the Dejap locations for those non-sprite SDD1 data blocks were 100% known, I felt it was strong enough to move forward.

In my work up until now, I saw that the ROMs normal pointers to the SDD1 data were grouped. I refereed to these groups as "pointer tables". My decision was to take the pointers that I already found when trying to relocate the 8 byte conflicts, go to their locations and just sift backwards through the ROM until I located the start of each pointer table. Once I had the start of a pointer table I could dump the entire table. I wrote some code to help assist in this, though due to the format of these pointers and my understanding of that format, I had to semi-manually/semi-automatically dump these tables. It would be a long boring job, but one I felt would be worth it in the end as I'd have a high degree of confidence in the completeness of the dumping.

Armed with a plan of attack, I set to work the very day after Christmas, on dumping these tables. A week later, I had them all dumped!

Errors and Regrets:

While in theory I had everything dumped, my first resulting patch of the reboot was definitely not without issue. I started my own play through of the game, and would hit a crash or soft-lock about every hour or so. But in doing so I'd learn the cause of that specific issue, then fix all other such related issues in the ROM.

Regrettably, it was at this point, a couple weeks into January, that I decided to contact the creator of a document I found. The document helped reveal some of the flaws in Dejaps pack a few weeks prior, and I felt I should be kind/generous enough to both thank him and provide him with a few corrections. The regrettable part is 2 things. One is that he sent me some un-needed, unsolicited files, which opened the door for him to make some false claims about him contributing to my work far FAR FAR more then is the truth. A bogus claim with which he's been using as a basis to slander me. And the other is that a few days into the exchanges, I decided to ask him some questions a bit prematurely when I encountered my first issue, before I just realized the cause myself. At the time I was happy to just exchange with someone who could understand things and relate. But, ignoring my help of submitting a dozen or so corrections for his document, it turns out that he felt that by responding to my questions, I was indebted to him and must give him all the exact details for him to completely reproduce my work himself. Granted, I didn't know the reality of his project(s) at the time, and made a mistake based off that. I gave him all the info that he asked for, but said I wouldn't help any further. From my perspective and understanding, his request came across as insulting, as it suggested to me that my work wasn't good enough. While the misunderstanding on my part was soon realized, and I fully admitted to it, I would have apologized for it if it wasn't for the fact that he refused to accept that I know I made a mistake. Add onto that, he took every chance that he got to legitimately insult me and diminish my work. He deflected all my attempts to appease the situation and return things to the civil discussion we shared prior. Conceding to his insistence on acting childish, I felt it was best to just stop communicating and go our own separate ways. He felt otherwise, and while the email exchanges stopped, he started sending his small army of trolls to harass me on my website.

Floored at how such civil enjoyable conversations could go so suddenly bad, and at the way he acted, I completely lost my desire to work on the project for a couple weeks. I was considering abandoning it and all my SNES project completely. I do this hobby for the fun. When someone sucks the fun out of it, and I just don't want to look at anything related the topic, why keep going?

Coming Back:

As sour as that last experience left me feeling, I ended up coming back to the project, motivated by the fact that I announced the project a few weeks prior. And I didn't have a ton left to do. I put in all this time and effort, it would be a massive shame to let some immature idiot cause it to never get finished and released.

With each fix I learned and saw commonality among the issues. And made a final rewrite to my tools to prevent all such crashes and soft-locks. So I soon just found myself playing the game. Not running into issues at all! But I have a harder time dedicating my time to such games than I used to. So progress towards my goal of completing a 100% play-through of the game, just wasn't going well. Thankfully I made a new friend, Robin64, who was willing to take up the task of beta testing it with a 100% play-through. This was much appreciated as it freed me up to dedicate my time more into SFROM Tool.

The End:

Fast forward a couple months... Maybe I stepped back and was hands off to much. I should have got this released earlier. A whopping one issue was found via Robin64 testing. And make no mistake, I'm not under any impression that he didn't look hard enough. While I fixed that issue, I'm still not sure why it existed. I'm quite satisfied with the routines I developed to assist me in generating the patch, so I don't see how any more issues could be present. But I wouldn't be terrible surprised to see someone discover something either.

My Thoughts:

Here's where things get interesting. At least to me.

As some know, I've put a LOT of time into understanding the Preset ID list, and canoe in general. In doing so I've been presented with evidence of different things. Subtle suggestions of what NERD was intending when they did something. For example, the major unlocking that happened with the Preset ID list was when I discovered the suggestion of a pattern to the order, and discovered that pattern matched up fairly well with the Product ID order of the games on the original Wii Virtual Console.

Here's my shocking conclusion on Star Ocean. I strongly think it was being considered/worked on for official release on the Super Famicom Classic! But ultimately got scrapped at least due to NERD seeing that the SDA pointer format used by SFA2 did not translate to Star Ocean very well. I suspect they concluded that the extra effort wasn't worth the time when they too ran into the 8 byte conflict issue.

But still, why do I think they worked on it? Well, that's because the fact that the Preset ID I use exists at all. No other Preset ID will boot Star Ocean. The ID sits in a region of the Preset ID list that was created for the SNESC. And in that region there are a few weird spaces. Why? Why would one of those unused spaces happen to work with SO, and be the only one that does? Also think about the fact that ExHiROM games don't seem to work, as when canoe tries to access data in the region beyond 4MB in such a game, it crashes. While SO with that Preset ID works perfectly fine coming in at 6MB.

To me that's enough evidence. That's why I chose a naming format for the patch that I did. The games first released on the SNESC in that region follow a similar Product ID format to their 2 unique characters. A "U" followed by a character from the name of the game. "S" being already taken, so "O" made the most sense. Altogether "WUP-JUOJ".

So I do have to thank them. Before I found the ID, I was coming to the conclusion that SO on canoe was not possible. My conclusion then was that it wouldn't work because each SFA2 region ROM only worked with 1 Preset ID. Each required that ID. And even if SO did boot, the ID would have to have special code to enable reading of the extend memory.

Now more personal. I'm happy with the project. I got to learn and grow my skills at dealing with ROM hacking. I went from a complete amateur, to what I think should be a intermediate skill. I now wonder what step I can take next to get to another level and what I could accomplish!