Triplis Developer Journal

This information is very helpful. Hope you will continue with the tutorials :smile:

Automation and Priorities

Had some stuff I thought might be interesting to talk about, so I figured I'd write another one of these. This is more of a story than anything else, so don't expect tutorial-like material. :)

It's been months since I've been actively modding for the game, but I've been trying to stay on top of keeping mods updated with new patches. With this comes the problem that much of the time I'm going back to fix things that are not at all fresh in my mind.

The Sorcerer mod was a particularly egregious offender in this way because I didn't even know for the longest time that you could name files like "S4_339BC5BD_00000000_EE1715C265488509_MotiveMagicalConnection" instead of "S4_339BC5BD_00000000_EE1715C265488509" and have them not only be read by S4PE (Sims 4 Package Editor - a modding tool for those who are unfamiliar) but S4PE will also sort them within the editor, making it relatively easy to know what is what at a glance.

So instead of giving the files themselves unique names, I had them all within uniquely named and carefully organized folders. But not carefully enough. As the mod grew and features got added on, some of my organizational choices became a matter of "remembering correctly," like knowing where things are in a messy room. This would not do at all for taking indeterminate long periods of time away from the project.

Fortunately, some of what I'd been spending my time doing not-modding was practicing C++. Modding was no longer a priority in my life, but practicing C++ was. I kept thinking about something Mike Acton (a video game programmer known for his talk on data-oriented design in programming) had said in a Q&A about the advantage indie developers have in being able to build extremely specialized tools. And I thought, "What if I can make some tools that help me with extremely specialized problems I'm facing in keeping mods up to date?" This would both help me with the more tedious problems and I'd be practicing C++ in the process. A win-win!

The first problem to tackle was the Sorcerer mod. I wanted to go through and give each file an identifiable name at the end of it (ex: S4_339BC5BD_00000000_EE1715C265488509_MotiveMagicalConnection). But if done manually, this would take hours and hours of mostly mindless tedium. Just the kind of problem programming was made for!

So I went to work making a program to help me rename the files. It was slow going at first. It was a new problem for me. I had some familiarity with writing and reading from files, but not specifically with renaming them. I didn't want to try to make a super generalized solution from the get-go, so I got very specific at first, to the point that my first working version of the program could only accurately rename one type of file (loot files). And only because I'd (mostly) stuck to a particular naming convention with loot files in their internal name, doing stuff like: Triplis_Sorcerer_Loot_ThingThatGetsLooted

I used the "Loot" part of the internal name as the part for the program to catch on and figure out what to append to the file's external name from there. For example, _ThingThatGetsLooted would get appended to the file's external name, making it something like: S4_0C772E27_00000000_8AF8CEE84A89C9C6_ThingThatGetsLooted

This method quickly broke down. A few loot files didn't even follow the convention and I had to edit their names manually. And more importantly, just about every other file for the mod did not follow a name convention like this.

Most of them were more like: Triplis_Sorcerer_ThingThatHappens_Interaction, or Triplis_Sorcerer_ThingThatHappens_Trait

After some tinkering around and the loot files renamed, I realized I could have it snag on Triplis_Sorcerer_ instead and get the right part of the internal file name from there. Everything surely followed *that* naming convention, right?

Well, most of them did. Some didn't. A few were named more like: Triplis_SorcererTraitChild_ThingHappens

I ended up editing those exceptions manually, for the most part, since there weren't many of them. There were a few that were like: Triplis_ThingThatHappens, too, and I just wrote in an extra scenario case to handle them.

Lessons learned in not following naming conventions, huh? But I got through it and without feeling like my brain was going to cave in from tedium. Victory! Luckily, most of the files for the mod were xmls files, which are easy to read with C++. Any files that were not were, well... I'm glad there weren't that many of them. There were some binary files where I researched a little how to read binary files to automate getting their internal names and then just gave up and did it manually. Would take too much upfront work for so few files renamed one time, I decided.

That was problem one.


The second problem to tackle was the "Lock Skills" mod. It's not a particularly complicated or large-scale mod, but there is a lot of repeating the same files, with one or two lines changed, depending on what skill needs to be locked or unlocked. I wanted a way that I could update it without painstakingly creating lots of new files by hand each time. Especially since I'd misssed a lot of skills over the past year or so and it was piling up.

My first goal was to separate out the files that were not already in the mod. Program one did this. Using searching in the extracted XML game files, I pulled out all of the skill files and put them in a separate folder (manually). My program then looked through that folder, compared it against a list of skills that were already part of the mod, and put all of the names of the ones that weren't already in the mod in a new text file.

So now I had a file something like this:

16718
136140
186703

etc.

I also wanted the string references from each file though, too. These would be used in each skill lock for its name that would show up in game. These went in a separate text file as well.

The next step was by far the most time-consuming. I put some template files in a folder (a copy of each needed file for a single skill being added: unlock/lock interaction, unlock/lock loot file, trait, and buff). And the program would need to, for each new skill, make a copy of those template files, put them in a named folder, then go into each one (in a very particular order!) and make some changes.

But I'm getting ahead of myself. Before I could get most of this done, I reached the biggest roadblock of the project. Hashes. Hashes, for those who are not familiar, are the unique IDs in sims 4 files that distinguish one from another. They look like a bunch of gibberish at a glance. Stuff like: 3051285741375902074

I had no idea how to generate these myself. I've always either made files through Zerbu's Mod Constructor tool or used ArtUrlWWW's FNV Hasher tool. Throughout most of my modding for this game, I barely even grasped what a hash was, much less had a concept of generating one myself!

But I pretty much had to do it for my program to work. In fact, getting new hashes from a tool like the FNV Hasher and putting them in the names of files has probably been one of the most time-consuming aspects of modding for me in the past and one of the primary reasons I wanted to automate the creation of these additional files in the first place.

The only alternative would be if I could somehow interface with ArtUrlWWW's Hasher from my program, but that would a whole other new thing to try to learn and may not even be possible with his specific program.

So I researched hashes. I read up on the FNV Hash at: http://isthe.com/chongo/tech/comp/fnv/

FNV Hash was, after all, what the FNV Hasher program used, so I figured that must be the right track.

I went at the problem from a variety of angles, from trying to read ArtUrlWWW's source code for the FNV Hasher on github (which unfortunately was written in C#, not C++, with which I'd been getting familiar), to searching google for C++ implementations of an FNV hash algorithm and praying that somebody had an example to learn from. Slowly, with lots of trial and error and reading and more trial and error and actually reading the description of the algorithm on the page of the website I listed above, I started to wrap my head around what the general idea was and managed to generate 64 bit hash values as needed. Success!

Er, well... not quite. I had generated the decimal form of hash values. I still needed to be able to transform the decimal version into hexadecimal to able to get the files working. So back to research I went and eventually ended up clueing in on an example of changing between the two in a video tutorial on youtube.

More trial and error and error and error later, I finally got it working and tried out a test example in game with a custom hash that I had generated. It worked! Huzzah!

Now I just had to do the rest of the program, which was most of it. But you know the saying, if Rome had been built in a day, they'd have been debugging for months. Or something like that...

So I uneventfully (well, uneventfully if you don't count debugging) and tediously wrote out a series of highly-specific instructions that would edit each type of file that needed to be edited and do it in a very particular order.


Now I will note some important lessons that were highlighted to me along the way. Lessons like:

If you're trying to understand how to use a new algorithm, it might help to read what the actual algorithm is on the source page where you found it, instead of trying to use a C library in C++ and hitting the pavement face-first.

A program can be so specific to the point that it may be difficult to reuse for other things without significant changes, but that's not necessarily a bad thing if its purpose was valuable for the thing it was made for.

Most everything in programming is simultaneously more simple than it looks and astronomically more complicated. By that I mean, examples are often terrible and overcomplicate everything, but the problem also often involves more steps than you'd ever assume.


The story has an end

As all stories do, this one has an end.

I updated the Sorcerer mod a few days ago, with the updating made much easier with the files organized by name.

And tonight, I uploaded an update for the Skills Lock mod with more skills now available to be lockable.

And now I will go to sleep. (This has no relevance to the story whatsoever, but it is an ending of a kind, so it will have to do.)