Endless Attack REMIX Development Journal #2

18 Jul 2022

At the end of Journal #1, I mentioned something about publishing another journal "a month or two from now". It's been *checks notes* 4 and a half months since then.

"Clearly you must have made a ton of progress during that time, right?"

...

"Right???"

...allow me to explain.

I published Journal #1 a couple months before the end of the semester, so from a few days after I had published it up until my very last exam, I avoided working on REMIX quite a bit in order to focus on assignments. Even after the semester was over, I somehow decided the following was much more productive:

• Ruining my sleep schedule twice
• Redoing my entire Debian setup
• Building a bookshelf?

...yeah, I don't really have an excuse for not working on the game during the solid month or two after classes had ended. I picked it back up a couple weeks ago and have been making somewhat steady progress ever since, however. And even between publishing Journal #1 and resuming development recently, I was still able to do a few things here and there. So, let's talk about all of that!

Saving Space

REMIX is a DOS game, and as most know, DOS games were typically distributed via floppy disks (or CD-ROM in the case of a fancy multi-media game). I really liked the idea of distributing the demo and/or final product via floppy, but at first it didn't seem like REMIX would be a good fit (literally). Although the game's compiled code only took up around 70KiB at the time I considered it, there was a huge problem: Allegro alone, unoptimized, takes up 900KiB- nearly an entire megabyte. Since the typical floppy disk only has 1.44MB to spare, that left me with practically 540KiB to fit the game into, which wasn't ideal considering I was already dangerously close to that limit with all of the spritesheets already in the game. So, I had to start finding ways to crunch the game down.

Obviously I could have simply compressed the entire game folder into a ZIP or something and called it a day, but I felt like that was the cheap way out. Since I had already been using the obvious compiler options (such as -s), I focused my attention towards Allegro itself. Allegro has a documentation page on space-saving techniques, and there were a couple simple ones I started out with back before I wrote Journal #1. The first involved preventing extraneous drivers and color depth support from being compiled into the game. In my main.c, I simply put something like this:

BEGIN_GFX_DRIVER_LIST
	GFX_DRIVER_VGA
END_GFX_DRIVER_LIST

BEGIN_COLOR_DEPTH_LIST
	COLOR_DEPTH_8
END_COLOR_DEPTH_LIST

BEGIN_MIDI_DRIVER_LIST
	MIDI_DRIVER_AWE32
	MIDI_DRIVER_ADLIB
	MIDI_DRIVER_MPU
	MIDI_DRIVER_SB_OUT
END_MIDI_DRIVER_LIST

BEGIN_DIGI_DRIVER_LIST
	DIGI_DRIVER_SOUNDSCAPE
   	DIGI_DRIVER_AUDIODRIVE
   	DIGI_DRIVER_WINSOUNDSYS
   	DIGI_DRIVER_SB
END_DIGI_DRIVER_LIST

Most of the space savings here come from the GFX drivers and color depths. Since the game only targets standard VGA on DOS, no other drivers really need to be added in. The screen rendering code is dead simple and portable to the other drivers though, so maybe I'll add them back in at some point. Without any real hardware to test the other GFX drivers though, I'm unsure of it. Again, I still need a period-appropriate PC!!

Tangent aside, the color depth is always going to be 8-bit, since that's all VGA mode 13h supports. No use in including 15, 16, 24, and 32-bit color if I can't use them. The only sound-related driver I ended up removing was Allegro's DIGMID system, which uses samples to play MIDI files on the sound card, instead of relying on an OPL chip or MIDI-out. I decided against using DIGMID since it takes up a large amount of memory to store the patchset, CPU time to play them back, and valuable sound card voices I'd rather use for sound effects.

With all that said and done, I saved a whopping... ~200KiB. Not a bad improvement, but Allegro was still taking up 700KiB. What else could I do?

The second space optimization Allegro's documentation mentions involves recompiling the library without support for certain color depths. But wait, didn't I disable those with the COLOR_DEPTH_LIST earlier? Well... not exactly. For some bizarre reason, the color depth list can only go so far in removing support for these color depths; there's extra stuff, still don't know what, that can only be removed by doing a recompile. This was easy though; just open include/allegro/alconfig.h, remove the #define ALLEGRO_COLOR defines that don't end in an 8, recompile the library, and there we go: a version of Allegro that can ONLY use 8-bit color! And how much did that save us?

~50KiB. Allegro was still using 650KiB.

I decided to live with this fact for a while. I did one more optimization after Journal #1, in which I restructured the game's source to avoid multiple defines. This saved quite a bit of space on the game's side- didn't really take a note of how much- but the 650KiB from Allegro continued to bother me. It was then that I realized I glossed over an important line in that documentation page:

"For all platforms, you can use an executable compressor called UPX, which is available at http://upx.sourceforge.net/. This usually manages a compression ratio of about 40%."

I took to my terminal and quickly fired a sudo apt install upx. Surprisingly, it worked. To make sure I didn't install some other tool that also happened to be named "upx", I ran it and sure enough: it was the same tool the Allegro documentation mentioned. I ran it on the latest EXE of the game I had at the time, and like magic, it shrunk down from 700KiB to around 280KiB! Sure, it'll introduce some waiting time when launching the game on 486 and Pentium-class systems, but that's a sacrifice I'm willing to make in order to fit the game onto a single diskette.

With around 1.2MB now at my disposal for the game assets, I had nothing to really worry about. But there was still more that I could do.

The Easiest, Most Obvious Space Saving Technique: use a better format

I mentioned in Journal #1 that the spritesheets for the game are all standard, 8-bit indexed BMP files. What I didn't realize at the time was that BMPs, in the context of doing graphics for Allegro-based games, are absolutely awful for space-savings. For reference, Allegro 4.2 only supports loading BMP, TGA, and 2 fringe formats: LBM and PCX. So while I knew from the start that PNG would be a more space-efficient option than BMP, there was no way I could load them in natively (unless I added in support for PNG myself, which I'd rather not). So I tried experimenting with the other formats.

Of the 4 formats Allegro supports, Aseprite only supports saving/loading TGA and PCX files (in addition to the BMPs I had already been working with); so I had to leave LBM off the table (sorry DPaint fans!). When comparing TGA and PCX versions of a spritesheet to the original BMP version, the choice was clear: move away from BMPs.

menu.bmp: 30.7KiB
menu.tga: 11.7KiB
menu.pcx: 10.4KiB

And for another comparison, with a larger spritesheet:

b_kspike.bmp: 131.0KiB
b_kspike.tga: 31.7KiB
b_kspike.pcx: 28.6KiB

(To be fair, this sheet shrinks better than it should since it's unfinished and there's a lot of empty space. I'll explain why in a bit.)

I ended up going with PCX since it's able to squeeze out a couple KiB compared to TGA. PCX is a truly niche file format now, but at the time it was the image format to use on DOS. Of all the programs currently installed on my workstation, only Aseprite and GIMP can open them; I suppose the only reason Aseprite can open them in the first place is due to its Allegro heritage. With all of the spritesheets converted over to PCX, I'm currently sitting at 189KiB of spritesheets, compared to 737KiB had they all been BMPs. Impressive!

If you're not that much of an image format nerd, you're probably wondering how PCX is able to shrink the sprites down so much compared to BMP, while still maintaining the same exact data. The key difference between BMP and PCX is that PCX actually makes a (rudimentary) attempt at compression, via run-length encoding, or RLE (hey, there's that acronym again!) for short. BMP stores each individual pixel of an image, making no attempt at consolidating them together if multiple pixels in a row have the same color. With RLE, if multiple pixels in a row share the same color (in the case of PCX, 3 or more pixels), the consecutive pixels are stored as a byte containing the number of pixels, then a byte containing the palette index.

There's a bit more nuance as to how PCX implements RLE compression, such as the fact that it can only do runlengths of up to 63, certain colors can only be stored as an RLE pair, etc. Here's some additional reading if you're interested.

RLE isn't necessarily a magic bullet for all cases. If you have highly-dithered sprites or art, RLE will do next to nothing. And in the case of PCX, it'll store the individual pixels much like BMP (or for certain palette indices, an RLE pair of length 1). The artstyle I'm using for REMIX just happens to have large runs of flat colors, so the spritesheets benefit from the compression quite well.

But with all that aside, surely this is the optimal way to store all of the spritesheets, right??

Well yes, but actually no

There is one final trick to squeezing down the spritesheets, and it's as far as I'm willing to go with space optimization:

Just make the spritesheets smaller.

I said this was a "final" trick, but it's more like a few tricks aimed at doing the same thing (reducing spritesheet size). The first trick involves rearranging the sprites within the sheet in order to reduce empty space as much as possible. This only really saves a couple KBs per sheet, but the savings can add up quickly. Take for instance the menu spritesheet; here it is before rearrangement:

And after:

Sure, the fun easter egg had to go, but did it really need to be there in the first place? Clearly the 'X' and the checkerboard could have fit there the whole time. We even get some space to add some animation to the savefile icon. The second trick is also demonstrated in the same spritesheet; only the left half of the selector animation is stored in the sheet, and is flipped on runtime in order to produce the other half. I don't use this technique as often as I'd like, since for smaller sprites (like enemies) it gets a bit tedious, and larger sprites tend to be non-symmetric. Maybe before the final release I'll do a thorough sweep of all the symmetric sprites. That brings us to the final technique:

If possible, don't store a sprite at all.

Instead of storing the sprite in a spritesheet, why not store the instructions to make the sprite? Take for instance the Sound Test screen:

Now take a look at its spritesheet:

Hey, where'd the speakers go? Ah, here they are!

rectfill(dblbuffer, 12, 58, 83, 198, 18);
rectfill(dblbuffer, 139, 58, 210, 198, 18);
circlefill(dblbuffer, 50, 81, 15, 17);
circlefill(dblbuffer, 50, 117, 15, 17);
circlefill(dblbuffer, 50, 166, 26, 17);
circlefill(dblbuffer, 177, 81, 15, 17);
circlefill(dblbuffer, 177, 117, 15, 17);
circlefill(dblbuffer, 177, 166, 26, 17);
int drum_offset = is_bgm_playing ? rand()%3 : 0;
draw_rle_sprite(dblbuffer, speaker_drums[1], 47 + drum_offset, 77);
draw_rle_sprite(dblbuffer, speaker_drums[1], 47 + drum_offset, 113);
draw_rle_sprite(dblbuffer, speaker_drums[0], 45 + drum_offset, 155);
draw_rle_sprite(dblbuffer, speaker_drums[1], 174 + drum_offset, 77);
draw_rle_sprite(dblbuffer, speaker_drums[1], 174 + drum_offset, 113);
draw_rle_sprite(dblbuffer, speaker_drums[0], 172 + drum_offset, 155);
rect(dblbuffer, -1, 57, 84, 199, 16);
rect(dblbuffer, 99, 57, 211, 199, 16);
rectfill(dblbuffer, 0, 58, 10, 198, 18);
rectfill(dblbuffer, 100, 58, 137, 198, 18);
fastline(dblbuffer, 11, 58, 11, 198, 17);
fastline(dblbuffer, 138, 58, 138, 198, 17);
drawing_mode(DRAW_MODE_MASKED_PATTERN, speaker_mesh, 0, 0);
rectfill(dblbuffer, 12, 58, 83, 198, 19);
drawing_mode(DRAW_MODE_MASKED_PATTERN, speaker_mesh, 1, 0);
rectfill(dblbuffer, 139, 58, 210, 198, 19);
solid_mode();
fastline(dblbuffer, 16, 58, 16, 198, 18);
fastline(dblbuffer, 17, 58, 17, 198, 18);
fastline(dblbuffer, 143, 58, 143, 198, 18);
fastline(dblbuffer, 144, 58, 144, 198, 18);

Sure, it's a lot less flexible and way more complex than simply storing a speaker sprite in the sheet. But considering the size and dither pattern on the speakers, drawing via code makes much more sense when aiming for space efficiency. Unfortunately, CPU usage concerns limit this technique to very specific areas (like menus, where not much is going on).

Disclaimer: I actually have no clue how much space the compiled code takes up. I'm guessing not much compared to a RLE-compressed sprite, but I haven't gone in with a disassembler to check.

I think this is really all I can do in regards to executable and sprite storage, without getting into extremely weird, esoteric techniques à la the demoscene. There's also the topic of sound storage: MIDIs can't be made any smaller than they already are (besides getting rid of extraneous events), and WAVs are simply a balancing act of size vs. quality, with no alternatives other than Creative's VOC format (of which I have doubts about cross-platform compatibility). Again, I can't use something like OGG, since Allegro didn't support that until well after DOS support had been removed.

Porting Progress

I mentioned in Journal #1 that I don't want to limit REMIX to DOS, and am working to provide native Windows, Mac, and Linux ports as well. Here's the progress and findings I've made for those platforms thus far.

Windows

Still haven't tried this port. Moving along!

Linux

This one is... interesting. For whatever reason, the Linux port of REMIX runs at a framerate that is half the refresh rate of the monitor. I can remove the call to vsync() that's made before drawing the framebuffer and the game runs unlocked as expected, but I still can't seem to figure out why vsync() waits for every other refresh.

macOS (<= 10.14)

I actually tried this port (and the Linux one as well) long before I had written Journal #1, just never mentioned it. It suffers from really bad performance issues, as well as (I think) the same vsync() problem that Linux has.

Haiku

Ok, fine; I didn't even mention this one.

The Haiku port BARELY works. It runs, don't get me wrong. But it absolutely struggles. It's a combination of the weird vsync problem that Linux has, PLUS a new problem: certain gamestates do not update the screen. Unless, of course, focus is removed and replaced on the window. It's really bizarre, especially considering the fact that only certain gamestates will do this. For instance, the Epocti logo is fine (albeit slow), but as soon as the intro starts, redraws stop. I don't know if this is an issue with my rendering code, or the port of Allegro to Haiku. So for now, I'm leaving it as-is until I fix the vsync issue first.

Of course, with each port I do, the Makefile is updated and build parameters are added accordingly, so I'm never retreading old ground. Once the initial porting work is done and the game is at least running, it's all down to fixing any weird lingering issues in the source.

Framerate Fun

In Journal #1, I mentioned that I had plans for something called "dynamic framerate switching", and proceeded to not explain what that meant. It's actually a terrible name; if I were to go back and edit the post, it should probably be called "game speed management" instead. It really has more to do with altering the speed of the game in response to the framerate of the game; framerate isn't something that can be "switched". But I digress.

Initially, The only way the game could tell what framerate it was running at was via a variable that was set to either 1 or 2, called "game_tick_step". If the value was 1, that meant the game was running at 60fps. If it was 2, that meant 30fps. Essentially, the game would simulate running everything at 2x speed when running at 30fps in order to avoid slowdown.

There were many problems with this approach. First of all, the variable wasn't indicative of the current framerate at all. Tons of sprites could be drawn to the screen to the point where slowdown was occurring, but as far as the game was aware, it was still running at 60fps. The only way to tell the game to properly run at 30fps was to change a setting in the setup menu. Only then it would always run at 30fps to maintain a constant framerate without slowdown. There was also the fact that "fake" timers were set up to check the number of ticks that had elapsed, and used remainders that had to be multiples of 2 in order to maintain compatibility with 30fps mode. And another issue: what happens if slowdown occurs while running at 30fps? What about running the game faster than 60fps?

My first idea (actually, the idea I mentioned in Journal #1) was more of a band-aid; it used the same tick_step system, but starting at 1 = 120fps instead. There was more to it though; the idea was that the game would track how many objects were on-screen at any given time, and through some sort of magic, determine if the game was experiencing slowdown. If slowdown was occurring, the tick_step would increase, and the number of objects were recorded. As the game would continue, it would compare the number of objects on-screen to the list, and switch game_tick_step accordingly. I was so confident in this idea.

Then it hit me. How do we determine slowdown? A clock function. What can we do to tell what framerate we're running at? Clock function before and after calling vsync(), then compare that to the time it takes for a single screen refresh.

And with a bit of math, I have the actual speed to run at, scalable across any refresh rate. It's not exactly perfect, though; DOSBox has some stuttering issues that cause the game speed to jump to 2x for a single frame at a time, and it's a bit jarring. I added some code to take the average speed based on framerate history, but the stuttering issues still manage to leak through a bit. I guess I'll just have to work on it a bit more.

I even added a graph! How fancy!

From the new frame timing system, I also learned that DOS games using VGA mode 13h run at 70fps, not 60. ¯\_(ツ)_/¯

Timer Troubles

Oh god. Where do I start.

With the new frame timing system, the tick-based system has to go. Unfortunately, I tied so much time-sensitive code to it, using remainders on the elapsed number of ticks for periodic timers and whatnot. So now I'm stuck trying to come up with a new timer system that'll be easy to work with.

Allegro has a timer system, based on interrupts. I'm trying to avoid using it since that would effectively make the game kinda multi-threaded, and that's the last thing I'd like to deal with in C. It doesn't help that Allegro's timer system messes with the FPU state a bit, and since I've found myself using more floating point variables ever since I added the new frame timing system, that sounds like a recipe for disaster.

The idea I currently have is a pair of functions called "timer_do_after" and "timer_do_every", for one-shot and recurring timers respectively. The idea is to call these in the update function of a gamestate (in other words, every frame) as part of an if statement. The functions would return "1" if the timer is complete and "0" otherwise, meaning they can control access to a portion of code without the need to define a separate function. It would look something like this:

void do_game() {
	// do some stuff
	
	if (timer_do_after(identifier, number_of_milliseconds)) {
		some_cool_value++;
		make_explosion(enemies[0]);
		kill_enemy(enemies[0]);
	}
	
	// do more stuff
}

Something like that.

Things get less simple once you turn your attention to the "identifier" parameter. The issue with this timer system is that the timers need to rely on a table of in-progress timers in order to track their progress and tell if the specified time has elapsed. Which normally wouldn't be an issue, until you realize what happens if a timer needs to be run on a per-struct basis. How do you differentiate those timers from other ones sharing the exact same program counter value? The idea I had was to generate a random identifier per each timer and use that to look them up, but the more I plan it out, the hairier it gets. Now I have to implement a hashmap? What happens if two timers coincidentally have the same generated identifier? How fast would it be to look up a timer based on its identifier? Does a one-shot timer remove its own entry after finishing? Then how would it know not to start again?

I'm very much open to alternative ideas for a simple, yet effective timer system like this. Maybe I should just bite the bullet and go with Allegro's own timer system. Who knows.

Ok, enough engine development talk. It's definitely not for everyone, and my head hurts just writing about it. How about the player-facing stuff; y'know, the game?

Shop Scheming

It's not implemented into the game yet, but here's what the shop will look like (besides the background, which I haven't started)!

Say hi to the shop worker, aka "rat dude working a dead-end retail job". That's actually the sole description I designed him around.

Speaking of the shop, I made a pretty big alteration to the gameplay plans. In the original Endless Attack, the player could purchase any item they so desired during gameplay, as long as they had the funds, via the hotbar. The same held true in the gameplay plans for Endless Attack X. The key difference being that, during cooldown periods, the player would be able to visit an "extended shop", and change which items would be purchase-able via the hotbar. When beginning work on REMIX, I kept the same shop premise, but recently I decided to explore other options.

Why? I worry that, with a fully-open shop, players will simply subscribe to a meta and only save up for/purchase items and weapons that have a great advantage for different groups of waves. Since I want players to experiment with different items and weapons instead of immediately going for "the known best loadout", I first thought of a sort-of "weapon tree". With it, the first weapon upgrade the player makes would "lock" them to that weapon type (and upgrades for it) for a while, before giving the option of switching to a different type (and locking to that for a while as well). But I felt like this was simply too convoluted, and didn't really address the meta issue. Players would just go for the better weapon type path immediately, try as I might to balance all of them.

That brings us to the current shop plan: Instead of giving the player free reign, do what some roguelike/roguelite games do and provide a limited, randomized choice of items/weapons each time the shop is accessible. The set of possible items to pick from will be based on the current wave, scaling with the gradually-increasing difficulty. There's the chance of a "shop re-roll" item, to get a new set of items to pick from without needing to wait until the next shop (every 5 waves). There's also the chance of a "shop extension" item, to increase the number of items available to pick from permanently... or at least until the run ends.

The previous shop made the hotbar serve two different purposes, and felt a bit cumbersome since it acted as both an inventory and mini-shop at the time. With the more limited shop, the hotbar will no longer be used for purchasing items, instead acting solely as an inventory, which I feel would be a lot easier for new players to understand.

Graphics Gimmicks

One day, I decided to choose violence. "What if there were post-processing effects in a DOS game?", I asked myself.

This was the result. I have no clue if I'll use this effect in the final game, I just wanted to do it because I could.

Moving away from weird experiments, here's this cool transition effect I made. It'll be used when entering/exiting the shop, and maybe a couple other places.

The "out" transition is a bit unpolished since I do a tiny bit of overdraw not visible with the "in" transition.

Conventional Conclusion

From this journal, it feels as if I'm writing a full on game engine at this point. Technically, I am. I'm hoping to keep the engine side of things as separate as possible from the game itself, so at some point I can turn the engine into its own library for others to use and make their own cross-platform DOS games (without all of the pain I'm having to go through first). Allegro is very fun to work with, and my goal is to eventually make it even more accessible!

The hope is that, once I'm done writing the timer system and the sprite-management systems I mentioned in Journal #1, development can go full steam ahead. I've nearly finalized the fictitious game design document (which exists only in my head), so I know precisely what I'll need to work on at this point. Expect another journal soon!

The header alliteration was originally unintentional. It was only after I realized I had done it twice that I decided to keep it going. I promise not to do it again.