This is a story how an old bug came to haunt me.

Once upon a time, I decided to work on the ddhx hex editor again.

I added some preparation work and additional unit tests for upcoming planned enhancements to the pattern engine. Tests on my machine are good, so let’s push the commit!

GitHub picks up the commit, start rolling out its Actions, and adds this lovely notification in my inbox:

dd86k/ddhx – Tests #303
Tests workflow run failed for master branch

That happens, let’s see which test.

ddhx (ubuntu-22.04, gdc) test failed

Alright, let’s check the logs.

Failed to throw with: [“x:00”, “INVALID:ff”]
core.exception.AssertError@src/ddhx/pattern.d(245): test_throw test failed

This test in particular checks if the pattern function throws on invalid input. A pattern consists of one or most patterns that create an array of bytes, as a needle, to be used with the search function.

A valid example includes x:00 01, this creates the array [0x00, 0x01]. A bad example would be something with an unknown prefix, like idk.

A pattern prefix determines the type of parsing to perform. x: and 0x will be hex, d: for decimal, etc.

In short, a new test I added fails with an old D compiler release.

I got to confirm it on my own machine:

• GDC 15: Test passes
• GDC 14: Test passes
• GDC 13: Test passes
• GDC 12: Test passes
• GDC 11: Test fails

Okay, makes sense. The gdc package, on Ubuntu 22.04, points to gdc-11, the default GNU D Compiler package. Here, it is specifically GDC version 11.4.

I have two options:

• Drop support for gdc-11 specifically, which is a sane thing to do.
• Investigate, because that’s my sort of thing.

Ah, you know me, we both know where we’re going.

Wrong Direction

I was rather perplexed at the test. It throws perfectly fine on LDC and DMD. But, why this specific combo of items on gdc-11?

The successful tests prior to the erroneous one:

• Empty values: [""], ["x:"], ["o:"], ["d:"], ["s:"], ["0x"], ["\""], ["00"]
• Unknown prefixes: ["INVALID:ff"], ["INVALID:"]

Curious to know if DMD was also affected, since it’s fine on GDC 12, I assumed it to be front-end related… Checking GDC 11’s front-end version using __VERSION__, I get 2076. 2.076 is quite old, we’re talking a 2017 release!

Armed with a virtual machine, I downloaded dmd-2.076.0-0_amd64.deb, installed, ran tests… Compiler is very unhappy about the code.

Right, I am using Add Expression-Based Contract Syntax (DIP1009). Let’s retry with 2.081, when this feature was introduced… Same story.

Wait, I remembered that GDC from that era has a custom front-end written in C++. Its reported version does not represent a proper DMD release. GDC only started using DMD as a front-end in version 12 and later. For example, GDC 12.4 reports 2.100 (2022).

As a rough estimate of a version, let’s go by its release year:

$ gdc-11 --version
gdc-11 (Ubuntu 11.5.0-1ubuntu1~24.04.1) 11.5.0
Copyright (C) 2021 Free Software Foundation, Inc

To see if this was a GDC-specific issue, these compilers were used, in order:

• dmd-2.095.1 (2021): Passing
• dmd-2.090.0 (2020): Passing
• dmd-2.084.0 (2019): Failing with the same message

Well, interesting. Not a GDC specific issue then! This at least eliminates anything related to the back-end.

“Bisecting” compiler versions:

• dmd-2.084.1: Failed
• dmd-2.085.0: Failed
• dmd-2.085.1: Failed
• dmd-2.086.0: Failed
• dmd-2.086.1: Failed
• dmd-2.087.0: Failed
• dmd-2.087.1: Failed
• dmd-2.088.0: Failed
• dmd-2.088.1: Failed
• dmd-2.089.0: Failed
• dmd-2.089.1: Failed
• dmd-2.090.0: Pass

Nice, we now know that something was fixed between 2.089.1 and 2.090.0. Let’s look at the list of bugfixes:

DMD Compiler bugs:
14. Bugzilla 20401: ref variable copied before return

Oh, maybe an issue related to Ref Parameters? That’s possible, let’s try remodelling the patternpfx function.

This function is in change of taking one parameter, slice out the prefix of an input, and return the prefix type. It has a signature of PatternType patternpfx(ref string input).

After reshaping it to return a structure, it has a new Prefix patternpfx(string input) signature, getting rid of the Ref Parameter.

At the very least, this change makes the API a little clear. We’re going from

string p0 = "0x00";
assert(patternpfx(p0) == PatternType.hex);
assert(p0 == "00");

to

assert(patternpfx("0x00") == Prefix("00", PatternType.hex));

Did the remodelling work? Nope, same issue.

Getting It

The pattern function returns a new array of bytes from the pattern it received.

The way it works is rather simple: For each parameter received, it detects the prefix, strips the prefix, parses the value depending on that prefix as a 64-bit number, slices it into a byte array, and appends the array into the output array.

If the former prefix is valid and the current parameter scanned has an invalid prefix (assuming it’s missing), the current value is retried using the former prefix type.

Up to this point, I’ve never checked what it actually produces, since it successfully returns data. But, enough guessing, let’s do some print debugging!

So, the arguments ["x:00", "INVALID:ff"] successfully return an array of [0, 0]. The first parameter makes sense, but not the second!

Next, let’s validate which path the function takes:

-- args=["x:00", "INVALID:ff"]
---- case: hex
---- unknown: retrying
---- case: hex

That seems normal, retrying with gdc-12…

-- args=["x:00", "INVALID:ff"]
---- case: hex
---- unknown: retrying
---- case: hex

Uh oh, same path with gdc-12 too? Wait a minute, am I relying on a specific behaviour of unformatValue?

unformatValue

When unformatValue fails to unformat (parse) a string value, it throws a ConvException. ConvException? Yes, unformatValueImpl uses parse (std.conv) for integer values!

Here’s the kicker: This is exactly what is happening here. The parse function is not throwing the expected exception.

Checking Phobos sources for std/conv.d with commits between tags v2.089.1 and v2.090.0, there is this lovely commit:

radix overload of std.conv.parse fails to throw on non-empty range without number

• ✅ We’re using the radix overload indirectly from unformatValueImpl
• ✅ We have a non-empty range without numbers (“INVALID”)
• ✅ It’s not throwing with versions prior to 2.090.0
• ✅ It’s fixed in 2.090.0-beta.1, included in 2.090.0

With these points in mind, our fix should be rather simple. When __VERSION__ < 2090, we should check for a range of characters that are only valid for the given prefix.

Here’s an example:

final switch (pfx.type) {
case PatternType.hex:
    static if (VERSION < 2090)
    {
        import ddhx.platform : assertion;
        assertion(
            (pfx.str[0] >= '0' && pfx.str[0] <= '9') ||
            (pfx.str[0] >= 'a' && pfx.str[0] <= 'f') ||
            (pfx.str[0] >= 'A' && pfx.str[0] <= 'F'),
            text("Not a hex number", pfx.str));
    }
    // Parse...

And finally…

$ dub test --compiler=gdc-11
             Generating test runner configuration 'ddhx-test-library' for 'library' (library).
    Starting Performing "unittest" build using gdc-11 for x86_64.
    Building ddhx 0.9.2+commit.4.g45bfd15: building configuration [ddhx-test-library]
     Running ddhx-test-library
All unit tests have been run successfully.

Nice! Only took me about 5 lazy hours.

Obviously, all characters should be checked. This is only to reduce some amount of future frustrations with a specific compiler version. No sweat about it.

I even took the time to use std.conv.parse directly, since I am already stripping any kind of prefix. Turning switch cases from

static immutable auto xspec = singleSpec("%x");
ulong b = unformatValue!ulong(arg, xspec);

to

ulong b = parse!ulong(pfx.str, 16);

Cleaner!

Now, I can rest easy. I was not particularly worried, because ddhx 0.9.2, the current static version for Linux, is built with a version of 2.110, which includes this fix.

Moral of the story?

Hm, it’s probably “make sure you test specific behaviours with valid and invalid data more often”. Always a big yay for more tests.

At the same time, it is only fair if you wish to stick supporting newer compiler releases. For a hex editor, I consider supporting older compilers more important to me than my other projects, like my recent vrcd project.

What fascinates me are other articles that go on debugging steaks that stretch for multiple weeks. Since I was inspired by these articles I have previously read in the past, I decided to write something similar.

That was fun! I hope you had some fun reading this. Have a good day.