This is the second part of the Flare-On 6 CTF WriteUp series.

2 - Overlong

The challenge reads

The secret of this next challenge is cleverly hidden. However, with the right approach, finding the solution will not take an overlong amount of time.

We have a PE file named Overlong.exe. Running, a MessageBox pops up.

It displays "I never broke the encoding". The binary doesn't exhibit any other behavior. Let's debug in x64dbg.

Analyzing the code at the entrypoint, we can see a call to sub_1231160 followed by a call to MessageBoxA@16. MessageBoxA is a WinAPI function which displays a modal dialog box on the screen. Hence the MessageBox we saw must be the result of calling this function.

As per the docs, the function takes 4 parameters. The second parameter LPCTSTR lpText contains a pointer to the string to be displayed. We set a breakpoint on the call and allow it to run as in Figure 2. Once the breakpoint hits, let's have a look at the function arguments window in x64dbg at the right.

Let's try to decode the arguments

HWND hWnd: NULL
LPCTSTR lpText: 0x001AFAAC (Pointer to "I never broke the encoding: "
LPCTSTR lpCaption:0x01233000 (Pointer to "Output")
UINT uType: 0 (MB_OK)

This explains the MessageBox call but what about the other function which was called before this. Reading assembly can often be time consuming. Let's load it in Ghidra which is an open source reverse engineering tool. Among other features, it sports a decompiler which converts the assembly listing to pseudo C, speeding up our analysis.

The decompiled code for the function at the entrypoint looks like Figure 4. local_88 is a array of 128 bytes located on the stack. This is passed to the function FUN_00401160 as the first argument. The second argument &DAT_00402008 is a pointer within the data section. The last argument is the integer 0x1c.

Navigating to 402008 we see a bunch of bytes.

It looks that this is an encrypted piece of data which the function is going to decrypt with the output written to the local_88 array. We don't know yet the purpose of the third argument 0x1c, it may be the key.

This function in turn calls FUN_00401000. However, it doesn't look like that the third parameter (0x1c) is the key as it is used as the loop upper bound. Interestingly, the number of characters in the string "I never broke the encoding: " is also 0x1c or in 28 decimal. Could it mean that it refers to the number of characters in the final output? Let's check FUN_00401000 before deciding.

This looks to be the function performing the actual decryption. However, it just takes two parameters - input and output. It does not take a key. This suggests that 0x1c is not the key but rather the length of the output. Let's run the program in x64dbg and try increasing the passed value (0x1c).

This is easy to do. Set a breakpoint at the call instruction. When the breakpoint hits, double-click on 0x1c on the stack window to open the Modify dialog. From here, we can set any value. Lets put 38 and run.

This time we get some piece of additional text at the end which looks to be a part of the flag. Our assumption that 0x1c is the length of the output is right. From Ghidra, we know that the maximum length of the output is 128 which is the length of the output array. Lets re-run the program changing the parameter to 128.

And that's the flag.

Flag I_a_M_t_h_e_e_n_C_o_D_i_n_g@flare-on.com