A system we're introducing into the organization is not using any traditional hashing function that I can find. I've been assigned to "approve" it by black-box testing.

I suspect passwords are simply being obfuscated or encrypted (not hashed) since the length of the result changes with password length (though there is some minimum, it seems - perhaps padding?). 

Here is some data:


password   : result
1          : TG3432WB7VU=
11         : r8ahLkJGWbY=
111        : E7fXdNqEWAA=
11111111   : FIcx3a00R4eGhFyHjD56qw==
1111111111 : FIcx3a00R4evxqEuQkZZtg==
2111111111 : GPwnH80qEACvxqEuQkZZtg==


The result is obviously base64, but doesn't decode to anything readable. Before I knew that the result's length changed, I tried taking the decoded bytes and treating them as MD5 and other hashing functions, but that obviously didn't pan out. Now that I know the result changes in length, I'm considering other, worse, alternatives. Of note are the two bold parts above: they're identical in two different passwords. So either each 8 bytes are being processed independently (why?) or there's some polyalphabetic substitution going on(?).

Update:

Any character, including Unicode characters, is accepted by the system for passwords. Repeating a 3-byte character 8 times does result in a 24 byte long "hash".

I highly suspect this is a self rolled, or at least very outdated method.  It is very weak by modern standards and should not be used to protect anything important.

Short passwords could be cracked with no more than 2^64 brute force attempts, which is quite possible with even a normal computer.  If both halves of the result are independent, even with fairly long passwords, 2*2^64 brute force attempts could crack it (so 2^65 attempts). There are likely further weaknesses in the algorithm that make it weaker than described here.

Another interesting point to test would be 2111111111 and see if the second part of the result remains the same.  If the second part doesn't change, this is definitely a weak algorithm.

Edit: 

Seeing that the results of the 2111111111 test are the same for the second half, this is definitely a weak algorithm and should not be used to protect anything sensitive!  I have included the relevant comparison below:


  
  1111111111 : FIcx3a00R4evxqEuQkZZtg==
  2111111111 : GPwnH80qEACvxqEuQkZZtg==
  


Edit:

Another interesting test would be what Ricky Demer suggests below:


  The next thing to check is that it handles the 8-byte blocks identically. ​ What are the "hashes" of aaaaaaaabbbbbbbbc and bbbbbbbbaaaaaaaac? ​ ​ ​ ​ – Ricky Demer Feb 25 at 6:44 


If it handled the 8-byte blocks identically, it is even worse for even long passwords, no matter what the length.  It would require just 2^64 brute force attempts as pointed out by Neil in the comments. Once someone made a table of what each possibility calculates to, this algorithm would be practically useless.  There are probably even more weaknesses to this algorithm remaining to be discovered.  

Bottom line:

I do not recommend using this algorithm regardless of the results of the last test.  We have already seen enough to know it is weak.  Something with a higher bit encryption / hash scheme, salt, long computation time etc... would be much better, and I would recommend going with one of the existing thoroughly tested methods.  Self rolled encryption / hashing has not had the benefit of extensive testing that most of the main stream methods have.
