New TS dictionary

[jfoug]

I have created a c++ file that generates a new input dictionary. I am hopeful that this dictionary will help put a bit more stress on JtR, especially on length boundary issues.

There are no passwords with a ':' char (colon)
there are no passwords that have a space and ( char together, i.e " (" (this is a problem for some new logic in the TS)
I tried to get the start of each password to be unique. This is so that JtR chopping off the word to a shorter length will not match anything.
All passwords are 4 bytes or more (except for a null pw, and a single letter password).
I tried to make some sections vary greatly between lengths of adjacent passwords.
I have areas where passwords grow or shrink in size each time.
I have areas where passwords grow or shrink every 5 passwords.
I have values that have high bits set (but it was much more controlled that before).

Please comment on this strategy NOTE, if we put the passwords in the input files, they will be larger, since we are using longer passwords for input.

580bc7d
59e8737

[magnumripper]

So we never test length 2 and 3? What is the rationale for that?

[jfoug]

None, other than it was not coded in, but there normally are not too many issues with smaller passes. It would take little to add some.

The changed files (only on my dev system, not checked on ) have found some issues in the newer code of the ts.

I am almost certain that if the input files are Regen, there would be several formats where we flush out problems.

I think best way to make input files, is to make them 1500 to longest possible plaintext Len, then put 1500 and then add the number of hashes that have shorter passes, like 1112 for 55 byte limited SIMD hashes. This allows full length testing of both types.

One thing I can think of, is to make sure certain key lengths are handled very early on the dict file, since some formats use same salts with the first few words of the dict so we need to have the first few be very well chosen lengths.

[jfoug]

Here are the plaintext lengths used in jtr (an SIMD build, and a 'any' build). NOTE, did a -list=format-details and sort/unique the 2nd field, for the 2 builds

7
8
12
14
15
16
19
20
22
23
24
25
27
30
31
32
35
38
39
40
46
47
48
51
53
55
56
63
64
70
72
78
80
81
86
87
90
94
95
99
101
107
108
110
111
120
125

If I remove the dynamic formats from length checking (since most dyna will NOT require a tiny pw file), I have these plaintext lengths:

7
8
12
14
15
16
19
20
22
23
24
25
27
30
31
32
35
38
39
40
46
47
51
55
63
64
70
72
80
81
87
95
99
107
110
111
120
125

[magnumripper]

I think we should test every possible length, including very short. We had a few problems in the past that only affected a certain length due to buffer cleaning or optimizations.

[magnumripper]

Length 56 is important too. And length 28.

[jfoug]

Dyna 1015 and dyna5 both use length 56 (for non SIMD builds). Why would you worry about length 56? normally for the 55 byte items those words would simply be truncated. Yes, I know they need testing, and will be for most TS input file builds. What is important about length 28?

One of the more problematic (still) long length items is handling longer lengths of multi-byte encoded items (utf8, etc), when JtR core handling only deals with byte length. Hopefully we have gotten most of those issues worked out by now, but I am not 100% confident.

[magnumripper]

Why would you worry about length 56?

Because it's a critical length for formats that do support two limbs of MD4/MD5/SHA1. We've had such bugs in the past. Length 28 is same same for UTF-16.

[jfoug]

Gotcha. So that is mostly for non simd 2 by hashing overflows. I do not remember those bugs. But will trust ur memory

[magnumripper]

The one I recall was one (or both?) of the SAP formats I implemented SIMD for. It had some flaw making only length 56-63 fail (or something like that) and it took a while before we noticed (very likely the TS caught it when you implemented SAP in there). At those lengths, the 0x80 still goes in the first limb but the length word goes in the 2nd and that ended up wrong IIRC.

[jfoug]

'most' true hash formats (except the really slow ones) will not matter at all, since there are all lengths, from 0 to 130 scattered in the file. The 'problems', is when there are ONLY a few passwords, due to such slow hashes. I will try to run 150 or so, with 10 salts for some, but some formats do not allow 'same-salt'. For those, I will run only the first 25 passwords, (or 50, etc), and then append the file over and over again with same 25 input words. This makes these 'first' hashes pretty important for length. The 'normal' hashes will not really matter. If we make 1500 count input files, and the max-len is 125/55, then there will be plenty of max length passwords scattered in the file.

[jfoug]

I have just done all dyna formats, and am finishing up a run on non-simd. I made a perl script named count finder. It will read the pw-new.dic file for 1500 items of a max length, and also count the seen items for a shorter length. That shorter length has to be added as a 'valid' length to the expected count line.

The only 'issue' I found in the dyna run, was that formspring hard codes plaintext length to 32, while dyna-61 handles 90 byte lengths. I will need to check other thin formats, and possibly make sure that plain length is properly being handled. It 'should' be used from the dyna side, EXCEPT for the semi-thin formats. In that case, we need to make sure that plain_len for the fat side of the format is as big or smaller than the plain length on the dyna side.

But without 'pushing' the limits, I would not have found this behavior. I think it is something we should look at addressing somehow.

I will move on to the other non-dyna formats shortly. I am not 100% sure I will check this in just yet. I may play with it, get things done, but in the end, I may re-do the pw-new.dic file. However, I do have scripts that will be done, so I can re-gen the input files very quickly, so not too much work will be lost.

[jfoug]

@magnum / @frank-dittrich , what are your ideas about having a lot more test cases added? I do not want to drop the original ones just yet (but at some time, they 'can' also go. I have about 200 or so new input files. Most are dynamic, but I have also started on some of the others

The reason I ask, is this is a pretty big addition. I just do not want to lose my work. Things are pretty hectic right now for me, so just do not want to leave things laying around too much.

[magnumripper]

I don't care about the size, especially not (semi-)temporary. As an alternative you could branch off and put them there instead, just for "storing" them in a safe place.