Adding Mersenne Twisters Impl.

[azret]

Adding numerically identical to torch rand utils for when we need to init_from_scratch #243
We can init on a CPU and mem copy to GPU.

Usage:

    mt19937_state state;
    manual_seed(&state, 137);
    printf("%u\n", randint32(&state));
    printf("%u\n", randint32(&state));
    printf("%u\n", randint32(&state));
    printf("%u\n", randint32(&state));
    printf("%u\n", randint32(&state));

    float t8[8];
    normal_(t8, 8, 0, 1, &state);
    for (int i = 0; i < 8; i++) {
        printf("%f\n", t8[i]);
    }
    printf("%u\n", randint32(&state));

    float t16[16];
    normal_(t16, 16, 0, 1, &state);
    for (int i = 0; i < 16; i++) {
        printf("%f\n", t16[i]);
    }
    printf("%u\n", randint32(&state));

    import torch
    torch.manual_seed(137)
    print(torch.randint(0, 0xFFFFFFFF, [1]).item())
    print(torch.randint(0, 0xFFFFFFFF, [1]).item())
    print(torch.randint(0, 0xFFFFFFFF, [1]).item())
    print(torch.randint(0, 0xFFFFFFFF, [1]).item())
    print(torch.randint(0, 0xFFFFFFFF, [1]).item())
    t = torch.zeros(8);
    t.normal_()
    for i in range(len(t)) :
        print(t[i].item())
    print(torch.randint(0, 0xFFFFFFFF, [1]).item())
    t = torch.zeros(16);
    t.normal_()
    for i in range(len(t)) :
        print(t[i].item())
    print(torch.randint(0, 0xFFFFFFFF, [1]).item())

Output for both:

    // 4053805790
    // 2173880614
    // 380293709
    // 1237255315
    // 2986595568
    // 0.7947664260864258
    // 1.4369317293167114
    // - 0.2292192131280899
    // 0.47556325793266296
    // - 0.6334410905838013
    // - 0.5791953802108765
    // - 0.0925704762339592
    // - 0.8659197092056274
    // 2186503452
    // - 1.2813878059387207
    // - 2.646395683288574
    // - 0.06569503247737885
    // 0.2180829495191574
    // - 0.46536165475845337
    // - 0.33108410239219666
    // 2.5485482215881348
    // 0.10425379872322083
    // 0.8460659980773926
    // 0.9462448358535767
    // - 0.2913765013217926
    // 0.34313806891441345
    // - 1.1186704635620117
    // - 0.18305328488349915
    // - 2.3153159618377686
    // 0.3961987793445587
    // 2756748748

[karpathy]

Nice! This is actually super convenient because it may mean that we could have tests for our training matching that of PyTorch from scratch, without having to save/load checkpoints. We just seed rng the same way and do the init the same way. I'll take a look shortly!

[ngc92]

at least for the cuda code, note that the C++ standard library directly has a mersenne twister implementation
https://cplusplus.com/reference/random/mt19937/