llama 2 70B not usable on tinybox green

[wozeparrot]

CUDA=1 python3 examples/llama.py --gen 2 --size 70B --shard 6 --prompt "Hello." --count 10 --temperature 0 --timing

Runs out of host memory at loading 90% of the weights.

[nimlgen]

Yeah, NV should be a bit better but also OOMs. Wrote a custom allocator to fit it better, but it OOM when creating graphs. Is there any way to remove this reserved memory (like it stops when it has >500ms and requesting ~80mb)?

[geohot]

It's not the GPU memory that's the problem AFAIK, it's the host memory not being freed.

[nimlgen]

Hmm, need to retest. I recall trying to set cuMemcpyHtoD_v2 in copyin and it still was an OOM during gpu buffer allocation