Update Llama config to use Llama block and RoPE lower precision

[2015aroras]

Updating the Llama config to use Llama block and RoPE lower precision, to match the behavior of bf16-autocast Llama more closely.

[dirkgr]

This is good, but I want to run that separately first to see if it makes a difference. We'll have to wait a bit to get cluster time.

[2015aroras]

This is good, but I want to run that separately first to see if it makes a difference. We'll have to wait a bit to get cluster time.

Do you want a separate config then, or should I just keep this PR on hold until you're ready to take it?

[dirkgr]

Let's keep this on hold for a bit, but if it gets too long, we'll merge it as a separate config.

[dirkgr]

I just ran this on Beaker, and it said this:

RuntimeError: When using the full_megatron init, every module must have a type.

coming from /home/dirkg/LLM/olmo/model.py:826.

Can you find some place to run this to avoid errors like this, even if it's a tiny batch and sequence length, just for a few batches?

[2015aroras]

I just ran this on Beaker, and it said this:

RuntimeError: When using the full_megatron init, every module must have a type.

coming from /home/dirkg/LLM/olmo/model.py:826.

Can you find some place to run this to avoid errors like this, even if it's a tiny batch and sequence length, just for a few batches?

That runtime error is now fixed on a local hackish setup I have. I'll try running it on beaker briefly to see if anything else shows up.

[2015aroras]

This seems to run fine on beaker (with reduced model size to adjust for lack of GPUs).