How does Transformers without PE Learn Positional Information?

Overview

This repository contains the code for our investigation of positional encoding in Transformers.

trian.py contains the training loop.
pe_info contains our implementation of the experiments
pe_info/model_nope.py this is the one and only file for the model, which has modifiable PE, SC, and more.
pe_info_experiments contains the experiments we have run.
pe_info_experiments/tuning.py contains the code for tuning the hyperparameters.

This codebase forked from teaching arithmetic

Name		Name	Last commit message	Last commit date
Latest commit History 117 Commits
data		data
pe_info		pe_info
pe_info_experiments		pe_info_experiments
previous_currently_unused		previous_currently_unused
prompts		prompts
utils		utils
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
all_r.csv		all_r.csv
all_r_max.csv		all_r_max.csv
configurator.py		configurator.py
delete_extras.py		delete_extras.py
environment.yml		environment.yml
evaluate_additions.py		evaluate_additions.py
evaluate_models.py		evaluate_models.py
main_utils.py		main_utils.py
meta_all_ascii_chars.pkl		meta_all_ascii_chars.pkl
paridy_2seeds.csv		paridy_2seeds.csv
paridy_3seeds.csv		paridy_3seeds.csv
paridy_nc_2seeds.csv		paridy_nc_2seeds.csv
parity_1seeds.csv		parity_1seeds.csv
parity_2seeds.csv		parity_2seeds.csv
parity_nc_1seeds.csv		parity_nc_1seeds.csv
parity_nc_2seeds.csv		parity_nc_2seeds.csv
requirements.txt		requirements.txt
train.py		train.py