Skip to content

Commit 0aedb80

Browse files
MountchickenMountchicken
committed
[Update] Add Accelerate
1 parent bde33fe commit 0aedb80

File tree

5 files changed

+122
-117
lines changed

5 files changed

+122
-117
lines changed

‎Efficient_Coding.md

Lines changed: 25 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -239,37 +239,37 @@ If you are not familiar with Git commands, just follow [this guide](https://lear
239239
240240
## 8. Accelerate from HuggingFace🤗
241241
- [Accelerate](https://github.com/huggingface/accelerate) is created for PyTorch users who like to write the training loop of PyTorch models but are reluctant to write and maintain the boilerplate code needed to use multi-GPUs/TPU/fp16. Accelerate abstracts exactly and only the boilerplate code related to multi-GPUs/TPU/fp16 and leaves the rest of your code unchanged.
242-
```python
243-
import torch
244-
import torch.nn.functional as F
245-
from datasets import load_dataset
246-
+ from accelerate import Accelerator
242+
```python
243+
import torch
244+
import torch.nn.functional as F
245+
from datasets import load_dataset
246+
+ from accelerate import Accelerator
247247
248-
+ accelerator = Accelerator()
249-
- device = 'cpu'
250-
+ device = accelerator.device
248+
+ accelerator = Accelerator()
249+
- device = 'cpu'
250+
+ device = accelerator.device
251251
252-
model = torch.nn.Transformer().to(device)
253-
optimizer = torch.optim.Adam(model.parameters())
252+
model = torch.nn.Transformer().to(device)
253+
optimizer = torch.optim.Adam(model.parameters())
254254
255-
dataset = load_dataset('my_dataset')
256-
data = torch.utils.data.DataLoader(dataset, shuffle=True)
255+
dataset = load_dataset('my_dataset')
256+
data = torch.utils.data.DataLoader(dataset, shuffle=True)
257257
258-
+ model, optimizer, data = accelerator.prepare(model, optimizer, data)
258+
+ model, optimizer, data = accelerator.prepare(model, optimizer, data)
259259
260-
model.train()
261-
for epoch in range(10):
262-
for source, targets in data:
263-
source = source.to(device)
264-
targets = targets.to(device)
260+
model.train()
261+
for epoch in range(10):
262+
for source, targets in data:
263+
source = source.to(device)
264+
targets = targets.to(device)
265265
266-
optimizer.zero_grad()
266+
optimizer.zero_grad()
267267
268-
output = model(source)
269-
loss = F.cross_entropy(output, targets)
268+
output = model(source)
269+
loss = F.cross_entropy(output, targets)
270270
271-
- loss.backward()
272-
+ accelerator.backward(loss)
271+
- loss.backward()
272+
+ accelerator.backward(loss)
273273
274-
optimizer.step()
275-
```
274+
optimizer.step()
275+
```

‎Efficient_GPUtilization.md

Lines changed: 23 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,24 @@
11
# Efficient GPU Utilization
22

3-
- [1. CUDA out of memory solutions](#1-cuda-out-of-memory-solutions)
4-
- [1.1. Use a smaller batch size](#11-use-a-smaller-batch-size)
5-
- [1.2. Check if there is any accumulated history across your training loop](#12-check-if-there-is-any-accumulated-history-across-your-training-loop)
6-
- [1.3. Avoid creating new variables](#13-avoid-creating-new-variables)
7-
- [1.4. Delete intermediate variables you don't need](#14-delete-intermediate-variables-you-dont-need)
8-
- [1.5. Check if you GPU memory is freed properly](#15-check-if-you-gpu-memory-is-freed-properly)
9-
- [1.6. Turn off gradient calculation during validation](#16-turn-off-gradient-calculation-during-validation)
10-
- [1.7. COM in Google Colab](#17-com-in-google-colab)
11-
- [2. GPU Memory Saving Tips](#2-gpu-memory-saving-tips)
12-
- [2.1. Automatic Mixed Precision (AMP)](#21-automatic-mixed-precisionamp)
13-
- [2.2. Gradient Accumulation](#22-gradient-accumulation)
14-
- [2.3. Gradient Checkpoint](#23-gradient-checkpoint)
15-
- [3. Multiple GPUs](#2-multiple-gpus)
16-
- [3.1. Distributed model training](#31-distributed-model-training)
3+
- [Efficient GPU Utilization](#efficient-gpu-utilization)
4+
- [1. CUDA out of memory solutions](#1-cuda-out-of-memory-solutions)
5+
- [1.1. Use a smaller batch size](#11-use-a-smaller-batch-size)
6+
- [1.2. Check if there is any accumulated history across your training loop](#12-check-if-there-is-any-accumulated-history-across-your-training-loop)
7+
- [1.3. Avoid creating new variables](#13-avoid-creating-new-variables)
8+
- [1.4. Delete intermediate variables you don't need](#14-delete-intermediate-variables-you-dont-need)
9+
- [1.5. Check if your GPU memory is freed properly](#15-check-if-your-gpu-memory-is-freed-properly)
10+
- [1.6. Turn off gradient calculation during validation](#16-turn-off-gradient-calculation-during-validation)
11+
- [1.7. COM in Google Colab](#17-com-in-google-colab)
12+
- [2. GPU Memory Saving Tips](#2-gpu-memory-saving-tips)
13+
- [2.1. Automatic Mixed Precision(AMP)](#21-automatic-mixed-precisionamp)
14+
- [2.2. Gradient Accumulation](#22-gradient-accumulation)
15+
- [2.3. Gradient Checkpoint](#23-gradient-checkpoint)
16+
- [3. Multiple GPUs](#3-multiple-gpus)
17+
- [3.1. Distributed model training](#31-distributed-model-training)
18+
- [What is distributed training](#what-is-distributed-training)
19+
- [Data parallelization training](#data-parallelization-training)
20+
- [Use Accelerate for Fast Implementation](#use-accelerate-for-fast-implementation)
21+
- [Reference](#reference)
1722

1823
## 1. CUDA out of memory solutions
1924

@@ -313,6 +318,10 @@
313318

314319
![img](images/dataparal.gif)
315320

321+
#### Use Accelerate for Fast Implementation
322+
- Convert your code from single GPU to multi GPU can be very tedious. Fortunately, you can implement distributed training in PyTorch with just a few lines of code using the [Accelerate](https://github.com/huggingface/accelerate) library.
323+
324+
316325
## Reference
317326

318327
- [Training larger-than-memory PyTorch models using gradient checkpointing](https://spell.ml/blog/gradient-checkpointing-pytorch-YGypLBAAACEAefHs)

‎Efficient_Tools.md

Lines changed: 6 additions & 72 deletions
Original file line numberDiff line numberDiff line change
@@ -4,15 +4,9 @@
44
- [1. Torchinfo: Visualize Network Architecture](#1-torchinfo-visualize-network-architecture)
55
- [2. drawio: Free graphing software](#2-drawio-free-graphing-software)
66
- [3. Octotree: Free gitHub code tree](#3-octotree-free-github-code-tree)
7-
- [4. ACRONYMIFY: Name your paper with a cool acronyms](#4-acronymify-name-your-paper-with-a-cool-acronyms)
8-
- [5. Linggle: Grammer checker](#5-linggle-grammer-checker)
9-
- [Search for the correct preposition pairing](#search-for-the-correct-preposition-pairing)
10-
- [What can be the verb before `war`?](#what-can-be-the-verb-before-war)
11-
- [What nouns can be used after `execute`?](#what-nouns-can-be-used-after-execute)
12-
- [Check the usage for authenticity](#check-the-usage-for-authenticity)
13-
- [6. AI pair programmer: Github Copilot](#6-ai-pair-programmer-github-copilot)
14-
- [7. PARSEC: Free Remote Desktop](#7-parsec-free-remote-desktop)
15-
- [8. gdown: Download Large files from Google Drive in command line](#8-gdown-download-large-files-from-google-drive-in-command-line)
7+
- [4. AI pair programmer: Github Copilot](#4-ai-pair-programmer-github-copilot)
8+
- [5. PARSEC: Free Remote Desktop](#5-parsec-free-remote-desktop)
9+
- [6. gdown: Download Large files from Google Drive in command line](#6-gdown-download-large-files-from-google-drive-in-command-line)
1610
- [Reference](#reference)
1711
## 1. Torchinfo: Visualize Network Architecture
1812

@@ -40,68 +34,8 @@
4034
<img src='images/octotree.gif' width=300 >
4135
</div>
4236

43-
## 4. ACRONYMIFY: Name your paper with a cool acronyms
4437

45-
- **Website**:[http://acronymify.com/](http://acronymify.com/)
46-
- A good paper naming not only motivates the reader to read, but also makes your paper memorable, for example by using abbreviations of words to construct an interesting word. A typical example is the Sesame Street Family in NLP (BERT, ELMO, ERNIE ...). [Acronymify](http://acronymify.com/search?q=Efficient+Deep+Learning) can automatically generate a series of acronyms for the title of your paper, and you pick the one you like
47-
48-
<div align=center>
49-
<img src='images/acronymify.JPG' width=500 >
50-
</div>
51-
52-
## 5. Linggle: Grammer checker
53-
- **Website**: [https://linggle.com/](https://linggle.com/)
54-
- When you are writing in English, have you ever encountered these similar problems:
55-
* `in the afternoon`, `at the afternoon`, which one is correct?
56-
* The preposition after `present a method` is `for` or `to`.
57-
* What can be the verb before `war`?
58-
* What nouns can be used after `execute`
59-
* I just made up an expression `avoid his coming`, trying to determine if it was authentic.
60-
- Let's see how linggle can help you with it
61-
#### Search for the correct preposition pairing
62-
63-
- **Use / to space out the prepositions**:
64-
65-
<div align=center>
66-
<img src='images/linggle1.JPG' width=500 >
67-
</div>
68-
69-
<div align=center>
70-
<img src='images/linggle2.JPG' width=500 >
71-
</div>
72-
73-
- Linggle can look up the usage of relevant prepositions in its corpus and tell you which ones are most commonly used, and will give you many examples
74-
75-
#### What can be the verb before `war`?
76-
- **We can use `v. a war` to query.**, `v. a war` means verb ahead war.
77-
78-
<div align=center>
79-
<img src='images/linggle3.JPG' width=500 >
80-
</div>
81-
82-
<div align=center>
83-
<img src='images/linggle4.JPG' width=500 >
84-
</div>
85-
86-
#### What nouns can be used after `execute`?
87-
- **`excute n.`**
88-
89-
<div align=center>
90-
<img src='images/linggle5.JPG' width=500 >
91-
</div>
92-
93-
#### Check the usage for authenticity
94-
95-
<div align=center>
96-
<img src='images/linggle6.JPG' width=500 >
97-
</div>
98-
99-
- Saddly, `avoid his coming` is not authentic.
100-
101-
- **Linggle can do much more than those, and you can see more usages on its website.**
102-
103-
104-
## 6. AI pair programmer: Github Copilot
38+
## 4. AI pair programmer: Github Copilot
10539

10640
<div align=center>
10741
<img src='images/copilot.png' width=100 >
@@ -119,7 +53,7 @@
11953
<img src='images/copilot.gif' width=800 >
12054
</div>
12155

122-
## 7. PARSEC: Free Remote Desktop
56+
## 5. PARSEC: Free Remote Desktop
12357

12458
<div align=center>
12559
<img src='images/parsec.png' width=8500 >
@@ -128,7 +62,7 @@
12862
- **Website**: [https://parsec.app](https://parsec.app)
12963
- Parsec is a free, HD, powerful remote desktop. It allows you to play games with your friends, watch movies with your family, or collaborate with colleagues from anywhere.
13064

131-
## 8. gdown: Download Large files from Google Drive in command line
65+
## 6. gdown: Download Large files from Google Drive in command line
13266

13367
<div align=center>
13468
<img src='images/gdown.png' width=500 >

‎Efficient_Writting.md

Lines changed: 62 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,4 +45,65 @@
4545
</div>
4646

4747
- **Website**: [https://www.tablesgenerator.com/](https://www.tablesgenerator.com/)
48-
- Tables are an important part of your paper. This website provides a tool for you to generate LaTeX tables. You can also use it to generate tables in other formats, such as HTML, Markdown, etc.
48+
- Tables are an important part of your paper. This website provides a tool for you to generate LaTeX tables. You can also use it to generate tables in other formats, such as HTML, Markdown, etc.
49+
50+
## 6. ACRONYMIFY: Name your paper with a cool acronyms
51+
52+
- **Website**:[http://acronymify.com/](http://acronymify.com/)
53+
- A good paper naming not only motivates the reader to read, but also makes your paper memorable, for example by using abbreviations of words to construct an interesting word. A typical example is the Sesame Street Family in NLP (BERT, ELMO, ERNIE ...). [Acronymify](http://acronymify.com/search?q=Efficient+Deep+Learning) can automatically generate a series of acronyms for the title of your paper, and you pick the one you like
54+
55+
<div align=center>
56+
<img src='images/acronymify.JPG' width=500 >
57+
</div>
58+
59+
## 7. Linggle: Grammer checker
60+
- **Website**: [https://linggle.com/](https://linggle.com/)
61+
- When you are writing in English, have you ever encountered these similar problems:
62+
* `in the afternoon`, `at the afternoon`, which one is correct?
63+
* The preposition after `present a method` is `for` or `to`.
64+
* What can be the verb before `war`?
65+
* What nouns can be used after `execute`
66+
* I just made up an expression `avoid his coming`, trying to determine if it was authentic.
67+
- Let's see how linggle can help you with it
68+
#### Search for the correct preposition pairing
69+
70+
- **Use / to space out the prepositions**:
71+
72+
<div align=center>
73+
<img src='images/linggle1.JPG' width=500 >
74+
</div>
75+
76+
<div align=center>
77+
<img src='images/linggle2.JPG' width=500 >
78+
</div>
79+
80+
- Linggle can look up the usage of relevant prepositions in its corpus and tell you which ones are most commonly used, and will give you many examples
81+
82+
#### What can be the verb before `war`?
83+
- **We can use `v. a war` to query.**, `v. a war` means verb ahead war.
84+
85+
<div align=center>
86+
<img src='images/linggle3.JPG' width=500 >
87+
</div>
88+
89+
<div align=center>
90+
<img src='images/linggle4.JPG' width=500 >
91+
</div>
92+
93+
#### What nouns can be used after `execute`?
94+
- **`excute n.`**
95+
96+
<div align=center>
97+
<img src='images/linggle5.JPG' width=500 >
98+
</div>
99+
100+
#### Check the usage for authenticity
101+
102+
<div align=center>
103+
<img src='images/linggle6.JPG' width=500 >
104+
</div>
105+
106+
- Saddly, `avoid his coming` is not authentic.
107+
108+
- **Linggle can do much more than those, and you can see more usages on its website.**
109+

‎README.md

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@
2323
- [StackOverflow](Efficient_Coding.md#6-search-on-stackoverflow-first)
2424
- [Auto docstring](Efficient_Coding.md#7-automatically-format-your-docstring)
2525
- [Accelerate](Efficient_Coding.md#8-accelerate)
26+
2627
## 3.Efficient Data Processing
2728

2829
- Strategies to speed up your data processing.
@@ -63,11 +64,9 @@
6364
- [Torchinfo: Visualize Network Architecture](Efficient_Tools.md#1-torchinfo-visualize-network-architecture)
6465
- [drawio: Free graphing software](Efficient_Tools.md#2-drawio-free-graphing-software)
6566
- [Octotree: Free gitHub code tree](Efficient_Tools.md#3-octotree-free-github-code-tree)
66-
- [ACRONYMIFY: Name your paper with a cool acronyms](Efficient_Tools.md#4-acronymify-name-your-paper-with-a-cool-acronyms)
67-
- [Linggle: Grammer checker](Efficient_Tools.md#5-linggle-grammer-checker)
68-
- [AI pair programmer: Github Copilot](Efficient_Tools.md#6-ai-pair-programmer-github-copilot)
69-
- [PARSEC: Free Remote Desktop](Efficient_Tools.md#7-parsec-free-remote-desktop)
70-
- [gdown: Download Large files from Google Drive in command line](Efficient_Tools.md#8-gdown-download-large-files-from-google-drive-in-command-line)
67+
- [AI pair programmer: Github Copilot](Efficient_Tools.md#4-ai-pair-programmer-github-copilot)
68+
- [PARSEC: Free Remote Desktop](Efficient_Tools.md#5-parsec-free-remote-desktop)
69+
- [gdown: Download Large files from Google Drive in command line](Efficient_Tools.md#6-gdown-download-large-files-from-google-drive-in-command-line)
7170
----
7271

7372
## 7.Efficient Writting
@@ -77,5 +76,7 @@
7776
- [I Love PDF: Online PDF Tools](Efficient_Writting.md#3-i-love-pdf-online-pdf-tools)
7877
- [Color Hunt: Color Palettes for Designers and Artists](Efficient_Writting.md#4-color-hunt-color-palettes-for-designers-and-artists)
7978
- [Tables Generator: Create LaTeX Tables](Efficient_Writting.md#5-tables-generator-create-latex-tables)
79+
- [ACRONYMIFY: Name your paper with a cool acronyms](Efficient_Tools.md#6-acronymify-name-your-paper-with-a-cool-acronyms)
80+
- [Linggle: Grammer checker](Efficient_Tools.md#7-linggle-grammer-checker)
8081
----
8182
- ***"The past decade has seen tremendous progress in the field of artificial intelligence thanks to the resurgence of neural networks through deep learning. This has helped improve the ability for computers to see, hear, and understand the world around them, leading to dramatic advances in the application of AI to many fields of science and other areas of human endeavor" ——Jeffrey Dean***

0 commit comments

Comments
 (0)