-
Notifications
You must be signed in to change notification settings - Fork 0
/
atom.xml
410 lines (198 loc) · 129 KB
/
atom.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>王路飞</title>
<link href="http://wongchen.github.io/atom.xml" rel="self"/>
<link href="http://wongchen.github.io/"/>
<updated>2023-07-16T07:07:58.787Z</updated>
<id>http://wongchen.github.io/</id>
<author>
<name>Wong Chen</name>
</author>
<generator uri="https://hexo.io/">Hexo</generator>
<entry>
<title>如何写好我们自己的vimrc❓</title>
<link href="http://wongchen.github.io/2019/11/30/vimrc-of-your-own/"/>
<id>http://wongchen.github.io/2019/11/30/vimrc-of-your-own/</id>
<published>2019-11-29T16:44:42.000Z</published>
<updated>2023-07-16T07:07:58.787Z</updated>
<content type="html"><![CDATA[<p>Vim可以说是个高度自定义化的编辑器,1000个vimer就有1000个vimrc:😀</p><p>怎么写好属于我们自己的vimrc呢, That is a question✔︎</p><p>我是如下分类整理和更新的.📁</p><h2 id="基础的配置"><a href="#基础的配置" class="headerlink" title="基础的配置"></a>基础的配置</h2><p>大概包括下面一些, </p><ul><li>文件类型(filetype)</li><li>界面配置</li><li>语法(syntax)</li><li>fold相关</li></ul><p>好的是,上面这些都可以使用vim的帮助命令去查看,<code>filetype</code>, <code>syntax</code>以及<code>fold</code>在帮助文档都有详细的描述,</p><p>当你发现需要某些功能的时候,就可以去搜索查找vim如何配置做到.比如:</p><p>切换buffer的时候会发现每次都会提示你当前buffer未保存, 那么如何做到呢?</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">set hidden</span><br></pre></td></tr></table></figure><p>就可以了.</p><p>这些命令也都是可以在<code>command-mode</code>下面执行的命令. 主要是</p><ul><li><code>set</code></li><li><code>filetype</code>, <code>syntax</code>, <code>fold*</code></li><li><code>call</code></li></ul><blockquote><p>提示:</p><p>在vim-user manual里, <code>C-]</code> 可以跳转到cusor下文件对应的目录.</p></blockquote><h2 id="自动命令autocmd-🚘"><a href="#自动命令autocmd-🚘" class="headerlink" title="自动命令autocmd 🚘"></a>自动命令autocmd 🚘</h2><p>首先需要编译vim时 <code>+autocmd</code>的支持.</p><p>简要来说就是在某种模式下,vim自动执行某些命令. 这样分类是因为</p><ul><li>autocmd很强大,实现许多功能</li><li>也可能出现一些副作用等问题,使用时最好先提前测试.</li></ul><p><code>autocmd</code>看手册学习就可以了,有需求的时候可以边学边造轮子. 拜托那些繁杂的操作,从现在开始.</p><p>简单来说了解模式之后,主要看一下<code><Event></code></p><ul><li>读buffer/file</li><li>写buffer/file</li><li>buffer/window/file切换</li><li>filetype/syntax等</li><li>cusor/界面相关</li></ul><blockquote><p>提示:</p><p>查看帮助文档的时候,(命令模式), 插入cusor当前字符使用 <code><C-r><C-w></code>.</p></blockquote><h2 id="插件🔌"><a href="#插件🔌" class="headerlink" title="插件🔌"></a>插件🔌</h2><p>主要包括添加了哪些插件, 每个插件的全局配置等等.</p><p>目前平时用的插件还是不太多的,vim自带的功能其实本身已经很强大了.自带的许多插件也很棒的像</p><ul><li>nerdtree</li><li>quickfix</li></ul><p>其他也就 <code>YouCompleteMe</code>和<code>vim-colorscheme</code>用的比较频繁.</p><h2 id="mapping-☑️"><a href="#mapping-☑️" class="headerlink" title="mapping ☑️"></a>mapping ☑️</h2><p>按键映射可以说是使用频繁的一个功能, 也使得vim更加高度自定义化和便于记忆. 最终达到的一个很棒的效果就是不用大脑思考,完全肌肉记忆.</p><p>首先<code><Leader></code>键还是很有用的,个人定义空格作为leader key. 个人常用的的mapping有:</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br></pre></td><td class="code"><pre><span class="line">let mapleader = "\<Space>"</span><br><span class="line">" key mapping for ale</span><br><span class="line">nmap <silent> <C-p> <Plug>(ale_previous_wrap)</span><br><span class="line">nmap <silent> <C-n> <Plug>(ale_next_wrap)</span><br><span class="line"></span><br><span class="line">" key mapping for g++</span><br><span class="line">nnoremap <Leader>rr :!g++ -std=c++11 % -o %:r<CR></span><br><span class="line">nnoremap <Leader>rg :!g++ -g -std=c++11 % -o %:r<CR></span><br><span class="line">nnoremap <Leader>ter :packadd termdebug<CR></span><br><span class="line">nnoremap <Leader>db :Termdebug %:r<CR></span><br><span class="line">inoremap jk <ESC>l</span><br><span class="line"></span><br><span class="line">map <S-e> $</span><br><span class="line">imap xz <ESC>o</span><br><span class="line">imap zx <Up><ESC>o</span><br><span class="line"></span><br><span class="line">" close buffer mapping</span><br><span class="line">nnoremap <Leader>bb :bd<CR></span><br><span class="line">nnoremap <Leader>zz :wq<CR></span><br><span class="line">nnoremap <Leader>w :w<CR></span><br><span class="line"></span><br><span class="line">" for python</span><br><span class="line">nnoremap <Leader>pp :!python3 %<CR></span><br><span class="line"></span><br><span class="line">" ctags</span><br><span class="line">nnoremap <silent> <Leader>tt :!ctags -R<CR></span><br><span class="line"></span><br><span class="line">" buffer switch</span><br><span class="line">nnoremap <silent> [b :bprevious<CR> </span><br><span class="line">nnoremap <silent> ]b :bnext<CR> </span><br><span class="line">nnoremap <silent> [B :bfirst<CR> </span><br><span class="line">nnoremap <silent> ]B :blast<CR></span><br><span class="line"></span><br><span class="line">" tab switch</span><br><span class="line">nnoremap [t :tabp<CR></span><br><span class="line">nnoremap ]t :tabn<CR></span><br><span class="line"></span><br></pre></td></tr></table></figure><h3 id="函数-⚱️"><a href="#函数-⚱️" class="headerlink" title="函数 ⚱️"></a>函数 ⚱️</h3><p>一些定义的函数可以放在一起.</p><p>这些就需要简单地学习一下vimscript去写了.</p><p>设定几个任务</p><ul><li><input disabled="" type="checkbox"> 细节部分稍微细致学一下,在写几篇笔记</li><li><input disabled="" type="checkbox"> vimsript好好看一下,争取可以做到,能实现自己想要的函数功能</li><li><input disabled="" type="checkbox"> 写一个插件</li></ul>]]></content>
<summary type="html"><p>Vim可以说是个高度自定义化的编辑器,1000个vimer就有1000个vimrc:😀</p>
<p>怎么写好属于我们自己的vimrc呢, That is a question✔︎</p>
<p>我是如下分类整理和更新的.📁</p>
<h2 id="基础的配置"><a </summary>
<category term="vim" scheme="http://wongchen.github.io/tags/vim/"/>
</entry>
<entry>
<title>Vim中的跳转</title>
<link href="http://wongchen.github.io/2019/11/18/vim-navigate/"/>
<id>http://wongchen.github.io/2019/11/18/vim-navigate/</id>
<published>2019-11-17T23:55:51.000Z</published>
<updated>2023-07-16T07:07:58.786Z</updated>
<content type="html"><![CDATA[<p>Vim中的跳转方式多样,熟练之后可以肌肉记忆💪做到灵活跳转,十分方便,脱🖱指南,就在这里🐶</p><p>一般我们move还是navigate的时候,大部分都是在<strong>Normal Mode</strong>下,<strong>Visual Mode</strong>也有一些.</p><p>Insert Mode只做好<strong>一件事情</strong>就可以了,那就是插入内容,当我们完成插入之后要记得</p><blockquote><p>Get Back To Normal Mode.</p></blockquote><p><code>Esc</code>, <code>C-[</code>, <code>C-o</code>都可以从Insert Mode到Normal Mode,我个人习惯将<code>jk</code>映射到<code>Esc</code>键.</p><h2 id="Movement-↪️-Navigation-➡️-In-Normal-Mode"><a href="#Movement-↪️-Navigation-➡️-In-Normal-Mode" class="headerlink" title="Movement ↪️ Navigation ➡️ In Normal Mode"></a>Movement ↪️ Navigation ➡️ In Normal Mode</h2><p>包含基础跳转, 查找跳转, 插件中的跳转, 寄存器跳转以及文件之间的跳转.</p><p>Vim基本的一个操作就是数字加上动作重复多次,比如<code>5k</code>就是上移动5行, <code>2@a</code>就是执行<code>a</code>寄存器里的宏两次.</p><h3 id="Basic-Movements-🚶"><a href="#Basic-Movements-🚶" class="headerlink" title="Basic Movements 🚶"></a>Basic Movements 🚶</h3><p>字符移动: <code>hjkl</code>, 就不多说了.</p><p>行移动:</p><ul><li><code>gg</code>, <code>G</code>, 🔢+<code>G</code>, 行首, 行尾, 指定行~ 使用频率不算高其实,可以设置相对行<code>set relativenumber</code>, 通过🔢+<code>jk</code>实现行的跳转.</li><li><code>+</code>, <code>-</code>分别是回到下一行和上一行的第一个非空白字符的位置.</li></ul><p>词汇移动:<code>wW/eE/bB/geE</code></p><p>前向/后向移动一个词汇到头/尾部, 许多符号也是被当做一个词汇看待的,如<code>[]()</code>但是<code>:</code>是不被当做额外的词汇的.</p><p>所以,</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">if</span> flag: <span class="comment"># 如果cursor在flag头部想要 ce 的时候,会把后面的 : 也带走</span></span><br><span class="line"> <span class="built_in">print</span>(<span class="string">"🐶"</span>)</span><br></pre></td></tr></table></figure><p>另外, <code>cw</code>其实就是<code>ce</code>, <code>change-word</code>不应该把后面的空格也带走,这样更合理一些🤔.</p><p>大小写(word/WORD)的定义的词汇的区别是:</p><ul><li>word, 一系列连续的字符(字母, 数字,下划线,等等, <strong>空格/tab/换行</strong>分隔, <strong>空行也被视为词汇</strong>)</li><li>WORD, 一系列连续的非空字符(<strong>空格</strong>分隔, <strong>空行也被视为词汇</strong>)</li></ul><p><img src="/images/word_move.png" alt="走起"></p><p> [{]}系列: 这个系列就比较多了, 在vim中 :h [查看,个人而言,使用较少,常用的有section 跳转和paragraph跳转.</p><p>marker系列: 通过<code>m/M</code>+<code>{a-zA-Z}</code>mark一个位置, 此后通过<code>'</code>或者` + <code>marker</code>返回到标记位置. ‘返回到该行, `会返回到该位置. 大写的标记是可以全局进行跳转的.</p><p>此外vim在我们进行某些操作的时候会自动进行一些标记✍️</p><table><thead><tr><th>按键</th><th>行为</th></tr></thead><tbody><tr><td>``</td><td>当前文件的上一次jump</td></tr><tr><td>`.</td><td>上一次change的地方</td></tr><tr><td>`^</td><td>上一次插入的地方</td></tr><tr><td>`[</td><td>上一次change或者yank的起始位置</td></tr><tr><td>`]</td><td>上一次change或者yank的结束位置</td></tr><tr><td>`</`></td><td>上一次可视化的起始/结束位置</td></tr></tbody></table><p>查找跳转:</p><p><code>f/F</code>+<code>char</code>: 行内正向/逆向查找某个字符, <code>;</code>next, <code>,</code> previous.</p><p><code>t/T</code>+<code>char</code>: 同上,但是cursor定位在所查找字符的前面(正向/逆向方向的前面).</p><p>最后就是<code>/</code> <code>#</code>查找命令了.</p><p>jump list: 🕺</p><p><code><C-i></code> 和 <code><C-o></code> 前向/后向切换,要查看jump list, 命令<code>:jumps</code></p><p>change list:</p><p>与jump list相似, 我们修改内容的时候,也会保留一个change list, 命令<code>:changes</code>可以查看.</p><p>当我们uodo redo的时候其实就是在change list中切换,同时cursor会跳转到相应位置.</p><p><code>g;</code>和<code>g,</code>可以前向和后向在change list中进行跳转.</p><p>其他:</p><p><code>$</code>和<code>^</code>分别是本行最后一个字符和第一个非空白字符.</p><p><code>%</code>匹配括号之间跳转.</p><p>在cursor处 <code>*</code>或<code>#</code>会后向/前向查找完全匹配cursor位置处的词. 很有用.😀</p><h3 id="页面相关"><a href="#页面相关" class="headerlink" title="页面相关:"></a>页面相关:</h3><p><code>HML</code>移动到页面的顶部/中间/底部.</p><p><code>C-d</code>, <code>C-u</code>, <code>C-b</code> 向下/向上半页.</p><p><code>C-y</code>, <code>C-e</code> 上下一行.</p><p><code>zz</code>: cursor所在行移动到页面中间. (<code>z</code>系列可以参考帮助文档,包含移动,folds以及spelling相关的命令).</p><p>此外fold之间的移动,也需要z系列, 分别是<code>zj</code>, <code>zk</code>, 在一个开启的folds中可以通过<code>[z</code>和<code>]z</code>上下跳转</p><p><code>scrolloff</code>: 这个属性指的是进行移动跳转时,cursor下面至少包含多少行内容.设置一个合理的值可以保证我们在移动的时候尽量让要查看的内容在屏幕的中间位置.</p><h2 id="Movement-↪️-Navigation-➡️-In-Visual-Mode"><a href="#Movement-↪️-Navigation-➡️-In-Visual-Mode" class="headerlink" title="Movement ↪️ Navigation ➡️ In Visual Mode"></a>Movement ↪️ Navigation ➡️ In Visual Mode</h2><p>visual mode进行选择的时候,上面那些跳转都是可行的.比如: 🌰</p><p><code>vfe</code>就选择了从当前cursor到下一个<code>e</code>字符的位置.</p><p>此外就是在selection中cursor的movements</p><p><code>o</code>: 反方向的最后位置, 来回切换</p><p><code>O/o</code>: 在block visualmode下, 可以四个角来回切换.</p><p>大概就这么多了,以后学习到新的还会来补充😝</p>]]></content>
<summary type="html"><p>Vim中的跳转方式多样,熟练之后可以肌肉记忆💪做到灵活跳转,十分方便,脱🖱指南,就在这里🐶</p>
<p>一般我们move还是navigate的时候,大部分都是在<strong>Normal Mode</strong>下,<strong>Visual Mode</st</summary>
<category term="vim" scheme="http://wongchen.github.io/tags/vim/"/>
</entry>
<entry>
<title>2020算法校招感悟</title>
<link href="http://wongchen.github.io/2019/11/13/2020-campus-recruitment-1/"/>
<id>http://wongchen.github.io/2019/11/13/2020-campus-recruitment-1/</id>
<published>2019-11-13T13:17:20.000Z</published>
<updated>2023-07-16T07:07:58.788Z</updated>
<content type="html"><![CDATA[<p>据网上统计,今年算法竞争激烈程度大概是540:1的样子,可以观摩一下这个知乎问题<a href="https://www.zhihu.com/question/342267611/answer/805334722">如何看待2020 届校招算法工程师岗位求职人数远大于招聘岗位的现象？</a></p><p>我的情况很尴尬:</p><ul><li>无大厂实习</li><li>无顶会</li><li>非科班</li></ul><p>在实验室做了三年ML/DL的工作,勤勤恳恳😹辛辛苦苦.</p><p>更尴尬的是8月份才开始准备找工作🙃</p><p>实验室做的主要是医学影像分析,最后写出来的简历大概就是主要面向<strong>智能医疗</strong>的简历了😅</p><h2 id="复习和投简历👻"><a href="#复习和投简历👻" class="headerlink" title="复习和投简历👻"></a>复习和投简历👻</h2><ul><li>C++: (Primer),</li><li>Python(流畅的Python, Python源码剖析)</li><li>数据结构(邓俊辉)</li><li>ML/DL,(百面机器学习, 西瓜书, 统计学习方法)</li></ul><p>以及项目细节, 论文细节,牛客是每日必刷高频app.</p><p>8月11号开始投简历,海投啊,最终大概投了一百以上了😹,基本投的都是算法,身边的同学有转开发的,有转测试的,也有算法大佬的还有和我一样在算法苦苦挣扎的…</p><p>我居然还花时间学习了新知识🤞现在想起来还有点佩服自己.</p><p>学了点NLP(CS224N), Scala和Spark(coursera上的课程,之前有看过Haskell, 所以scala还好)…</p><h3 id="感悟🧐"><a href="#感悟🧐" class="headerlink" title="感悟🧐"></a>感悟🧐</h3><ol><li>早点准备,早点实习,顶会论文的话更好了,平时基础一定要好,踏踏实实.</li><li>抓住提前批的机会(67月份吧)</li><li>找力所能及的公司和岗位去投递,否则就是浪费自己的时间了,转岗要趁早,非要在算法这棵树上吊着掉死就不好受了,而且开发的薪资和算法的差距也在缩小,相信之后几年应该会达到一个平衡不会像今年这么爆炸💥了吧</li><li>多逛牛客和其他平台吧,我最终拿到的offer就是牛客内推的,没有笔试的~</li><li>心态一定要稳, 相信一半靠实力一半的一半靠颜值,一半的一半还是靠运气的.</li></ol><h2 id="面试💻"><a href="#面试💻" class="headerlink" title="面试💻"></a>面试💻</h2><p>投了那么多,要么杳无音信,要么面完杳无音信,有时候还会受到非科班的歧视,最后面得十几个吧~拿到offer的就更少了.</p><p>面试也是在一步步成长,逐渐积累经验,期间也发现自己许多知识学习还不是很扎实,项目许多细节还是有待考量,仔细总结的话还是可以学习到很多的.</p><h3 id="感悟🧐-1"><a href="#感悟🧐-1" class="headerlink" title="感悟🧐"></a>感悟🧐</h3><ol><li>有所取舍, 有些不值得的公司就没必要去面了,有时候面试体验不好的话还是很影响心态的</li><li>面试前心态要好,尽量不要紧张,提前看看面经挺有帮助的</li><li>面试时要自信,当受到质疑的时候要有理有据辩驳</li><li>所以简历上千万不要虚假,否则很容易被问到,想自信也莫得办法.</li><li>面试后一定要好好总结</li></ol><p>期间心态崩了无数次,身边好多人也都是这种状态,掉层皮呐,还好最后上岸了😬</p><p>我太难了😤😤😤</p><p><img src="https://img.doutugou.net/2019/08/006dMd5bgy1g62osjo7hzj308c08cdg1.jpg"></p>]]></content>
<summary type="html"><p>据网上统计,今年算法竞争激烈程度大概是540:1的样子,可以观摩一下这个知乎问题<a href="https://www.zhihu.com/question/342267611/answer/805334722">如何看待2020 届校招算法工程师岗位求职人数远大于招聘岗</summary>
<category term="总结" scheme="http://wongchen.github.io/categories/%E6%80%BB%E7%BB%93/"/>
<category term="总结" scheme="http://wongchen.github.io/tags/%E6%80%BB%E7%BB%93/"/>
</entry>
<entry>
<title>TensorFlow 使用Dataset 简单读取数据</title>
<link href="http://wongchen.github.io/2018/01/21/tf-dynamic-dataset/"/>
<id>http://wongchen.github.io/2018/01/21/tf-dynamic-dataset/</id>
<published>2018-01-21T06:40:43.000Z</published>
<updated>2023-07-16T07:07:58.768Z</updated>
<content type="html"><![CDATA[<p>上一篇说了pytorch下利用其<code>Dataset</code>和<code>DataLoader</code>api可以很方便地实现动态读取数据,不需要将图片保存到本地也可以快速取样本.</p><p>在TensorFlow下之前写过常用的数据读取方式,在TF1.3版本新添加了<code>Dataset</code>模块,1.3版本中其位于</p><p><code>tf.contrib.data.Dataset</code>,1.4版本后移到了’tf.data.Dataset’.</p><p>之前的tfrecord队列读取数据的方式有一些不足:</p><ul><li>需要保存为tfrecord文件格式,官方doc有些晦涩难懂=,学习成本较高(相比于pytorch dataset的实现)</li><li>需要保存为tfrecord文件,占用一定硬盘空间</li><li>其内部机制有些难懂,最后如果结果不对,不能保证是不是样本读取代码这里出了问题</li></ul><h2 id="构建一个Dataset"><a href="#构建一个Dataset" class="headerlink" title="构建一个Dataset"></a>构建一个Dataset</h2><p>首先,Dataset结构:</p><p>从这里也可以看到,跟pytorch下的很像,一个好处就是相比之前的队列或者feed_dict来说要简单,简介许多,代码易读性也增加了.</p><ul><li>Dataset模块: 包含基本的方法,如创建dataset, 随机(shuffle), 变换(transformation), 批处理(batch)等</li><li>子类: 针对特定数据的一些方便的子类,就像pytorch中的labeledImageDataset等</li><li>Dataset的方法就是实例化一个Iterator对象,保证每次取到一个dataset中的一个/批样本.</li></ul><p><img src="https://4.bp.blogspot.com/-Av_QQUdRvZI/WbfwxtE0cjI/AAAAAAAAD1s/bjWNV9siuoI92Qo3TaeFaumb2s5BVfYgwCLcBGAs/s1600/image7.jpg"></p><p>从<a href="https://www.tensorflow.org/api_docs/python/tf/data/Dataset">官方doc</a>我们可以看到,构建一个Dataset的大致的几个方法有:</p><ul><li><code>from_generator</code>:调用了<code>py_func</code>方法,可以将任意的python代码转化为tf图中的operation节点;使用指定生成器中的每个元素作为dataset中的元素</li><li><code>from_sparse_tensor_slices</code></li><li><code>from_tensor_slices</code></li></ul><h2 id="example"><a href="#example" class="headerlink" title="example"></a>example</h2><p>利用<code>from_generator</code>方法来实现之前的动态生成训练和测试样本.</p><blockquote><p>从一些3D图片中随机取一些[128, 128, 128]大小的样本, 原图和金标准图index要一致,每个样本包含(data, target), 为了方便,提前保存好了这些样本的行列坐标.</p></blockquote><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> tensorflow <span class="keyword">as</span> tf</span><br><span class="line"><span class="keyword">import</span> os</span><br><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"><span class="keyword">import</span> SimpleITK <span class="keyword">as</span> sitk</span><br><span class="line"><span class="keyword">import</span> pickle <span class="keyword">as</span> pk</span><br><span class="line"></span><br><span class="line"><span class="comment"># 实现生成器,这里由于生成器需要接受几个参数,from_generator中gen参数对象需要callable而且返回一个可迭代对象,</span></span><br><span class="line"><span class="comment"># 可以写成闭包的形式如下,也可以写成一个类(支持callable)或者lambda也是可以的</span></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">gen</span>(<span class="params">src_list, std_list, pickle_file</span>):</span><br><span class="line"> <span class="string">'''</span></span><br><span class="line"><span class="string"> src_list,std_list 图像array数据</span></span><br><span class="line"><span class="string"> 返回一个生成器函数</span></span><br><span class="line"><span class="string"> '''</span></span><br><span class="line"> <span class="keyword">with</span> <span class="built_in">open</span>(pickle_file, <span class="string">'rb'</span>) <span class="keyword">as</span> f:</span><br><span class="line"> points = pk.load(f)</span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">func</span>():</span><br><span class="line"> <span class="keyword">for</span> j <span class="keyword">in</span> <span class="built_in">range</span>(<span class="built_in">len</span>(points)):</span><br><span class="line"> i, c, h, w = points[j]</span><br><span class="line"> src = src_list[i][c-<span class="number">64</span>:c+<span class="number">64</span>, h-<span class="number">64</span>:h+<span class="number">64</span>, w-<span class="number">64</span>:w+<span class="number">64</span>] </span><br><span class="line"> std = std_list[i][c-<span class="number">64</span>:c+<span class="number">64</span>, h-<span class="number">64</span>:h+<span class="number">64</span>, w-<span class="number">64</span>:w+<span class="number">64</span>] </span><br><span class="line"> <span class="keyword">yield</span> (src, std)</span><br><span class="line"> <span class="keyword">return</span> func</span><br><span class="line"></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">main</span>():</span><br><span class="line"> .....</span><br><span class="line"> </span><br><span class="line"> ds = tf.data.Dataset.from_generator(gen(src_list, std_list, pickle_file), \</span><br><span class="line"> (tf.int16, tf.int16)).batch(<span class="number">2</span>)</span><br><span class="line"> ge = gen(src_list, std_list, pickle_file)</span><br><span class="line"> value = ds.make_one_shot_iterator().get_next()</span><br><span class="line"> sess = tf.Session()</span><br><span class="line"> <span class="keyword">for</span> _ <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">9</span>):</span><br><span class="line"> <span class="built_in">print</span> sess.run(value)[<span class="number">0</span>].shape <span class="comment"># [2, 128, 128, 128]</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">"__main__"</span>:</span><br><span class="line"> main()</span><br><span class="line"></span><br></pre></td></tr></table></figure><p>之后可以从时间较多对比下~,其实之前做过一些测试,发现tfrecord和pytorch下Dataset&DataLoader时间相差无几.</p>]]></content>
<summary type="html"><p>上一篇说了pytorch下利用其<code>Dataset</code>和<code>DataLoader</code>api可以很方便地实现动态读取数据,不需要将图片保存到本地也可以快速取样本.</p>
<p>在TensorFlow下之前写过常用的数据读取方式,在TF1.</summary>
<category term="TensorFlow" scheme="http://wongchen.github.io/categories/TensorFlow/"/>
<category term="TensorFlow" scheme="http://wongchen.github.io/tags/TensorFlow/"/>
<category term="Dataset" scheme="http://wongchen.github.io/tags/Dataset/"/>
</entry>
<entry>
<title>深度学习中动态加载样本</title>
<link href="http://wongchen.github.io/2018/01/17/pytorch-dynamic-sampling/"/>
<id>http://wongchen.github.io/2018/01/17/pytorch-dynamic-sampling/</id>
<published>2018-01-17T12:48:34.000Z</published>
<updated>2023-07-16T07:07:58.777Z</updated>
<content type="html"><![CDATA[<p>在用CNN处理图像的时候,经常我们不是用一整张完整的图像去训练,而是经常会采用取一个图像块也就是patch来输入网络进行训练,构成我们的训练集样本和测试集样本.</p><p>一种常用的方法就是先把他们保存到本地硬盘中,然后每次训练时一张张去读取(或者将他们转换为<code>tfrecord</code>格式使用TensorFlow中的队列读取这些文件),最近在做3D图像的操作,直接对3D图像进行卷积,卷积核也是3维的,因此图像尺寸不可能取到很大,也是voxel patch.</p><p>但是3D数据是较大的,硬盘空间有时候没那么大,尤其是固态,因为固态下的文件读取速度要比机械盘快很多,所以能放到固态下的话是个不错的选择,但是对于这个问题来说,完全不够,只能放到机械盘下面.</p><p>还有一种想法就是: “把鸡蛋放到一个篮子里.🥚🥚🥚”</p><p>就是在训练时把所有数据一次性读取加载到内存中,然后训练的时候从这些图片中随机取样本,要求就是这些图片可以被同时读取到内存中,需要一定大小的内存,尤其是在一些用来跑程序的主机上,内存占用很低.</p><h2 id="Example"><a href="#Example" class="headerlink" title="Example"></a>Example</h2><p>例如可以把要提取的patch的点以pickle文件保存为list对象,用pytorch实现为:</p><p>将所有的读取到图像以及要提取样本的关键点传入MyDataset类, 在<code>__getitem__</code>方法中对已有的图像进行切片就能得到需要的样本.</p><p>这样的好处就是节省硬盘空间,从内存获取数据速度较快.</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line">from torch.utils.data import Dataset</span><br><span class="line">import numpy as np</span><br><span class="line">import pickle</span><br><span class="line"></span><br><span class="line">class MyDataset(Dataset):</span><br><span class="line">def __init__(self, src_img_list, std_img_list, points_pickle):</span><br><span class="line"> self.src_img_list = src_img_list</span><br><span class="line"> self.std_img_list = std_img_list</span><br><span class="line"> with open(points_pickle, 'rb') as f:</span><br><span class="line"> self.points = pickle.load(f)</span><br><span class="line"> </span><br><span class="line"> def __len__(self):</span><br><span class="line"> return len(self.points)</span><br><span class="line"> </span><br><span class="line"> def __getitem__(self, idx):</span><br><span class="line"> i, c, h, w = self.points[idx]</span><br><span class="line"> data = self.src_img_list[i][c-64:c+64, h-64:h+64, w-64:w+64].reshape([1, 128, , 128]).</span><br><span class="line"> target = self.std_img_list[i][c-64:c+64, h-64:h+64, w-64:w+64].reshape([1, , 128, 128]</span><br><span class="line"> return data, target</span><br></pre></td></tr></table></figure>]]></content>
<summary type="html"><p>在用CNN处理图像的时候,经常我们不是用一整张完整的图像去训练,而是经常会采用取一个图像块也就是patch来输入网络进行训练,构成我们的训练集样本和测试集样本.</p>
<p>一种常用的方法就是先把他们保存到本地硬盘中,然后每次训练时一张张去读取(或者将他们转换为<code</summary>
<category term="Deep Learning" scheme="http://wongchen.github.io/categories/Deep-Learning/"/>
<category term="pytorch" scheme="http://wongchen.github.io/tags/pytorch/"/>
<category term="deep learning" scheme="http://wongchen.github.io/tags/deep-learning/"/>
</entry>
<entry>
<title>安装TensorFlow1.5+cuda9.0+cudnn7.0</title>
<link href="http://wongchen.github.io/2018/01/17/cuda9-cudnn7/"/>
<id>http://wongchen.github.io/2018/01/17/cuda9-cudnn7/</id>
<published>2018-01-17T12:17:11.000Z</published>
<updated>2023-07-16T07:07:58.784Z</updated>
<content type="html"><![CDATA[<p>之前使用的一直是python2.7+TensorFlow1.1+cuda8.0+cudnn6.0,用了有一段时间了,python3是一直打算换过来的,毕竟py3比2增加了许多实用性的功能,而且py2逐渐不被许多库所支持,官方也声明在2020年放弃对py2的维护.</p><p>今天看到TensorFlow1.5版本的发布以及cuda9的发布想更新一波.</p><h2 id="py"><a href="#py" class="headerlink" title="py"></a>py</h2><p>由于目前还会短暂使用py2,所以需要在一台电脑上同时使用py2和py3.</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">ps -aux | grep python</span><br></pre></td></tr></table></figure><p>可以看到系统当前调用python的进程,看到python和python3都有.</p><p>可行的方法包括:</p><ol><li>virtualenv,不过暂时不想用这个.</li><li>最简单就是在用3的时候后面加3就好了, <code>python3</code>, <code>pip3</code></li><li>也可以去<a href="https://github.com/pyenv/pyenv">pyenv</a>看看</li></ol><h2 id="CUDA-CUDNN"><a href="#CUDA-CUDNN" class="headerlink" title="CUDA & CUDNN"></a>CUDA & CUDNN</h2><p>官网下载cuda时选择9.0的版本,当前最新为9.1,刚开始安装了9.1的cuda toolkit,发现还是不支持,后来换到9.0版本就好了.</p><p>CUDNN也是官网下载deb包安装就可以.</p><p>驱动需要384以上,一个经验就是当电脑驱动出问题的时候,(经常登录界面无线循环,低分辨率等问题),最简单的解决办法就是把原来的使用linux apt安装的驱动包都卸载掉</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">sudo apt-get remove nvidia*</span><br></pre></td></tr></table></figure><p>然后<a href="http://www.nvidia.com/Download/index.aspx?lang=cn">官网</a>下载相应版本的driver, <code>.run</code>文件,用它来安装,一般可以解决驱动的问题.</p><h2 id="TensorFlow1-5"><a href="#TensorFlow1-5" class="headerlink" title="TensorFlow1.5"></a>TensorFlow1.5</h2><p>本来打算源码安装的,进行到<code>bazel build</code>这一步的时候一个跟网络和bazel有关的问题解决不了,也没必要这么装了.</p><p>索性还是pip:</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">pip install tensorflow-gpu==1.5rc1 --user</span><br></pre></td></tr></table></figure><p>这样原来的python2也是还可以继续用的,新的python3也ok.</p>]]></content>
<summary type="html"><p>之前使用的一直是python2.7+TensorFlow1.1+cuda8.0+cudnn6.0,用了有一段时间了,python3是一直打算换过来的,毕竟py3比2增加了许多实用性的功能,而且py2逐渐不被许多库所支持,官方也声明在2020年放弃对py2的维护.</p>
<</summary>
<category term="TensorFlow" scheme="http://wongchen.github.io/categories/TensorFlow/"/>
<category term="cuda" scheme="http://wongchen.github.io/tags/cuda/"/>
<category term="TensorFlow" scheme="http://wongchen.github.io/tags/TensorFlow/"/>
</entry>
<entry>
<title>summary2017</title>
<link href="http://wongchen.github.io/2017/12/31/summary2017/"/>
<id>http://wongchen.github.io/2017/12/31/summary2017/</id>
<published>2017-12-31T07:26:51.000Z</published>
<updated>2023-07-16T07:07:58.791Z</updated>
<content type="html"><![CDATA[<h2 id="读书"><a href="#读书" class="headerlink" title="读书"></a>读书</h2><p>不得不说kindle真的是个好东西,现在看书除了理工科的一些书都在kindle下看,对眼睛好也方便,其他书籍公式较多或者代码较多还的在电脑上看</p><h3 id="文学"><a href="#文学" class="headerlink" title="文学"></a>文学</h3><ol><li>人类简史</li></ol><p><img src="https://img3.doubanio.com/lpic/s27814883.jpg"></p><p>起源是在Ted上看了一篇作者的演讲,兴起就打算看这两本书:人类简史与未来简史.结果只看了一篇,还是跳着看的,其中的观点还是很新颖的,不能同意更多.</p><ol start="2"><li>拉普拉斯的魔女</li></ol><p><img src="https://img3.doubanio.com/lpic/s29195703.jpg"></p><p>一直都很喜欢看东野的推理小说,之前也看过了嫌疑人, 放学后, 幻夜……白夜行看的电视剧是个遗憾,书确实太长了啊!</p><p>这本书在我看的时候还是很入胜的,不过更多的还是关于人性跟情感以及AI, 少了些许推理的部分,难怪知乎豆瓣评论一致不高.</p><ol start="3"><li>追风筝的人</li></ol><p><img src="https://img3.doubanio.com/lpic/s1727290.jpg"></p><blockquote><p>为你,千千万万遍~</p></blockquote><p>读完不禁感慨战争难民的不幸,美丽又残忍的故事,经典~</p><p>已经几乎忘记它带给我的最深的感受了,读过~</p><h3 id="DL-Programming"><a href="#DL-Programming" class="headerlink" title="DL/Programming"></a>DL/Programming</h3><ol><li>Deep Learning (50%)</li></ol><p><img src="https://img3.doubanio.com/lpic/s29133163.jpg"></p><p>传说中的花书,还没读完,读到后面感觉知识储备有些不足,就先暂停下来了,书涉及的范围比较广,刚开始英文一章一章读的,还是有些吃力,后来又读了一遍中文,现在还落着好几章的总结没有做~ :fire:</p><ol start="2"><li>PRML(in progress…. ) 🚲🚲</li></ol><p><img src="https://img1.doubanio.com/lpic/s4254558.jpg"></p><p>经典式教科书,读起来还是有点难,打算每天想起来,闲下来的时候拿起来读一读就好……慢慢啃吧☠️☠️</p><p>目前中英文混着读</p><ol start="3"><li>PGM (in progress)🚲🚲</li></ol><p><img src="https://img3.doubanio.com/lpic/s28057016.jpg"></p><p>读这本书是跟着coursera上的一个课程同步学习的,可惜还没读多少,明年继续….👊👊</p><ol start="4"><li>Cocoa Programming for OS X (5th Edition)</li></ol><p><img src="https://img3.doubanio.com/lpic/s28041282.jpg"></p><p>暑假放假回家呆了两周,断断续续看了那么些时间,跟着学了一些简单的操作,应该差不多读了50%吧,跟着书操作还是比较简单的,就是版本稍微有点旧了,许多操作及api需要改一下💪💪</p><ol start="5"><li>Python 源码剖析</li></ol><p><img src="https://img3.doubanio.com/lpic/s3435132.jpg"></p><p>我觉得这本书很棒,可以让我理解到python的内部的一些很好的设计,我没有仔细读C的部分,没必要,需要的时候会再认真读一遍.</p><ol start="6"><li>流畅的Python(90%)</li></ol><p><img src="https://img3.doubanio.com/lpic/s29434304.jpg"></p><p>可以说是2017年最佳Python进阶书目, 多个技术大咖强烈推荐,从我菜鸡的角度来看还是要花些时间去读的,豆瓣上那些花70小时读完的简直了,读书还要以小时来计时吗~~🙈🙈</p><p>明年准备都换到Python3下去工作,正式放弃Python2</p><blockquote><p>把相关的主题与java比较可以体会到更多乐趣</p></blockquote><p>可惜我还不会java</p><p>其他应该没了,读书工具一般就是kindle for mobi, iBook for epub, pdf expert for pdf.</p><h2 id="公开课"><a href="#公开课" class="headerlink" title="公开课"></a>公开课</h2><p>coursera 上看了几门</p><ol><li>第一个就是Ng的深度学习系列课程,分为五个部分,学起来还是比较简单的,但是没有仔细做记录也无所谓了.</li></ol><p><img src="https://ws3.sinaimg.cn/large/006tNc79ly1fn00pu433cj30m609u0tw.jpg"></p><ol start="2"><li>PGM/RL,CS249-112/CS231n2017</li></ol><ul><li><input disabled="" type="checkbox"> 还没学完,有点难,又堆到明年了</li><li><input disabled="" type="checkbox"> 概率图模型</li><li><input disabled="" type="checkbox"> 强化学习</li></ul><h2 id="学术"><a href="#学术" class="headerlink" title="学术"></a>学术</h2><p>读了许多论文,差不多100篇吧~,这里就不多说了</p><ul><li><input disabled="" type="checkbox"> 明年争取努力发篇论文</li></ul><h3 id="🔑🔗📌"><a href="#🔑🔗📌" class="headerlink" title="🔑🔗📌"></a>🔑🔗📌</h3><ul><li>加深了TensorFlow的学习</li><li>简单使用caffe, chainer和MXNet,</li><li>熟练pytorch</li><li>换了pixel手机❌📱</li><li>管理实验室Linux计算服务器</li></ul><h3 id="🔜🔜🔜"><a href="#🔜🔜🔜" class="headerlink" title="🔜🔜🔜"></a>🔜🔜🔜</h3><p>明年还有许多东西要做,压力更大了,加油</p><ul><li><input disabled="" type="checkbox"> 学习kotlin语言(Android 官方钦定)</li><li><input disabled="" type="checkbox"> 多读书,多看报,多打球</li><li><input disabled="" type="checkbox"> TensorFlow mobile</li><li><input disabled="" type="checkbox"> golang, Ruby, Haskell, Docker</li><li><input disabled="" type="checkbox"> python2 ⏩python3</li></ul>]]></content>
<summary type="html"><h2 id="读书"><a href="#读书" class="headerlink" title="读书"></a>读书</h2><p>不得不说kindle真的是个好东西,现在看书除了理工科的一些书都在kindle下看,对眼睛好也方便,其他书籍公式较多或者代码较多还的在电脑上</summary>
<category term="总结" scheme="http://wongchen.github.io/categories/%E6%80%BB%E7%BB%93/"/>
<category term="总结" scheme="http://wongchen.github.io/tags/%E6%80%BB%E7%BB%93/"/>
<category term="2017" scheme="http://wongchen.github.io/tags/2017/"/>
</entry>
<entry>
<title>TensorFlow 数据输入方式</title>
<link href="http://wongchen.github.io/2017/09/11/tf-datareader/"/>
<id>http://wongchen.github.io/2017/09/11/tf-datareader/</id>
<published>2017-09-11T10:00:53.000Z</published>
<updated>2023-07-16T07:07:58.790Z</updated>
<content type="html"><![CDATA[<p>最近使用TensorFlow时候发现读入图片数据的时候很难受,自己写的一些代码又感觉很蹩脚,官方的文档易读性太低,太费劲了.<br>打算好好整理一下数据输入这部分代码.</p><h2 id="方式"><a href="#方式" class="headerlink" title="方式"></a>方式</h2><h3 id="直接加载到内存中"><a href="#直接加载到内存中" class="headerlink" title="直接加载到内存中"></a>直接加载到内存中</h3><p>这种方式简直暴力,老早之前我数据没有很多,就是这么做的,把数据先用<code>pickle</code>,将其保存下来,运行的时候直接从保存的文件中获取,这种方式只适合一些小型的文本文件数据,图片数据还是算了,电脑会卡死的.</p><h3 id="动态输入placeholder"><a href="#动态输入placeholder" class="headerlink" title="动态输入placeholder"></a>动态输入placeholder</h3><p>在每次训练时加载一些数据到内存当中,以placeholder的形式feed到graph当中.<br>这种方式需要自己写一些代码,比如</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">class</span> <span class="title class_">ImageData</span>:</span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">___init__</span>(<span class="params">self, batch_size</span>):</span><br><span class="line"><span class="comment"># define some attr</span></span><br><span class="line"> self.batch_size = batch_size</span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">next_batch</span>(<span class="params">self</span>):</span><br><span class="line"> ....</span><br><span class="line"><span class="keyword">return</span> np.array([imread(<span class="string">'name%s.png'</span>%i <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(self.batch_seze))]), labels</span><br><span class="line"> ...</span><br><span class="line"> </span><br><span class="line">image_reader = ImageData(batch_size)</span><br><span class="line"><span class="keyword">for</span> step <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">10001</span>):</span><br><span class="line"> batch = image_reader.next_batch()</span><br><span class="line"> sess.run(batch, feed_dict={x: batch[<span class="number">0</span>], y: batch[<span class="number">1</span>]})</span><br><span class="line"> ...</span><br></pre></td></tr></table></figure><p>在训练时实例化相应的类, 每次运行加载一次,需要说明的是,这样需要大量地重复读写多个小文件,所以把这些文件放到固态硬盘下面对提升读取速度是个不错的选择.</p><p>这个方法是比较类似于pytorch的<code>dataset</code>和<code>dataloader</code>的,同时采用pytorch的这两个类来实现TF中的数据输入也是很可行的.</p><p>详情见<a href="http://pytorch.org/tutorials/beginner/data_loading_tutorial.html">Data Loading and Processing Tutorial</a>.</p><h3 id="tfrecord"><a href="#tfrecord" class="headerlink" title="tfrecord"></a>tfrecord</h3><p>TF推荐的一种文件格式为tfrecord,把多个零碎的原始文件编码为一个较大的tfrecord文件,运行时通过解码器将其转化为<code>Tensors</code>, <a href="https://www.tensorflow.org/api_guides/python/reading_data">这篇tutorial</a>解释了TF中读入数据的方法,这里代码已经写得很好了,但读完发现还是不会……</p><p>可能用到的TF对象和方法, (先大致看名字了解一下,详细后面会涉及到):</p><ul><li><a href="https://www.tensorflow.org/api_docs/python/tf/TFRecordReader"><code>tf.TFRecordReader</code></a></li><li><code> tf.train.string_input_producer</code></li><li><code> tf.decode_raw()</code></li><li><code>tf.train.shuffle_batch()</code></li><li><a href="https://www.tensorflow.org/api_docs/python/tf/parse_single_example"><code>tf.parse_single_example</code></a></li><li><code>tf.train.Coordinator()</code></li><li><code>tf.train.start_queue_runners</code></li></ul><h3 id="数据转化"><a href="#数据转化" class="headerlink" title="数据转化"></a>数据转化</h3><p>首先通过一个<code>tf.TFRecordWriter</code>对象将原始零碎小文件转化为tfrecord格式文件,代码参考某个地方的:</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">writer = tf.python_io.TFRecordWriter(<span class="string">'./data/test_deepmatching.tfrecords'</span>) <span class="comment"># 初始化TFRecordWriter对象</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># write exanple</span></span><br><span class="line">img_raw = img.tobytes()</span><br><span class="line">example = tf.train.Example(features=tf.train.Features(feature={</span><br><span class="line"> <span class="string">'label'</span>: tf.train.Feature(int64_list=tf.train.Int64List(value=[<span class="number">1</span>])),</span><br><span class="line"> <span class="string">'img_raw'</span>: tf.train.Feature(bytes_list=tf.train.BytesList(value=[img_raw]))</span><br><span class="line"> }))</span><br><span class="line">writer.write(example.SerializeToString())</span><br><span class="line"></span><br><span class="line">....</span><br><span class="line">writer.close()</span><br></pre></td></tr></table></figure><h3 id="读入tfrecord"><a href="#读入tfrecord" class="headerlink" title="读入tfrecord"></a>读入tfrecord</h3><p>TF将文件名生成队列,并行解码读取里面的Tensors.</p><p><img src="https://www.tensorflow.org/images/AnimatedFileQueues.gif"></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">def</span> <span class="title function_">read_and_decode</span>(<span class="params">filename</span>):</span><br><span class="line"> filename_queue = tf.train.string_input_producer([filename])</span><br><span class="line"> reader = tf.TFRecordReader()</span><br><span class="line"> _, serialized_example = reader.read(filename_queue) </span><br><span class="line"> features = tf.parse_single_example(serialized_example,</span><br><span class="line"> features={</span><br><span class="line"> <span class="string">'label'</span>: tf.FixedLenFeature([], tf.int64),</span><br><span class="line"> <span class="string">'img_raw'</span> : tf.FixedLenFeature([], tf.string),</span><br><span class="line"> })</span><br><span class="line"></span><br><span class="line"> img = tf.decode_raw(features[<span class="string">'img_raw'</span>], tf.uint8)</span><br><span class="line"> <span class="built_in">print</span> img.shape</span><br><span class="line"> img = tf.reshape(img, [<span class="number">65</span>, <span class="number">130</span>, <span class="number">1</span>])</span><br><span class="line"> <span class="comment"># img = tf.cast(img, tf.float32) * (1. / 255) - 0.5</span></span><br><span class="line"> <span class="comment"># pre-processing</span></span><br><span class="line"> label = tf.cast(features[<span class="string">'label'</span>], tf.int32)</span><br><span class="line"> <span class="keyword">return</span> img, label</span><br><span class="line"></span><br></pre></td></tr></table></figure><p>上述函数返回我们的img 和label <code>ops</code>,运行时候还需要run一下这几个ops.</p><p>为了实现并行读取,我们需要创建多个线程,使用<code>QueueRunner</code>兑现来运行.</p><p>需要一个<code>Coordnator</code>对象来管理这些线程.</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">coord = tf.train.Coordinator()</span><br><span class="line">threads = tf.train.start_queue_runners(sess=sess, coord=coord)</span><br></pre></td></tr></table></figure><p>训练时,</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">for step in range(10001):</span><br><span class="line"> batch = sess.run([img_batch, label_batch])</span><br><span class="line"> sess.run(train_op)</span><br><span class="line"> ...</span><br><span class="line"> </span><br><span class="line">coord.request_stop()</span><br></pre></td></tr></table></figure><p>上面这种读取不是很好,可能会遇到一些错误,导致线程无法正常关闭,还在等待读取队列,必须强行kill掉才可以,可以采用官方的代码:</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">try:</span><br><span class="line"> while not coord.should_stop():</span><br><span class="line"> # Run training steps or whatever</span><br><span class="line"> sess.run(train_op)</span><br><span class="line"></span><br><span class="line">except tf.errors.OutOfRangeError:</span><br><span class="line"> print('Done training -- epoch limit reached')</span><br><span class="line">finally:</span><br><span class="line"> # When done, ask the threads to stop.</span><br><span class="line"> coord.request_stop()</span><br><span class="line"></span><br><span class="line"># Wait for threads to finish.</span><br><span class="line">coord.join(threads)</span><br><span class="line">sess.close()</span><br></pre></td></tr></table></figure>]]></content>
<summary type="html"><p>最近使用TensorFlow时候发现读入图片数据的时候很难受,自己写的一些代码又感觉很蹩脚,官方的文档易读性太低,太费劲了.<br>打算好好整理一下数据输入这部分代码.</p>
<h2 id="方式"><a href="#方式" class="headerlink" tit</summary>
<category term="TensorFlow" scheme="http://wongchen.github.io/categories/TensorFlow/"/>
<category term="TensorFlow" scheme="http://wongchen.github.io/tags/TensorFlow/"/>
</entry>
<entry>
<title>python之context manager学习</title>
<link href="http://wongchen.github.io/2017/09/09/python-context-manager/"/>
<id>http://wongchen.github.io/2017/09/09/python-context-manager/</id>
<published>2017-09-08T16:43:18.000Z</published>
<updated>2023-07-16T07:07:58.789Z</updated>
<content type="html"><![CDATA[<h2 id="What-is-context-manager"><a href="#What-is-context-manager" class="headerlink" title="What is context manager?"></a>What is context manager?</h2><blockquote><p>A basic issue in programming is <a href="https://en.wikipedia.org/wiki/Resource_management_(computing)">resource management</a>: a <a href="https://en.wikipedia.org/wiki/Resource_(computing)">resource</a> is anything in limited supply, notably file handles, <a href="https://en.wikipedia.org/wiki/network_socket">network sockets</a>, locks, etc., and a key problem is making sure these are <em>released</em> after they are <em>acquired.</em> If they are not released, you have a <a href="https://en.wikipedia.org/wiki/resource_leak">resource leak</a>, and the system may slow down or crash. More generally, you may want cleanup actions to always be done, other than simply releasing resources.</p></blockquote><p>就像其名字一样,context manager负责管理一定的上下文环境,按照你的设定,及时<strong>分配</strong>一定的资源,在完成相应的工作之后,及时<strong>释放</strong>这些资源. 而且不止是释放资源,你还可以做其他你想的事情……</p><p>Python提供了一种特定的语法结构可以用来做如此工作,许多类型都支持context text,比如buit-in <code>File</code>对象.</p><p>常用的操作</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">with open(file_name, 'rb') as f:</span><br><span class="line">lines = f.readlines()</span><br><span class="line"># other opoerations</span><br><span class="line"></span><br><span class="line">print f.closed # True</span><br></pre></td></tr></table></figure><p>i.e.在这个例子里context manager会自动关闭f文件对象,不需要用户显示地执行代码关闭,更加安全.</p><p>仔细研究会发现,这类对象,需要实现<code>__enter__</code>和<code>__exit__</code>两个方法属性,上面的例子等价于</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">f = <span class="built_in">open</span>(file_name, <span class="string">'rb'</span>)</span><br><span class="line">f.__enter__()</span><br><span class="line"><span class="keyword">try</span>:</span><br><span class="line"> BLOCK Operations</span><br><span class="line"><span class="keyword">finally</span>:</span><br><span class="line"> f.__exit__()</span><br><span class="line"> </span><br></pre></td></tr></table></figure><h2 id="How-to-implement"><a href="#How-to-implement" class="headerlink" title="How to implement?"></a>How to implement?</h2><p>首先,通过实现<code>__enter__</code>和<code>__exit__</code>这两个协议方法,我们可以使用<code>with expre [as var]:</code>语法来使用context manager.如:</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">class File(object):</span><br><span class="line"> def __init__(self, file_name, method):</span><br><span class="line"> self.file_obj = open(file_name, method)</span><br><span class="line"> def __enter__(self):</span><br><span class="line"> return self.file_obj</span><br><span class="line"> def __exit__(self, type, value, traceback):</span><br><span class="line"> self.file_obj.close()</span><br></pre></td></tr></table></figure><p><code>__enter__</code>将返回值传入<code>as</code>后面的<code>var</code>变量,<code>block</code>退出时执行<code>__exit__</code>方法,更通常一些,<code>__exit__</code>方法应该返回一个<code>bool</code>值表示是否发生异常. 基于以上<code>File</code>类的定义可以使用下面的<code>with</code>statement.</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">with File('demo.txt', 'w') as opened_file:</span><br><span class="line"> opened_file.write('Hola!')</span><br></pre></td></tr></table></figure><p>处理异常也基本同理.</p><h2 id="contextlib-Utilities-for-with-statement-contexts"><a href="#contextlib-Utilities-for-with-statement-contexts" class="headerlink" title="contextlib: Utilities for with-statement contexts"></a>contextlib: Utilities for <a href="https://docs.python.org/2/reference/compound_stmts.html#with"><code>with</code></a>-statement contexts</h2><p>看一下这个built-in moudule<code>contextlib</code></p><blockquote><p>This module provides utilities for common tasks involving the <a href="https://docs.python.org/2/reference/compound_stmts.html#with"><code>with</code></a> statement. For more information see also <a href="https://docs.python.org/2/library/stdtypes.html#typecontextmanager">Context Manager Types</a> and <a href="https://docs.python.org/2/reference/datamodel.html#context-managers">With Statement Context Managers</a>.</p></blockquote><h3 id="contextlib-contextmanager"><a href="#contextlib-contextmanager" class="headerlink" title="contextlib.contextmanager"></a>contextlib.contextmanager</h3><p>官方的例子是:</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">from contextlib import contextmanager</span><br><span class="line"></span><br><span class="line">@contextmanager</span><br><span class="line">def tag(name):</span><br><span class="line"> print "<%s>" % name</span><br><span class="line"> yield</span><br><span class="line"> print "</%s>" % name</span><br><span class="line"></span><br><span class="line">>>> with tag("h1"):</span><br><span class="line">... print "foo"</span><br><span class="line">...</span><br><span class="line"><h1></span><br><span class="line">foo</span><br><span class="line"></h1></span><br></pre></td></tr></table></figure><p>这个函数是个decorator函数,参考之前的<a href="https://wongchen.github.io/2017/08/19/python-decorator/">一篇</a>,它可以将函数包装为context manager,无需自己显示为其实现<code>__enter__</code>和<code>__exit__</code>方法.</p><p>这个函数返回一个iterator,可以使用<code>yield</code>关键字</p><blockquote><p>At the point where the generator yields, the block nested in the <a href="https://docs.python.org/2/reference/compound_stmts.html#with"><code>with</code></a> statement is executed.</p></blockquote><p>yield之后,便会执行块下的内容.</p><p>改写上面的例子为</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">@contextmanager</span><br><span class="line">def opened(filename, mode="r"):</span><br><span class="line"> f = open(filename, mode)</span><br><span class="line"> try:</span><br><span class="line"> yield f</span><br><span class="line"> finally:</span><br><span class="line"> f.close()</span><br></pre></td></tr></table></figure>]]></content>
<summary type="html"><h2 id="What-is-context-manager"><a href="#What-is-context-manager" class="headerlink" title="What is context manager?"></a>What is context </summary>
<category term="python" scheme="http://wongchen.github.io/categories/python/"/>
<category term="python" scheme="http://wongchen.github.io/tags/python/"/>
<category term="content manager" scheme="http://wongchen.github.io/tags/content-manager/"/>
</entry>
<entry>
<title>python之generator学习</title>
<link href="http://wongchen.github.io/2017/08/20/python-generator/"/>
<id>http://wongchen.github.io/2017/08/20/python-generator/</id>
<published>2017-08-19T16:43:18.000Z</published>
<updated>2023-07-16T07:07:58.785Z</updated>
<content type="html"><![CDATA[<h2 id="动机"><a href="#动机" class="headerlink" title="动机"></a>动机</h2><p>先看一个问题: 假设我们遇到一个需要长时间响应才能得到返回值的情境,比如深度学习中的大计算或者网络较长时间的响应等(这里我们用<code>sleep()</code>函数来代替),每次需要较长时间的计算才能得到结果.</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> time <span class="keyword">import</span> sleep</span><br><span class="line"><span class="keyword">def</span> <span class="title function_">long_time_compute</span>():</span><br><span class="line"> rv = <span class="built_in">list</span>()</span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">10</span>):</span><br><span class="line"> sleep(<span class="number">2</span>)</span><br><span class="line"> rv.append(i)</span><br><span class="line"> <span class="keyword">return</span> rv</span><br></pre></td></tr></table></figure><p>如上面所述,假如我想得到的最终结果含10个值,我需要20秒才能观察到最后结果,但是假如包含更长的值(如100呢)而且我想观察前几个值呢? 因为从前几个值就能发现一些有意义的东西,我不需要观察全部的值就能得到我想要的,怎么办?</p><p>明显要等到全部值计算完都返回是不符合逻辑的,如果我们只是观察结果不用到返回值的情况可以每次都<code>print</code>打印出每次的结果,但是这也不能解决所有的情况. 幸运的是我们可以通过<code>generator</code>来解决这种问题.</p><h2 id="What-is-python-Generator"><a href="#What-is-python-Generator" class="headerlink" title="What is python Generator?"></a>What is python Generator?</h2><blockquote><p>Introduced in Python 2.2 as an optional feature and finalized in version 2.3, [generators][1] are Python’s mechanism for [lazy evaluation][2] of a function that would otherwise return a space-prohibitive or computationally intensive list.</p></blockquote><blockquote><p>A Python <em>generator</em> is a function which returns a <em>generator iterator</em> (just an object we can iterate over) by calling <code>yield</code>. <code>yield</code> may be called with a value, in which case that value is treated as the “generated” value. The <em>next time</em><code>next()</code> is called on the <em>generator iterator</em> (i.e. in the next step in a <code>for</code> loop, for example),the generator resumes execution <em>from where it called yield</em>, not from the beginning of the function. All of the state, like the values of local variables, is recovered and the generator contiues to execute until the next call to <code>yield</code>.</p></blockquote><p>generator可以不像一般的函数那样,一次返回一个值,下一次还在上次执行的地方继续运行程序返回下一个值,可以说是返回一系列值(One at a time). </p><h3 id="Generator-expressions"><a href="#Generator-expressions" class="headerlink" title="Generator expressions"></a>Generator expressions</h3><blockquote><p>Introduced in Python 2.4, generator expressions are the lazy evaluation equivalent of list comprehensions. </p></blockquote><p>不同于生成器,生成器表达式是<code>list comprehensions</code>的generator形式表达.与<code>list comprehensions</code>长得很像,但是返回一个generator而不是list.</p><p>此外,generator不能用下标来索引.</p><p>如下, 详细后面叙述.</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">>>> a = (x for x in range(8))</span><br><span class="line">>>> a</span><br><span class="line"><generator object <genexpr> at 0x104684960></span><br><span class="line">>>> a[2]</span><br><span class="line">Traceback (most recent call last):</span><br><span class="line"> File "<stdin>", line 1, in <module></span><br><span class="line">TypeError: 'generator' object has no attribute '__getitem__'</span><br></pre></td></tr></table></figure><h2 id="如何"><a href="#如何" class="headerlink" title="如何"></a>如何</h2><p>对于开始提出的问题,如何使用generator来解决呢?</p><p>我们最后想要的是一个可以迭代的对象,大致为</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">x = <span class="built_in">iter</span>(rv)</span><br><span class="line">next_value = <span class="built_in">next</span>(x)</span><br></pre></td></tr></table></figure><p>我们知道python中许多方法和属性都是有对应的double underscore属性和方法的(如<code>__add__</code>).上面的对应为<code>__iter__</code>(python2中是<code>next()</code>)和<code>__next__</code>方法.</p><p>于是我们可以使用一个类,实现上述两种方法来实现generator.</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">class</span> <span class="title class_">ComputerLong</span>:</span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">__iter__</span>(<span class="params">self</span>):</span><br><span class="line"> self.last = <span class="number">0</span></span><br><span class="line"> <span class="keyword">return</span> self</span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">__next__</span>(<span class="params">self</span>):</span><br><span class="line"> rv = self.last</span><br><span class="line"> self.last += <span class="number">1</span></span><br><span class="line"> <span class="keyword">if</span> self.last > <span class="number">10</span>:</span><br><span class="line"> <span class="keyword">raise</span> StopIteration()</span><br><span class="line"> sleep(<span class="number">2</span>)</span><br><span class="line"> <span class="keyword">return</span> rv</span><br></pre></td></tr></table></figure><p>但是这样读起来太费劲了,正常点的写法还是应该像上面函数那样. 更Generator的写法应该使用<code>yield</code>.</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> time <span class="keyword">import</span> sleep</span><br><span class="line"><span class="keyword">def</span> <span class="title function_">long_time_compute</span>():</span><br><span class="line"> rv = <span class="built_in">list</span>()</span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">10</span>):</span><br><span class="line"> sleep(<span class="number">2</span>)</span><br><span class="line"> <span class="keyword">yield</span> i</span><br><span class="line"> <span class="comment"># rv.append(i)</span></span><br><span class="line"> <span class="comment"># return rv</span></span><br></pre></td></tr></table></figure><p>Generator每次生成一个新的值时,采用<code>yield</code>关键字代替原来return的位置,相当于返回一个需要即可取的值.</p><p>Generator返回是一个可迭代的对象,因此可以使用<code>for</code>循环:</p><p>每次<code>for</code>循环中需要一个值时,便会调用generator对象的next()方法,取回一个值.</p><h2 id="generator结束"><a href="#generator结束" class="headerlink" title="generator结束"></a>generator结束</h2><p>当generator函数中使用return或者generator迭代到最后,将会返回一个<code>StopIteration</code> exception,当next到最后一个值时将会报错,因此generator只能够循环一次.</p><p>![][image-1]</p><p>最后:</p><ul><li>generator可以用来返回一个迭代器,可以返回一个序列(每次一个值而不是直接全部的返回值).</li><li>yield可以看做next()方法return一个值并记住这个位置,下次在这个位置利用next()方法仍旧如此.</li><li>python2中generator next()方法为<code>g.next()</code>,而python3为<code>g.__next__()</code></li></ul><p>[1]:<a href="https://en.wikipedia.org/wiki/Generator_(computer_science)">https://en.wikipedia.org/wiki/Generator_(computer_science)</a><br>[2]:<a href="https://en.wikipedia.org/wiki/Lazy_evaluation">https://en.wikipedia.org/wiki/Lazy_evaluation</a></p><p>[image-1]:/images/Screenshot2017-08-23_%E4%B8%8B%E5%8D%8810.27.24.png</p>]]></content>
<summary type="html"><h2 id="动机"><a href="#动机" class="headerlink" title="动机"></a>动机</h2><p>先看一个问题: 假设我们遇到一个需要长时间响应才能得到返回值的情境,比如深度学习中的大计算或者网络较长时间的响应等(这里我们用<code>s</summary>
<category term="python" scheme="http://wongchen.github.io/categories/python/"/>
<category term="python" scheme="http://wongchen.github.io/tags/python/"/>
<category term="generator" scheme="http://wongchen.github.io/tags/generator/"/>
</entry>
<entry>
<title>学习python中的Decorator</title>
<link href="http://wongchen.github.io/2017/08/19/python-decorator/"/>
<id>http://wongchen.github.io/2017/08/19/python-decorator/</id>
<published>2017-08-19T12:28:27.000Z</published>
<updated>2023-07-16T07:07:58.783Z</updated>
<content type="html"><![CDATA[<h2 id="What-is-a-Decorator"><a href="#What-is-a-Decorator" class="headerlink" title="What is a Decorator?"></a>What is a Decorator?</h2><p>参考<a href="https://en.wikipedia.org/wiki/Decorator_pattern">Decorator Pattern wiki</a>,(这个维基词条多讲得是Java/C#/C++相关,但是与python中的实现的功能类似.)</p><blockquote><p>A decorator is the name used for a software design pattern. Decorators dynamically alter the functionality of a function, method, or class without having to directly use subclasses or change the source code of the function being decorated.</p></blockquote><blockquote><p><strong>修饰模式</strong>，是<a href="https://zh.wikipedia.org/wiki/%E9%9D%A2%E5%90%91%E5%AF%B9%E8%B1%A1%E7%BC%96%E7%A8%8B">面向对象编程</a>领域中，一种动态地往一个类中添加新的行为的<a href="https://zh.wikipedia.org/wiki/%E8%BD%AF%E4%BB%B6%E8%AE%BE%E8%AE%A1%E6%A8%A1%E5%BC%8F">设计模式</a>。就功能而言，修饰模式相比生成<a href="https://zh.wikipedia.org/wiki/%E5%AD%90%E7%B1%BB">子类</a>更为灵活，这样可以给某个对象而不是整个类添加一些功能。</p><p>通过使用修饰模式，可以在运行时扩充一个类的功能。原理是：增加一个修饰类包裹原来的类，包裹的方式一般是通过在将原来的对象作为修饰类的构造函数的参数。装饰类实现新的功能，但是，在不需要用到新功能的地方，它可以直接调用原来的类中的方法。修饰类必须和原来的类有相同的接口。</p></blockquote><p><a href="https://en.wikipedia.org/wiki/Python_syntax_and_semantics#Decorators">Python Decorator</a>:</p><blockquote><p>A decorator is any callable Python object that is used to modify a function, method or class definition. A decorator is passed the original object being defined and returns a modified object, which is then bound to the name in the definition. Python decorators were inspired in part by <a href="https://en.wikipedia.org/wiki/Java_annotation">Java annotations</a>, and have a similar syntax; the decorator syntax is pure <a href="https://en.wikipedia.org/wiki/Syntactic_sugar">syntactic sugar</a>, using <code>@</code> as the keyword:</p></blockquote><p>在python中,Decorator与<a href="https://en.wikipedia.org/wiki/Functional_programming">函数式编程(Functional Programming)</a>和<a href="https://en.wikipedia.org/wiki/Metaprogramming">元编程(Metaprogramming)</a>息息相关,可以用Decorator来修改一个函数,类以及类的定义.</p><h2 id="动机"><a href="#动机" class="headerlink" title="动机"></a>动机</h2><p>接下来举几个简单例子说明python中为何要使用Decorator.</p><ol><li><p>更加简明,易读</p><p>如下面两个例子,分别要实现一个classmethod方法和测试函数试行时间的功能,但是函数的定义和最后要实现的功能处出现了分离,对函数的操作变换代码出现在了函数的后面位置,不管是代码复杂性和逻辑都有问题.</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> time <span class="keyword">import</span> time</span><br><span class="line"><span class="comment"># example 1: a classmethod function</span></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">foo</span>(<span class="params">self</span>):</span><br><span class="line">do_something</span><br><span class="line">foo = <span class="built_in">classmethod</span>(foo)</span><br><span class="line"></span><br><span class="line"><span class="comment"># example 2: timer for a function</span></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">foo</span>():</span><br><span class="line">do_something</span><br><span class="line"><span class="keyword">def</span> <span class="title function_">timer</span>(<span class="params">func</span>):</span><br><span class="line"><span class="keyword">def</span> <span class="title function_">decor_fun</span>():</span><br><span class="line">start_time = time()</span><br><span class="line">func()</span><br><span class="line"><span class="built_in">print</span>(<span class="string">"Cost time: "</span>, time()-start_time)</span><br><span class="line"><span class="keyword">return</span> decor_fun</span><br><span class="line">foo = timer(foo)</span><br></pre></td></tr></table></figure><p>通过Decorator更加直观,明了</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> time <span class="keyword">import</span> time</span><br><span class="line"><span class="comment"># example 1</span></span><br><span class="line"><span class="meta">@classmethod</span></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">foo</span>(<span class="params">cls</span>):</span><br><span class="line"> do_something</span><br><span class="line"> </span><br><span class="line"><span class="comment"># example 2</span></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">timer</span>(<span class="params">func</span>):</span><br><span class="line"><span class="keyword">def</span> <span class="title function_">decor_fun</span>():</span><br><span class="line">start_time = time()</span><br><span class="line">func()</span><br><span class="line"><span class="built_in">print</span>(<span class="string">"Cost time: "</span>, time()-start_time)</span><br><span class="line"><span class="keyword">return</span> decor_fun</span><br><span class="line"> </span><br><span class="line"><span class="meta">@timer</span></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">foo</span>():</span><br><span class="line"> do_something</span><br></pre></td></tr></table></figure></li><li><p>实现特定的功能</p><p>参考<a href="https://www.zhihu.com/question/20021164">这个知乎问题:Python 中的 classmethod 和 staticmethod 有什么具体用途?</a>,还可以结合函数式编程,某个函数功能需要被其他多个函数重复实现时,可以考虑Decorator,如: example 2测试函数运行时间, 输出log日志等等.</p></li></ol><h2 id="如何使用"><a href="#如何使用" class="headerlink" title="如何使用"></a>如何使用</h2><p>通过example 2以及<a href="https://en.wikipedia.org/wiki/Python_syntax_and_semantics#Decorators">python Decorator Wiki</a>不难看出<code>@</code>符号其实是以要定义的函数<code>foo</code>作为参数传入<code>@</code>后面的函数<code>bar</code>中,返回一个新的同时实现两者功能的函数.</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@bar</span></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">foo</span>():</span><br><span class="line"> <span class="keyword">return</span> <span class="string">"foo"</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 等价于以下的 bar(foo)</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">foo</span>():</span><br><span class="line"> <span class="built_in">print</span>(<span class="string">"foo"</span>)</span><br><span class="line"> </span><br><span class="line"><span class="keyword">def</span> <span class="title function_">bar</span>(<span class="params">func</span>):</span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">func_decor</span>():</span><br><span class="line"> <span class="built_in">print</span>(<span class="string">"bar/"</span>+func()) </span><br><span class="line"> <span class="keyword">return</span> func_decor</span><br><span class="line"></span><br></pre></td></tr></table></figure><h3 id="带参数呢"><a href="#带参数呢" class="headerlink" title="带参数呢?"></a>带参数呢?</h3><p>如果接触过函数式编程的话,不难解决这个问题,更普遍的方法可以使用<code>*args</code>和<code>**kwargs</code>关键字,针对带有默认参数和不带有默认参数的函数都可以解决.</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> time <span class="keyword">import</span> time</span><br><span class="line"><span class="keyword">def</span> <span class="title function_">timer</span>(<span class="params">func, *args, **kwargs</span>):</span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">decor_func</span>(<span class="params">*args, **kwargs</span>):</span><br><span class="line"> start_time = time()</span><br><span class="line"> ans = func(*args, **kwargs)</span><br><span class="line"> <span class="built_in">print</span>(<span class="string">"cost time: "</span>, time()-start_time)</span><br><span class="line"> <span class="keyword">return</span> ans</span><br><span class="line"> <span class="keyword">return</span> decor_func</span><br><span class="line"></span><br><span class="line"><span class="meta">@timer</span></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">add2</span>(<span class="params">x, y=<span class="number">66</span></span>):</span><br><span class="line"> <span class="keyword">return</span> x + y</span><br><span class="line"></span><br></pre></td></tr></table></figure><p>另外,Decorator也是可以输入参数的,还是上一个例子,实现调用n次函数.</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">def</span> <span class="title function_">ntimes</span>(<span class="params">n</span>):</span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">innner</span>(<span class="params">func</span>)</span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">wrapper</span>(<span class="params">*args, **kwargs</span>):</span><br><span class="line"> <span class="keyword">for</span> _ <span class="keyword">in</span> <span class="built_in">range</span>(n):</span><br><span class="line"> <span class="built_in">print</span>(<span class="string">"calling function %s"</span>%func.__name__)</span><br><span class="line"> ans = func(*args, **kwargs)</span><br><span class="line"> <span class="keyword">return</span> ans</span><br><span class="line"> <span class="keyword">return</span> wrapper</span><br><span class="line"> <span class="keyword">return</span> inner</span><br><span class="line">n = <span class="number">10</span></span><br><span class="line"><span class="meta">@ntimes(<span class="params"><span class="number">10</span></span>)</span></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">add2</span>(<span class="params">x, y=<span class="number">66</span></span>):</span><br><span class="line"> <span class="keyword">return</span> x + y</span><br></pre></td></tr></table></figure><p>本例中,从函数式编程角度看,深度更深了一层,从<code>@</code>本质理解,其后符号ntimes也是函数,故也可接受参数,一般Decorator不会比这个更深,所以了解即可. 详情可以了解<a href="https://en.wikipedia.org/wiki/Python_syntax_and_semantics#Closures">python中的闭包</a>.</p><h3 id="多个Decorator"><a href="#多个Decorator" class="headerlink" title="多个Decorator"></a>多个Decorator</h3><p>道理都是类似的,语法为</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@dec2</span></span><br><span class="line"><span class="meta">@dec1</span></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">func</span>(<span class="params">arg1, arg2, ...</span>):</span><br><span class="line"> <span class="keyword">pass</span></span><br></pre></td></tr></table></figure><h2 id="Class-Decorator"><a href="#Class-Decorator" class="headerlink" title="Class Decorator"></a>Class Decorator</h2><blockquote><p>In Python prior to version 2.6, decorators apply to functions and methods, but not to classes. Decorating a (dummy) <code>__new__</code> method can modify a class, however.[<a href="https://en.wikipedia.org/wiki/Python_syntax_and_semantics#cite_note-28">25]</a>Class decorators are supported[<a href="https://en.wikipedia.org/wiki/Python_syntax_and_semantics#cite_note-29">26]</a> starting with Python 2.6.</p></blockquote><p>Python 2.6之后才支持class decorators,之前都是通过metaclass来实现相应功能,<code>__new__</code>方法修改类.</p><p>按照之前的解释,Class Decorator与Function Decorator应该是基本相同的,只是装饰的对象不同而已.</p><h3 id="Decorator-作用整个类"><a href="#Decorator-作用整个类" class="headerlink" title="Decorator 作用整个类"></a>Decorator 作用整个类</h3><p>可以看到此时的decorator接受一个类作为函数参数,返回一个修饰之后的类,跟之前一样的道理.</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">def</span> <span class="title function_">decorator</span>(<span class="params">cls</span>):</span><br><span class="line"> <span class="keyword">class</span> <span class="title class_">Wrapper</span>(<span class="title class_ inherited__">object</span>):</span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">__init__</span>(<span class="params">self, *args</span>):</span><br><span class="line"> self.wrapped = cls(*args)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">__getattr__</span>(<span class="params">self, name</span>):</span><br><span class="line"> <span class="built_in">print</span>(<span class="string">'Getting the {} of {}'</span>.<span class="built_in">format</span>(name, self.wrapped))</span><br><span class="line"> <span class="keyword">return</span> <span class="built_in">getattr</span>(self.wrapped, name)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> Wrapper</span><br><span class="line"></span><br><span class="line"><span class="meta">@decorator</span></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">C</span>(<span class="title class_ inherited__">object</span>):</span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">__init__</span>(<span class="params">self, x, y</span>):</span><br><span class="line"> self.x = x</span><br><span class="line"> self.y = y</span><br></pre></td></tr></table></figure><h3 id="Decorator本身是类"><a href="#Decorator本身是类" class="headerlink" title="Decorator本身是类"></a>Decorator本身是类</h3><p>如下面例子所示,</p><p><code>__init__</code>方法接受要修饰的函数为参数.</p><p><code>__call__</code>方法在调用被修饰函数时调用.</p><p>此外还有<code>__set__</code>, <code>__get__</code>, <code>__getattribute__</code>等方法均可修改.</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">class</span> <span class="title class_">decorator</span>(<span class="title class_ inherited__">object</span>):</span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">__init__</span>(<span class="params">self, func</span>):</span><br><span class="line"> self.func = func</span><br><span class="line"></span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">__call__</span>(<span class="params">self, *args</span>):</span><br><span class="line"> <span class="built_in">print</span>(<span class="string">'Called {func} with args: {args}'</span>.<span class="built_in">format</span>(func=self.func.func_name,</span><br><span class="line"> args=args))</span><br><span class="line"> <span class="keyword">return</span> self.func(*args)</span><br><span class="line"></span><br><span class="line"><span class="meta">@decorator</span></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">func</span>(<span class="params">x,y</span>):</span><br><span class="line"> <span class="keyword">return</span> x + y</span><br><span class="line"></span><br><span class="line">func(<span class="number">1</span>, <span class="number">3</span>)</span><br></pre></td></tr></table></figure><p>从这个角度看,Decorator可以实现修改调用模型的这一功能. 可以参考<a href="https://www.ibm.com/developerworks/cn/linux/l-cpdecor.html">修改调用模型</a>.</p><h2 id="改进"><a href="#改进" class="headerlink" title="改进"></a>改进</h2><p>仔细研读细节就会发现,使用decorator可以完成期望的功能,取得成效,但是也会带来代码细节与原本出现偏差的问题,毕竟对了一层调用,改进方法可以使用 <code>functools </code>的<code>wraps</code>Decorator.</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> functools <span class="keyword">import</span> wraps</span><br><span class="line"></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">some_decorator</span>(<span class="params">func</span>):</span><br><span class="line"><span class="meta"> @wraps(<span class="params">f</span>)</span></span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">wrapper</span>(<span class="params">*args, **kwargs</span>):</span><br><span class="line"> func(*args, **kwargs)</span><br><span class="line"> <span class="keyword">return</span> wrapper</span><br></pre></td></tr></table></figure><p>也可以使用decorator.</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> decorator <span class="keyword">import</span> decorator</span><br><span class="line"></span><br><span class="line"><span class="meta">@decorator</span></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">some_decorator</span>(<span class="params">func</span>):</span><br><span class="line"> <span class="keyword">def</span> <span class="title function_">wrapper</span>(<span class="params">*args, **kwargs</span>):</span><br><span class="line"> func(*args, **kwargs)</span><br><span class="line"> <span class="keyword">return</span> wrapper</span><br></pre></td></tr></table></figure>]]></content>
<summary type="html"><h2 id="What-is-a-Decorator"><a href="#What-is-a-Decorator" class="headerlink" title="What is a Decorator?"></a>What is a Decorator?</h2><p></summary>
<category term="python" scheme="http://wongchen.github.io/categories/python/"/>
<category term="python" scheme="http://wongchen.github.io/tags/python/"/>
<category term="decorator" scheme="http://wongchen.github.io/tags/decorator/"/>
</entry>
<entry>
<title>我的KeyboardMaestro宏</title>
<link href="http://wongchen.github.io/2016/12/09/myKeyboardMaestroActions/"/>
<id>http://wongchen.github.io/2016/12/09/myKeyboardMaestroActions/</id>
<published>2016-12-09T08:16:00.000Z</published>
<updated>2023-07-16T07:07:58.782Z</updated>
<content type="html"><![CDATA[<p><a href="https://www.keyboardmaestro.com/main/">Keyboard Maestro</a>作为Mac下出名的效率软件,有着不菲的名声,是<strong>Winner of a Macworld Editors’ Choice Award for 2013.</strong></p><p>官网的介绍是:</p><blockquote><p>Whether you are a power user or a grandparent (or both!), your time is precious. So why waste it when Keyboard Maestro can help improve almost every aspect of using your Mac. Even the simplest things, like typing your email address, or going to Gmail or Facebook, launching Pages, or duplicating a line, all take time and add frustration. <strong>Let Keyboard Maestro help make your Mac life more pleasant and efficient.</strong></p></blockquote><p>工欲善其事必先利其器,在使用Mac工作学习的过程中发现一些跟之前使用习惯不适应和”稍微反人类的”一些操作,比如在Finder中的*”在当前目录打开终端”*等操作,因此想到了Keyboard Maestro,通过它来简化一些操作将大大提高效率,也更加简便.</p><p>然后在这里简单介绍我常用的一些宏(<strong>常更新哦</strong>)</p><h2 id="Finder"><a href="#Finder" class="headerlink" title="Finder"></a>Finder</h2><p>Finder作为使用率最高的软件之一,首先介绍Finder相关的宏.由于快捷键太难记忆而决定采用Palette,Finder不会占据太大的空间,因此Palette也不会出现遮挡Finder窗口的问题.</p><p><img src="/images/palette.png" alt="finder palette"></p><p>显示为</p><p><img src="/images/finder_palette.png" alt="finder palette display"></p><ul><li>在Finder中显示”在terminal中打开”,很简单如下图实现</li></ul><p><img src="/images/finder_terminal.png" alt="finder terminal"></p><ul><li>新建文件(<strong>文件名自行输入</strong>)</li></ul><p>由于Keyboard Maestro支持Apple Script,因此可以通过如下代码实现:</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">tell application "Finder"</span><br><span class="line">set annoying_path to the target of the front window as text</span><br><span class="line">set actual_path to POSIX path of annoying_path</span><br><span class="line">set something to display dialog "File name?" default answer ""</span><br><span class="line">set textReturned to text returned of something</span><br><span class="line">set newFileName to actual_path & textReturned</span><br><span class="line">set accessNum to open for access newFileName with write permission</span><br><span class="line">end tell</span><br></pre></td></tr></table></figure><p>点击<code>"新建文件"</code>时,会提示窗口输入文件名</p><p><img src="/images/fider_new_file.png" alt="new file"></p><ul><li>复制文件名</li></ul><p>只需如下脚本:</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">tell application "Finder"</span><br><span class="line">set theFile to the selection</span><br><span class="line">set fileName to name of item 1 of theFile</span><br><span class="line">set the clipboard to fileName</span><br><span class="line">end tell</span><br></pre></td></tr></table></figure><p>另外Mac自带的<strong>脚本编辑器</strong>,可以编写运行Apple Script,在写脚本时可以用它写完测试ok后再复制到Keyboard Maestro中.</p><ul><li>复制文件绝对路径</li></ul><p>有时在终端下工作时,或者其他环境中需要某一文件的路径,这时候使用绝对路径会是有效可靠且方便的,同样脚本为:</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">tell application "Finder"</span><br><span class="line">set sel to the selection</span><br><span class="line">set the clipboard to POSIX path of (sel as text)</span><br><span class="line">end tell</span><br></pre></td></tr></table></figure><p>结合Apple Script, Keyboard Maestro可以发挥更加强大的功能,慢慢通过不断的需求和学习可以开发出更多新的功能</p><h2 id="Launch-Apps"><a href="#Launch-Apps" class="headerlink" title="Launch Apps"></a>Launch Apps</h2><p>对于一些常用的应用程序可以设定某些快捷键,方便快速启动,就像在Linux下面,<code>Alt</code> <code>+</code> <code>control</code> <code>+</code> <code>T</code> 开启终端,win下面, <code>win</code> <code>+</code> <code>D</code>开启文件管理器.</p><p>我常用的如:</p><p><img src="/images/launch_app_lists.png" alt="launch apps"></p><p>设定自己习惯,容易记忆的快捷键即可.</p><p>另外,发现每次截图之后想要寻找截图照片变得繁杂,因此每次截图后主动在Finder下显示截图目录</p><p><img src="/images/screeshots_finder.png" alt="截图 弹窗"></p><p>(逐渐更新中….)</p>]]></content>
<summary type="html"><p><a href="https://www.keyboardmaestro.com/main/">Keyboard Maestro</a>作为Mac下出名的效率软件,有着不菲的名声,是<strong>Winner of a Macworld Editors’ Choice A</summary>
<category term="Mac配置" scheme="http://wongchen.github.io/categories/Mac%E9%85%8D%E7%BD%AE/"/>
<category term="Keyboard Maestro" scheme="http://wongchen.github.io/tags/Keyboard-Maestro/"/>
</entry>
<entry>
<title>Mac下安装软件包遇到的错误总结</title>
<link href="http://wongchen.github.io/2016/11/22/mac-uninstall-pip-packages/"/>
<id>http://wongchen.github.io/2016/11/22/mac-uninstall-pip-packages/</id>
<published>2016-11-21T16:30:37.000Z</published>
<updated>2023-07-16T07:07:58.780Z</updated>
<content type="html"><![CDATA[<p>最近在装一些软件包的时候,总遇到一些很难解决的错误,总结下:</p><h2 id="scipy-seaborn"><a href="#scipy-seaborn" class="headerlink" title="scipy, seaborn"></a>scipy, seaborn</h2><p>Python的包我都是采用pip来安装的</p><blockquote><p>SciPy (pronounced “Sigh Pie”) is a Python-based ecosystem of open-source software for mathematics, science, and engineering. In particular, these are some of the core packages:</p></blockquote><ul><li>NumPy</li><li>SciPy</li><li>Matplotlib</li><li>IPython</li><li>Sympy</li><li>pandas</li></ul><p>Seaborn的话,</p><blockquote><p>Seaborn is a Python visualization library based on matplotlib. It provides a high-level interface for drawing attractive statistical graphics.</p></blockquote><p>但是,我用<code>pip install seaborn</code>后,import时,出现<font color=red>ERROR</font></p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">from . import futil</span><br><span class="line">ImportError: cannot import name futil</span><br></pre></td></tr></table></figure><p>在这之前还有个错误,这个跟个人习惯有关,我在本地创建的一个<code>new.py</code>的文件,正好seaborn导入时会用到<code>new.py</code>,由Python包导入顺序此时本地的<code>new</code>要优先于seaborn的<code>new</code>,这个”案子”提示我们自我创建文件时</p><ol><li>目录要准确</li><li>名字不要太容易起冲突</li></ol><p>不过这类错误一眼也能看出来.<br>回到刚那个错误,搜索很久,发现安装的<code>SciPy</code>应该是一个Broken的版本(到那个目录查看的话,会发现<code>futil.so</code>这个文件),更新下就好了.然而,这货根本卸载不掉</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">OSError: [Errno 1] Operation not permitted: '/tmp/pip-JPaA5b-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy-0.13.0b1-py2.7.egg-info'</span><br></pre></td></tr></table></figure><p>参考<a href="http://xiaorui.cc/2016/03/27/%E8%A7%A3%E5%86%B3mac-osx%E4%B8%8Bpip%E5%AE%89%E8%A3%85ipython%E6%9D%83%E9%99%90%E7%9A%84%E9%97%AE%E9%A2%98/">解决mac osx下pip安装ipython权限的问题</a>,终于解决了.</p><blockquote><p>由于El Capitan引入了SIP机制(System Integrity Protection),默认下系统启用SIP系统完整性保护机制，无论是对于硬盘还是运行时的进程限制对系统目录的写操作。<br> 现在的解决办法是取消SIP机制，具体做法是：<br>重启电脑，按住Command+R(直到出现苹果标志)进入Recovery Mode(恢复模式)<br>左上角菜单里找到实用工具 -> 终端<br>输入csrutil disable回车<br>重启Mac即可<br>如果想重新启动SIP机制重复上述步骤改用csrutil enable即可.</p></blockquote><p>这下好了,然而运行下面这段代码时发现,</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">sns.set_style("whitegrid")</span><br><span class="line">data = np.random.normal(size=(20, 6)) + np.arange(6) / 2</span><br><span class="line">sns.boxplot(data=data);</span><br></pre></td></tr></table></figure><p>这根本什么都没有,什么都没有,其实这个问题很早就有了……</p><p><a href="https://www.zhihu.com/question/35169072">知乎</a><br><a href="http://stackoverflow.com/questions/2512225/matplotlib-not-showing-up-in-mac-osx">stack<strong>overflow</strong></a></p><p>都这么长时间了,真是的,尝试了下面的所有方法,只有<code>plt.show()</code>这个方法能使.但是,这个”小火箭”,便退不出去了<img src="/images/python_sns.png" alt="python sns"><br>不过<code>sns.plt</code>倒是有很多功能,应该能够实现我们想要的.</p><h2 id="VTK"><a href="#VTK" class="headerlink" title="VTK"></a>VTK</h2><p>全称是<strong>The Visulition ToolKit</strong>, 有<a href="http://www.vtk.org/Wiki/Main_Page">wiki</a>.</p><p>安装完遇到的第一个问题便是几个examples的cxx版本,几乎什么都没有<br><img src="http://g.recordit.co/TWfNFkfk2y.gif" alt="NoneVtk"><br>似乎是VTK版本更新后,出现了这些问题,我不知道了.不过还是有其他例子可以显示的.</p><p>有些例子中包含vtkRenderWindowInteractor`即VTK交互渲染窗口是可以正常显示的,但是其他就不行了.</p><p>后来发现解决方法:</p><p>举例来说,如这个<a href="https://github.com/Kitware/VTK/blob/master/Examples/Tutorial/Step1/Cxx/Cone.cxx">例子</a>是VTK官方的Toturial中的step1,在cxx源代码中加入</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">#include <unistd.h></span><br><span class="line">// <...step1 code...></span><br><span class="line"> for (i = 0; i < 360; ++i) //每次渲染窗口时通过usleep函数,可以让窗口保留显示一段时间</span><br><span class="line"> {</span><br><span class="line"> usleep(microseconds);</span><br><span class="line"></span><br><span class="line"> // render the image</span><br><span class="line"> renWin->Render();</span><br><span class="line"> // rotate the active camera by one degree</span><br><span class="line"> ren1->GetActiveCamera()->Azimuth( 1 );</span><br><span class="line"> }</span><br></pre></td></tr></table></figure><h2 id="其他"><a href="#其他" class="headerlink" title="其他"></a>其他</h2><ul><li>安装MITK时,用Cmake编译好久,错误好多,无法解决</li><li>安装Qt5,遇到错误</li></ul><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">Project ERROR: Xcode not set up properly. You may need to confirm the license agreement by running /usr/bin/xcodebuild</span><br></pre></td></tr></table></figure><p>后来发现是个某个版本的bug,换了其他方式安装合适的版本后ok了</p>]]></content>
<summary type="html"><p>最近在装一些软件包的时候,总遇到一些很难解决的错误,总结下:</p>
<h2 id="scipy-seaborn"><a href="#scipy-seaborn" class="headerlink" title="scipy, seaborn"></a>scipy, s</summary>
<category term="Mac配置" scheme="http://wongchen.github.io/categories/Mac%E9%85%8D%E7%BD%AE/"/>
<category term="mac" scheme="http://wongchen.github.io/tags/mac/"/>
<category term="pip" scheme="http://wongchen.github.io/tags/pip/"/>
<category term="vtk" scheme="http://wongchen.github.io/tags/vtk/"/>
<category term="seaborn" scheme="http://wongchen.github.io/tags/seaborn/"/>
</entry>
<entry>
<title>IPv6环境使用Goagent代理免校园网流量</title>
<link href="http://wongchen.github.io/2016/11/13/gae-proxy-ipv6/"/>
<id>http://wongchen.github.io/2016/11/13/gae-proxy-ipv6/</id>
<published>2016-11-13T14:49:58.000Z</published>
<updated>2023-07-16T07:07:58.790Z</updated>
<content type="html"><![CDATA[<h2 id="ipv6"><a href="#ipv6" class="headerlink" title="ipv6"></a>ipv6</h2><p>If you ask Google “What is ipv6 ?”, the first entry it will display says that</p><blockquote><p>Internet Protocol Version 6 (IPv6) is a network layer protocol that enables data communications over a packet switched network. Packet switching involves the sending and receiving of data in packets between two nodes in a network. </p></blockquote><p>用中文讲,就是</p><blockquote><p>互联网通信协议第6版（英文：Internet Protocol version 6，缩写：IPv6）是互联网协议的最新版本，用于分组交换互联网络的网络层协议，旨在解决IPv4地址枯竭问题。<br> IPv6意图替换IPv4，而IPv4在2014年5月仍然在网络交通上占有96%。[1][2][3]在2014年10月，通过IPv6使用Google服务的用户百分率首次超过5%。</p></blockquote><p> 教育网由于采用IPv6,有它固有的优势</p><ul><li>地址容量大大扩展，由原来的32位扩充到128位，彻底解决IPv4地址不足的问题；支持分层地址结构，从而更易于寻址；扩展支持组播和任意播地址，这使得数据包可以发送给任何一个或一组节点；</li><li>大容量的地址空间能够真正的实现无状态地址自动配置，使IPv6终端能够快速连接到网络上，无需人工配置，实现了真正的即插即用；</li><li>…… 其实我是不懂的</li></ul><h2 id="GoAgent"><a href="#GoAgent" class="headerlink" title="GoAgent"></a>GoAgent</h2><p>From <a href="https://zh.wikipedia.org/wiki/GoAgent">Wikepidia</a>,</p><blockquote><p>GoAgent是使用跨平台语言Python开发、基于GPL自由软件协议的代理软件。它利用Google App Engine（GAE）的服务器充当代理。从2015年8月以后，GoAgent已停止维护，并被开发者删除。不过，GoAgent作者phuslu于2015年3月开始开发的另一个翻墙软件GoProxy仍处于活跃维护状态。</p></blockquote><p>虽然那会儿停止维护了,但是我发现虽然IPv4的IP不能使了,IPv6还是OK的.于是我就大部分时间一直都在用它,用到现在.</p><p>具体怎么用就不说了,BTW,需要开启IPv6,在<code>local/proxy.ini</code>配置文件中,修改<code>ipv6=1</code><br><img src="/images/ipv6.png" alt="ipv6"></p><h3 id="浏览器使用"><a href="#浏览器使用" class="headerlink" title="浏览器使用"></a>浏览器使用</h3><p><code>chrome</code> 配合插件 <code>SwitchyOmega</code>,修改规则<br>代理类型<em>HTTP</em>,服务器 <em>127.0.0.1</em> ,端口 <em>8087</em>,也可以使用备份文件恢复.<br>需要注意的是,完了需要信任其证书.</p><p>在线看YouTube还是很快的,经常看Apple/Google 发布会 Live stream,还有NBA 的Live,还有Udacity和Coursera上面的课程……资源还是很多的,速度也基本ok.<br><img src="/images/youtube.png" alt="youtube"></p><h3 id="Terminal使用"><a href="#Terminal使用" class="headerlink" title="Terminal使用"></a>Terminal使用</h3><p>修改变量<code>http_proxy</code>和<code>https_proxy</code>即可,如下</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">export http_proxy='http://localhost:8087' </span><br><span class="line">export https_proxy='http://localhost:8087'</span><br></pre></td></tr></table></figure><p>总之,在Terminal下,代理后速度比不用代理要<strong>快了</strong>几倍,还是很可以的.</p><p>此外,还可以设置git代理, clone不再慢~</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">git config --global http.proxy 'http://127.0.0.1:8087'</span><br><span class="line">git config --global https.proxy 'http://127.0.0.1:8087'</span><br></pre></td></tr></table></figure><p>在用<code>wget</code>下载时需要添加<code>--no-check-certificate</code>,否则会<br><code>ERROR: cannot verify github.com's certificate, issued by ‘CN=GoAgent,OU=GoAgent,O=GoAgent,L=Cernet,ST=Internet,C=CN’: Unable to locally verify the issuer's authority.</code> </p><p> 之后找到解决办法再来更新</p><p>其他app也是可以,像迅雷,百度云,spotify这些,很省流量啊</p>]]></content>
<summary type="html"><h2 id="ipv6"><a href="#ipv6" class="headerlink" title="ipv6"></a>ipv6</h2><p>If you ask Google “What is ipv6 ?”, the first entry it will di</summary>
<category term="Mac配置" scheme="http://wongchen.github.io/categories/Mac%E9%85%8D%E7%BD%AE/"/>
<category term="IPv6" scheme="http://wongchen.github.io/tags/IPv6/"/>
<category term="GAE" scheme="http://wongchen.github.io/tags/GAE/"/>
</entry>
<entry>
<title>我的Vim配置</title>
<link href="http://wongchen.github.io/2016/11/13/%E6%88%91%E7%9A%84MacVim%E9%85%8D%E7%BD%AE/"/>
<id>http://wongchen.github.io/2016/11/13/%E6%88%91%E7%9A%84MacVim%E9%85%8D%E7%BD%AE/</id>
<published>2016-11-13T05:14:35.000Z</published>
<updated>2023-07-16T07:07:58.781Z</updated>
<content type="html"><,像goto declarations, definitions, usages, etc这种,根据语意重命名变量(rename variables)等等,使用体验试试吧</p><h4 id="安装"><a href="#安装" class="headerlink" title="安装"></a>安装</h4><ul><li>使用mvim(Vim binary that is inside the MacVim.app package )更好体验,将vim链接到mvim</li></ul><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">ln -s /usr/local/bin/mvim vim</span><br></pre></td></tr></table></figure><ul><li>vundle cloneYCM后,到YCM目录编译,需要用到<a href="https://cmake.org/download/">Cmake</a></li><li>编译</li></ul><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">cd ~/.vim/bundle/YouCompleteMe</span><br><span class="line">./install.py --[xxx]</span><br></pre></td></tr></table></figure><p>可选参数有:</p><ul><li>–clang-completer,添加C-family支持</li><li>–all, everything</li></ul><h3 id="auto-pairs"><a href="#auto-pairs" class="headerlink" title="auto-pairs"></a>auto-pairs</h3><p>添加引号,括号配对补全,不多说了.</p><h2 id="nerdcommenter"><a href="#nerdcommenter" class="headerlink" title="nerdcommenter"></a>nerdcommenter</h2><p>添加/解除注释,支持多种语言,默认操作为:</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><leader>cc 加注释</span><br><span class="line"><leader>cu 解开注释</span><br></pre></td></tr></table></figure><h2 id="vim学习"><a href="#vim学习" class="headerlink" title="vim学习"></a>vim学习</h2><ul><li><code>vimtutor</code>是最基础的老师了.</li><li>在推荐一个<a href="https://getpocket.com/a/read/43816131">vim 学习的链接</a>,看完这个应该能学到好多</li><li>话说新MacBook Pro with TouchBar取消了<code>esc</code>键后,vim使用便比较尴尬</li></ul><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">inoremap jk <ESC></span><br></pre></td></tr></table></figure><p>环境搭建好之后,强迫自己多使用学习到的vim高端操作,逐渐熟悉,尤其是在远程CLI环境中,vim很有用,慢慢你会熟悉掌握并爱上vim.</p><p>映射到自己喜欢的按键吧.<br>等等.</p>]]></content>
<summary type="html"><p>网上关于Vim配置的讲很多，一键配置也很多。但是我们使用Vim不是因为它的插件实现的功能，自动补全，提示有多厉害，而是它自身的逻辑，自身的快速定位，格式化代码。</p>
<p>所以自己使用方便，好用的配置才是最好的！</p>
<p>我起初就是写<code>python</c</summary>
<category term="Mac配置" scheme="http://wongchen.github.io/categories/Mac%E9%85%8D%E7%BD%AE/"/>
<category term="mac" scheme="http://wongchen.github.io/tags/mac/"/>
<category term="vim" scheme="http://wongchen.github.io/tags/vim/"/>
</entry>
</feed>