Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: enable bytes read tracking #1074

Merged
merged 31 commits into from May 13, 2020
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
818f150
feat: enable bytesrRead tracking on uploads
AVaksman Jan 7, 2020
72dbea6
refactor: only attach listener if flagged by user
AVaksman Jan 14, 2020
00aea08
lint: lint
AVaksman Feb 13, 2020
edf63fa
refactor: forward progress event from resumable or simple upload to user
AVaksman Mar 2, 2020
9131ec5
lint: lint
AVaksman Mar 2, 2020
a19deee
fix: add listener on a condition
AVaksman Mar 2, 2020
32c1637
Merge branch 'master' into enable_bytesRead_tracking
AVaksman Mar 2, 2020
039a768
Merge branch 'master' into enable_bytesRead_tracking
jkwlui Mar 2, 2020
f063ae9
refactor: only register progress listener if onUploadProgress param i…
AVaksman Mar 4, 2020
f5496a1
feat: add onUploadProgress to file.save()
AVaksman Mar 4, 2020
1116127
Merge branch 'master' into enable_bytesRead_tracking
AVaksman Mar 4, 2020
52dcc37
lint: lint
AVaksman Mar 4, 2020
aa0fcdd
Merge branch 'master' into enable_bytesRead_tracking
AVaksman Mar 11, 2020
04d9c15
Merge branch 'master' into enable_bytesRead_tracking
jkwlui Mar 13, 2020
b74521d
Merge branch 'master' into enable_bytesRead_tracking
AVaksman Mar 16, 2020
2c515c6
Merge branch 'master' into enable_bytesRead_tracking
AVaksman Mar 23, 2020
e55cf34
refactor: readability
AVaksman Mar 23, 2020
bfe3e3b
chore: move event handler assignment on it own line
AVaksman Apr 2, 2020
84c08a1
fix(deps): update dependency @google-cloud/common to v3
AVaksman Apr 2, 2020
f960bcb
Merge branch 'master' into enable_bytesRead_tracking
AVaksman Apr 2, 2020
f6a9c1e
Merge branch 'master' into enable_bytesRead_tracking
AVaksman Apr 7, 2020
ed711f7
Merge branch 'master' into enable_bytesRead_tracking
AVaksman Apr 20, 2020
c80bf75
Merge branch 'master' into enable_bytesRead_tracking
AVaksman Apr 27, 2020
5fc5975
Merge branch 'master' into enable_bytesRead_tracking
AVaksman Apr 27, 2020
ae665f9
test: typo
AVaksman Apr 27, 2020
38841f7
Merge branch 'master' into enable_bytesRead_tracking
AVaksman May 4, 2020
67bacf2
Merge branch 'master' into enable_bytesRead_tracking
AVaksman May 4, 2020
2f9055f
Merge branch 'master' into enable_bytesRead_tracking
AVaksman May 7, 2020
512352d
Merge branch 'master' into enable_bytesRead_tracking
stephenplusplus May 12, 2020
66a76e0
Merge branch 'master' into enable_bytesRead_tracking
stephenplusplus May 12, 2020
195cd46
Merge branch 'master' into enable_bytesRead_tracking
stephenplusplus May 12, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
31 changes: 30 additions & 1 deletion src/file.ts
Expand Up @@ -35,7 +35,7 @@ import * as once from 'onetime';
import * as os from 'os';
const pumpify = require('pumpify');
import * as resumableUpload from 'gcs-resumable-upload';
import {Duplex, Writable, Readable} from 'stream';
import {Duplex, Writable, Readable, Transform} from 'stream';
import * as streamEvents from 'stream-events';
import * as through from 'through2';
import * as xdgBasedir from 'xdg-basedir';
Expand Down Expand Up @@ -198,6 +198,8 @@ export interface CreateWriteStreamOptions extends CreateResumableUploadOptions {
gzip?: string | boolean;
resumable?: boolean;
validation?: string | boolean;
// tslint:disable-next-line:no-any
onUploadProgress?: (progressEvent: any) => void;
}

export interface MakeFilePrivateOptions {
Expand Down Expand Up @@ -1752,12 +1754,20 @@ class File extends ServiceObject<File> {
md5,
});

const progressStream = new ProgressStream();
if (options.onUploadProgress) {
progressStream.on('progress', bytesRead => {
options.onUploadProgress!({bytesRead});
AVaksman marked this conversation as resolved.
Show resolved Hide resolved
AVaksman marked this conversation as resolved.
Show resolved Hide resolved
});
}

const fileWriteStream = duplexify();

const stream = streamEvents(
pumpify([
gzip ? zlib.createGzip() : through(),
validateStream,
progressStream,
AVaksman marked this conversation as resolved.
Show resolved Hide resolved
fileWriteStream,
])
) as Duplex;
Expand Down Expand Up @@ -3489,6 +3499,25 @@ class File extends ServiceObject<File> {
}
}

/**
* Basic Passthrough Stream that records the number of bytes read
* every time the cursor is moved.
*/
class ProgressStream extends Transform {
bytesRead: number;
constructor() {
super(...arguments);
this.bytesRead = 0;
}
// tslint:disable-next-line: no-any
_transform(chunk: any, encoding: string, callback: Function) {
this.bytesRead += chunk.length;
this.emit('progress', this.bytesRead);
this.push(chunk);
callback();
}
}

/*! Developer Documentation
*
* All async methods (except for streams) will return a Promise in the event
Expand Down
16 changes: 16 additions & 0 deletions system-test/storage.ts
Expand Up @@ -93,6 +93,7 @@ import {
Iam,
} from '../src';
import * as nock from 'nock';
import * as readline from 'readline';

interface ErrorCallbackFunction {
(err: Error | null): void;
Expand Down Expand Up @@ -2791,6 +2792,21 @@ describe('storage', () => {
});
});

describe('bucket upload with progress', () => {
it('show bytes sent', async () => {
AVaksman marked this conversation as resolved.
Show resolved Hide resolved
const fileSize = fs.statSync(FILES.big.path).size;
await bucket.upload(FILES.big.path, {
onUploadProgress: evt => {
const progress = (evt.bytesRead / fileSize) * 100;

readline.clearLine(process.stdout, 0);
readline.cursorTo(process.stdout, 0, 0);
process.stdout.write(`${Math.round(progress)}% complete`);
},
});
});
});

describe('channels', () => {
it('should create a channel', done => {
const config = {
Expand Down