New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Introduce StorageManager and an Hdfs Implementation for Storage #90
Conversation
cluster/storage/src/main/java/com/linkedin/openhouse/cluster/storage/StorageManager.java
Show resolved
Hide resolved
cluster/storage/src/main/java/com/linkedin/openhouse/cluster/storage/BaseStorage.java
Show resolved
Hide resolved
...ter/storage/src/main/java/com/linkedin/openhouse/cluster/storage/hdfs/HdfsStorageClient.java
Outdated
Show resolved
Hide resolved
cluster/storage/src/main/java/com/linkedin/openhouse/cluster/storage/BaseStorage.java
Outdated
Show resolved
Hide resolved
cluster/storage/src/main/java/com/linkedin/openhouse/cluster/storage/StorageManager.java
Outdated
Show resolved
Hide resolved
cluster/storage/src/main/java/com/linkedin/openhouse/cluster/storage/StorageManager.java
Outdated
Show resolved
Hide resolved
cluster/storage/src/main/java/com/linkedin/openhouse/cluster/storage/StorageManager.java
Outdated
Show resolved
Hide resolved
cluster/storage/src/main/java/com/linkedin/openhouse/cluster/storage/StorageManager.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @HotSushi . Added some comments.
cluster/storage/src/main/java/com/linkedin/openhouse/cluster/storage/StorageManager.java
Outdated
Show resolved
Hide resolved
...ter/storage/src/main/java/com/linkedin/openhouse/cluster/storage/hdfs/HdfsStorageClient.java
Outdated
Show resolved
Hide resolved
...ter/storage/src/main/java/com/linkedin/openhouse/cluster/storage/hdfs/HdfsStorageClient.java
Outdated
Show resolved
Hide resolved
cluster/storage/src/main/java/com/linkedin/openhouse/cluster/storage/StorageManager.java
Outdated
Show resolved
Hide resolved
cluster/storage/src/main/java/com/linkedin/openhouse/cluster/storage/StorageManager.java
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Left a few comments. But looks good otherwise.
cluster/storage/src/main/java/com/linkedin/openhouse/cluster/storage/StorageManager.java
Show resolved
Hide resolved
cluster/storage/src/main/java/com/linkedin/openhouse/cluster/storage/StorageManager.java
Outdated
Show resolved
Hide resolved
cluster/storage/src/main/java/com/linkedin/openhouse/cluster/storage/StorageManager.java
Outdated
Show resolved
Hide resolved
cluster/storage/src/main/java/com/linkedin/openhouse/cluster/storage/StorageManager.java
Outdated
Show resolved
Hide resolved
...ter/storage/src/main/java/com/linkedin/openhouse/cluster/storage/hdfs/HdfsStorageClient.java
Show resolved
Hide resolved
...ter/storage/src/main/java/com/linkedin/openhouse/cluster/storage/hdfs/HdfsStorageClient.java
Outdated
Show resolved
Hide resolved
...ter/storage/src/main/java/com/linkedin/openhouse/cluster/storage/hdfs/HdfsStorageClient.java
Show resolved
Hide resolved
cluster/storage/src/main/java/com/linkedin/openhouse/cluster/storage/StorageManager.java
Outdated
Show resolved
Hide resolved
cluster/storage/src/main/java/com/linkedin/openhouse/cluster/storage/hdfs/HdfsStorage.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Thanks for addressing the comments/questions!
20af749
Summary
Laying foundations for storage part 3:
StorageManager
and an implementation for Hdfs:HdfsStorage
.StorageManager interface looks like
It runs through all configured Storages and returns the appropriate storage for default or for particular type.
HdfsStorage
is yet another implementation forStorage
similar toLocalStorage
.We've abstracted the common code for all storage-impls into BaseStorage
Changes
Other related PRs:
Add cluster.yaml, StorageProperties, StorageType enum for supporting multiple storages #76
Add interfaces for StorageClient, Storage and its Implementation for Local #82
Testing Done