using data selector as wrapper over raw data #815
Conversation
| @@ -0,0 +1,75 @@ | |||
| package zingg.common.core.data.df; | |||
|
|
||
| public static final Log LOG = LogFactory.getLog(ZFrameDataSelector.class); | ||
|
|
||
| public ZFrameDataSelector(ZFrame<D, R, C> rawData, IArguments args, Context<S,D,R,C,T> context,StopWordsRemover<S, D, R, C, T> stopWordsRemover) { |
There was a problem hiding this comment.
this class shuld not be aware of the blocking tree loading and context etc. it should get the blocking tree and have a method to apply it.
|
|
||
| public ZFrame<D,R,C> getBlocked() throws Exception, ZinggClientException{ | ||
| if (blockedData==null) { | ||
| preprocessedRepartitionedData = stopWordsRemover.preprocessForStopWords(getFieldDefColumnsDS()); |
| //testData = dropDuplicates(testData); | ||
| long count = preprocessedRepartitionedData.count(); | ||
| LOG.info("Read " + count); | ||
| Analytics.track(Metric.DATA_COUNT, count, args.getCollectMetrics()); |
There was a problem hiding this comment.
count should be a method on this and Analytics should not be a part of this object - it should be driven from the executor
|
|
||
| import zingg.common.client.ZFrame; | ||
|
|
||
| public interface IZFrameProcessor<S, D, R, C, T> { |
There was a problem hiding this comment.
BlockingTreeUtil<S, D,R,C,T>
There was a problem hiding this comment.
StopWordsRemover<S,D,R,C,T>
There was a problem hiding this comment.
so we should have one frame<D,R,C> and then another that extends this and adds S and T.
There was a problem hiding this comment.
what benefit it would give? ZData in any case has <S,D,R,C,T>
There was a problem hiding this comment.
unnecessarily two very similar interfaces to be maintained whose objects can't be assigned to one another
| @@ -0,0 +1,11 @@ | |||
| package zingg.common.core.data.df; | |||
There was a problem hiding this comment.
it may be better to introduce a method process() so that the logic is not in the constructor.
| super(); | ||
| this.originalDF = originalDF; | ||
| this.args = args; | ||
| this.processedDF = getOriginalDF().select(new ZidAndFieldDefSelector(args.getFieldDefinition()).getCols()); |
There was a problem hiding this comment.
better to have an overloaded constructor here which can take the fielddefselectors. we have a few so may need to pass those in the future
|
|
||
| public static final Log LOG = LogFactory.getLog(RepartitionFrame.class); | ||
|
|
||
| public RepartitionFrame(ZFrame<D, R, C> originalDF, IArguments args) { |
There was a problem hiding this comment.
we should pass the numberofPartitions and column on which we need to partition
| import zingg.common.client.ZinggClientException; | ||
| import zingg.common.core.preprocess.StopWordsRemover; | ||
|
|
||
| public class PreprocessedFrame<S, D, R, C, T> implements IZFrameProcessor<S, D, R, C, T> { |
There was a problem hiding this comment.
change stopword design - have a processor there. so preprocessedframe can be passed any processor(stopwords, tomorrow lowercase etc)
context.getBlockingTreeUtil().readBlockingTree(args) => other places args can be avoided but blockedframe needs it |
using data selector as wrapper over raw data