CDK-928: Utility to generate events to existing table. #23
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
DennisDawson
changed the title
| import org.apache.hadoop.conf.Configured; | ||
| import org.apache.hadoop.util.Tool; | ||
|
|
||
| public abstract class BaseEventsTool extends Configured implements Tool { |
There was a problem hiding this comment.
It doesn't look like any of the code in this class is used, so it would be better to remove it and make GenerateEvents implement Tool directly.
| baseTimestamp = System.currentTimeMillis(); | ||
|
|
||
| View<StandardEvent> events = Datasets.load( | ||
| (args.length==1 ? args[0] : "dataset:hive:events"), StandardEvent.class); |
There was a problem hiding this comment.
I noted this elsewhere, but I think it would be better to use a variable rather than the inline test here.
There was a problem hiding this comment.
Is this wrong, or just different? Are you suggesting that the test should set the variable before the load method? If the argument is invalid, does it change the result by setting it outside the load method? If the code must change before publication, please provide the acceptable alternate code, rather than have me guess at what I should do.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With an eye toward modularization, I've repurposed CreateEvents.java from the Spark example and placed it in
org/kitesdk/examples/data. This lets the customer create the events dataset using the CLI, then populate it with a substantial number of records using the Java utility. The same dataset can be used for the Flume and Spark examples, without having to delete them after running their respective jobs.In GenerateEvents, I essentially swapped the CreateEvents
create()method withload(). I added the Avro plug-in topom.xml, copied theavrofolder withstandard_event.avscinto themaindirectory, and copiedBaseEventsTool.javatoorg/kitesdk/examples/data.In my environment, it compiles, runs, and populates the events table as expected.
**Update
The random records were a little too random: if the user_id, session_id, and ip are different each time, when the Crunch utility runs, there are no sessions to aggregate. I revised the run method to first generate the user_id, session_id, and ip, then used a for loop to generate 1-25 random events. I also modified the randomTimestamp method to increase the base length of time and add random padding to create more realistic session duration.
I'm happy to incorporate any changes that make the code more elegant, my changes just make it work.