Fields with a state of DROP on normalization failure #2267
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…to the index but are still preserved in the event.
rdhayes68
changed the title
…lean up old images
* Enforce term limits during EdgeQueries * Update error message * Fix EdgeQuery maxQueryTerms setting
* Added remote edge query logic and tests * factored the BaseRemoteQueryLogic out of the remote edge and event instances
* Updates to try and avoid hanging ivarators * fixed configuration issues of the thread pools * Updates to allow value to be set via properties.
* Cleaning noisy logs in start-loader * Removing tracer log calls * Adjusting logs for additional loader * Small adjustment to echo --------- Co-authored-by: pcagbu <pcagbu@ip-10-113-15-132.evoforge.org>
- added git pre-push hook to ensure code is formatted properly Co-authored-by: Christopher Gross cogross <christopher.gross@red-drum.com>
* Enrich 'i' and 'ri' rows in metadata table with event date Modify the generation of 'i' (indexed rows) and 'ri' (reverse indexed rows) in the metadata table such that the column qualifier contains the event date. This is required as a first step to support efforts for issue #825 so that we can identify dates when an event was ingested and included in a frequency count for an associated 'f' row, but was not indexed. * Add counts to 'i' and 'ri' rows
* fix javadoc warnings --------- Co-authored-by: Christopher Gross cogross <christopher.gross@red-drum.com>
* 2170 - update dependencies --------- Co-authored-by: Christopher Gross cogross <christopher.gross@red-drum.com>
* Replaced custom Tuple2 implementation with ImmutablePair from commons-lang3 * Includes complete ingest testframework for SSDeep data that includes both shard data and ssdeepIndex data. * Cleans up some sins of the past related to the faceted query logic leaking into the test framework. * Adds protoype datawave ingest SSDeepIndexHandler, good enough for testing at this point. * Trimmed pom for ssdeep-ingest to minimum dependency set * Reorganized package structure to keep ssdeep code/tests together * Moved SSDeepQueryTest to common package * Fix parent pom version for ingest-ssdeep and static version for ssdeep-common * Addressed comments from code review
…odes.children() method
* @jwomeara observed that the `QueryTestTableHelper` was no longer using the `RebuildingScannerTestHelper`, and was using a regular `InMemoryInstance` instead, so we were not testing teardowns and interrupts. * Updated `RebuildingScannerTestHelper` to use `AccumuloClient` instead of `AccumuloConnection` incorporating changes from the `feature/queryMicroservices` branch * The only test broken by this is the `SSDeepIngestQueryTest`, so `@Ignore`-ing for now until that test can be fixed by me. Co-authored-by: Whitney O'Meara <jwomeara@users.noreply.github.com>
…iter and vintage dependencies
add junit-jupiter-engine to warehouse-common remove jupiter-api from warehouse pom restore jupitier-api to ws-client pom
…n pom, add jupiter-engine and vintage-engine to failsafe plugins in webservices and warehouse pom
#2271) * Created an event predicate for discarding events during ingest processing * Created a discard interval and future discard interval predicates * Updated tests to test hit and miss on the interval predicates * Add default for DISCARD_FUTURE_INTERVAL event Co-authored-by: foster33 <lukefos62@gmail.com> Co-authored-by: Keith Ratcliffe <ratcliffe.keith@gmail.com> Co-authored-by: hlgp <hlgp@users.noreply.github.com> Co-authored-by: palindrome <31748527+hlgp@users.noreply.github.com> Co-authored-by: matthpeterson <matt.peterson@tibertechnologies.com>
…2283) required in the main datawave builds
…to the index but are still preserved in the event.
…lure-incorrectly-removes-the-field-from-the-event-as-well
…n-failure-incorrectly-removes-the-field-from-the-event-as-well' into 2266-drop-state-for-normalization-failure-incorrectly-removes-the-field-from-the-event-as-well
rdhayes68
deleted the
2266-drop-state-for-normalization-failure-incorrectly-removes-the-field-from-the-event-as-well
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…to the index but are still preserved in the event.