From 845da554fb8bb679fa39713e2c9ab94be7934f70 Mon Sep 17 00:00:00 2001 From: Wylie Conlon Date: Thu, 10 Jun 2021 10:09:16 -0400 Subject: [PATCH] [Lens] Formula editor (#99297) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * :lipstick: Hack to fix suggestion box * :bug: Fix validation messages * :bug: Relax operations check for managedReferences * Change completion params * :label: Fix missing arg issue * :sparkles: Add more tinymath fns * :bug: Improved validation around math operations + multiple named arguments * :bug: Use new onError feature in math expression * :recycle: Refactor namedArguments validation * :bug: Fix circular dependency issue in tests + minor fixes * Move formula into a tab * :fire: Leftovers from previous merge * :sparkles: Move over namedArgs from previous function * :white_check_mark: Add tests for transferable scenarios * :white_check_mark: Fixed broken test * :sparkles: Use custom label for axis * Allow switching back and forth to formula tab * Add a section for the function reference * Add modal editor and markdown docs * Change the way math nodes are validated * Use custom portal to fix monaco positioning * Fix model sharing issues * Provide signature help * :bug: Fix small test issue * :bug: Mark pow arguments as required * :bug: validate on first render only if a formula is present * :fire: Remove log10 fn for now * :sparkles: Improved math validation + add tests for math functions * Fix mount/unmount issues with Monaco * [Lens] Fully unmount React when flyout closes * Fix bug with editor frame unmounting * Fix type * Add tests for monaco providers, add hover provider * Add test for last_value * Usability improvements * Add KQL and Lucene named parameters * Add kql, lucene completion and validation * Fix autocomplete on weird characters and properly connect KQL * Highlight functions that have additional requirements after validating * Fix type error and move help text to popover * Fix escape characters inside KQL * :bug: Fix dataType issue when moving over to Formula * Automatically insert single quotes on every named param * Only insert single quotes when typing kql= or lucene= * Reorganize help popover * Fix merge issues * Update grammar for formulas * Fix bad merge * Rough fullscreen mode * Type updates * Pass through fullscreen state * Remove more chrome from full screen mode * Fix minor bugs in formula typing * 🐛 Decouple column order of references and output * 🔧 Fix tests and types * ✅ Add first functional test * Fix copying formulas and empty formula * Trigger suggestion prompt when hitting enter on function or typing kql= * 🐛 Prevent flyout from closing while interacting with monaco * refactoring * move main column generation into parse module * fix tests * refactor small formula styles and markup * documentation * adjustments in formula footer * Formula refactoring (#12) * refactoring * move main column generation into parse module * fix tests * more style and markup tweak for custom formula * Fix tests * [Expressions] Use table column ID instead of name when set * [Lens] Create managedReference type for formulas * Fix test failures * Fix i18n types * fix fullscreen flex issues * Delete managedReference when replacing * refactor css and markup; add button placeholders * [Lens] Formulas * Tests for formula Co-authored-by: Marco Liberati * added error count placeholder * Add tooltips * Refactoring from code review * Fix some editor issues * Update ID matching to match by name sometimes * Improve performance of Monaco, fix formulas with 0, update labels * Improve performance of full screen toggle * Fix formula tests * fix stuff * Add an extra case to prevent insertion of duplicate column * Simplify logic and add test for output ID * add telemetry for Lens formula (#15) * Respond to review comments * :sparkles: Improve the signatures with better documentation and examples * adjust border styles to account for docs collapse * refactor docs markup; restructure docs obj; styles * Fix formula auto reordering (#18) * fix formula auto reordering * add unit test * Fix and improve suggestion experience in Formula (#19) * :sparkles: Revisit documentation and suggestions * :ok_hand: Integrated feedback * :sparkles: Add query validation for quotes * Usability updates & type fixes * add search to formula * fix form styles to match designs * fix text styles; revert to Markdown for control * :ok_hand: Integrated more feedback * improve search * improve suggestions * improve suggestions even more * :bug: Fix i18n issues (#22) * Persist formula on leave, fix fullscreen and popovers * Fix documentation tests * :label: fix type issue * :bug: Remove hidden operations from valid functions list * :bug: Fix empty string query edge case * :bug: Enable more suggestions + extends validation * Fix tests that depended on setState being called without function * Error state and text wrapping updates * :sparkles: Add new module to CodeEditor for brackets matching (#25) * Fix type * show warning * keep current quick function * :sparkles: Improve suggestions within kql query * :camera: Fix snapshot editor test * :bug: Improved suggestion for single quote and refactored debounce * Fix lodash usage * Fix tests * Revert "keep current quick function" This reverts commit ed477054c5560b50ede4245106954509617d65db. * Improve performance of dispatch by using timeout * Improve memoization of datapanel * Fix escape characters * fix reduced suggestions * fix responsiveness * fix unit test * Fix autocomplete on nested math * Show errors and warnings on first render * fix transposing column crash * Update comment * :bug: Fix field error message * fix test types * :memo: Fix i18n name * :lipstick: Manage wordwrap via react component * Fix selector for palettes that interferes with quick functions * Use word wrapping by default * Errors for managed references are handled at the top level * :bug: Move the cursor just next to new inserted text * :alembic: First pass for performance * :bug: Fix unwanted change * :zap: Memoize as many combobox props as possible * :zap: More memoization * Show errors in hover * Use temporary invalid state when moving away from formula * Remove setActiveDimension and shouldClose, fixed by async setters * Fix test dependency * do not show quick functions tab * increase documentation popover width * fix functional test * Call setActiveDimension when updating visualization * Simplify handling of flyout with incomplete columns * Fix test issues * add description to formula telemetry * fix schema * Update from design feedback * More review comments * Hide callout border from v7 theme Co-authored-by: dej611 Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Joe Reuter Co-authored-by: Michael Marcialis Co-authored-by: Joe Reuter Co-authored-by: Marco Liberati Co-authored-by: Marco Liberati --- packages/kbn-monaco/src/monaco_imports.ts | 1 + packages/kbn-tinymath/grammar/grammar.peggy | 44 +- packages/kbn-tinymath/index.d.ts | 6 +- packages/kbn-tinymath/test/library.test.js | 31 + .../server/usage/dashboard_telemetry.test.ts | 17 + .../server/usage/dashboard_telemetry.ts | 23 + .../expression_functions/specs/map_column.ts | 6 +- .../specs/tests/map_column.test.ts | 6 +- .../__snapshots__/code_editor.test.tsx.snap | 2 + .../public/code_editor/code_editor.tsx | 8 +- .../shareable_runtime/webpack.config.js | 2 +- .../lens/public/app_plugin/lens_top_nav.tsx | 251 +++--- .../components/columns.tsx | 2 +- .../config_panel/config_panel.test.tsx | 8 +- .../config_panel/config_panel.tsx | 70 +- .../config_panel/dimension_container.scss | 15 + .../config_panel/dimension_container.tsx | 29 +- .../config_panel/layer_panel.test.tsx | 110 ++- .../editor_frame/config_panel/layer_panel.tsx | 86 +- .../editor_frame/config_panel/types.ts | 2 + .../editor_frame/editor_frame.tsx | 60 +- .../editor_frame/frame_layout.scss | 19 + .../editor_frame/frame_layout.tsx | 28 +- .../editor_frame/state_management.ts | 6 + .../editor_frame/suggestion_panel.tsx | 3 +- .../workspace_panel/workspace_panel.test.tsx | 2 + .../workspace_panel/workspace_panel.tsx | 13 +- .../workspace_panel_wrapper.scss | 8 + .../workspace_panel_wrapper.test.tsx | 2 + .../workspace_panel_wrapper.tsx | 73 +- .../public/editor_frame_service/mocks.tsx | 1 + .../dimension_panel/dimension_editor.scss | 36 +- .../dimension_panel/dimension_editor.tsx | 270 ++++-- .../dimension_panel/dimension_panel.test.tsx | 111 +-- .../droppable/droppable.test.ts | 2 + .../dimension_panel/format_selector.tsx | 96 ++- .../dimension_panel/reference_editor.test.tsx | 3 + .../dimension_panel/reference_editor.tsx | 10 + .../indexpattern_datasource/indexpattern.tsx | 5 + .../indexpattern_suggestions.test.tsx | 211 ++++- .../indexpattern_suggestions.ts | 74 +- .../operations/__mocks__/index.ts | 3 + .../definitions/calculations/counter_rate.tsx | 18 + .../calculations/cumulative_sum.tsx | 16 + .../definitions/calculations/differences.tsx | 17 + .../calculations/moving_average.tsx | 26 +- .../operations/definitions/cardinality.tsx | 17 + .../operations/definitions/count.tsx | 15 + .../definitions/date_histogram.test.tsx | 3 + .../definitions/filters/filters.test.tsx | 3 + .../definitions/formula/editor/formula.scss | 167 ++++ .../formula/editor/formula_editor.tsx | 791 ++++++++++++++++++ .../formula/editor/formula_help.tsx | 469 +++++++++++ .../definitions/formula/editor/index.ts | 8 + .../formula/editor/math_completion.test.ts | 386 +++++++++ .../formula/editor/math_completion.ts | 594 +++++++++++++ .../formula/editor/math_tokenization.tsx | 66 ++ .../definitions/formula/formula.test.tsx | 192 ++++- .../definitions/formula/formula.tsx | 64 +- .../definitions/formula/math_examples.md | 28 + .../operations/definitions/formula/parse.ts | 24 +- .../operations/definitions/formula/util.ts | 273 ++++-- .../definitions/formula/validation.ts | 157 +++- .../operations/definitions/index.ts | 9 + .../definitions/last_value.test.tsx | 3 + .../operations/definitions/last_value.tsx | 16 + .../operations/definitions/metrics.tsx | 43 + .../definitions/percentile.test.tsx | 3 + .../operations/definitions/percentile.tsx | 18 +- .../definitions/ranges/ranges.test.tsx | 3 + .../definitions/terms/terms.test.tsx | 3 + .../operations/layer_helpers.test.ts | 79 +- .../operations/layer_helpers.ts | 57 +- .../operations/mocks.ts | 25 + .../public/indexpattern_datasource/types.ts | 2 + x-pack/plugins/lens/public/mocks.tsx | 3 + x-pack/plugins/lens/public/types.ts | 11 +- x-pack/plugins/lens/server/usage/schema.ts | 16 + .../lens/server/usage/visualization_counts.ts | 36 +- .../schema/xpack_plugins.json | 42 + x-pack/test/functional/apps/lens/formula.ts | 198 +++++ x-pack/test/functional/apps/lens/index.ts | 1 + .../test/functional/apps/lens/smokescreen.ts | 54 +- .../test/functional/page_objects/lens_page.ts | 65 +- 84 files changed, 5117 insertions(+), 659 deletions(-) create mode 100644 x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/formula.scss create mode 100644 x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/formula_editor.tsx create mode 100644 x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/formula_help.tsx create mode 100644 x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/index.ts create mode 100644 x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/math_completion.test.ts create mode 100644 x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/math_completion.ts create mode 100644 x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/math_tokenization.tsx create mode 100644 x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/math_examples.md create mode 100644 x-pack/test/functional/apps/lens/formula.ts diff --git a/packages/kbn-monaco/src/monaco_imports.ts b/packages/kbn-monaco/src/monaco_imports.ts index 872ac46352cf31..92ea23347c374c 100644 --- a/packages/kbn-monaco/src/monaco_imports.ts +++ b/packages/kbn-monaco/src/monaco_imports.ts @@ -21,5 +21,6 @@ import 'monaco-editor/esm/vs/editor/contrib/folding/folding.js'; // Needed for f import 'monaco-editor/esm/vs/editor/contrib/suggest/suggestController.js'; // Needed for suggestions import 'monaco-editor/esm/vs/editor/contrib/hover/hover.js'; // Needed for hover import 'monaco-editor/esm/vs/editor/contrib/parameterHints/parameterHints.js'; // Needed for signature +import 'monaco-editor/esm/vs/editor/contrib/bracketMatching/bracketMatching.js'; // Needed for brackets matching highlight export { monaco }; diff --git a/packages/kbn-tinymath/grammar/grammar.peggy b/packages/kbn-tinymath/grammar/grammar.peggy index cbcb0b91bfea90..1c6f8c3334c234 100644 --- a/packages/kbn-tinymath/grammar/grammar.peggy +++ b/packages/kbn-tinymath/grammar/grammar.peggy @@ -1,16 +1,16 @@ // tinymath parsing grammar { - function simpleLocation (location) { - // Returns an object representing the position of the function within the expression, - // demarcated by the position of its first character and last character. We calculate these values - // using the offset because the expression could span multiple lines, and we don't want to deal - // with column and line values. - return { - min: location.start.offset, - max: location.end.offset + function simpleLocation (location) { + // Returns an object representing the position of the function within the expression, + // demarcated by the position of its first character and last character. We calculate these values + // using the offset because the expression could span multiple lines, and we don't want to deal + // with column and line values. + return { + min: location.start.offset, + max: location.end.offset + } } - } } start @@ -74,26 +74,34 @@ Expression = AddSubtract AddSubtract - = _ left:MultiplyDivide rest:(('+' / '-') MultiplyDivide)* _ { - return rest.reduce((acc, curr) => ({ + = _ left:MultiplyDivide rest:(('+' / '-') MultiplyDivide)+ _ { + const topLevel = rest.reduce((acc, curr) => ({ type: 'function', name: curr[0] === '+' ? 'add' : 'subtract', args: [acc, curr[1]], - location: simpleLocation(location()), - text: text() - }), left) + }), left); + if (typeof topLevel === 'object') { + topLevel.location = simpleLocation(location()); + topLevel.text = text(); + } + return topLevel; } + / MultiplyDivide MultiplyDivide = _ left:Factor rest:(('*' / '/') Factor)* _ { - return rest.reduce((acc, curr) => ({ + const topLevel = rest.reduce((acc, curr) => ({ type: 'function', name: curr[0] === '*' ? 'multiply' : 'divide', args: [acc, curr[1]], - location: simpleLocation(location()), - text: text() - }), left) + }), left); + if (typeof topLevel === 'object') { + topLevel.location = simpleLocation(location()); + topLevel.text = text(); + } + return topLevel; } + / Factor Factor = Group diff --git a/packages/kbn-tinymath/index.d.ts b/packages/kbn-tinymath/index.d.ts index c3c32a59fa15ad..8e15d86c88fc84 100644 --- a/packages/kbn-tinymath/index.d.ts +++ b/packages/kbn-tinymath/index.d.ts @@ -24,9 +24,11 @@ export interface TinymathLocation { export interface TinymathFunction { type: 'function'; name: string; - text: string; args: TinymathAST[]; - location: TinymathLocation; + // Location is not guaranteed because PEG grammars are not left-recursive + location?: TinymathLocation; + // Text is not guaranteed because PEG grammars are not left-recursive + text?: string; } export interface TinymathVariable { diff --git a/packages/kbn-tinymath/test/library.test.js b/packages/kbn-tinymath/test/library.test.js index bf1c7a9dbc5fb5..bbc8503684fd40 100644 --- a/packages/kbn-tinymath/test/library.test.js +++ b/packages/kbn-tinymath/test/library.test.js @@ -41,6 +41,35 @@ describe('Parser', () => { }); }); + describe('Math', () => { + it('converts basic symbols into left-to-right pairs', () => { + expect(parse('a + b + c - d')).toEqual({ + args: [ + { + name: 'add', + type: 'function', + args: [ + { + name: 'add', + type: 'function', + args: [ + expect.objectContaining({ location: { min: 0, max: 2 } }), + expect.objectContaining({ location: { min: 3, max: 6 } }), + ], + }, + expect.objectContaining({ location: { min: 7, max: 10 } }), + ], + }, + expect.objectContaining({ location: { min: 11, max: 13 } }), + ], + name: 'subtract', + type: 'function', + text: 'a + b + c - d', + location: { min: 0, max: 13 }, + }); + }); + }); + describe('Variables', () => { it('strings', () => { expect(parse('f')).toEqual(variableEqual('f')); @@ -263,6 +292,8 @@ describe('Evaluate', () => { expect(evaluate('5/20')).toEqual(0.25); expect(evaluate('1 + 1 + 2 + 3 + 12')).toEqual(19); expect(evaluate('100 / 10 / 10')).toEqual(1); + expect(evaluate('0 * 1 - 100 / 10 / 10')).toEqual(-1); + expect(evaluate('100 / (10 / 10)')).toEqual(100); }); it('equations with functions', () => { diff --git a/src/plugins/dashboard/server/usage/dashboard_telemetry.test.ts b/src/plugins/dashboard/server/usage/dashboard_telemetry.test.ts index 1cfa9d862e6b95..60f1f7eb0955c0 100644 --- a/src/plugins/dashboard/server/usage/dashboard_telemetry.test.ts +++ b/src/plugins/dashboard/server/usage/dashboard_telemetry.test.ts @@ -72,6 +72,22 @@ const lensXYSeriesB = ({ visualization: { preferredSeriesType: 'seriesB', }, + datasourceStates: { + indexpattern: { + layers: { + first: { + columns: { + first: { + operationType: 'terms', + }, + second: { + operationType: 'formula', + }, + }, + }, + }, + }, + }, }, }, }, @@ -144,6 +160,7 @@ describe('dashboard telemetry', () => { expect(collectorData.lensByValue.a).toBe(3); expect(collectorData.lensByValue.seriesA).toBe(2); expect(collectorData.lensByValue.seriesB).toBe(1); + expect(collectorData.lensByValue.formula).toBe(1); }); it('handles misshapen lens panels', () => { diff --git a/src/plugins/dashboard/server/usage/dashboard_telemetry.ts b/src/plugins/dashboard/server/usage/dashboard_telemetry.ts index 912dc04d16d092..fb1ddff469f578 100644 --- a/src/plugins/dashboard/server/usage/dashboard_telemetry.ts +++ b/src/plugins/dashboard/server/usage/dashboard_telemetry.ts @@ -27,6 +27,16 @@ interface LensPanel extends SavedDashboardPanel730ToLatest { visualization?: { preferredSeriesType?: string; }; + datasourceStates?: { + indexpattern?: { + layers: Record< + string, + { + columns: Record; + } + >; + }; + }; }; }; }; @@ -109,6 +119,19 @@ export const collectByValueLensInfo: DashboardCollectorFunction = (panels, colle } collectorData.lensByValue[type] = collectorData.lensByValue[type] + 1; + + const hasFormula = Object.values( + lensPanel.embeddableConfig.attributes.state?.datasourceStates?.indexpattern?.layers || {} + ).some((layer) => + Object.values(layer.columns).some((column) => column.operationType === 'formula') + ); + + if (hasFormula && !collectorData.lensByValue.formula) { + collectorData.lensByValue.formula = 0; + } + if (hasFormula) { + collectorData.lensByValue.formula++; + } } } }; diff --git a/src/plugins/expressions/common/expression_functions/specs/map_column.ts b/src/plugins/expressions/common/expression_functions/specs/map_column.ts index 7939441ff0d602..d6af19d9dbf531 100644 --- a/src/plugins/expressions/common/expression_functions/specs/map_column.ts +++ b/src/plugins/expressions/common/expression_functions/specs/map_column.ts @@ -10,7 +10,7 @@ import { Observable } from 'rxjs'; import { take } from 'rxjs/operators'; import { i18n } from '@kbn/i18n'; import { ExpressionFunctionDefinition } from '../types'; -import { Datatable, getType } from '../../expression_types'; +import { Datatable, DatatableColumn, getType } from '../../expression_types'; export interface MapColumnArguments { id?: string | null; @@ -110,10 +110,10 @@ export const mapColumn: ExpressionFunctionDefinition< return Promise.all(rowPromises).then((rows) => { const type = rows.length ? getType(rows[0][columnId]) : 'null'; - const newColumn = { + const newColumn: DatatableColumn = { id: columnId, name: args.name, - meta: { type }, + meta: { type, params: { id: type } }, }; if (args.copyMetaFrom) { const metaSourceFrom = columns.find(({ id }) => id === args.copyMetaFrom); diff --git a/src/plugins/expressions/common/expression_functions/specs/tests/map_column.test.ts b/src/plugins/expressions/common/expression_functions/specs/tests/map_column.test.ts index f5c1f3838f66c7..bb4e6303e90b7d 100644 --- a/src/plugins/expressions/common/expression_functions/specs/tests/map_column.test.ts +++ b/src/plugins/expressions/common/expression_functions/specs/tests/map_column.test.ts @@ -29,7 +29,11 @@ describe('mapColumn', () => { expect(result.type).toBe('datatable'); expect(result.columns).toEqual([ ...testTable.columns, - { id: 'pricePlusTwo', name: 'pricePlusTwo', meta: { type: 'number' } }, + { + id: 'pricePlusTwo', + name: 'pricePlusTwo', + meta: { type: 'number', params: { id: 'number' } }, + }, ]); expect(result.columns[result.columns.length - 1]).toHaveProperty('name', 'pricePlusTwo'); expect(result.rows[arbitraryRowIndex]).toHaveProperty('pricePlusTwo'); diff --git a/src/plugins/kibana_react/public/code_editor/__snapshots__/code_editor.test.tsx.snap b/src/plugins/kibana_react/public/code_editor/__snapshots__/code_editor.test.tsx.snap index a779ef540d72ec..d4fb5a708e4405 100644 --- a/src/plugins/kibana_react/public/code_editor/__snapshots__/code_editor.test.tsx.snap +++ b/src/plugins/kibana_react/public/code_editor/__snapshots__/code_editor.test.tsx.snap @@ -11,6 +11,7 @@ exports[`is rendered 1`] = ` onChange={[Function]} options={ Object { + "matchBrackets": "never", "minimap": Object { "enabled": false, }, @@ -39,6 +40,7 @@ exports[`is rendered 1`] = ` nodeType="div" onResize={[Function]} querySelector={null} + refreshMode="debounce" refreshRate={1000} skipOnMount={false} targetDomEl={null} diff --git a/src/plugins/kibana_react/public/code_editor/code_editor.tsx b/src/plugins/kibana_react/public/code_editor/code_editor.tsx index 55e10e7861e518..251f05950da227 100644 --- a/src/plugins/kibana_react/public/code_editor/code_editor.tsx +++ b/src/plugins/kibana_react/public/code_editor/code_editor.tsx @@ -187,10 +187,16 @@ export class CodeEditor extends React.Component { wordBasedSuggestions: false, wordWrap: 'on', wrappingIndent: 'indent', + matchBrackets: 'never', ...options, }} /> - + ); } diff --git a/x-pack/plugins/canvas/shareable_runtime/webpack.config.js b/x-pack/plugins/canvas/shareable_runtime/webpack.config.js index e85840e8734308..d0bdc292619d89 100644 --- a/x-pack/plugins/canvas/shareable_runtime/webpack.config.js +++ b/x-pack/plugins/canvas/shareable_runtime/webpack.config.js @@ -38,7 +38,7 @@ module.exports = { 'src/plugins/data/public/expressions/interpreter' ), 'kbn/interpreter': path.resolve(KIBANA_ROOT, 'packages/kbn-interpreter/target/common'), - tinymath: path.resolve(KIBANA_ROOT, 'node_modules/tinymath/lib/tinymath.es5.js'), + tinymath: path.resolve(KIBANA_ROOT, 'node_modules/tinymath/lib/tinymath.min.js'), core_app_image_assets: path.resolve(KIBANA_ROOT, 'src/core/public/core_app/images'), }, extensions: ['.js', '.json', '.ts', '.tsx', '.scss'], diff --git a/x-pack/plugins/lens/public/app_plugin/lens_top_nav.tsx b/x-pack/plugins/lens/public/app_plugin/lens_top_nav.tsx index 245e964bbd2e67..ecaae04232f8af 100644 --- a/x-pack/plugins/lens/public/app_plugin/lens_top_nav.tsx +++ b/x-pack/plugins/lens/public/app_plugin/lens_top_nav.tsx @@ -7,7 +7,7 @@ import { isEqual } from 'lodash'; import { i18n } from '@kbn/i18n'; -import React from 'react'; +import React, { useCallback, useMemo } from 'react'; import { TopNavMenuData } from '../../../../../src/plugins/navigation/public'; import { LensAppServices, LensTopNavActions, LensTopNavMenuProps } from './types'; import { downloadMultipleAs } from '../../../../../src/plugins/share/public'; @@ -164,79 +164,152 @@ export const LensTopNavMenu = ({ const unsavedTitle = i18n.translate('xpack.lens.app.unsavedFilename', { defaultMessage: 'unsaved', }); - const topNavConfig = getLensTopNavConfig({ - showSaveAndReturn: Boolean( - isLinkedToOriginatingApp && - // Temporarily required until the 'by value' paradigm is default. - (dashboardFeatureFlag.allowByValueEmbeddables || Boolean(initialInput)) - ), - enableExportToCSV: Boolean(isSaveable && activeData && Object.keys(activeData).length), - isByValueMode: getIsByValueMode(), - allowByValue: dashboardFeatureFlag.allowByValueEmbeddables, - showCancel: Boolean(isLinkedToOriginatingApp), - savingToLibraryPermitted, - savingToDashboardPermitted, - actions: { - exportToCSV: () => { - if (!activeData) { - return; - } - const datatables = Object.values(activeData); - const content = datatables.reduce>( - (memo, datatable, i) => { - // skip empty datatables - if (datatable) { - const postFix = datatables.length > 1 ? `-${i + 1}` : ''; + const topNavConfig = useMemo( + () => + getLensTopNavConfig({ + showSaveAndReturn: Boolean( + isLinkedToOriginatingApp && + // Temporarily required until the 'by value' paradigm is default. + (dashboardFeatureFlag.allowByValueEmbeddables || Boolean(initialInput)) + ), + enableExportToCSV: Boolean(isSaveable && activeData && Object.keys(activeData).length), + isByValueMode: getIsByValueMode(), + allowByValue: dashboardFeatureFlag.allowByValueEmbeddables, + showCancel: Boolean(isLinkedToOriginatingApp), + savingToLibraryPermitted, + savingToDashboardPermitted, + actions: { + exportToCSV: () => { + if (!activeData) { + return; + } + const datatables = Object.values(activeData); + const content = datatables.reduce>( + (memo, datatable, i) => { + // skip empty datatables + if (datatable) { + const postFix = datatables.length > 1 ? `-${i + 1}` : ''; - memo[`${lastKnownDoc?.title || unsavedTitle}${postFix}.csv`] = { - content: exporters.datatableToCSV(datatable, { - csvSeparator: uiSettings.get('csv:separator', ','), - quoteValues: uiSettings.get('csv:quoteValues', true), - formatFactory: data.fieldFormats.deserialize, - }), - type: exporters.CSV_MIME_TYPE, - }; + memo[`${lastKnownDoc?.title || unsavedTitle}${postFix}.csv`] = { + content: exporters.datatableToCSV(datatable, { + csvSeparator: uiSettings.get('csv:separator', ','), + quoteValues: uiSettings.get('csv:quoteValues', true), + formatFactory: data.fieldFormats.deserialize, + }), + type: exporters.CSV_MIME_TYPE, + }; + } + return memo; + }, + {} + ); + if (content) { + downloadMultipleAs(content); } - return memo; }, - {} - ); - if (content) { - downloadMultipleAs(content); - } - }, - saveAndReturn: () => { - if (savingToDashboardPermitted && lastKnownDoc) { - // disabling the validation on app leave because the document has been saved. - onAppLeave((actions) => { - return actions.default(); - }); - runSave( - { - newTitle: lastKnownDoc.title, - newCopyOnSave: false, - isTitleDuplicateConfirmed: false, - returnToOrigin: true, - }, - { - saveToLibrary: - (initialInput && attributeService.inputIsRefType(initialInput)) ?? false, + saveAndReturn: () => { + if (savingToDashboardPermitted && lastKnownDoc) { + // disabling the validation on app leave because the document has been saved. + onAppLeave((actions) => { + return actions.default(); + }); + runSave( + { + newTitle: lastKnownDoc.title, + newCopyOnSave: false, + isTitleDuplicateConfirmed: false, + returnToOrigin: true, + }, + { + saveToLibrary: + (initialInput && attributeService.inputIsRefType(initialInput)) ?? false, + } + ); } - ); - } - }, - showSaveModal: () => { - if (savingToDashboardPermitted || savingToLibraryPermitted) { - setIsSaveModalVisible(true); - } - }, - cancel: () => { - if (redirectToOrigin) { - redirectToOrigin(); + }, + showSaveModal: () => { + if (savingToDashboardPermitted || savingToLibraryPermitted) { + setIsSaveModalVisible(true); + } + }, + cancel: () => { + if (redirectToOrigin) { + redirectToOrigin(); + } + }, + }, + }), + [ + activeData, + attributeService, + dashboardFeatureFlag.allowByValueEmbeddables, + data.fieldFormats.deserialize, + getIsByValueMode, + initialInput, + isLinkedToOriginatingApp, + isSaveable, + lastKnownDoc, + onAppLeave, + redirectToOrigin, + runSave, + savingToDashboardPermitted, + savingToLibraryPermitted, + setIsSaveModalVisible, + uiSettings, + unsavedTitle, + ] + ); + + const onQuerySubmitWrapped = useCallback( + (payload) => { + const { dateRange, query: newQuery } = payload; + const currentRange = data.query.timefilter.timefilter.getTime(); + if (dateRange.from !== currentRange.from || dateRange.to !== currentRange.to) { + data.query.timefilter.timefilter.setTime(dateRange); + trackUiEvent('app_date_change'); + } else { + // Query has changed, renew the session id. + // Time change will be picked up by the time subscription + dispatchSetState({ searchSessionId: data.search.session.start() }); + trackUiEvent('app_query_change'); + } + if (newQuery) { + if (!isEqual(newQuery, query)) { + dispatchSetState({ query: newQuery }); } - }, + } }, - }); + [data.query.timefilter.timefilter, data.search.session, dispatchSetState, query] + ); + + const onSavedWrapped = useCallback( + (newSavedQuery) => { + dispatchSetState({ savedQuery: newSavedQuery }); + }, + [dispatchSetState] + ); + + const onSavedQueryUpdatedWrapped = useCallback( + (newSavedQuery) => { + const savedQueryFilters = newSavedQuery.attributes.filters || []; + const globalFilters = data.query.filterManager.getGlobalFilters(); + data.query.filterManager.setFilters([...globalFilters, ...savedQueryFilters]); + dispatchSetState({ + query: newSavedQuery.attributes.query, + savedQuery: { ...newSavedQuery }, + }); // Shallow query for reference issues + }, + [data.query.filterManager, dispatchSetState] + ); + + const onClearSavedQueryWrapped = useCallback(() => { + data.query.filterManager.setFilters(data.query.filterManager.getGlobalFilters()); + dispatchSetState({ + filters: data.query.filterManager.getGlobalFilters(), + query: data.query.queryString.getDefaultQuery(), + savedQuery: undefined, + }); + }, [data.query.filterManager, data.query.queryString, dispatchSetState]); return ( { - const { dateRange, query: newQuery } = payload; - const currentRange = data.query.timefilter.timefilter.getTime(); - if (dateRange.from !== currentRange.from || dateRange.to !== currentRange.to) { - data.query.timefilter.timefilter.setTime(dateRange); - trackUiEvent('app_date_change'); - } else { - // Query has changed, renew the session id. - // Time change will be picked up by the time subscription - dispatchSetState({ searchSessionId: data.search.session.start() }); - trackUiEvent('app_query_change'); - } - if (newQuery) { - if (!isEqual(newQuery, query)) { - dispatchSetState({ query: newQuery }); - } - } - }} - onSaved={(newSavedQuery) => { - dispatchSetState({ savedQuery: newSavedQuery }); - }} - onSavedQueryUpdated={(newSavedQuery) => { - const savedQueryFilters = newSavedQuery.attributes.filters || []; - const globalFilters = data.query.filterManager.getGlobalFilters(); - data.query.filterManager.setFilters([...globalFilters, ...savedQueryFilters]); - dispatchSetState({ - query: newSavedQuery.attributes.query, - savedQuery: { ...newSavedQuery }, - }); // Shallow query for reference issues - }} - onClearSavedQuery={() => { - data.query.filterManager.setFilters(data.query.filterManager.getGlobalFilters()); - dispatchSetState({ - filters: data.query.filterManager.getGlobalFilters(), - query: data.query.queryString.getDefaultQuery(), - savedQuery: undefined, - }); - }} + onQuerySubmit={onQuerySubmitWrapped} + onSaved={onSavedWrapped} + onSavedQueryUpdated={onSavedQueryUpdatedWrapped} + onClearSavedQuery={onClearSavedQueryWrapped} indexPatterns={indexPatternsForTopNav} query={query} dateRangeFrom={from} diff --git a/x-pack/plugins/lens/public/datatable_visualization/components/columns.tsx b/x-pack/plugins/lens/public/datatable_visualization/components/columns.tsx index ba24da8309ed7c..5c53d40f999b77 100644 --- a/x-pack/plugins/lens/public/datatable_visualization/components/columns.tsx +++ b/x-pack/plugins/lens/public/datatable_visualization/components/columns.tsx @@ -51,7 +51,7 @@ export const createGridColumns = ( columnId, }: Pick) => { const rowValue = table.rows[rowIndex][columnId]; - const column = columnsReverseLookup[columnId]; + const column = columnsReverseLookup?.[columnId]; const contentsIsDefined = rowValue != null; const cellContent = formatFactory(column?.meta?.params).convert(rowValue); diff --git a/x-pack/plugins/lens/public/editor_frame_service/editor_frame/config_panel/config_panel.test.tsx b/x-pack/plugins/lens/public/editor_frame_service/editor_frame/config_panel/config_panel.test.tsx index 3936fb9e1a1b12..1ec48f516bd32d 100644 --- a/x-pack/plugins/lens/public/editor_frame_service/editor_frame/config_panel/config_panel.test.tsx +++ b/x-pack/plugins/lens/public/editor_frame_service/editor_frame/config_panel/config_panel.test.tsx @@ -76,6 +76,8 @@ describe('ConfigPanel', () => { framePublicAPI: frame, dispatch: jest.fn(), core: coreMock.createStart(), + isFullscreen: false, + toggleFullscreen: jest.fn(), }; } @@ -119,19 +121,23 @@ describe('ConfigPanel', () => { expect(component.find(LayerPanel).exists()).toBe(false); }); - it('allow datasources and visualizations to use setters', () => { + it('allow datasources and visualizations to use setters', async () => { const props = getDefaultProps(); const component = mountWithIntl(); const { updateDatasource, updateAll } = component.find(LayerPanel).props(); const updater = () => 'updated'; updateDatasource('ds1', updater); + // wait for one tick so async updater has a chance to trigger + await new Promise((r) => setTimeout(r, 0)); expect(props.dispatch).toHaveBeenCalledTimes(1); expect(props.dispatch.mock.calls[0][0].updater(props.datasourceStates.ds1.state)).toEqual( 'updated' ); updateAll('ds1', updater, props.visualizationState); + // wait for one tick so async updater has a chance to trigger + await new Promise((r) => setTimeout(r, 0)); expect(props.dispatch).toHaveBeenCalledTimes(2); expect(props.dispatch.mock.calls[0][0].updater(props.datasourceStates.ds1.state)).toEqual( 'updated' diff --git a/x-pack/plugins/lens/public/editor_frame_service/editor_frame/config_panel/config_panel.tsx b/x-pack/plugins/lens/public/editor_frame_service/editor_frame/config_panel/config_panel.tsx index c1ab2b4586ab33..81c044af532fbf 100644 --- a/x-pack/plugins/lens/public/editor_frame_service/editor_frame/config_panel/config_panel.tsx +++ b/x-pack/plugins/lens/public/editor_frame_service/editor_frame/config_panel/config_panel.tsx @@ -71,32 +71,54 @@ export function LayerPanels( }, [dispatch] ); + const updateDatasourceAsync = useMemo( + () => (datasourceId: string, newState: unknown) => { + // React will synchronously update if this is triggered from a third party component, + // which we don't want. The timeout lets user interaction have priority, then React updates. + setTimeout(() => { + updateDatasource(datasourceId, newState); + }, 0); + }, + [updateDatasource] + ); const updateAll = useMemo( () => (datasourceId: string, newDatasourceState: unknown, newVisualizationState: unknown) => { - dispatch({ - type: 'UPDATE_STATE', - subType: 'UPDATE_ALL_STATES', - updater: (prevState) => { - const updatedDatasourceState = - typeof newDatasourceState === 'function' - ? newDatasourceState(prevState.datasourceStates[datasourceId].state) - : newDatasourceState; - return { - ...prevState, - datasourceStates: { - ...prevState.datasourceStates, - [datasourceId]: { - state: updatedDatasourceState, - isLoading: false, + // React will synchronously update if this is triggered from a third party component, + // which we don't want. The timeout lets user interaction have priority, then React updates. + setTimeout(() => { + dispatch({ + type: 'UPDATE_STATE', + subType: 'UPDATE_ALL_STATES', + updater: (prevState) => { + const updatedDatasourceState = + typeof newDatasourceState === 'function' + ? newDatasourceState(prevState.datasourceStates[datasourceId].state) + : newDatasourceState; + return { + ...prevState, + datasourceStates: { + ...prevState.datasourceStates, + [datasourceId]: { + state: updatedDatasourceState, + isLoading: false, + }, + }, + visualization: { + ...prevState.visualization, + state: newVisualizationState, }, - }, - visualization: { - ...prevState.visualization, - state: newVisualizationState, - }, - stagedPreview: undefined, - }; - }, + stagedPreview: undefined, + }; + }, + }); + }, 0); + }, + [dispatch] + ); + const toggleFullscreen = useMemo( + () => () => { + dispatch({ + type: 'TOGGLE_FULLSCREEN', }); }, [dispatch] @@ -118,6 +140,7 @@ export function LayerPanels( visualizationState={visualizationState} updateVisualization={setVisualizationState} updateDatasource={updateDatasource} + updateDatasourceAsync={updateDatasourceAsync} updateAll={updateAll} isOnlyLayer={layerIds.length === 1} onRemoveLayer={() => { @@ -135,6 +158,7 @@ export function LayerPanels( }); removeLayerRef(layerId); }} + toggleFullscreen={toggleFullscreen} /> ) : null )} diff --git a/x-pack/plugins/lens/public/editor_frame_service/editor_frame/config_panel/dimension_container.scss b/x-pack/plugins/lens/public/editor_frame_service/editor_frame/config_panel/dimension_container.scss index 91cd706ea77d11..135286fc2172bb 100644 --- a/x-pack/plugins/lens/public/editor_frame_service/editor_frame/config_panel/dimension_container.scss +++ b/x-pack/plugins/lens/public/editor_frame_service/editor_frame/config_panel/dimension_container.scss @@ -8,21 +8,36 @@ position: absolute; left: 0; animation: euiFlyout $euiAnimSpeedNormal $euiAnimSlightResistance; + @include euiBreakpoint('l', 'xl') { top: 0 !important; height: 100% !important; } + @include euiBreakpoint('xs', 's', 'm') { @include euiFlyout; } + + .lnsFrameLayout__sidebar-isFullscreen & { + border-left: $euiBorderThin; // Force border regardless of theme in fullscreen + box-shadow: none; + } } .lnsDimensionContainer__footer { padding: $euiSizeS; + + .lnsFrameLayout__sidebar-isFullscreen & { + display: none; + } } .lnsDimensionContainer__header { padding: $euiSizeS $euiSizeXS; + + .lnsFrameLayout__sidebar-isFullscreen & { + display: none; + } } .lnsDimensionContainer__headerTitle { diff --git a/x-pack/plugins/lens/public/editor_frame_service/editor_frame/config_panel/dimension_container.tsx b/x-pack/plugins/lens/public/editor_frame_service/editor_frame/config_panel/dimension_container.tsx index b14d391c2c9696..2f3eb5043d6106 100644 --- a/x-pack/plugins/lens/public/editor_frame_service/editor_frame/config_panel/dimension_container.tsx +++ b/x-pack/plugins/lens/public/editor_frame_service/editor_frame/config_panel/dimension_container.tsx @@ -29,26 +29,33 @@ export function DimensionContainer({ groupLabel, handleClose, panel, + isFullscreen, panelRef, }: { isOpen: boolean; - handleClose: () => void; - panel: React.ReactElement; + handleClose: () => boolean; + panel: React.ReactElement | null; groupLabel: string; + isFullscreen: boolean; panelRef: (el: HTMLDivElement) => void; }) { const [focusTrapIsEnabled, setFocusTrapIsEnabled] = useState(false); const closeFlyout = useCallback(() => { - handleClose(); - setFocusTrapIsEnabled(false); + const canClose = handleClose(); + if (canClose) { + setFocusTrapIsEnabled(false); + } + return canClose; }, [handleClose]); const closeOnEscape = useCallback( (event: KeyboardEvent) => { if (event.key === keys.ESCAPE) { - event.preventDefault(); - closeFlyout(); + const canClose = closeFlyout(); + if (canClose) { + event.preventDefault(); + } } }, [closeFlyout] @@ -69,7 +76,15 @@ export function DimensionContainer({
- + { + if (isFullscreen) { + return; + } + closeFlyout(); + }} + isDisabled={!isOpen} + >
{ visualizationState: 'state', updateVisualization: jest.fn(), updateDatasource: jest.fn(), + updateDatasourceAsync: jest.fn(), updateAll: jest.fn(), framePublicAPI: frame, isOnlyLayer: true, @@ -86,6 +87,8 @@ describe('LayerPanel', () => { core: coreMock.createStart(), layerIndex: 0, registerNewLayerRef: jest.fn(), + isFullscreen: false, + toggleFullscreen: jest.fn(), }; } @@ -255,7 +258,7 @@ describe('LayerPanel', () => { it('should not update the visualization if the datasource is incomplete', () => { (generateId as jest.Mock).mockReturnValue(`newid`); const updateAll = jest.fn(); - const updateDatasource = jest.fn(); + const updateDatasourceAsync = jest.fn(); mockVisualization.getConfiguration.mockReturnValue({ groups: [ @@ -273,7 +276,7 @@ describe('LayerPanel', () => { const component = mountWithIntl( ); @@ -292,15 +295,88 @@ describe('LayerPanel', () => { mockDatasource.renderDimensionEditor.mock.calls.length - 1 ][1].setState; + act(() => { + stateFn( + { + indexPatternId: '1', + columns: {}, + columnOrder: [], + incompleteColumns: { newId: { operationType: 'count' } }, + }, + { isDimensionComplete: false } + ); + }); + expect(updateAll).not.toHaveBeenCalled(); + expect(updateDatasourceAsync).toHaveBeenCalled(); + act(() => { stateFn({ indexPatternId: '1', columns: {}, columnOrder: [], - incompleteColumns: { newId: { operationType: 'count' } }, }); }); - expect(updateAll).not.toHaveBeenCalled(); + expect(updateAll).toHaveBeenCalled(); + }); + + it('should remove the dimension when the datasource marks it as removed', () => { + const updateAll = jest.fn(); + const updateDatasource = jest.fn(); + + mockVisualization.getConfiguration.mockReturnValue({ + groups: [ + { + groupLabel: 'A', + groupId: 'a', + accessors: [{ columnId: 'y' }], + filterOperations: () => true, + supportsMoreColumns: true, + dataTestSubj: 'lnsGroup', + }, + ], + }); + + const component = mountWithIntl( + + ); + + act(() => { + component.find('[data-test-subj="lnsLayerPanel-dimensionLink"]').first().simulate('click'); + }); + component.update(); + + expect(mockDatasource.renderDimensionEditor).toHaveBeenCalledWith( + expect.any(Element), + expect.objectContaining({ columnId: 'y' }) + ); + const stateFn = + mockDatasource.renderDimensionEditor.mock.calls[ + mockDatasource.renderDimensionEditor.mock.calls.length - 1 + ][1].setState; act(() => { stateFn( @@ -308,11 +384,19 @@ describe('LayerPanel', () => { indexPatternId: '1', columns: {}, columnOrder: [], + incompleteColumns: { y: { operationType: 'average' } }, }, - { shouldReplaceDimension: true } + { + isDimensionComplete: false, + } ); }); expect(updateAll).toHaveBeenCalled(); + expect(mockVisualization.removeDimension).toHaveBeenCalledWith( + expect.objectContaining({ + columnId: 'y', + }) + ); }); it('should keep the DimensionContainer open when configuring a new dimension', () => { @@ -331,6 +415,7 @@ describe('LayerPanel', () => { accessors: [], filterOperations: () => true, supportsMoreColumns: true, + enableDimensionEditor: true, dataTestSubj: 'lnsGroup', }, ], @@ -345,6 +430,7 @@ describe('LayerPanel', () => { accessors: [{ columnId: 'newid' }], filterOperations: () => true, supportsMoreColumns: false, + enableDimensionEditor: true, dataTestSubj: 'lnsGroup', }, ], @@ -357,6 +443,20 @@ describe('LayerPanel', () => { component.update(); expect(component.find('EuiFlyoutHeader').exists()).toBe(true); + + const lastArgs = + mockDatasource.renderDimensionEditor.mock.calls[ + mockDatasource.renderDimensionEditor.mock.calls.length - 1 + ][1]; + + // Simulate what is called by the dimension editor + act(() => { + lastArgs.setState(lastArgs.state, { + isDimensionComplete: true, + }); + }); + + expect(mockVisualization.renderDimensionEditor).toHaveBeenCalled(); }); it('should close the DimensionContainer when the active visualization changes', () => { diff --git a/x-pack/plugins/lens/public/editor_frame_service/editor_frame/config_panel/layer_panel.tsx b/x-pack/plugins/lens/public/editor_frame_service/editor_frame/config_panel/layer_panel.tsx index a605a94a346468..3a299de0fca6a3 100644 --- a/x-pack/plugins/lens/public/editor_frame_service/editor_frame/config_panel/layer_panel.tsx +++ b/x-pack/plugins/lens/public/editor_frame_service/editor_frame/config_panel/layer_panel.tsx @@ -42,6 +42,7 @@ export function LayerPanel( isOnlyLayer: boolean; updateVisualization: StateSetter; updateDatasource: (datasourceId: string, newState: unknown) => void; + updateDatasourceAsync: (datasourceId: string, newState: unknown) => void; updateAll: ( datasourceId: string, newDatasourcestate: unknown, @@ -49,6 +50,8 @@ export function LayerPanel( ) => void; onRemoveLayer: () => void; registerNewLayerRef: (layerId: string, instance: HTMLDivElement | null) => void; + toggleFullscreen: () => void; + isFullscreen: boolean; } ) { const [activeDimension, setActiveDimension] = useState( @@ -65,6 +68,8 @@ export function LayerPanel( activeVisualization, updateVisualization, updateDatasource, + toggleFullscreen, + isFullscreen, } = props; const datasourcePublicAPI = framePublicAPI.datasourceLayers[layerId]; @@ -197,9 +202,16 @@ export function LayerPanel( setNextFocusedButtonId, ]); + const isDimensionPanelOpen = Boolean(activeId); + return ( <> -
+
@@ -407,9 +419,16 @@ export function LayerPanel( (panelRef.current = el)} - isOpen={!!activeId} + isOpen={isDimensionPanelOpen} + isFullscreen={isFullscreen} groupLabel={activeGroup?.groupLabel || ''} handleClose={() => { + if ( + layerDatasource.canCloseDimensionEditor && + !layerDatasource.canCloseDimensionEditor(layerDatasourceState) + ) { + return false; + } if (layerDatasource.updateStateOnCloseDimension) { const newState = layerDatasource.updateStateOnCloseDimension({ state: layerDatasourceState, @@ -421,9 +440,13 @@ export function LayerPanel( } } setActiveDimension(initialActiveDimensionState); + if (isFullscreen) { + toggleFullscreen(); + } + return true; }} panel={ - <> +
{activeGroup && activeId && ( { - if (shouldReplaceDimension || shouldRemoveDimension) { + if (allAccessors.includes(activeId)) { + if (isDimensionComplete) { + props.updateDatasourceAsync(datasourceId, newState); + } else { + // The datasource can indicate that the previously-valid column is no longer + // complete, which clears the visualization. This keeps the flyout open and reuses + // the previous columnId + props.updateAll( + datasourceId, + newState, + activeVisualization.removeDimension({ + layerId, + columnId: activeId, + prevState: props.visualizationState, + }) + ); + } + } else if (isDimensionComplete) { props.updateAll( datasourceId, newState, - shouldRemoveDimension - ? activeVisualization.removeDimension({ - layerId, - columnId: activeId, - prevState: props.visualizationState, - }) - : activeVisualization.setDimension({ - layerId, - groupId: activeGroup.groupId, - columnId: activeId, - prevState: props.visualizationState, - }) + activeVisualization.setDimension({ + layerId, + groupId: activeGroup.groupId, + columnId: activeId, + prevState: props.visualizationState, + }) ); + setActiveDimension({ ...activeDimension, isNew: false }); } else { - props.updateDatasource(datasourceId, newState); + props.updateDatasourceAsync(datasourceId, newState); } - setActiveDimension({ - ...activeDimension, - isNew: false, - }); }, }} /> )} {activeGroup && activeId && + !isFullscreen && !activeDimension.isNew && activeVisualization.renderDimensionEditor && activeGroup?.enableDimensionEditor && ( @@ -491,7 +519,7 @@ export function LayerPanel( />
)} - +
} /> diff --git a/x-pack/plugins/lens/public/editor_frame_service/editor_frame/config_panel/types.ts b/x-pack/plugins/lens/public/editor_frame_service/editor_frame/config_panel/types.ts index 37b2198cfd51f9..1af8c16fa1395f 100644 --- a/x-pack/plugins/lens/public/editor_frame_service/editor_frame/config_panel/types.ts +++ b/x-pack/plugins/lens/public/editor_frame_service/editor_frame/config_panel/types.ts @@ -29,6 +29,7 @@ export interface ConfigPanelWrapperProps { } >; core: DatasourceDimensionEditorProps['core']; + isFullscreen: boolean; } export interface LayerPanelProps { @@ -46,6 +47,7 @@ export interface LayerPanelProps { } >; core: DatasourceDimensionEditorProps['core']; + isFullscreen: boolean; } export interface LayerDatasourceDropProps { diff --git a/x-pack/plugins/lens/public/editor_frame_service/editor_frame/editor_frame.tsx b/x-pack/plugins/lens/public/editor_frame_service/editor_frame/editor_frame.tsx index 4710e03d336bcf..161b0125a172a3 100644 --- a/x-pack/plugins/lens/public/editor_frame_service/editor_frame/editor_frame.tsx +++ b/x-pack/plugins/lens/public/editor_frame_service/editor_frame/editor_frame.tsx @@ -5,7 +5,7 @@ * 2.0. */ -import React, { useEffect, useReducer, useState, useCallback } from 'react'; +import React, { useEffect, useReducer, useState, useCallback, useRef } from 'react'; import { CoreStart } from 'kibana/public'; import { isEqual } from 'lodash'; import { PaletteRegistry } from 'src/plugins/charts/public'; @@ -30,6 +30,7 @@ import { applyVisualizeFieldSuggestions, getTopSuggestionForField, switchToSuggestion, + Suggestion, } from './suggestion_helpers'; import { trackUiEvent } from '../../lens_ui_telemetry'; import { @@ -327,45 +328,37 @@ export function EditorFrame(props: EditorFrameProps) { ] ); - const getSuggestionForField = React.useCallback( - (field: DragDropIdentifier) => { - const { activeDatasourceId, datasourceStates } = state; - const activeVisualizationId = state.visualization.activeId; - const visualizationState = state.visualization.state; - const { visualizationMap, datasourceMap } = props; + // Using a ref to prevent rerenders in the child components while keeping the latest state + const getSuggestionForField = useRef<(field: DragDropIdentifier) => Suggestion | undefined>(); + getSuggestionForField.current = (field: DragDropIdentifier) => { + const { activeDatasourceId, datasourceStates } = state; + const activeVisualizationId = state.visualization.activeId; + const visualizationState = state.visualization.state; + const { visualizationMap, datasourceMap } = props; - if (!field || !activeDatasourceId) { - return; - } + if (!field || !activeDatasourceId) { + return; + } - return getTopSuggestionForField( - datasourceLayers, - activeVisualizationId, - visualizationMap, - visualizationState, - datasourceMap[activeDatasourceId], - datasourceStates, - field - ); - }, - // eslint-disable-next-line react-hooks/exhaustive-deps - [ - state.visualization.state, - props.datasourceMap, - props.visualizationMap, - state.activeDatasourceId, - state.datasourceStates, - ] - ); + return getTopSuggestionForField( + datasourceLayers, + activeVisualizationId, + visualizationMap, + visualizationState, + datasourceMap[activeDatasourceId], + datasourceStates, + field + ); + }; const hasSuggestionForField = useCallback( - (field: DragDropIdentifier) => getSuggestionForField(field) !== undefined, + (field: DragDropIdentifier) => getSuggestionForField.current!(field) !== undefined, [getSuggestionForField] ); const dropOntoWorkspace = useCallback( (field) => { - const suggestion = getSuggestionForField(field); + const suggestion = getSuggestionForField.current!(field); if (suggestion) { trackUiEvent('drop_onto_workspace'); switchToSuggestion(dispatch, suggestion, 'SWITCH_VISUALIZATION'); @@ -377,6 +370,7 @@ export function EditorFrame(props: EditorFrameProps) { return ( ) } @@ -429,11 +424,12 @@ export function EditorFrame(props: EditorFrameProps) { visualizationState={state.visualization.state} visualizationMap={props.visualizationMap} dispatch={dispatch} + isFullscreen={Boolean(state.isFullscreenDatasource)} ExpressionRenderer={props.ExpressionRenderer} core={props.core} plugins={props.plugins} visualizeTriggerFieldContext={visualizeTriggerFieldContext} - getSuggestionForField={getSuggestionForField} + getSuggestionForField={getSuggestionForField.current} /> ) } diff --git a/x-pack/plugins/lens/public/editor_frame_service/editor_frame/frame_layout.scss b/x-pack/plugins/lens/public/editor_frame_service/editor_frame/frame_layout.scss index 0756c13f6999bf..282e69cd7636c7 100644 --- a/x-pack/plugins/lens/public/editor_frame_service/editor_frame/frame_layout.scss +++ b/x-pack/plugins/lens/public/editor_frame_service/editor_frame/frame_layout.scss @@ -67,9 +67,16 @@ a tilemap in an iframe: https://github.com/elastic/kibana/issues/16457 */ padding: $euiSize $euiSize 0; position: relative; z-index: $lnsZLevel1; + &:first-child { padding-left: $euiSize; } + + &.lnsFrameLayout__pageBody-isFullscreen { + background: $euiColorEmptyShade; + flex: 1; + padding: 0; + } } .lnsFrameLayout__sidebar { @@ -81,6 +88,13 @@ a tilemap in an iframe: https://github.com/elastic/kibana/issues/16457 */ position: relative; } +.lnsFrameLayout-isFullscreen .lnsFrameLayout__sidebar--left, +.lnsFrameLayout-isFullscreen .lnsFrameLayout__suggestionPanel { + // Hide the datapanel and suggestions in fullscreen mode. Using display: none does trigger + // a rerender when the container becomes visible again, maybe pushing offscreen is better + display: none; +} + .lnsFrameLayout__sidebar--right { flex-basis: 25%; background-color: lightOrDarkTheme($euiColorLightestShade, $euiColorInk); @@ -106,3 +120,8 @@ a tilemap in an iframe: https://github.com/elastic/kibana/issues/16457 */ } } } + +.lnsFrameLayout__sidebar-isFullscreen { + flex: 1; + max-width: none; +} diff --git a/x-pack/plugins/lens/public/editor_frame_service/editor_frame/frame_layout.tsx b/x-pack/plugins/lens/public/editor_frame_service/editor_frame/frame_layout.tsx index a54901a2a2fe1d..f27e0f9c24d7bd 100644 --- a/x-pack/plugins/lens/public/editor_frame_service/editor_frame/frame_layout.tsx +++ b/x-pack/plugins/lens/public/editor_frame_service/editor_frame/frame_layout.tsx @@ -10,23 +10,32 @@ import './frame_layout.scss'; import React from 'react'; import { EuiPage, EuiPageBody, EuiScreenReaderOnly } from '@elastic/eui'; import { i18n } from '@kbn/i18n'; +import classNames from 'classnames'; export interface FrameLayoutProps { dataPanel: React.ReactNode; configPanel?: React.ReactNode; suggestionsPanel?: React.ReactNode; workspacePanel?: React.ReactNode; + isFullscreen?: boolean; } export function FrameLayout(props: FrameLayoutProps) { return ( - + -
+

{i18n.translate('xpack.lens.section.dataPanelLabel', { @@ -36,7 +45,13 @@ export function FrameLayout(props: FrameLayoutProps) { {props.dataPanel}

-
+

{i18n.translate('xpack.lens.section.workspaceLabel', { @@ -45,10 +60,13 @@ export function FrameLayout(props: FrameLayoutProps) {

{props.workspacePanel} - {props.suggestionsPanel} +
{props.suggestionsPanel}
diff --git a/x-pack/plugins/lens/public/editor_frame_service/editor_frame/state_management.ts b/x-pack/plugins/lens/public/editor_frame_service/editor_frame/state_management.ts index aa365d1e66d6c5..a87aa7a2cb4282 100644 --- a/x-pack/plugins/lens/public/editor_frame_service/editor_frame/state_management.ts +++ b/x-pack/plugins/lens/public/editor_frame_service/editor_frame/state_management.ts @@ -22,6 +22,7 @@ export interface EditorFrameState extends PreviewState { description?: string; stagedPreview?: PreviewState; activeDatasourceId: string | null; + isFullscreenDatasource?: boolean; } export type Action = @@ -90,6 +91,9 @@ export type Action = | { type: 'SWITCH_DATASOURCE'; newDatasourceId: string; + } + | { + type: 'TOGGLE_FULLSCREEN'; }; export function getActiveDatasourceIdFromDoc(doc?: Document) { @@ -281,6 +285,8 @@ export const reducer = (state: EditorFrameState, action: Action): EditorFrameSta }, stagedPreview: action.clearStagedPreview ? undefined : state.stagedPreview, }; + case 'TOGGLE_FULLSCREEN': + return { ...state, isFullscreenDatasource: !state.isFullscreenDatasource }; default: return state; } diff --git a/x-pack/plugins/lens/public/editor_frame_service/editor_frame/suggestion_panel.tsx b/x-pack/plugins/lens/public/editor_frame_service/editor_frame/suggestion_panel.tsx index 0c2eb4f39d8959..8107b6646500d5 100644 --- a/x-pack/plugins/lens/public/editor_frame_service/editor_frame/suggestion_panel.tsx +++ b/x-pack/plugins/lens/public/editor_frame_service/editor_frame/suggestion_panel.tsx @@ -200,15 +200,16 @@ export function SuggestionPanel({ visualizationState: currentVisualizationState, activeData: frame.activeData, }) - .filter((suggestion) => !suggestion.hide) .filter( ({ + hide, visualizationId, visualizationState: suggestionVisualizationState, datasourceState: suggestionDatasourceState, datasourceId: suggetionDatasourceId, }) => { return ( + !hide && validateDatasourceAndVisualization( suggetionDatasourceId ? datasourceMap[suggetionDatasourceId] : null, suggestionDatasourceState, diff --git a/x-pack/plugins/lens/public/editor_frame_service/editor_frame/workspace_panel/workspace_panel.test.tsx b/x-pack/plugins/lens/public/editor_frame_service/editor_frame/workspace_panel/workspace_panel.test.tsx index 1d248c4411023c..38e9bb868b26aa 100644 --- a/x-pack/plugins/lens/public/editor_frame_service/editor_frame/workspace_panel/workspace_panel.test.tsx +++ b/x-pack/plugins/lens/public/editor_frame_service/editor_frame/workspace_panel/workspace_panel.test.tsx @@ -64,6 +64,8 @@ const defaultProps = { data: mockDataPlugin(), }, getSuggestionForField: () => undefined, + isFullscreen: false, + toggleFullscreen: jest.fn(), }; describe('workspace_panel', () => { diff --git a/x-pack/plugins/lens/public/editor_frame_service/editor_frame/workspace_panel/workspace_panel.tsx b/x-pack/plugins/lens/public/editor_frame_service/editor_frame/workspace_panel/workspace_panel.tsx index 45abbf120042d0..01d4e84ec4374d 100644 --- a/x-pack/plugins/lens/public/editor_frame_service/editor_frame/workspace_panel/workspace_panel.tsx +++ b/x-pack/plugins/lens/public/editor_frame_service/editor_frame/workspace_panel/workspace_panel.tsx @@ -79,6 +79,7 @@ export interface WorkspacePanelProps { title?: string; visualizeTriggerFieldContext?: VisualizeFieldContext; getSuggestionForField: (field: DragDropIdentifier) => Suggestion | undefined; + isFullscreen: boolean; } interface WorkspaceState { @@ -134,6 +135,7 @@ export const InnerWorkspacePanel = React.memo(function InnerWorkspacePanel({ title, visualizeTriggerFieldContext, suggestionForDraggedField, + isFullscreen, }: Omit & { suggestionForDraggedField: Suggestion | undefined; }) { @@ -346,6 +348,8 @@ export const InnerWorkspacePanel = React.memo(function InnerWorkspacePanel({ ); }; + const element = expression !== null ? renderVisualization() : renderEmptyWorkspace(); + const dragDropContext = useContext(DragContext); const renderDragDrop = () => { @@ -363,7 +367,10 @@ export const InnerWorkspacePanel = React.memo(function InnerWorkspacePanel({ customWorkspaceRenderer() ) : ( - {renderVisualization()} - {Boolean(suggestionForDraggedField) && expression !== null && renderEmptyWorkspace()} + {element} ); @@ -389,6 +395,7 @@ export const InnerWorkspacePanel = React.memo(function InnerWorkspacePanel({ datasourceStates={datasourceStates} datasourceMap={datasourceMap} visualizationMap={visualizationMap} + isFullscreen={isFullscreen} > {renderDragDrop()} diff --git a/x-pack/plugins/lens/public/editor_frame_service/editor_frame/workspace_panel/workspace_panel_wrapper.scss b/x-pack/plugins/lens/public/editor_frame_service/editor_frame/workspace_panel/workspace_panel_wrapper.scss index e687e478cd3680..21e3f9aa36674b 100644 --- a/x-pack/plugins/lens/public/editor_frame_service/editor_frame/workspace_panel/workspace_panel_wrapper.scss +++ b/x-pack/plugins/lens/public/editor_frame_service/editor_frame/workspace_panel/workspace_panel_wrapper.scss @@ -31,6 +31,10 @@ overflow: hidden; } } + + &.lnsWorkspacePanelWrapper--fullscreen { + margin-bottom: 0; + } } .lnsWorkspacePanel__dragDrop { @@ -62,6 +66,10 @@ animation: lnsWorkspacePanel__illustrationPulseContinuous 1.5s ease-in-out 0s infinite normal forwards; } } + + &.lnsWorkspacePanel__dragDrop--fullscreen { + border: none; + } } .lnsWorkspacePanel__emptyContent { diff --git a/x-pack/plugins/lens/public/editor_frame_service/editor_frame/workspace_panel/workspace_panel_wrapper.test.tsx b/x-pack/plugins/lens/public/editor_frame_service/editor_frame/workspace_panel/workspace_panel_wrapper.test.tsx index 7bb467df9ab0e1..c18b362e2faa4e 100644 --- a/x-pack/plugins/lens/public/editor_frame_service/editor_frame/workspace_panel/workspace_panel_wrapper.test.tsx +++ b/x-pack/plugins/lens/public/editor_frame_service/editor_frame/workspace_panel/workspace_panel_wrapper.test.tsx @@ -37,6 +37,7 @@ describe('workspace_panel_wrapper', () => { visualizationMap={{ myVis: mockVisualization }} datasourceMap={{}} datasourceStates={{}} + isFullscreen={false} > @@ -58,6 +59,7 @@ describe('workspace_panel_wrapper', () => { visualizationMap={{ myVis: { ...mockVisualization, renderToolbar: renderToolbarMock } }} datasourceMap={{}} datasourceStates={{}} + isFullscreen={false} /> ); diff --git a/x-pack/plugins/lens/public/editor_frame_service/editor_frame/workspace_panel/workspace_panel_wrapper.tsx b/x-pack/plugins/lens/public/editor_frame_service/editor_frame/workspace_panel/workspace_panel_wrapper.tsx index ec12e9e4002039..6724002d23e0bb 100644 --- a/x-pack/plugins/lens/public/editor_frame_service/editor_frame/workspace_panel/workspace_panel_wrapper.tsx +++ b/x-pack/plugins/lens/public/editor_frame_service/editor_frame/workspace_panel/workspace_panel_wrapper.tsx @@ -10,6 +10,7 @@ import './workspace_panel_wrapper.scss'; import React, { useCallback } from 'react'; import { i18n } from '@kbn/i18n'; import { EuiPageContent, EuiFlexGroup, EuiFlexItem, EuiScreenReaderOnly } from '@elastic/eui'; +import classNames from 'classnames'; import { Datasource, FramePublicAPI, Visualization } from '../../../types'; import { NativeRenderer } from '../../../native_renderer'; import { Action } from '../state_management'; @@ -32,6 +33,7 @@ export interface WorkspacePanelWrapperProps { state: unknown; } >; + isFullscreen: boolean; } export function WorkspacePanelWrapper({ @@ -44,6 +46,7 @@ export function WorkspacePanelWrapper({ visualizationMap, datasourceMap, datasourceStates, + isFullscreen, }: WorkspacePanelWrapperProps) { const activeVisualization = visualizationId ? visualizationMap[visualizationId] : null; const setVisualizationState = useCallback( @@ -85,40 +88,42 @@ export function WorkspacePanelWrapper({ wrap={true} justifyContent="spaceBetween" > - - - - - - {activeVisualization && activeVisualization.renderToolbar && ( + {!isFullscreen ? ( + + - - )} - - + {activeVisualization && activeVisualization.renderToolbar && ( + + + + )} + + + ) : null} {warningMessages && warningMessages.length ? ( {warningMessages} @@ -126,7 +131,11 @@ export function WorkspacePanelWrapper({
- +

{title || diff --git a/x-pack/plugins/lens/public/editor_frame_service/mocks.tsx b/x-pack/plugins/lens/public/editor_frame_service/mocks.tsx index 38669d72474df1..1762e7ff20fabb 100644 --- a/x-pack/plugins/lens/public/editor_frame_service/mocks.tsx +++ b/x-pack/plugins/lens/public/editor_frame_service/mocks.tsx @@ -57,6 +57,7 @@ export function createMockVisualization(): jest.Mocked { setDimension: jest.fn(), removeDimension: jest.fn(), getErrorMessages: jest.fn((_state) => undefined), + renderDimensionEditor: jest.fn(), }; } diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/dimension_editor.scss b/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/dimension_editor.scss index bf833c4a369325..874291ae25e341 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/dimension_editor.scss +++ b/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/dimension_editor.scss @@ -2,7 +2,27 @@ height: 100%; } -.lnsIndexPatternDimensionEditor__section { +.lnsIndexPatternDimensionEditor__header { + position: sticky; + top: 0; + background: $euiColorEmptyShade; + // Raise it above the elements that are after it in DOM order + z-index: $euiZLevel1; +} + +.lnsIndexPatternDimensionEditor-isFullscreen { + position: absolute; + left: 0; + right: 0; + top: 0; + bottom: 0; + + .lnsIndexPatternDimensionEditor__section { + height: 100%; + } +} + +.lnsIndexPatternDimensionEditor__section--padded { padding: $euiSizeS; } @@ -10,6 +30,14 @@ background-color: $euiColorLightestShade; } +.lnsIndexPatternDimensionEditor__section--top { + border-bottom: $euiBorderThin; +} + +.lnsIndexPatternDimensionEditor__section--bottom { + border-top: $euiBorderThin; +} + .lnsIndexPatternDimensionEditor__columns { column-count: 2; column-gap: $euiSizeXL; @@ -29,3 +57,9 @@ padding-top: 0; padding-bottom: 0; } + +.lnsIndexPatternDimensionEditor__warning { + @include kbnThemeStyle('v7') { + border: none; + } +} diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/dimension_editor.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/dimension_editor.tsx index 2ae7b9403a46db..3dd2d4a4ba3f50 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/dimension_editor.tsx +++ b/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/dimension_editor.tsx @@ -6,7 +6,7 @@ */ import './dimension_editor.scss'; -import React, { useState, useMemo } from 'react'; +import React, { useState, useMemo, useCallback } from 'react'; import { i18n } from '@kbn/i18n'; import { EuiListGroup, @@ -17,6 +17,9 @@ import { EuiFormLabel, EuiToolTip, EuiText, + EuiTabs, + EuiTab, + EuiCallOut, } from '@elastic/eui'; import { IndexPatternDimensionEditorProps } from './dimension_panel'; import { OperationSupportMatrix } from './operation_support'; @@ -91,6 +94,8 @@ export function DimensionEditor(props: DimensionEditorProps) { hideGrouping, dateRange, dimensionGroups, + toggleFullscreen, + isFullscreen, } = props; const services = { data: props.data, @@ -101,30 +106,34 @@ export function DimensionEditor(props: DimensionEditorProps) { }; const { fieldByOperation, operationWithoutField } = operationSupportMatrix; + const selectedOperationDefinition = + selectedColumn && operationDefinitionMap[selectedColumn.operationType]; + const setStateWrapper = ( setter: IndexPatternLayer | ((prevLayer: IndexPatternLayer) => IndexPatternLayer) ) => { + const prevOperationType = + operationDefinitionMap[state.layers[layerId].columns[columnId]?.operationType]?.input; + const hypotheticalLayer = typeof setter === 'function' ? setter(state.layers[layerId]) : setter; const hasIncompleteColumns = Boolean(hypotheticalLayer.incompleteColumns?.[columnId]); - const prevOperationType = - operationDefinitionMap[hypotheticalLayer.columns[columnId]?.operationType]?.input; setState( (prevState) => { const layer = typeof setter === 'function' ? setter(prevState.layers[layerId]) : setter; return mergeLayer({ state: prevState, layerId, newLayer: layer }); }, { - shouldReplaceDimension: Boolean(hypotheticalLayer.columns[columnId]), - // clear the dimension if there's an incomplete column pending && previous operation was a fullReference operation - shouldRemoveDimension: Boolean( - hasIncompleteColumns && prevOperationType === 'fullReference' - ), + isDimensionComplete: + prevOperationType === 'fullReference' + ? !hasIncompleteColumns + : Boolean(hypotheticalLayer.columns[columnId]), } ); }; - const selectedOperationDefinition = - selectedColumn && operationDefinitionMap[selectedColumn.operationType]; + const setIsCloseable = (isCloseable: boolean) => { + setState((prevState) => ({ ...prevState, isDimensionClosePrevented: !isCloseable })); + }; const incompleteInfo = (state.layers[layerId].incompleteColumns ?? {})[columnId]; const incompleteOperation = incompleteInfo?.operationType; @@ -132,14 +141,16 @@ export function DimensionEditor(props: DimensionEditorProps) { const ParamEditor = selectedOperationDefinition?.paramEditor; + const [temporaryQuickFunction, setQuickFunction] = useState(false); + const possibleOperations = useMemo(() => { return Object.values(operationDefinitionMap) .filter(({ hidden }) => !hidden) + .filter(({ type }) => fieldByOperation[type]?.size || operationWithoutField.has(type)) .sort((op1, op2) => { return op1.displayName.localeCompare(op2.displayName); }) - .map((def) => def.type) - .filter((type) => fieldByOperation[type]?.size || operationWithoutField.has(type)); + .map((def) => def.type); }, [fieldByOperation, operationWithoutField]); const [filterByOpenInitially, setFilterByOpenInitally] = useState(false); @@ -245,37 +256,44 @@ export function DimensionEditor(props: DimensionEditorProps) { visualizationGroups: dimensionGroups, targetGroup: props.groupId, }); + if (temporaryQuickFunction && newLayer.columns[columnId].operationType !== 'formula') { + // Only switch the tab once the formula is fully removed + setQuickFunction(false); + } setStateWrapper(newLayer); trackUiEvent(`indexpattern_dimension_operation_${operationType}`); return; } else if (!selectedColumn || !compatibleWithCurrentField) { const possibleFields = fieldByOperation[operationType] || new Set(); + let newLayer: IndexPatternLayer; if (possibleFields.size === 1) { - setStateWrapper( - insertOrReplaceColumn({ - layer: props.state.layers[props.layerId], - indexPattern: currentIndexPattern, - columnId, - op: operationType, - field: currentIndexPattern.getFieldByName(possibleFields.values().next().value), - visualizationGroups: dimensionGroups, - targetGroup: props.groupId, - }) - ); + newLayer = insertOrReplaceColumn({ + layer: props.state.layers[props.layerId], + indexPattern: currentIndexPattern, + columnId, + op: operationType, + field: currentIndexPattern.getFieldByName(possibleFields.values().next().value), + visualizationGroups: dimensionGroups, + targetGroup: props.groupId, + }); } else { - setStateWrapper( - insertOrReplaceColumn({ - layer: props.state.layers[props.layerId], - indexPattern: currentIndexPattern, - columnId, - op: operationType, - field: undefined, - visualizationGroups: dimensionGroups, - targetGroup: props.groupId, - }) - ); + newLayer = insertOrReplaceColumn({ + layer: props.state.layers[props.layerId], + indexPattern: currentIndexPattern, + columnId, + op: operationType, + field: undefined, + visualizationGroups: dimensionGroups, + targetGroup: props.groupId, + }); + // ); + } + if (temporaryQuickFunction && newLayer.columns[columnId].operationType !== 'formula') { + // Only switch the tab once the formula is fully removed + setQuickFunction(false); } + setStateWrapper(newLayer); trackUiEvent(`indexpattern_dimension_operation_${operationType}`); return; } @@ -287,6 +305,9 @@ export function DimensionEditor(props: DimensionEditorProps) { return; } + if (temporaryQuickFunction) { + setQuickFunction(false); + } const newLayer = replaceColumn({ layer: props.state.layers[props.layerId], indexPattern: currentIndexPattern, @@ -315,9 +336,34 @@ export function DimensionEditor(props: DimensionEditorProps) { currentFieldIsInvalid ); - return ( -
-
+ const shouldDisplayExtraOptions = + !currentFieldIsInvalid && + !incompleteInfo && + selectedColumn && + selectedColumn.operationType !== 'formula'; + + const quickFunctions = ( + <> + {temporaryQuickFunction && selectedColumn?.operationType === 'formula' && ( + <> + +

+ {i18n.translate('xpack.lens.indexPattern.formulaWarningText', { + defaultMessage: 'To overwrite your formula, select a quick function', + })} +

+ + + )} +
{i18n.translate('xpack.lens.indexPattern.functionsLabel', { defaultMessage: 'Select a function', @@ -336,7 +382,7 @@ export function DimensionEditor(props: DimensionEditorProps) { />
-
+
{!incompleteInfo && selectedColumn && 'references' in selectedColumn && @@ -375,6 +421,9 @@ export function DimensionEditor(props: DimensionEditorProps) { currentColumn: state.layers[layerId].columns[columnId], })} dimensionGroups={dimensionGroups} + isFullscreen={isFullscreen} + toggleFullscreen={toggleFullscreen} + setIsCloseable={setIsCloseable} {...services} /> ); @@ -385,7 +434,8 @@ export function DimensionEditor(props: DimensionEditorProps) { {!selectedColumn || selectedOperationDefinition?.input === 'field' || - (incompleteOperation && operationDefinitionMap[incompleteOperation].input === 'field') ? ( + (incompleteOperation && operationDefinitionMap[incompleteOperation].input === 'field') || + temporaryQuickFunction ? ( ) : null} - {!currentFieldIsInvalid && !incompleteInfo && selectedColumn && ParamEditor && ( - <> - - + {shouldDisplayExtraOptions && ParamEditor && ( + )} {!currentFieldIsInvalid && !incompleteInfo && selectedColumn && ( @@ -546,9 +597,96 @@ export function DimensionEditor(props: DimensionEditorProps) {
+ + ); - {!currentFieldIsInvalid && ( -
+ const formulaTab = ParamEditor ? ( + + ) : null; + + const onFormatChange = useCallback( + (newFormat) => { + setState( + mergeLayer({ + state, + layerId, + newLayer: updateColumnParam({ + layer: state.layers[layerId], + columnId, + paramName: 'format', + value: newFormat, + }), + }) + ); + }, + [columnId, layerId, setState, state] + ); + + return ( +
+ {!isFullscreen && operationSupportMatrix.operationWithoutField.has('formula') ? ( + + { + if (selectedColumn?.operationType === 'formula') { + setQuickFunction(true); + } + }} + > + {i18n.translate('xpack.lens.indexPattern.quickFunctionsLabel', { + defaultMessage: 'Quick functions', + })} + + { + if (selectedColumn?.operationType !== 'formula') { + setQuickFunction(false); + const newLayer = insertOrReplaceColumn({ + layer: props.state.layers[props.layerId], + indexPattern: currentIndexPattern, + columnId, + op: 'formula', + visualizationGroups: dimensionGroups, + }); + setStateWrapper(newLayer); + trackUiEvent(`indexpattern_dimension_operation_formula`); + return; + } else { + setQuickFunction(false); + } + }} + > + {i18n.translate('xpack.lens.indexPattern.formulaLabel', { + defaultMessage: 'Formula', + })} + + + ) : null} + + {isFullscreen + ? formulaTab + : selectedOperationDefinition?.type === 'formula' && !temporaryQuickFunction + ? formulaTab + : quickFunctions} + + {!isFullscreen && !currentFieldIsInvalid && !temporaryQuickFunction && ( +
{!incompleteInfo && selectedColumn && ( )} - {!incompleteInfo && !hideGrouping && ( + {!isFullscreen && !incompleteInfo && !hideGrouping && ( )} - {selectedColumn && + {!isFullscreen && + selectedColumn && (selectedColumn.dataType === 'number' || selectedColumn.operationType === 'range') ? ( - { - setState( - mergeLayer({ - state, - layerId, - newLayer: updateColumnParam({ - layer: state.layers[layerId], - columnId, - paramName: 'format', - value: newFormat, - }), - }) - ); - }} - /> + ) : null}
)}
); } + function getErrorMessage( selectedColumn: IndexPatternColumn | undefined, incompleteOperation: boolean, diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/dimension_panel.test.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/dimension_panel.test.tsx index 03db6141b917f9..7e45b295215631 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/dimension_panel.test.tsx +++ b/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/dimension_panel.test.tsx @@ -208,6 +208,8 @@ describe('IndexPatternDimensionEditorPanel', () => { core: {} as CoreSetup, dimensionGroups: [], groupId: 'a', + isFullscreen: false, + toggleFullscreen: jest.fn(), }; jest.clearAllMocks(); @@ -500,10 +502,7 @@ describe('IndexPatternDimensionEditorPanel', () => { comboBox.prop('onChange')!([option]); }); - expect(setState.mock.calls[0]).toEqual([ - expect.any(Function), - { shouldRemoveDimension: false, shouldReplaceDimension: true }, - ]); + expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]); expect(setState.mock.calls[0][0](defaultProps.state)).toEqual({ ...initialState, layers: { @@ -535,10 +534,7 @@ describe('IndexPatternDimensionEditorPanel', () => { comboBox.prop('onChange')!([option]); }); - expect(setState.mock.calls[0]).toEqual([ - expect.any(Function), - { shouldRemoveDimension: false, shouldReplaceDimension: true }, - ]); + expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]); expect(setState.mock.calls[0][0](defaultProps.state)).toEqual({ ...state, layers: { @@ -569,10 +565,7 @@ describe('IndexPatternDimensionEditorPanel', () => { wrapper.find('button[data-test-subj="lns-indexPatternDimension-min"]').simulate('click'); }); - expect(setState.mock.calls[0]).toEqual([ - expect.any(Function), - { shouldRemoveDimension: false, shouldReplaceDimension: true }, - ]); + expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]); expect(setState.mock.calls[0][0](state)).toEqual({ ...state, layers: { @@ -643,10 +636,7 @@ describe('IndexPatternDimensionEditorPanel', () => { wrapper.find('button[data-test-subj="lns-indexPatternDimension-min"]').simulate('click'); }); - expect(setState.mock.calls[0]).toEqual([ - expect.any(Function), - { shouldRemoveDimension: false, shouldReplaceDimension: true }, - ]); + expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]); expect(setState.mock.calls[0][0](state)).toEqual({ ...state, layers: { @@ -681,10 +671,7 @@ describe('IndexPatternDimensionEditorPanel', () => { wrapper.find('button[data-test-subj="lns-indexPatternDimension-min"]').simulate('click'); }); - expect(setState.mock.calls[0]).toEqual([ - expect.any(Function), - { shouldRemoveDimension: false, shouldReplaceDimension: true }, - ]); + expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]); expect(setState.mock.calls[0][0](state)).toEqual({ ...state, layers: { @@ -750,10 +737,7 @@ describe('IndexPatternDimensionEditorPanel', () => { .simulate('click'); }); - expect(setState.mock.calls[0]).toEqual([ - expect.any(Function), - { shouldRemoveDimension: false, shouldReplaceDimension: true }, - ]); + expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]); expect(setState.mock.calls[0][0](state)).toEqual({ ...state, layers: { @@ -879,7 +863,7 @@ describe('IndexPatternDimensionEditorPanel', () => { expect(setState.mock.calls[0]).toEqual([ expect.any(Function), - { shouldRemoveDimension: false, shouldReplaceDimension: false }, + { isDimensionComplete: false }, ]); expect(setState.mock.calls[0][0](state)).toEqual({ ...state, @@ -948,7 +932,7 @@ describe('IndexPatternDimensionEditorPanel', () => { // Now check that the dimension gets cleaned up on state update expect(setState.mock.calls[0]).toEqual([ expect.any(Function), - { shouldRemoveDimension: false, shouldReplaceDimension: false }, + { isDimensionComplete: false }, ]); expect(setState.mock.calls[0][0](state)).toEqual({ ...state, @@ -1042,10 +1026,7 @@ describe('IndexPatternDimensionEditorPanel', () => { }); expect(setState.mock.calls.length).toEqual(2); - expect(setState.mock.calls[1]).toEqual([ - expect.any(Function), - { shouldRemoveDimension: false, shouldReplaceDimension: true }, - ]); + expect(setState.mock.calls[1]).toEqual([expect.any(Function), { isDimensionComplete: true }]); expect(setState.mock.calls[1][0](state)).toEqual({ ...state, layers: { @@ -1143,10 +1124,7 @@ describe('IndexPatternDimensionEditorPanel', () => { .find('[data-test-subj="indexPattern-time-scaling-enable"]') .hostNodes() .simulate('click'); - expect(setState.mock.calls[0]).toEqual([ - expect.any(Function), - { shouldRemoveDimension: false, shouldReplaceDimension: true }, - ]); + expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]); expect(setState.mock.calls[0][0](props.state)).toEqual({ ...props.state, layers: { @@ -1175,10 +1153,7 @@ describe('IndexPatternDimensionEditorPanel', () => { wrapper .find('button[data-test-subj="lns-indexPatternDimension-count incompatible"]') .simulate('click'); - expect(setState.mock.calls[0]).toEqual([ - expect.any(Function), - { shouldRemoveDimension: false, shouldReplaceDimension: true }, - ]); + expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]); expect(setState.mock.calls[0][0](props.state)).toEqual({ ...props.state, layers: { @@ -1205,10 +1180,7 @@ describe('IndexPatternDimensionEditorPanel', () => { }); wrapper = mount(); wrapper.find('button[data-test-subj="lns-indexPatternDimension-average"]').simulate('click'); - expect(setState.mock.calls[0]).toEqual([ - expect.any(Function), - { shouldRemoveDimension: false, shouldReplaceDimension: true }, - ]); + expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]); expect(setState.mock.calls[0][0](props.state)).toEqual({ ...props.state, layers: { @@ -1239,10 +1211,7 @@ describe('IndexPatternDimensionEditorPanel', () => { .prop('onChange')!(({ target: { value: 'h' }, } as unknown) as ChangeEvent); - expect(setState.mock.calls[0]).toEqual([ - expect.any(Function), - { shouldRemoveDimension: false, shouldReplaceDimension: true }, - ]); + expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]); expect(setState.mock.calls[0][0](props.state)).toEqual({ ...props.state, layers: { @@ -1269,10 +1238,7 @@ describe('IndexPatternDimensionEditorPanel', () => { .prop('onChange')!(({ target: { value: 'h' }, } as unknown) as ChangeEvent); - expect(setState.mock.calls[0]).toEqual([ - expect.any(Function), - { shouldRemoveDimension: false, shouldReplaceDimension: true }, - ]); + expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]); expect(setState.mock.calls[0][0](props.state)).toEqual({ ...props.state, layers: { @@ -1300,10 +1266,7 @@ describe('IndexPatternDimensionEditorPanel', () => { // eslint-disable-next-line @typescript-eslint/no-explicit-any {} as any ); - expect(setState.mock.calls[0]).toEqual([ - expect.any(Function), - { shouldRemoveDimension: false, shouldReplaceDimension: true }, - ]); + expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]); expect(setState.mock.calls[0][0](props.state)).toEqual({ ...props.state, layers: { @@ -1593,10 +1556,7 @@ describe('IndexPatternDimensionEditorPanel', () => { .find('[data-test-subj="indexPattern-filter-by-enable"]') .hostNodes() .simulate('click'); - expect(setState.mock.calls[0]).toEqual([ - expect.any(Function), - { shouldRemoveDimension: false, shouldReplaceDimension: true }, - ]); + expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]); expect(setState.mock.calls[0][0](props.state)).toEqual({ ...props.state, layers: { @@ -1627,10 +1587,7 @@ describe('IndexPatternDimensionEditorPanel', () => { wrapper .find('button[data-test-subj="lns-indexPatternDimension-count incompatible"]') .simulate('click'); - expect(setState.mock.calls[0]).toEqual([ - expect.any(Function), - { shouldRemoveDimension: false, shouldReplaceDimension: true }, - ]); + expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]); expect(setState.mock.calls[0][0](props.state)).toEqual({ ...props.state, layers: { @@ -1656,10 +1613,7 @@ describe('IndexPatternDimensionEditorPanel', () => { language: 'kuery', query: 'c: d', }); - expect(setState.mock.calls[0]).toEqual([ - expect.any(Function), - { shouldRemoveDimension: false, shouldReplaceDimension: true }, - ]); + expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]); expect(setState.mock.calls[0][0](props.state)).toEqual({ ...props.state, layers: { @@ -1688,10 +1642,7 @@ describe('IndexPatternDimensionEditorPanel', () => { // eslint-disable-next-line @typescript-eslint/no-explicit-any {} as any ); - expect(setState.mock.calls[0]).toEqual([ - expect.any(Function), - { shouldRemoveDimension: false, shouldReplaceDimension: true }, - ]); + expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]); expect(setState.mock.calls[0][0](props.state)).toEqual({ ...props.state, layers: { @@ -1743,10 +1694,7 @@ describe('IndexPatternDimensionEditorPanel', () => { wrapper.find('button[data-test-subj="lns-indexPatternDimension-average"]').simulate('click'); - expect(setState.mock.calls[0]).toEqual([ - expect.any(Function), - { shouldRemoveDimension: false, shouldReplaceDimension: false }, - ]); + expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: false }]); expect(setState.mock.calls[0][0](defaultProps.state)).toEqual({ ...state, layers: { @@ -1810,10 +1758,7 @@ describe('IndexPatternDimensionEditorPanel', () => { wrapper.find('button[data-test-subj="lns-indexPatternDimension-average"]').simulate('click'); - expect(setState.mock.calls[0]).toEqual([ - expect.any(Function), - { shouldRemoveDimension: false, shouldReplaceDimension: true }, - ]); + expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]); expect(setState.mock.calls[0][0](initialState)).toEqual({ ...initialState, layers: { @@ -1838,10 +1783,7 @@ describe('IndexPatternDimensionEditorPanel', () => { wrapper.find('button[data-test-subj="lns-indexPatternDimension-count"]').simulate('click'); - expect(setState.mock.calls[0]).toEqual([ - expect.any(Function), - { shouldRemoveDimension: false, shouldReplaceDimension: true }, - ]); + expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]); expect(setState.mock.calls[0][0](state)).toEqual({ ...state, layers: { @@ -1975,10 +1917,7 @@ describe('IndexPatternDimensionEditorPanel', () => { comboBox.prop('onChange')!([option]); }); - expect(setState.mock.calls[0]).toEqual([ - expect.any(Function), - { shouldRemoveDimension: false, shouldReplaceDimension: true }, - ]); + expect(setState.mock.calls[0]).toEqual([expect.any(Function), { isDimensionComplete: true }]); expect(setState.mock.calls[0][0](defaultProps.state)).toEqual({ ...state, layers: { diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/droppable/droppable.test.ts b/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/droppable/droppable.test.ts index a77a980257c88c..56d255ec02227c 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/droppable/droppable.test.ts +++ b/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/droppable/droppable.test.ts @@ -284,6 +284,8 @@ describe('IndexPatternDimensionEditorPanel', () => { } as unknown) as DataPublicPluginStart, core: {} as CoreSetup, dimensionGroups: [], + isFullscreen: false, + toggleFullscreen: () => {}, }; jest.clearAllMocks(); diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/format_selector.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/format_selector.tsx index 3a57579583c90c..ff10810208e706 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/format_selector.tsx +++ b/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/format_selector.tsx @@ -5,7 +5,7 @@ * 2.0. */ -import React, { useState } from 'react'; +import React, { useCallback, useMemo, useState } from 'react'; import { i18n } from '@kbn/i18n'; import { EuiFormRow, EuiComboBox, EuiSpacer, EuiRange } from '@elastic/eui'; import { IndexPatternColumn } from '../indexpattern'; @@ -28,6 +28,13 @@ const supportedFormats: Record = { }, }; +const defaultOption = { + value: '', + label: i18n.translate('xpack.lens.indexPattern.defaultFormatLabel', { + defaultMessage: 'Default', + }), +}; + interface FormatSelectorProps { selectedColumn: IndexPatternColumn; onChange: (newFormat?: { id: string; params?: Record }) => void; @@ -37,6 +44,8 @@ interface State { decimalPlaces: number; } +const singleSelectionOption = { asPlainText: true }; + export function FormatSelector(props: FormatSelectorProps) { const { selectedColumn, onChange } = props; @@ -51,13 +60,6 @@ export function FormatSelector(props: FormatSelectorProps) { const selectedFormat = currentFormat?.id ? supportedFormats[currentFormat.id] : undefined; - const defaultOption = { - value: '', - label: i18n.translate('xpack.lens.indexPattern.defaultFormatLabel', { - defaultMessage: 'Default', - }), - }; - const label = i18n.translate('xpack.lens.indexPattern.columnFormatLabel', { defaultMessage: 'Value format', }); @@ -66,6 +68,48 @@ export function FormatSelector(props: FormatSelectorProps) { defaultMessage: 'Decimals', }); + const stableOptions = useMemo( + () => [ + defaultOption, + ...Object.entries(supportedFormats).map(([id, format]) => ({ + value: id, + label: format.title ?? id, + })), + ], + [] + ); + + const onChangeWrapped = useCallback( + (choices) => { + if (choices.length === 0) { + return; + } + + if (!choices[0].value) { + onChange(); + return; + } + onChange({ + id: choices[0].value, + params: { decimals: state.decimalPlaces }, + }); + }, + [onChange, state.decimalPlaces] + ); + + const currentOption = useMemo( + () => + currentFormat + ? [ + { + value: currentFormat.id, + label: selectedFormat?.title ?? currentFormat.id, + }, + ] + : [defaultOption], + [currentFormat, selectedFormat?.title] + ); + return ( <> @@ -76,38 +120,10 @@ export function FormatSelector(props: FormatSelectorProps) { isClearable={false} data-test-subj="indexPattern-dimension-format" aria-label={label} - singleSelection={{ asPlainText: true }} - options={[ - defaultOption, - ...Object.entries(supportedFormats).map(([id, format]) => ({ - value: id, - label: format.title ?? id, - })), - ]} - selectedOptions={ - currentFormat - ? [ - { - value: currentFormat.id, - label: selectedFormat?.title ?? currentFormat.id, - }, - ] - : [defaultOption] - } - onChange={(choices) => { - if (choices.length === 0) { - return; - } - - if (!choices[0].value) { - onChange(); - return; - } - onChange({ - id: choices[0].value, - params: { decimals: state.decimalPlaces }, - }); - }} + singleSelection={singleSelectionOption} + options={stableOptions} + selectedOptions={currentOption} + onChange={onChangeWrapped} /> {currentFormat ? ( <> diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/reference_editor.test.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/reference_editor.test.tsx index 645b6bfe70a97c..fd3ad9a4e5dd54 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/reference_editor.test.tsx +++ b/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/reference_editor.test.tsx @@ -51,6 +51,9 @@ describe('reference editor', () => { http: {} as HttpSetup, data: {} as DataPublicPluginStart, dimensionGroups: [], + isFullscreen: false, + toggleFullscreen: jest.fn(), + setIsCloseable: jest.fn(), }; } diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/reference_editor.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/reference_editor.tsx index c473be05ba3154..b0cdf96f928f93 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/reference_editor.tsx +++ b/x-pack/plugins/lens/public/indexpattern_datasource/dimension_panel/reference_editor.tsx @@ -47,10 +47,14 @@ export interface ReferenceEditorProps { setter: IndexPatternLayer | ((prevLayer: IndexPatternLayer) => IndexPatternLayer) ) => void; currentIndexPattern: IndexPattern; + existingFields: IndexPatternPrivateState['existingFields']; dateRange: DateRange; labelAppend?: EuiFormRowProps['labelAppend']; dimensionGroups: VisualizationDimensionGroupConfig[]; + isFullscreen: boolean; + toggleFullscreen: () => void; + setIsCloseable: (isCloseable: boolean) => void; // Services uiSettings: IUiSettingsClient; @@ -72,6 +76,9 @@ export function ReferenceEditor(props: ReferenceEditorProps) { dateRange, labelAppend, dimensionGroups, + isFullscreen, + toggleFullscreen, + setIsCloseable, ...services } = props; @@ -347,6 +354,9 @@ export function ReferenceEditor(props: ReferenceEditorProps) { indexPattern={currentIndexPattern} dateRange={dateRange} operationDefinitionMap={operationDefinitionMap} + isFullscreen={isFullscreen} + toggleFullscreen={toggleFullscreen} + setIsCloseable={setIsCloseable} {...services} /> diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/indexpattern.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/indexpattern.tsx index 7cb49de15c0665..bc2184bd9edb7b 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/indexpattern.tsx +++ b/x-pack/plugins/lens/public/indexpattern_datasource/indexpattern.tsx @@ -323,6 +323,11 @@ export function getIndexPatternDatasource({ domElement ); }, + + canCloseDimensionEditor: (state) => { + return !state.isDimensionClosePrevented; + }, + getDropProps, onDrop, diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/indexpattern_suggestions.test.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/indexpattern_suggestions.test.tsx index 864a3a6f089db7..93ea3069894d32 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/indexpattern_suggestions.test.tsx +++ b/x-pack/plugins/lens/public/indexpattern_datasource/indexpattern_suggestions.test.tsx @@ -16,6 +16,7 @@ import { } from './indexpattern_suggestions'; import { documentField } from './document_field'; import { getFieldByNameFactory } from './pure_helpers'; +import { isEqual } from 'lodash'; jest.mock('./loader'); jest.mock('../id_generator'); @@ -867,10 +868,7 @@ describe('IndexPattern Data Source suggestions', () => { searchable: true, }); - expect(suggestions).toHaveLength(1); - // Check that the suggestion is a single metric - expect(suggestions[0].table.columns).toHaveLength(1); - expect(suggestions[0].table.columns[0].operation.isBucketed).toBeFalsy(); + expect(suggestions).toHaveLength(0); }); it('appends a terms column with default size on string field', () => { @@ -1025,6 +1023,24 @@ describe('IndexPattern Data Source suggestions', () => { expect(suggestions).not.toContain(expect.objectContaining({ changeType: 'extended' })); }); + it('skips metric only suggestion when the field is already in use', () => { + const initialState = stateWithNonEmptyTables(); + const suggestions = getDatasourceSuggestionsForField(initialState, '1', { + name: 'bytes', + displayName: 'bytes', + type: 'number', + aggregatable: true, + searchable: true, + }); + + expect( + suggestions.some( + (suggestion) => + suggestion.table.changeType === 'initial' && suggestion.table.columns.length === 1 + ) + ).toBeFalsy(); + }); + it('skips duplicates when the document-specific field is already in use', () => { const initialState = stateWithNonEmptyTables(); const modifiedState: IndexPatternPrivateState = { @@ -2344,7 +2360,7 @@ describe('IndexPattern Data Source suggestions', () => { ); }); - it('will skip a reduced suggestion when handling multiple references', () => { + it('will create reduced suggestions with all referenced children when handling references', () => { const initialState = testInitialState(); const state: IndexPatternPrivateState = { ...initialState, @@ -2352,7 +2368,17 @@ describe('IndexPattern Data Source suggestions', () => { ...initialState.layers, first: { ...initialState.layers.first, - columnOrder: ['date', 'metric', 'metric2', 'ref', 'ref2'], + columnOrder: [ + 'date', + 'metric', + 'metric2', + 'ref', + 'ref2', + 'ref3', + 'ref4', + 'metric3', + 'metric4', + ], columns: { date: { @@ -2384,6 +2410,20 @@ describe('IndexPattern Data Source suggestions', () => { operationType: 'count', sourceField: 'Records', }, + metric3: { + label: '', + dataType: 'number', + isBucketed: false, + operationType: 'count', + sourceField: 'Records', + }, + metric4: { + label: '', + dataType: 'number', + isBucketed: false, + operationType: 'count', + sourceField: 'Records', + }, ref2: { label: '', dataType: 'number', @@ -2391,22 +2431,163 @@ describe('IndexPattern Data Source suggestions', () => { operationType: 'cumulative_sum', references: ['metric2'], }, + ref3: { + label: '', + dataType: 'number', + isBucketed: false, + operationType: 'math', + references: ['ref4', 'metric3'], + params: { + tinymathAst: '', + }, + }, + ref4: { + label: '', + dataType: 'number', + isBucketed: false, + operationType: 'math', + references: ['metric4'], + params: { + tinymathAst: '', + }, + }, }, }, }, }; - const result = getSuggestionSubset(getDatasourceSuggestionsFromCurrentState(state)); - - expect(result).not.toContainEqual( - expect.objectContaining({ - table: expect.objectContaining({ - changeType: 'reduced', - }), - }) - ); + const result = getDatasourceSuggestionsFromCurrentState(state); + + // only generate suggestions for top level metrics + expect( + result.filter((suggestion) => suggestion.table.changeType === 'reduced').length + ).toEqual(3); + + // top level "ref" column + expect( + result.some( + (suggestion) => + suggestion.table.changeType === 'reduced' && + isEqual(suggestion.state.layers.first.columnOrder, ['ref', 'metric']) + ) + ).toBeTruthy(); + + // top level "ref2" column + expect( + result.some( + (suggestion) => + suggestion.table.changeType === 'reduced' && + isEqual(suggestion.state.layers.first.columnOrder, ['ref2', 'metric2']) + ) + ).toBeTruthy(); + + // top level "ref3" column + expect( + result.some( + (suggestion) => + suggestion.table.changeType === 'reduced' && + isEqual(suggestion.state.layers.first.columnOrder, [ + 'ref3', + 'ref4', + 'metric3', + 'metric4', + ]) + ) + ).toBeTruthy(); }); }); + + it('will leave dangling references in place', () => { + const initialState = testInitialState(); + const state: IndexPatternPrivateState = { + ...initialState, + layers: { + ...initialState.layers, + first: { + ...initialState.layers.first, + columnOrder: ['date', 'ref'], + + columns: { + date: { + label: '', + dataType: 'date', + isBucketed: true, + operationType: 'date_histogram', + sourceField: 'timestamp', + params: { interval: 'auto' }, + }, + ref: { + label: '', + dataType: 'number', + isBucketed: false, + operationType: 'cumulative_sum', + references: ['non_existing_metric'], + }, + }, + }, + }, + }; + + const result = getDatasourceSuggestionsFromCurrentState(state); + + // only generate suggestions for top level metrics + expect( + result.filter((suggestion) => suggestion.table.changeType === 'reduced').length + ).toEqual(1); + + // top level "ref" column + expect( + result.some( + (suggestion) => + suggestion.table.changeType === 'reduced' && + isEqual(suggestion.state.layers.first.columnOrder, ['ref']) + ) + ).toBeTruthy(); + }); + + it('will not suggest reduced tables if there is just a referenced top level metric', () => { + const initialState = testInitialState(); + const state: IndexPatternPrivateState = { + ...initialState, + layers: { + ...initialState.layers, + first: { + ...initialState.layers.first, + columnOrder: ['ref', 'metric'], + + columns: { + ref: { + label: '', + dataType: 'number', + isBucketed: false, + operationType: 'math', + params: { + tinymathAst: '', + }, + references: ['metric'], + }, + metric: { + label: '', + dataType: 'number', + isBucketed: false, + operationType: 'count', + sourceField: 'Records', + }, + }, + }, + }, + }; + + const result = getDatasourceSuggestionsFromCurrentState(state); + + expect( + result.filter((suggestion) => suggestion.table.changeType === 'unchanged').length + ).toEqual(1); + + expect( + result.filter((suggestion) => suggestion.table.changeType === 'reduced').length + ).toEqual(0); + }); }); }); diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/indexpattern_suggestions.ts b/x-pack/plugins/lens/public/indexpattern_datasource/indexpattern_suggestions.ts index 803ba9f5bae5dc..cff036db4813bb 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/indexpattern_suggestions.ts +++ b/x-pack/plugins/lens/public/indexpattern_datasource/indexpattern_suggestions.ts @@ -5,7 +5,7 @@ * 2.0. */ -import { flatten, minBy, pick, mapValues } from 'lodash'; +import { flatten, minBy, pick, mapValues, partition } from 'lodash'; import { i18n } from '@kbn/i18n'; import { generateId } from '../id_generator'; import { DatasourceSuggestion, TableChangeType } from '../types'; @@ -20,6 +20,7 @@ import { OperationType, getExistingColumnGroups, isReferenced, + getReferencedColumnIds, } from './operations'; import { hasField } from './utils'; import { @@ -254,9 +255,11 @@ function getExistingLayerSuggestionsForField( } } - const metricSuggestion = createMetricSuggestion(indexPattern, layerId, state, field); - if (metricSuggestion) { - suggestions.push(metricSuggestion); + if (!fieldInUse) { + const metricSuggestion = createMetricSuggestion(indexPattern, layerId, state, field); + if (metricSuggestion) { + suggestions.push(metricSuggestion); + } } return suggestions; @@ -514,8 +517,11 @@ function createAlternativeMetricSuggestions( ) { const layer = state.layers[layerId]; const suggestions: Array> = []; + const topLevelMetricColumns = layer.columnOrder.filter( + (columnId) => !isReferenced(layer, columnId) + ); - layer.columnOrder.forEach((columnId) => { + topLevelMetricColumns.forEach((columnId) => { const column = layer.columns[columnId]; if (!hasField(column)) { return; @@ -580,11 +586,13 @@ function createSuggestionWithDefaultDateHistogram( function createSimplifiedTableSuggestions(state: IndexPatternPrivateState, layerId: string) { const layer = state.layers[layerId]; - const [ - availableBucketedColumns, - availableMetricColumns, - availableReferenceColumns, - ] = getExistingColumnGroups(layer); + const [availableBucketedColumns, availableMetricColumns] = partition( + layer.columnOrder, + (colId) => layer.columns[colId].isBucketed + ); + const topLevelMetricColumns = availableMetricColumns.filter( + (columnId) => !isReferenced(layer, columnId) + ); return flatten( availableBucketedColumns.map((_col, index) => { @@ -593,46 +601,60 @@ function createSimplifiedTableSuggestions(state: IndexPatternPrivateState, layer const allMetricsSuggestion = { ...layer, columnOrder: [...bucketedColumns, ...availableMetricColumns], + noBuckets: false, }; - if (availableBucketedColumns.length <= 1 || availableReferenceColumns.length) { - // Don't simplify when dealing with single-bucket table. Also don't break - // reference-based columns by removing buckets. + if (availableBucketedColumns.length <= 1) { + // Don't simplify when dealing with single-bucket table. return []; - } else if (availableMetricColumns.length > 1) { - return [{ ...layer, columnOrder: [...bucketedColumns, availableMetricColumns[0]] }]; + } else if (topLevelMetricColumns.length > 1) { + return [ + { + ...layer, + columnOrder: [ + ...bucketedColumns, + topLevelMetricColumns[0], + ...getReferencedColumnIds(layer, topLevelMetricColumns[0]), + ], + noBuckets: false, + }, + ]; } else { return allMetricsSuggestion; } }) ) .concat( - availableReferenceColumns.length - ? [] - : availableMetricColumns.map((columnId) => { - return { ...layer, columnOrder: [columnId] }; + // if there is just a single top level metric, the unchanged suggestion will take care of this case - only split up if there are multiple metrics or at least one bucket + availableBucketedColumns.length > 0 || topLevelMetricColumns.length > 1 + ? topLevelMetricColumns.map((columnId) => { + return { + ...layer, + columnOrder: [columnId, ...getReferencedColumnIds(layer, columnId)], + noBuckets: true, + }; }) + : [] ) - .map((updatedLayer) => { + .map(({ noBuckets, ...updatedLayer }) => { return buildSuggestion({ state, layerId, updatedLayer, changeType: 'reduced', - label: - updatedLayer.columnOrder.length === 1 - ? getMetricSuggestionTitle(updatedLayer, availableMetricColumns.length === 1) - : undefined, + label: noBuckets + ? getMetricSuggestionTitle(updatedLayer, availableMetricColumns.length === 1) + : undefined, }); }); } -function getMetricSuggestionTitle(layer: IndexPatternLayer, onlyMetric: boolean) { +function getMetricSuggestionTitle(layer: IndexPatternLayer, onlySimpleMetric: boolean) { const { operationType, label } = layer.columns[layer.columnOrder[0]]; return i18n.translate('xpack.lens.indexpattern.suggestions.overallLabel', { defaultMessage: '{operation} overall', values: { - operation: onlyMetric ? operationDefinitionMap[operationType].displayName : label, + operation: onlySimpleMetric ? operationDefinitionMap[operationType].displayName : label, }, description: 'Title of a suggested chart containing only a single numerical metric calculated over all available data', diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/__mocks__/index.ts b/x-pack/plugins/lens/public/indexpattern_datasource/operations/__mocks__/index.ts index 40d7e3ef94ad68..d6429fb67e9a1d 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/operations/__mocks__/index.ts +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/__mocks__/index.ts @@ -17,6 +17,7 @@ jest.spyOn(actualHelpers, 'insertOrReplaceColumn'); jest.spyOn(actualHelpers, 'insertNewColumn'); jest.spyOn(actualHelpers, 'replaceColumn'); jest.spyOn(actualHelpers, 'getErrorMessages'); +jest.spyOn(actualHelpers, 'getColumnOrder'); export const { getAvailableOperationsByMetadata, @@ -48,6 +49,8 @@ export const { resetIncomplete, isOperationAllowedAsReference, canTransition, + isColumnValidAsReference, + getManagedColumnsFrom, } = actualHelpers; export const { adjustTimeScaleLabelSuffix, DEFAULT_TIME_SCALE } = actualTimeScaleUtils; diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/calculations/counter_rate.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/calculations/counter_rate.tsx index 823ec3eb58a924..396eae9b39c412 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/calculations/counter_rate.tsx +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/calculations/counter_rate.tsx @@ -121,5 +121,23 @@ export const counterRateOperation: OperationDefinition< }, timeScalingMode: 'mandatory', filterable: true, + documentation: { + section: 'calculation', + signature: i18n.translate('xpack.lens.indexPattern.counterRate.signature', { + defaultMessage: 'metric: number', + }), + description: i18n.translate('xpack.lens.indexPattern.counterRate.documentation', { + defaultMessage: ` +Calculates the rate of an ever increasing counter. This function will only yield helpful results on counter metric fields which contain a measurement of some kind monotonically growing over time. +If the value does get smaller, it will interpret this as a counter reset. To get most precise results, \`counter_rate\` should be calculated on the \`max\` of a field. + +This calculation will be done separately for separate series defined by filters or top values dimensions. +It uses the current interval when used in Formula. + +Example: Visualize the rate of bytes received over time by a memcached server: +\`counter_rate(max(memcached.stats.read.bytes))\` + `, + }), + }, shiftable: true, }; diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/calculations/cumulative_sum.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/calculations/cumulative_sum.tsx index c4f01e27be886f..f39e5587b398c0 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/calculations/cumulative_sum.tsx +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/calculations/cumulative_sum.tsx @@ -117,5 +117,21 @@ export const cumulativeSumOperation: OperationDefinition< )?.join(', '); }, filterable: true, + documentation: { + section: 'calculation', + signature: i18n.translate('xpack.lens.indexPattern.cumulative_sum.signature', { + defaultMessage: 'metric: number', + }), + description: i18n.translate('xpack.lens.indexPattern.cumulativeSum.documentation', { + defaultMessage: ` +Calculates the cumulative sum of a metric over time, adding all previous values of a series to each value. To use this function, you need to configure a date histogram dimension as well. + +This calculation will be done separately for separate series defined by filters or top values dimensions. + +Example: Visualize the received bytes accumulated over time: +\`cumulative_sum(sum(bytes))\` + `, + }), + }, shiftable: true, }; diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/calculations/differences.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/calculations/differences.tsx index 7c48b5742b8dbb..e103acd9ab6779 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/calculations/differences.tsx +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/calculations/differences.tsx @@ -109,5 +109,22 @@ export const derivativeOperation: OperationDefinition< }, timeScalingMode: 'optional', filterable: true, + documentation: { + section: 'calculation', + signature: i18n.translate('xpack.lens.indexPattern.differences.signature', { + defaultMessage: 'metric: number', + }), + description: i18n.translate('xpack.lens.indexPattern.differences.documentation', { + defaultMessage: ` +Calculates the difference to the last value of a metric over time. To use this function, you need to configure a date histogram dimension as well. +Differences requires the data to be sequential. If your data is empty when using differences, try increasing the date histogram interval. + +This calculation will be done separately for separate series defined by filters or top values dimensions. + +Example: Visualize the change in bytes received over time: +\`differences(sum(bytes))\` + `, + }), + }, shiftable: true, }; diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/calculations/moving_average.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/calculations/moving_average.tsx index a3d0241d4887e9..ee305bc043f1be 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/calculations/moving_average.tsx +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/calculations/moving_average.tsx @@ -65,7 +65,9 @@ export const movingAverageOperation: OperationDefinition< validateMetadata: (meta) => meta.dataType === 'number' && !meta.isBucketed, }, ], - operationParams: [{ name: 'window', type: 'number', required: true }], + operationParams: [ + { name: 'window', type: 'number', required: false, defaultValue: WINDOW_DEFAULT_VALUE }, + ], getPossibleOperation: (indexPattern) => { if (hasDateField(indexPattern)) { return { @@ -130,6 +132,28 @@ export const movingAverageOperation: OperationDefinition< }, timeScalingMode: 'optional', filterable: true, + documentation: { + section: 'calculation', + signature: i18n.translate('xpack.lens.indexPattern.moving_average.signature', { + defaultMessage: 'metric: number, [window]: number', + }), + description: i18n.translate('xpack.lens.indexPattern.movingAverage.documentation', { + defaultMessage: ` +Calculates the moving average of a metric over time, averaging the last n-th values to calculate the current value. To use this function, you need to configure a date histogram dimension as well. +The default window value is {defaultValue}. + +This calculation will be done separately for separate series defined by filters or top values dimensions. + +Takes a named parameter \`window\` which specifies how many last values to include in the average calculation for the current value. + +Example: Smooth a line of measurements: +\`moving_average(sum(bytes), window=5)\` + `, + values: { + defaultValue: WINDOW_DEFAULT_VALUE, + }, + }), + }, shiftable: true, }; diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/cardinality.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/cardinality.tsx index 1911af0a6f679b..4da8a3ca0eb24e 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/cardinality.tsx +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/cardinality.tsx @@ -116,4 +116,21 @@ export const cardinalityOperation: OperationDefinition { diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/filters/filters.test.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/filters/filters.test.tsx index 46fddd9b1ffbf4..75068817c61237 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/filters/filters.test.tsx +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/filters/filters.test.tsx @@ -28,6 +28,9 @@ const defaultProps = { http: {} as HttpSetup, indexPattern: createMockedIndexPattern(), operationDefinitionMap: {}, + isFullscreen: false, + toggleFullscreen: jest.fn(), + setIsCloseable: jest.fn(), }; // mocking random id generator function diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/formula.scss b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/formula.scss new file mode 100644 index 00000000000000..14b3fc33efb4e6 --- /dev/null +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/formula.scss @@ -0,0 +1,167 @@ +.lnsFormula { + display: flex; + flex-direction: column; + + .lnsIndexPatternDimensionEditor-isFullscreen & { + height: 100%; + } + + & > * { + flex: 1; + min-height: 0; + } + + & > * + * { + border-top: $euiBorderThin; + } +} + +.lnsFormula__editor { + border-bottom: $euiBorderThin; + + .lnsIndexPatternDimensionEditor-isFullscreen & { + border-bottom: none; + display: flex; + flex-direction: column; + } + + & > * + * { + border-top: $euiBorderThin; + } +} + +.lnsFormula__editorHeader, +.lnsFormula__editorFooter { + padding: $euiSizeS; +} + +.lnsFormula__editorFooter { + // make sure docs are rendered in front of monaco + z-index: 1; + background-color: $euiColorLightestShade; +} + +.lnsFormula__editorHeaderGroup, +.lnsFormula__editorFooterGroup { + display: block; // Overrides EUI's styling of `display: flex` on `EuiFlexItem` components +} + +.lnsFormula__editorContent { + position: relative; + height: 201px; +} + +.lnsFormula__editorPlaceholder { + position: absolute; + top: 0; + left: $euiSize; + right: 0; + color: $euiTextSubduedColor; + // Matches monaco editor + font-family: Menlo, Monaco, 'Courier New', monospace; + pointer-events: none; +} + +.lnsIndexPatternDimensionEditor-isFullscreen .lnsFormula__editorContent { + flex: 1; + min-height: 201px; +} + +.lnsFormula__warningText + .lnsFormula__warningText { + margin-top: $euiSizeS; + border-top: $euiBorderThin; + padding-top: $euiSizeS; +} + +.lnsFormula__editorHelp--inline { + align-items: center; + display: flex; + padding: $euiSizeXS; + + & > * + * { + margin-left: $euiSizeXS; + } +} + +.lnsFormula__editorError { + white-space: nowrap; +} + +.lnsFormula__docs { + background: $euiColorEmptyShade; +} + +.lnsFormula__docs--inline { + display: flex; + flex-direction: column; + // make sure docs are rendered in front of monaco + z-index: 1; +} + +.lnsFormula__docsContent { + .lnsFormula__docs--overlay & { + height: 40vh; + width: #{'min(75vh, 90vw)'}; + } + + .lnsFormula__docs--inline & { + flex: 1; + min-height: 0; + } + + & > * + * { + border-left: $euiBorderThin; + } +} + +.lnsFormula__docsSidebar { + background: $euiColorLightestShade; +} + +.lnsFormula__docsSidebarInner { + min-height: 0; + + & > * + * { + border-top: $euiBorderThin; + } +} + +.lnsFormula__docsSearch { + padding: $euiSizeS; +} + +.lnsFormula__docsNav { + @include euiYScroll; +} + +.lnsFormula__docsNavGroup { + padding: $euiSizeS; + + & + & { + border-top: $euiBorderThin; + } +} + +.lnsFormula__docsNavGroupLink { + font-weight: inherit; +} + +.lnsFormula__docsText { + @include euiYScroll; + padding: $euiSize; +} + +.lnsFormula__docsTextGroup, +.lnsFormula__docsTextItem { + margin-top: $euiSizeXXL; +} + +.lnsFormula__docsTextGroup { + border-top: $euiBorderThin; + padding-top: $euiSizeXXL; +} + +.lnsFormulaOverflow { + // Needs to be higher than the modal and all flyouts + z-index: $euiZLevel9 + 1; +} diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/formula_editor.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/formula_editor.tsx new file mode 100644 index 00000000000000..312ceb116dcede --- /dev/null +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/formula_editor.tsx @@ -0,0 +1,791 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import React, { useCallback, useEffect, useState, useMemo } from 'react'; +import { i18n } from '@kbn/i18n'; +import { + EuiButtonIcon, + EuiButtonEmpty, + EuiFlexGroup, + EuiFlexItem, + EuiIcon, + EuiLink, + EuiPopover, + EuiText, + EuiToolTip, + EuiSpacer, +} from '@elastic/eui'; +import useUnmount from 'react-use/lib/useUnmount'; +import { monaco } from '@kbn/monaco'; +import classNames from 'classnames'; +import { CodeEditor } from '../../../../../../../../../src/plugins/kibana_react/public'; +import type { CodeEditorProps } from '../../../../../../../../../src/plugins/kibana_react/public'; +import { useDebounceWithOptions } from '../../../../../shared_components'; +import { ParamEditorProps } from '../../index'; +import { getManagedColumnsFrom } from '../../../layer_helpers'; +import { ErrorWrapper, runASTValidation, tryToParse } from '../validation'; +import { + LensMathSuggestion, + SUGGESTION_TYPE, + suggest, + getSuggestion, + getSignatureHelp, + getHover, + getTokenInfo, + offsetToRowColumn, + monacoPositionToOffset, +} from './math_completion'; +import { LANGUAGE_ID } from './math_tokenization'; +import { MemoizedFormulaHelp } from './formula_help'; +import { trackUiEvent } from '../../../../../lens_ui_telemetry'; + +import './formula.scss'; +import { FormulaIndexPatternColumn } from '../formula'; +import { regenerateLayerFromAst } from '../parse'; +import { filterByVisibleOperation } from '../util'; + +export const MemoizedFormulaEditor = React.memo(FormulaEditor); + +export function FormulaEditor({ + layer, + updateLayer, + currentColumn, + columnId, + indexPattern, + operationDefinitionMap, + data, + toggleFullscreen, + isFullscreen, + setIsCloseable, +}: ParamEditorProps) { + const [text, setText] = useState(currentColumn.params.formula); + const [warnings, setWarnings] = useState< + Array<{ severity: monaco.MarkerSeverity; message: string }> + >([]); + const [isHelpOpen, setIsHelpOpen] = useState(isFullscreen); + const [isWarningOpen, setIsWarningOpen] = useState(false); + const [isWordWrapped, toggleWordWrap] = useState(true); + const editorModel = React.useRef(); + const overflowDiv1 = React.useRef(); + const disposables = React.useRef([]); + const editor1 = React.useRef(); + + const visibleOperationsMap = useMemo(() => filterByVisibleOperation(operationDefinitionMap), [ + operationDefinitionMap, + ]); + + // The Monaco editor needs to have the overflowDiv in the first render. Using an effect + // requires a second render to work, so we are using an if statement to guarantee it happens + // on first render + if (!overflowDiv1?.current) { + const node1 = (overflowDiv1.current = document.createElement('div')); + node1.setAttribute('data-test-subj', 'lnsFormulaWidget'); + // Monaco CSS is targeted on the monaco-editor class + node1.classList.add('lnsFormulaOverflow', 'monaco-editor'); + document.body.appendChild(node1); + } + + // Clean up the monaco editor and DOM on unmount + useEffect(() => { + const model = editorModel; + const allDisposables = disposables; + const editor1ref = editor1; + return () => { + model.current?.dispose(); + overflowDiv1.current?.parentNode?.removeChild(overflowDiv1.current); + editor1ref.current?.dispose(); + allDisposables.current?.forEach((d) => d.dispose()); + }; + }, []); + + useUnmount(() => { + setIsCloseable(true); + // If the text is not synced, update the column. + if (text !== currentColumn.params.formula) { + updateLayer((prevLayer) => { + return regenerateLayerFromAst( + text || '', + prevLayer, + columnId, + currentColumn, + indexPattern, + operationDefinitionMap + ).newLayer; + }); + } + }); + + useDebounceWithOptions( + () => { + if (!editorModel.current) return; + + if (!text) { + setWarnings([]); + monaco.editor.setModelMarkers(editorModel.current, 'LENS', []); + if (currentColumn.params.formula) { + // Only submit if valid + const { newLayer } = regenerateLayerFromAst( + text || '', + layer, + columnId, + currentColumn, + indexPattern, + operationDefinitionMap + ); + updateLayer(newLayer); + } + + return; + } + + let errors: ErrorWrapper[] = []; + + const { root, error } = tryToParse(text, visibleOperationsMap); + if (error) { + errors = [error]; + } else if (root) { + const validationErrors = runASTValidation(root, layer, indexPattern, visibleOperationsMap); + if (validationErrors.length) { + errors = validationErrors; + } + } + + if (errors.length) { + if (currentColumn.params.isFormulaBroken) { + // If the formula is already broken, show the latest error message in the workspace + if (currentColumn.params.formula !== text) { + updateLayer( + regenerateLayerFromAst( + text || '', + layer, + columnId, + currentColumn, + indexPattern, + visibleOperationsMap + ).newLayer + ); + } + } + + const markers = errors.flatMap((innerError) => { + if (innerError.locations.length) { + return innerError.locations.map((location) => { + const startPosition = offsetToRowColumn(text, location.min); + const endPosition = offsetToRowColumn(text, location.max); + return { + message: innerError.message, + startColumn: startPosition.column + 1, + startLineNumber: startPosition.lineNumber, + endColumn: endPosition.column + 1, + endLineNumber: endPosition.lineNumber, + severity: + innerError.severity === 'warning' + ? monaco.MarkerSeverity.Warning + : monaco.MarkerSeverity.Error, + }; + }); + } else { + // Parse errors return no location info + const startPosition = offsetToRowColumn(text, 0); + const endPosition = offsetToRowColumn(text, text.length - 1); + return [ + { + message: innerError.message, + startColumn: startPosition.column + 1, + startLineNumber: startPosition.lineNumber, + endColumn: endPosition.column + 1, + endLineNumber: endPosition.lineNumber, + severity: + innerError.severity === 'warning' + ? monaco.MarkerSeverity.Warning + : monaco.MarkerSeverity.Error, + }, + ]; + } + }); + + monaco.editor.setModelMarkers(editorModel.current, 'LENS', markers); + setWarnings(markers.map(({ severity, message }) => ({ severity, message }))); + } else { + monaco.editor.setModelMarkers(editorModel.current, 'LENS', []); + + // Only submit if valid + const { newLayer, locations } = regenerateLayerFromAst( + text || '', + layer, + columnId, + currentColumn, + indexPattern, + visibleOperationsMap + ); + updateLayer(newLayer); + + const managedColumns = getManagedColumnsFrom(columnId, newLayer.columns); + const markers: monaco.editor.IMarkerData[] = managedColumns + .flatMap(([id, column]) => { + if (locations[id]) { + const def = visibleOperationsMap[column.operationType]; + if (def.getErrorMessage) { + const messages = def.getErrorMessage( + newLayer, + id, + indexPattern, + visibleOperationsMap + ); + if (messages) { + const startPosition = offsetToRowColumn(text, locations[id].min); + const endPosition = offsetToRowColumn(text, locations[id].max); + return [ + { + message: messages.join(', '), + startColumn: startPosition.column + 1, + startLineNumber: startPosition.lineNumber, + endColumn: endPosition.column + 1, + endLineNumber: endPosition.lineNumber, + severity: monaco.MarkerSeverity.Warning, + }, + ]; + } + } + } + return []; + }) + .filter((marker) => marker); + setWarnings(markers.map(({ severity, message }) => ({ severity, message }))); + monaco.editor.setModelMarkers(editorModel.current, 'LENS', markers); + } + }, + // Make it validate on flyout open in case of a broken formula left over + // from a previous edit + { skipFirstRender: false }, + 256, + [text] + ); + + const errorCount = warnings.filter((marker) => marker.severity === monaco.MarkerSeverity.Error) + .length; + const warningCount = warnings.filter( + (marker) => marker.severity === monaco.MarkerSeverity.Warning + ).length; + + /** + * The way that Monaco requests autocompletion is not intuitive, but the way we use it + * we fetch new suggestions in these scenarios: + * + * - If the user types one of the trigger characters, suggestions are always fetched + * - When the user selects the kql= suggestion, we tell Monaco to trigger new suggestions after + * - When the user types the first character into an empty text box, Monaco requests suggestions + * + * Monaco also triggers suggestions automatically when there are no suggestions being displayed + * and the user types a non-whitespace character. + * + * While suggestions are being displayed, Monaco uses an in-memory cache of the last known suggestions. + */ + const provideCompletionItems = useCallback( + async ( + model: monaco.editor.ITextModel, + position: monaco.Position, + context: monaco.languages.CompletionContext + ) => { + const innerText = model.getValue(); + let aSuggestions: { list: LensMathSuggestion[]; type: SUGGESTION_TYPE } = { + list: [], + type: SUGGESTION_TYPE.FIELD, + }; + const offset = monacoPositionToOffset(innerText, position); + + if (context.triggerCharacter === '(') { + // Monaco usually inserts the end quote and reports the position is after the end quote + if (innerText.slice(offset - 1, offset + 1) === '()') { + position = position.delta(0, -1); + } + const wordUntil = model.getWordAtPosition(position.delta(0, -3)); + if (wordUntil) { + // Retrieve suggestions for subexpressions + aSuggestions = await suggest({ + expression: innerText, + zeroIndexedOffset: offset, + context, + indexPattern, + operationDefinitionMap: visibleOperationsMap, + data, + }); + } + } else { + aSuggestions = await suggest({ + expression: innerText, + zeroIndexedOffset: offset, + context, + indexPattern, + operationDefinitionMap: visibleOperationsMap, + data, + }); + } + + return { + suggestions: aSuggestions.list.map((s) => + getSuggestion(s, aSuggestions.type, visibleOperationsMap, context.triggerCharacter) + ), + }; + }, + [indexPattern, visibleOperationsMap, data] + ); + + const provideSignatureHelp = useCallback( + async ( + model: monaco.editor.ITextModel, + position: monaco.Position, + token: monaco.CancellationToken, + context: monaco.languages.SignatureHelpContext + ) => { + const innerText = model.getValue(); + const textRange = model.getFullModelRange(); + + const lengthAfterPosition = model.getValueLengthInRange({ + startLineNumber: position.lineNumber, + startColumn: position.column, + endLineNumber: textRange.endLineNumber, + endColumn: textRange.endColumn, + }); + return getSignatureHelp( + model.getValue(), + innerText.length - lengthAfterPosition, + visibleOperationsMap + ); + }, + [visibleOperationsMap] + ); + + const provideHover = useCallback( + async ( + model: monaco.editor.ITextModel, + position: monaco.Position, + token: monaco.CancellationToken + ) => { + const innerText = model.getValue(); + const textRange = model.getFullModelRange(); + + const lengthAfterPosition = model.getValueLengthInRange({ + startLineNumber: position.lineNumber, + startColumn: position.column, + endLineNumber: textRange.endLineNumber, + endColumn: textRange.endColumn, + }); + return getHover( + model.getValue(), + innerText.length - lengthAfterPosition, + visibleOperationsMap + ); + }, + [visibleOperationsMap] + ); + + const onTypeHandler = useCallback( + (e: monaco.editor.IModelContentChangedEvent, editor: monaco.editor.IStandaloneCodeEditor) => { + if (e.isFlush || e.isRedoing || e.isUndoing) { + return; + } + if (e.changes.length === 1) { + const char = e.changes[0].text; + if (char !== '=' && char !== "'") { + return; + } + const currentPosition = e.changes[0].range; + if (currentPosition) { + const currentText = editor.getValue(); + const offset = monacoPositionToOffset( + currentText, + new monaco.Position(currentPosition.startLineNumber, currentPosition.startColumn) + ); + let tokenInfo = getTokenInfo(currentText, offset + 1); + + if (!tokenInfo && char === "'") { + // try again this time replacing the current quote with an escaped quote + const line = currentText; + const lineEscaped = line.substring(0, offset) + "\\'" + line.substring(offset + 1); + tokenInfo = getTokenInfo(lineEscaped, offset + 2); + } + + const isSingleQuoteCase = /'LENS_MATH_MARKER/; + // Make sure that we are only adding kql='' or lucene='', and also + // check that the = sign isn't inside the KQL expression like kql='=' + if ( + !tokenInfo || + typeof tokenInfo.ast === 'number' || + tokenInfo.ast.type !== 'namedArgument' || + (tokenInfo.ast.name !== 'kql' && tokenInfo.ast.name !== 'lucene') || + (tokenInfo.ast.value !== 'LENS_MATH_MARKER' && + !isSingleQuoteCase.test(tokenInfo.ast.value)) + ) { + return; + } + + let editOperation: monaco.editor.IIdentifiedSingleEditOperation | null = null; + const cursorOffset = 2; + if (char === '=') { + editOperation = { + range: { + ...currentPosition, + // Insert after the current char + startColumn: currentPosition.startColumn + 1, + endColumn: currentPosition.startColumn + 1, + }, + text: `''`, + }; + } + if (char === "'") { + editOperation = { + range: { + ...currentPosition, + // Insert after the current char + startColumn: currentPosition.startColumn, + endColumn: currentPosition.startColumn + 1, + }, + text: `\\'`, + }; + } + + if (editOperation) { + setTimeout(() => { + editor.executeEdits( + 'LENS', + [editOperation!], + [ + // After inserting, move the cursor in between the single quotes or after the escaped quote + new monaco.Selection( + currentPosition.startLineNumber, + currentPosition.startColumn + cursorOffset, + currentPosition.startLineNumber, + currentPosition.startColumn + cursorOffset + ), + ] + ); + + // Need to move these sync to prevent race conditions between a fast user typing a single quote + // after an = char + // Timeout is required because otherwise the cursor position is not updated. + editor.setPosition({ + column: currentPosition.startColumn + cursorOffset, + lineNumber: currentPosition.startLineNumber, + }); + editor.trigger('lens', 'editor.action.triggerSuggest', {}); + }, 0); + } + } + } + }, + [] + ); + + const codeEditorOptions: CodeEditorProps = { + languageId: LANGUAGE_ID, + value: text ?? '', + onChange: setText, + options: { + automaticLayout: false, + fontSize: 14, + folding: false, + lineNumbers: 'off', + scrollBeyondLastLine: false, + minimap: { enabled: false }, + wordWrap: isWordWrapped ? 'on' : 'off', + // Disable suggestions that appear when we don't provide a default suggestion + wordBasedSuggestions: false, + autoIndent: 'brackets', + wrappingIndent: 'none', + dimension: { width: 320, height: 200 }, + fixedOverflowWidgets: true, + matchBrackets: 'always', + }, + }; + + useEffect(() => { + // Because the monaco model is owned by Lens, we need to manually attach and remove handlers + const { dispose: dispose1 } = monaco.languages.registerCompletionItemProvider(LANGUAGE_ID, { + triggerCharacters: ['.', '(', '=', ' ', ':', `'`], + provideCompletionItems, + }); + const { dispose: dispose2 } = monaco.languages.registerSignatureHelpProvider(LANGUAGE_ID, { + signatureHelpTriggerCharacters: ['(', '='], + provideSignatureHelp, + }); + const { dispose: dispose3 } = monaco.languages.registerHoverProvider(LANGUAGE_ID, { + provideHover, + }); + return () => { + dispose1(); + dispose2(); + dispose3(); + }; + }, [provideCompletionItems, provideSignatureHelp, provideHover]); + + // The Monaco editor will lazily load Monaco, which takes a render cycle to trigger. This can cause differences + // in the behavior of Monaco when it's first loaded and then reloaded. + return ( +
+
+
+
+
+ + + {/* TODO: Replace `bolt` with `wordWrap` icon (after latest EUI is deployed) and hook up button to enable/disable word wrapping. */} + + { + editor1.current?.updateOptions({ + wordWrap: isWordWrapped ? 'off' : 'on', + }); + toggleWordWrap(!isWordWrapped); + }} + /> + + + + + {/* TODO: Replace `bolt` with `fullScreenExit` icon (after latest EUI is deployed). */} + { + toggleFullscreen(); + // Help text opens when entering full screen, and closes when leaving full screen + setIsHelpOpen(!isFullscreen); + trackUiEvent('toggle_formula_fullscreen'); + }} + iconType={isFullscreen ? 'bolt' : 'fullScreen'} + size="xs" + color="text" + flush="right" + data-test-subj="lnsFormula-fullscreen" + > + {isFullscreen + ? i18n.translate('xpack.lens.formula.fullScreenExitLabel', { + defaultMessage: 'Collapse', + }) + : i18n.translate('xpack.lens.formula.fullScreenEnterLabel', { + defaultMessage: 'Expand', + })} + + + +
+ +
+ { + editor1.current = editor; + const model = editor.getModel(); + if (model) { + editorModel.current = model; + } + disposables.current.push( + editor.onDidFocusEditorWidget(() => { + setTimeout(() => { + setIsCloseable(false); + }); + }) + ); + disposables.current.push( + editor.onDidBlurEditorWidget(() => { + setTimeout(() => { + setIsCloseable(true); + }); + }) + ); + // If we ever introduce a second Monaco editor, we need to toggle + // the typing handler to the active editor to maintain the cursor + disposables.current.push( + editor.onDidChangeModelContent((e) => { + onTypeHandler(e, editor); + }) + ); + }} + /> + + {!text ? ( +
+ + {i18n.translate('xpack.lens.formulaPlaceholderText', { + defaultMessage: 'Type a formula by combining functions with math, like:', + })} + + + 
count() + 1
+
+ ) : null} +
+ +
+ + + {isFullscreen ? ( + + setIsHelpOpen(!isHelpOpen)} + > + + + + + ) : ( + + setIsHelpOpen(false)} + ownFocus={false} + button={ + setIsHelpOpen(!isHelpOpen)} + iconType="help" + color="text" + size="s" + aria-label={i18n.translate( + 'xpack.lens.formula.editorHelpInlineShowToolTip', + { + defaultMessage: 'Show function reference', + } + )} + /> + } + > + + + + )} + + + {errorCount || warningCount ? ( + + setIsWarningOpen(false)} + button={ + { + setIsWarningOpen(!isWarningOpen); + }} + > + {errorCount + ? i18n.translate('xpack.lens.formulaErrorCount', { + defaultMessage: + '{count} {count, plural, one {error} other {errors}}', + values: { count: errorCount }, + }) + : null} + {warningCount + ? i18n.translate('xpack.lens.formulaWarningCount', { + defaultMessage: + '{count} {count, plural, one {warning} other {warnings}}', + values: { count: warningCount }, + }) + : null} + + } + > + {warnings.map(({ message, severity }, index) => ( +
+ + {message} + +
+ ))} + + + ) : null} + +
+
+ + {isFullscreen && isHelpOpen ? ( +
+ +
+ ) : null} +
+
+
+ ); +} diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/formula_help.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/formula_help.tsx new file mode 100644 index 00000000000000..afe5471666b22b --- /dev/null +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/formula_help.tsx @@ -0,0 +1,469 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import React, { useEffect, useRef, useState, useMemo } from 'react'; +import { i18n } from '@kbn/i18n'; +import { + EuiFlexGroup, + EuiFlexItem, + EuiLink, + EuiPopoverTitle, + EuiText, + EuiListGroupItem, + EuiListGroup, + EuiTitle, + EuiFieldSearch, + EuiHighlight, +} from '@elastic/eui'; +import { Markdown } from '../../../../../../../../../src/plugins/kibana_react/public'; +import { IndexPattern } from '../../../../types'; +import { tinymathFunctions } from '../util'; +import { getPossibleFunctions } from './math_completion'; +import { hasFunctionFieldArgument } from '../validation'; + +import type { + GenericOperationDefinition, + IndexPatternColumn, + OperationDefinition, + ParamEditorProps, +} from '../../index'; +import type { FormulaIndexPatternColumn } from '../formula'; + +function FormulaHelp({ + indexPattern, + operationDefinitionMap, + isFullscreen, +}: { + indexPattern: IndexPattern; + operationDefinitionMap: Record; + isFullscreen: boolean; +}) { + const [selectedFunction, setSelectedFunction] = useState(); + const scrollTargets = useRef>({}); + + useEffect(() => { + if (selectedFunction && scrollTargets.current[selectedFunction]) { + scrollTargets.current[selectedFunction].scrollIntoView(); + } + }, [selectedFunction]); + + const helpGroups: Array<{ + label: string; + description?: string; + items: Array<{ label: string; description?: JSX.Element }>; + }> = []; + + helpGroups.push({ + label: i18n.translate('xpack.lens.formulaDocumentationHeading', { + defaultMessage: 'How it works', + }), + items: [], + }); + + helpGroups.push({ + label: i18n.translate('xpack.lens.formulaDocumentation.elasticsearchSection', { + defaultMessage: 'Elasticsearch', + }), + description: i18n.translate('xpack.lens.formulaDocumentation.elasticsearchSectionDescription', { + defaultMessage: + 'These functions will be executed on the raw documents for each row of the resulting table, aggregating all documents matching the break down dimensions into a single value.', + }), + items: [], + }); + + const availableFunctions = getPossibleFunctions(indexPattern); + + // Es aggs + helpGroups[1].items.push( + ...availableFunctions + .filter( + (key) => + key in operationDefinitionMap && + operationDefinitionMap[key].documentation?.section === 'elasticsearch' + ) + .sort() + .map((key) => ({ + label: key, + description: ( + <> +

+ {key}({operationDefinitionMap[key].documentation?.signature}) +

+ + {operationDefinitionMap[key].documentation?.description ? ( + + ) : null} + + ), + })) + ); + + helpGroups.push({ + label: i18n.translate('xpack.lens.formulaDocumentation.columnCalculationSection', { + defaultMessage: 'Column-wise calculation', + }), + description: i18n.translate( + 'xpack.lens.formulaDocumentation.columnCalculationSectionDescription', + { + defaultMessage: + 'These functions will be executed for reach row of the resulting table, using data from cells from other rows as well as the current value.', + } + ), + items: [], + }); + + // Calculations aggs + helpGroups[2].items.push( + ...availableFunctions + .filter( + (key) => + key in operationDefinitionMap && + operationDefinitionMap[key].documentation?.section === 'calculation' + ) + .sort() + .map((key) => ({ + label: key, + description: ( + <> +

+ {key}({operationDefinitionMap[key].documentation?.signature}) +

+ + {operationDefinitionMap[key].documentation?.description ? ( + + ) : null} + + ), + checked: + selectedFunction === `${key}: ${operationDefinitionMap[key].displayName}` + ? ('on' as const) + : undefined, + })) + ); + + helpGroups.push({ + label: i18n.translate('xpack.lens.formulaDocumentation.mathSection', { + defaultMessage: 'Math', + }), + description: i18n.translate('xpack.lens.formulaDocumentation.mathSectionDescription', { + defaultMessage: + 'These functions will be executed for reach row of the resulting table using single values from the same row calculated using other functions.', + }), + items: [], + }); + + const tinymathFns = useMemo(() => { + return getPossibleFunctions(indexPattern) + .filter((key) => key in tinymathFunctions) + .sort() + .map((key) => { + const [description, examples] = tinymathFunctions[key].help.split(`\`\`\``); + return { + label: key, + description: description.replace(/\n/g, '\n\n'), + examples: examples ? `\`\`\`${examples}\`\`\`` : '', + }; + }); + }, [indexPattern]); + + helpGroups[3].items.push( + ...tinymathFns.map(({ label, description, examples }) => { + return { + label, + description: ( + <> +

{getFunctionSignatureLabel(label, operationDefinitionMap)}

+ + + + ), + }; + }) + ); + + const [searchText, setSearchText] = useState(''); + + const normalizedSearchText = searchText.trim().toLocaleLowerCase(); + + const filteredHelpGroups = helpGroups + .map((group) => { + const items = group.items.filter((helpItem) => { + return ( + !normalizedSearchText || helpItem.label.toLocaleLowerCase().includes(normalizedSearchText) + ); + }); + return { ...group, items }; + }) + .filter((group) => { + if (group.items.length > 0 || !normalizedSearchText) { + return true; + } + return group.label.toLocaleLowerCase().includes(normalizedSearchText); + }); + + return ( + <> + + {i18n.translate('xpack.lens.formulaDocumentation.header', { + defaultMessage: 'Formula reference', + })} + + + + + + + { + setSearchText(e.target.value); + }} + placeholder={i18n.translate('xpack.lens.formulaSearchPlaceholder', { + defaultMessage: 'Search functions', + })} + /> + + + + {filteredHelpGroups.map((helpGroup, index) => { + return ( + + ); + })} + + + + + + +
{ + if (el) { + scrollTargets.current[helpGroups[0].label] = el; + } + }} + > + +
+ + {helpGroups.slice(1).map((helpGroup, index) => { + return ( +
{ + if (el) { + scrollTargets.current[helpGroup.label] = el; + } + }} + > +

{helpGroup.label}

+ +

{helpGroup.description}

+ + {helpGroups[index + 1].items.map((helpItem) => { + return ( +
{ + if (el) { + scrollTargets.current[helpItem.label] = el; + } + }} + > + {helpItem.description} +
+ ); + })} +
+ ); + })} + + + + + ); +} + +export const MemoizedFormulaHelp = React.memo(FormulaHelp); + +export function getFunctionSignatureLabel( + name: string, + operationDefinitionMap: ParamEditorProps['operationDefinitionMap'], + firstParam?: { label: string | [number, number] } | null +): string { + if (tinymathFunctions[name]) { + return `${name}(${tinymathFunctions[name].positionalArguments + .map(({ name: argName, optional, type }) => `[${argName}]${optional ? '?' : ''}: ${type}`) + .join(', ')})`; + } + if (operationDefinitionMap[name]) { + const def = operationDefinitionMap[name]; + let extraArgs = ''; + if (def.filterable) { + extraArgs += hasFunctionFieldArgument(name) || 'operationParams' in def ? ',' : ''; + extraArgs += i18n.translate('xpack.lens.formula.kqlExtraArguments', { + defaultMessage: '[kql]?: string, [lucene]?: string', + }); + } + return `${name}(${def.documentation?.signature}${extraArgs})`; + } + return ''; +} + +function getFunctionArgumentsStringified( + params: Required< + OperationDefinition + >['operationParams'] +) { + return params + .map( + ({ name, type: argType, defaultValue = 5 }) => + `${name}=${argType === 'string' ? `"${defaultValue}"` : defaultValue}` + ) + .join(', '); +} + +/** + * Get an array of strings containing all possible information about a specific + * operation type: examples and infos. + */ +export function getHelpTextContent( + type: string, + operationDefinitionMap: ParamEditorProps['operationDefinitionMap'] +): { description: string; examples: string[] } { + const definition = operationDefinitionMap[type]; + const description = definition.documentation?.description ?? ''; + + // as for the time being just add examples text. + // Later will enrich with more information taken from the operation definitions. + const examples: string[] = []; + // If the description already contain examples skip it + if (!/Example/.test(description)) { + if (!hasFunctionFieldArgument(type)) { + // ideally this should have the same example automation as the operations below + examples.push(`${type}()`); + return { description, examples }; + } + if (definition.input === 'field') { + const mandatoryArgs = definition.operationParams?.filter(({ required }) => required) || []; + if (mandatoryArgs.length === 0) { + examples.push(`${type}(bytes)`); + } + if (mandatoryArgs.length) { + const additionalArgs = getFunctionArgumentsStringified(mandatoryArgs); + examples.push(`${type}(bytes, ${additionalArgs})`); + } + if ( + definition.operationParams && + mandatoryArgs.length !== definition.operationParams.length + ) { + const additionalArgs = getFunctionArgumentsStringified(definition.operationParams); + examples.push(`${type}(bytes, ${additionalArgs})`); + } + } + if (definition.input === 'fullReference') { + const mandatoryArgs = definition.operationParams?.filter(({ required }) => required) || []; + if (mandatoryArgs.length === 0) { + examples.push(`${type}(sum(bytes))`); + } + if (mandatoryArgs.length) { + const additionalArgs = getFunctionArgumentsStringified(mandatoryArgs); + examples.push(`${type}(sum(bytes), ${additionalArgs})`); + } + if ( + definition.operationParams && + mandatoryArgs.length !== definition.operationParams.length + ) { + const additionalArgs = getFunctionArgumentsStringified(definition.operationParams); + examples.push(`${type}(sum(bytes), ${additionalArgs})`); + } + } + } + return { description, examples }; +} diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/index.ts b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/index.ts new file mode 100644 index 00000000000000..4b6acefa6b30ad --- /dev/null +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/index.ts @@ -0,0 +1,8 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +export * from './formula_editor'; diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/math_completion.test.ts b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/math_completion.test.ts new file mode 100644 index 00000000000000..9cd748f5759c98 --- /dev/null +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/math_completion.test.ts @@ -0,0 +1,386 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { parse } from '@kbn/tinymath'; +import { monaco } from '@kbn/monaco'; +import { createMockedIndexPattern } from '../../../../mocks'; +import { GenericOperationDefinition } from '../../index'; +import type { IndexPatternField } from '../../../../types'; +import type { OperationMetadata } from '../../../../../types'; +import { dataPluginMock } from '../../../../../../../../../src/plugins/data/public/mocks'; +import { tinymathFunctions } from '../util'; +import { + getSignatureHelp, + getHover, + suggest, + monacoPositionToOffset, + getInfoAtZeroIndexedPosition, +} from './math_completion'; + +const buildGenericColumn = (type: string) => { + return ({ field }: { field?: IndexPatternField }) => { + return { + label: type, + dataType: 'number', + operationType: type, + sourceField: field?.name ?? undefined, + isBucketed: false, + scale: 'ratio', + timeScale: false, + }; + }; +}; + +const numericOperation = () => ({ dataType: 'number', isBucketed: false }); +const stringOperation = () => ({ dataType: 'string', isBucketed: true }); + +// Only one of each type is needed +const operationDefinitionMap: Record = { + sum: ({ + type: 'sum', + input: 'field', + buildColumn: buildGenericColumn('sum'), + getPossibleOperationForField: (field: IndexPatternField) => + field.type === 'number' ? numericOperation() : null, + documentation: { + section: 'elasticsearch', + signature: 'field: string', + description: 'description', + }, + } as unknown) as GenericOperationDefinition, + count: ({ + type: 'count', + input: 'field', + buildColumn: buildGenericColumn('count'), + getPossibleOperationForField: (field: IndexPatternField) => + field.name === 'Records' ? numericOperation() : null, + } as unknown) as GenericOperationDefinition, + last_value: ({ + type: 'last_value', + input: 'field', + buildColumn: buildGenericColumn('last_value'), + getPossibleOperationForField: (field: IndexPatternField) => ({ + dataType: field.type, + isBucketed: false, + }), + } as unknown) as GenericOperationDefinition, + moving_average: ({ + type: 'moving_average', + input: 'fullReference', + requiredReferences: [ + { + input: ['field', 'managedReference'], + validateMetadata: (meta: OperationMetadata) => + meta.dataType === 'number' && !meta.isBucketed, + }, + ], + operationParams: [{ name: 'window', type: 'number', required: true }], + buildColumn: buildGenericColumn('moving_average'), + getPossibleOperation: numericOperation, + } as unknown) as GenericOperationDefinition, + cumulative_sum: ({ + type: 'cumulative_sum', + input: 'fullReference', + buildColumn: buildGenericColumn('cumulative_sum'), + getPossibleOperation: numericOperation, + } as unknown) as GenericOperationDefinition, + terms: ({ + type: 'terms', + input: 'field', + getPossibleOperationForField: stringOperation, + } as unknown) as GenericOperationDefinition, +}; + +describe('math completion', () => { + describe('signature help', () => { + function unwrapSignatures(signatureResult: monaco.languages.SignatureHelpResult) { + return signatureResult.value.signatures[0]; + } + + it('should silently handle parse errors', () => { + expect(unwrapSignatures(getSignatureHelp('sum(', 4, operationDefinitionMap))).toBeUndefined(); + }); + + it('should return a signature for a field-based ES function', () => { + expect(unwrapSignatures(getSignatureHelp('sum()', 4, operationDefinitionMap))).toEqual({ + label: 'sum(field: string)', + documentation: { value: 'description' }, + parameters: [{ label: 'field' }], + }); + }); + + it('should return a signature for count', () => { + expect(unwrapSignatures(getSignatureHelp('count()', 6, operationDefinitionMap))).toEqual({ + label: 'count(undefined)', + documentation: { value: '' }, + parameters: [], + }); + }); + + it('should return a signature for a function with named parameters', () => { + expect( + unwrapSignatures( + getSignatureHelp('2 * moving_average(count(), window=)', 35, operationDefinitionMap) + ) + ).toEqual({ + label: expect.stringContaining('moving_average('), + documentation: { value: '' }, + parameters: [ + { label: 'function' }, + { + label: 'window=number', + documentation: 'Required', + }, + ], + }); + }); + + it('should return a signature for an inner function', () => { + expect( + unwrapSignatures( + getSignatureHelp('2 * moving_average(count())', 25, operationDefinitionMap) + ) + ).toEqual({ + label: expect.stringContaining('count('), + parameters: [], + documentation: { value: '' }, + }); + }); + + it('should return a signature for a complex tinymath function', () => { + // 15 is the whitespace between the two arguments + expect( + unwrapSignatures(getSignatureHelp('clamp(count(), 5)', 15, operationDefinitionMap)) + ).toEqual({ + label: expect.stringContaining('clamp('), + documentation: { value: '' }, + parameters: [ + { label: 'value', documentation: '' }, + { label: 'min', documentation: '' }, + { label: 'max', documentation: '' }, + ], + }); + }); + }); + + describe('hover provider', () => { + it('should silently handle parse errors', () => { + expect(getHover('sum(', 2, operationDefinitionMap)).toEqual({ contents: [] }); + }); + + it('should show signature for a field-based ES function', () => { + expect(getHover('sum()', 2, operationDefinitionMap)).toEqual({ + contents: [{ value: 'sum(field: string)' }], + }); + }); + + it('should show signature for count', () => { + expect(getHover('count()', 2, operationDefinitionMap)).toEqual({ + contents: [{ value: expect.stringContaining('count(') }], + }); + }); + + it('should show signature for a function with named parameters', () => { + expect(getHover('2 * moving_average(count())', 10, operationDefinitionMap)).toEqual({ + contents: [{ value: expect.stringContaining('moving_average(') }], + }); + }); + + it('should show signature for an inner function', () => { + expect(getHover('2 * moving_average(count())', 22, operationDefinitionMap)).toEqual({ + contents: [{ value: expect.stringContaining('count(') }], + }); + }); + + it('should show signature for a complex tinymath function', () => { + expect(getHover('clamp(count(), 5)', 2, operationDefinitionMap)).toEqual({ + contents: [{ value: expect.stringContaining('clamp([value]: number') }], + }); + }); + }); + + describe('autocomplete', () => { + it('should list all valid functions at the top level (fake test)', async () => { + // This test forces an invalid scenario, since the autocomplete actually requires + // some typing + const results = await suggest({ + expression: '', + zeroIndexedOffset: 1, + context: { + triggerKind: monaco.languages.CompletionTriggerKind.TriggerCharacter, + triggerCharacter: '', + }, + indexPattern: createMockedIndexPattern(), + operationDefinitionMap, + data: dataPluginMock.createStartContract(), + }); + expect(results.list).toHaveLength(4 + Object.keys(tinymathFunctions).length); + ['sum', 'moving_average', 'cumulative_sum', 'last_value'].forEach((key) => { + expect(results.list).toEqual(expect.arrayContaining([{ label: key, type: 'operation' }])); + }); + Object.keys(tinymathFunctions).forEach((key) => { + expect(results.list).toEqual(expect.arrayContaining([{ label: key, type: 'operation' }])); + }); + }); + + it('should list all valid sub-functions for a fullReference', async () => { + const results = await suggest({ + expression: 'moving_average()', + zeroIndexedOffset: 15, + context: { + triggerKind: monaco.languages.CompletionTriggerKind.TriggerCharacter, + triggerCharacter: '(', + }, + indexPattern: createMockedIndexPattern(), + operationDefinitionMap, + data: dataPluginMock.createStartContract(), + }); + expect(results.list).toHaveLength(2); + ['sum', 'last_value'].forEach((key) => { + expect(results.list).toEqual(expect.arrayContaining([{ label: key, type: 'operation' }])); + }); + }); + + it('should list all valid named arguments for a fullReference', async () => { + const results = await suggest({ + expression: 'moving_average(count(),)', + zeroIndexedOffset: 23, + context: { + triggerKind: monaco.languages.CompletionTriggerKind.TriggerCharacter, + triggerCharacter: ',', + }, + indexPattern: createMockedIndexPattern(), + operationDefinitionMap, + data: dataPluginMock.createStartContract(), + }); + expect(results.list).toEqual(['window']); + }); + + it('should not list named arguments when they are already in use', async () => { + const results = await suggest({ + expression: 'moving_average(count(), window=5, )', + zeroIndexedOffset: 34, + context: { + triggerKind: monaco.languages.CompletionTriggerKind.TriggerCharacter, + triggerCharacter: ',', + }, + indexPattern: createMockedIndexPattern(), + operationDefinitionMap, + data: dataPluginMock.createStartContract(), + }); + expect(results.list).toEqual([]); + }); + + it('should list all valid positional arguments for a tinymath function used by name', async () => { + const results = await suggest({ + expression: 'divide(count(), )', + zeroIndexedOffset: 16, + context: { + triggerKind: monaco.languages.CompletionTriggerKind.TriggerCharacter, + triggerCharacter: ',', + }, + indexPattern: createMockedIndexPattern(), + operationDefinitionMap, + data: dataPluginMock.createStartContract(), + }); + expect(results.list).toHaveLength(4 + Object.keys(tinymathFunctions).length); + ['sum', 'moving_average', 'cumulative_sum', 'last_value'].forEach((key) => { + expect(results.list).toEqual(expect.arrayContaining([{ label: key, type: 'math' }])); + }); + Object.keys(tinymathFunctions).forEach((key) => { + expect(results.list).toEqual(expect.arrayContaining([{ label: key, type: 'math' }])); + }); + }); + + it('should list all valid positional arguments for a tinymath function used with alias', async () => { + const results = await suggest({ + expression: 'count() / ', + zeroIndexedOffset: 10, + context: { + triggerKind: monaco.languages.CompletionTriggerKind.TriggerCharacter, + triggerCharacter: ',', + }, + indexPattern: createMockedIndexPattern(), + operationDefinitionMap, + data: dataPluginMock.createStartContract(), + }); + expect(results.list).toHaveLength(4 + Object.keys(tinymathFunctions).length); + ['sum', 'moving_average', 'cumulative_sum', 'last_value'].forEach((key) => { + expect(results.list).toEqual(expect.arrayContaining([{ label: key, type: 'math' }])); + }); + Object.keys(tinymathFunctions).forEach((key) => { + expect(results.list).toEqual(expect.arrayContaining([{ label: key, type: 'math' }])); + }); + }); + + it('should not autocomplete any fields for the count function', async () => { + const results = await suggest({ + expression: 'count()', + zeroIndexedOffset: 6, + context: { + triggerKind: monaco.languages.CompletionTriggerKind.TriggerCharacter, + triggerCharacter: '(', + }, + indexPattern: createMockedIndexPattern(), + operationDefinitionMap, + data: dataPluginMock.createStartContract(), + }); + expect(results.list).toHaveLength(0); + }); + + it('should autocomplete and validate the right type of field', async () => { + const results = await suggest({ + expression: 'sum()', + zeroIndexedOffset: 4, + context: { + triggerKind: monaco.languages.CompletionTriggerKind.TriggerCharacter, + triggerCharacter: '(', + }, + indexPattern: createMockedIndexPattern(), + operationDefinitionMap, + data: dataPluginMock.createStartContract(), + }); + expect(results.list).toEqual(['bytes', 'memory']); + }); + + it('should autocomplete only operations that provide numeric output', async () => { + const results = await suggest({ + expression: 'last_value()', + zeroIndexedOffset: 11, + context: { + triggerKind: monaco.languages.CompletionTriggerKind.TriggerCharacter, + triggerCharacter: '(', + }, + indexPattern: createMockedIndexPattern(), + operationDefinitionMap, + data: dataPluginMock.createStartContract(), + }); + expect(results.list).toEqual(['bytes', 'memory']); + }); + }); + + describe('monacoPositionToOffset', () => { + it('should work with multi-line strings accounting for newline characters', () => { + const input = `012 +456 +89')`; + expect(input[monacoPositionToOffset(input, new monaco.Position(1, 1))]).toEqual('0'); + expect(input[monacoPositionToOffset(input, new monaco.Position(3, 2))]).toEqual('9'); + }); + }); + + describe('getInfoAtZeroIndexedPosition', () => { + it('should return the location for a function inside multiple levels of math', () => { + const expression = `count() + 5 + average(LENS_MATH_MARKER)`; + const ast = parse(expression); + expect(getInfoAtZeroIndexedPosition(ast, 22)).toEqual({ + ast: expect.objectContaining({ value: 'LENS_MATH_MARKER' }), + parent: expect.objectContaining({ name: 'average' }), + }); + }); + }); +}); diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/math_completion.ts b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/math_completion.ts new file mode 100644 index 00000000000000..df747e532b38a0 --- /dev/null +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/math_completion.ts @@ -0,0 +1,594 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { uniq, startsWith } from 'lodash'; +import { i18n } from '@kbn/i18n'; +import { monaco } from '@kbn/monaco'; +import { + parse, + TinymathLocation, + TinymathAST, + TinymathFunction, + TinymathNamedArgument, +} from '@kbn/tinymath'; +import type { + DataPublicPluginStart, + QuerySuggestion, +} from '../../../../../../../../../src/plugins/data/public'; +import { IndexPattern } from '../../../../types'; +import { memoizedGetAvailableOperationsByMetadata } from '../../../operations'; +import { tinymathFunctions, groupArgsByType } from '../util'; +import type { GenericOperationDefinition } from '../..'; +import { getFunctionSignatureLabel, getHelpTextContent } from './formula_help'; +import { hasFunctionFieldArgument } from '../validation'; + +export enum SUGGESTION_TYPE { + FIELD = 'field', + NAMED_ARGUMENT = 'named_argument', + FUNCTIONS = 'functions', + KQL = 'kql', +} + +export type LensMathSuggestion = + | string + | { + label: string; + type: 'operation' | 'math'; + } + | QuerySuggestion; + +export interface LensMathSuggestions { + list: LensMathSuggestion[]; + type: SUGGESTION_TYPE; +} + +function inLocation(cursorPosition: number, location: TinymathLocation) { + return cursorPosition >= location.min && cursorPosition < location.max; +} + +const MARKER = 'LENS_MATH_MARKER'; + +export function getInfoAtZeroIndexedPosition( + ast: TinymathAST, + zeroIndexedPosition: number, + parent?: TinymathFunction +): undefined | { ast: TinymathAST; parent?: TinymathFunction } { + if (typeof ast === 'number') { + return; + } + // +, -, *, and / do not have location any more + if (ast.location && !inLocation(zeroIndexedPosition, ast.location)) { + return; + } + if (ast.type === 'function') { + const [match] = ast.args + .map((arg) => getInfoAtZeroIndexedPosition(arg, zeroIndexedPosition, ast)) + .filter((a) => a); + if (match) { + return match; + } else if (ast.location) { + return { ast }; + } else { + // None of the arguments match, but we don't know the position so it's not a match + return; + } + } + return { + ast, + parent, + }; +} + +export function offsetToRowColumn(expression: string, offset: number): monaco.Position { + const lines = expression.split(/\n/); + let remainingChars = offset; + let lineNumber = 1; + for (const line of lines) { + if (line.length >= remainingChars) { + return new monaco.Position(lineNumber, remainingChars); + } + remainingChars -= line.length + 1; + lineNumber++; + } + + throw new Error('Algorithm failure'); +} + +export function monacoPositionToOffset(expression: string, position: monaco.Position): number { + const lines = expression.split(/\n/); + return lines + .slice(0, position.lineNumber) + .reduce( + (prev, current, index) => + prev + (index === position.lineNumber - 1 ? position.column - 1 : current.length + 1), + 0 + ); +} + +export async function suggest({ + expression, + zeroIndexedOffset, + context, + indexPattern, + operationDefinitionMap, + data, +}: { + expression: string; + zeroIndexedOffset: number; + context: monaco.languages.CompletionContext; + indexPattern: IndexPattern; + operationDefinitionMap: Record; + data: DataPublicPluginStart; +}): Promise<{ list: LensMathSuggestion[]; type: SUGGESTION_TYPE }> { + const text = + expression.substr(0, zeroIndexedOffset) + MARKER + expression.substr(zeroIndexedOffset); + try { + const ast = parse(text); + + const tokenInfo = getInfoAtZeroIndexedPosition(ast, zeroIndexedOffset); + const tokenAst = tokenInfo?.ast; + + const isNamedArgument = + tokenInfo?.parent && + typeof tokenAst !== 'number' && + tokenAst && + 'type' in tokenAst && + tokenAst.type === 'namedArgument'; + if (tokenInfo?.parent && (context.triggerCharacter === '=' || isNamedArgument)) { + return await getNamedArgumentSuggestions({ + ast: tokenAst as TinymathNamedArgument, + data, + indexPattern, + }); + } else if (tokenInfo?.parent) { + return getArgumentSuggestions( + tokenInfo.parent, + tokenInfo.parent.args.findIndex((a) => a === tokenAst), + indexPattern, + operationDefinitionMap + ); + } + if ( + typeof tokenAst === 'object' && + Boolean(tokenAst.type === 'variable' || tokenAst.type === 'function') + ) { + const nameWithMarker = tokenAst.type === 'function' ? tokenAst.name : tokenAst.value; + return getFunctionSuggestions( + nameWithMarker.split(MARKER)[0], + indexPattern, + operationDefinitionMap + ); + } + } catch (e) { + // Fail silently + } + return { list: [], type: SUGGESTION_TYPE.FIELD }; +} + +export function getPossibleFunctions( + indexPattern: IndexPattern, + operationDefinitionMap?: Record +) { + const available = memoizedGetAvailableOperationsByMetadata(indexPattern, operationDefinitionMap); + const possibleOperationNames: string[] = []; + available.forEach((a) => { + if (a.operationMetaData.dataType === 'number' && !a.operationMetaData.isBucketed) { + possibleOperationNames.push( + ...a.operations.filter((o) => o.type !== 'managedReference').map((o) => o.operationType) + ); + } + }); + + return [...uniq(possibleOperationNames), ...Object.keys(tinymathFunctions)]; +} + +function getFunctionSuggestions( + prefix: string, + indexPattern: IndexPattern, + operationDefinitionMap: Record +) { + return { + list: uniq( + getPossibleFunctions(indexPattern, operationDefinitionMap).filter((func) => + startsWith(func, prefix) + ) + ).map((func) => ({ label: func, type: 'operation' as const })), + type: SUGGESTION_TYPE.FUNCTIONS, + }; +} + +function getArgumentSuggestions( + ast: TinymathFunction, + position: number, + indexPattern: IndexPattern, + operationDefinitionMap: Record +) { + const { name } = ast; + const operation = operationDefinitionMap[name]; + if (!operation && !tinymathFunctions[name]) { + return { list: [], type: SUGGESTION_TYPE.FIELD }; + } + + const tinymathFunction = tinymathFunctions[name]; + if (tinymathFunction) { + if (tinymathFunction.positionalArguments[position]) { + return { + list: uniq(getPossibleFunctions(indexPattern, operationDefinitionMap)).map((f) => ({ + type: 'math' as const, + label: f, + })), + type: SUGGESTION_TYPE.FUNCTIONS, + }; + } + return { list: [], type: SUGGESTION_TYPE.FIELD }; + } + + if (position > 0 || !hasFunctionFieldArgument(operation.type)) { + const { namedArguments } = groupArgsByType(ast.args); + const list = []; + if (operation.filterable) { + if (!namedArguments.find((arg) => arg.name === 'kql')) { + list.push('kql'); + } + if (!namedArguments.find((arg) => arg.name === 'lucene')) { + list.push('lucene'); + } + } + if ('operationParams' in operation) { + // Exclude any previously used named args + list.push( + ...operation + .operationParams!.filter( + (param) => + // Keep the param if it's the first use + !namedArguments.find((arg) => arg.name === param.name) + ) + .map((p) => p.name) + ); + } + return { list, type: SUGGESTION_TYPE.NAMED_ARGUMENT }; + } + + if (operation.input === 'field' && position === 0) { + const available = memoizedGetAvailableOperationsByMetadata( + indexPattern, + operationDefinitionMap + ); + // TODO: This only allow numeric functions, will reject last_value(string) for example. + const validOperation = available.find( + ({ operationMetaData }) => + operationMetaData.dataType === 'number' && !operationMetaData.isBucketed + ); + if (validOperation) { + const fields = validOperation.operations + .filter((op) => op.operationType === operation.type) + .map((op) => ('field' in op ? op.field : undefined)) + .filter((field) => field); + return { list: fields as string[], type: SUGGESTION_TYPE.FIELD }; + } else { + return { list: [], type: SUGGESTION_TYPE.FIELD }; + } + } + + if (operation.input === 'fullReference') { + const available = memoizedGetAvailableOperationsByMetadata( + indexPattern, + operationDefinitionMap + ); + const possibleOperationNames: string[] = []; + available.forEach((a) => { + if ( + operation.requiredReferences.some((requirement) => + requirement.validateMetadata(a.operationMetaData) + ) + ) { + possibleOperationNames.push( + ...a.operations + .filter((o) => + operation.requiredReferences.some((requirement) => requirement.input.includes(o.type)) + ) + .map((o) => o.operationType) + ); + } + }); + return { + list: uniq(possibleOperationNames).map((n) => ({ label: n, type: 'operation' as const })), + type: SUGGESTION_TYPE.FUNCTIONS, + }; + } + + return { list: [], type: SUGGESTION_TYPE.FIELD }; +} + +export async function getNamedArgumentSuggestions({ + ast, + data, + indexPattern, +}: { + ast: TinymathNamedArgument; + indexPattern: IndexPattern; + data: DataPublicPluginStart; +}) { + if (ast.name !== 'kql' && ast.name !== 'lucene') { + return { list: [], type: SUGGESTION_TYPE.KQL }; + } + if (!data.autocomplete.hasQuerySuggestions(ast.name === 'kql' ? 'kuery' : 'lucene')) { + return { list: [], type: SUGGESTION_TYPE.KQL }; + } + + const query = ast.value.split(MARKER)[0]; + const position = ast.value.indexOf(MARKER) + 1; + + const suggestions = await data.autocomplete.getQuerySuggestions({ + language: ast.name === 'kql' ? 'kuery' : 'lucene', + query, + selectionStart: position, + selectionEnd: position, + indexPatterns: [indexPattern], + boolFilter: [], + }); + return { + list: suggestions ?? [], + type: SUGGESTION_TYPE.KQL, + }; +} + +const TRIGGER_SUGGESTION_COMMAND = { + title: 'Trigger Suggestion Dialog', + id: 'editor.action.triggerSuggest', +}; + +export function getSuggestion( + suggestion: LensMathSuggestion, + type: SUGGESTION_TYPE, + operationDefinitionMap: Record, + triggerChar: string | undefined +): monaco.languages.CompletionItem { + let kind: monaco.languages.CompletionItemKind = monaco.languages.CompletionItemKind.Method; + let label: string = + typeof suggestion === 'string' + ? suggestion + : 'label' in suggestion + ? suggestion.label + : suggestion.text; + let insertText: string | undefined; + let insertTextRules: monaco.languages.CompletionItem['insertTextRules']; + let detail: string = ''; + let command: monaco.languages.CompletionItem['command']; + let sortText: string = ''; + const filterText: string = label; + + switch (type) { + case SUGGESTION_TYPE.FIELD: + kind = monaco.languages.CompletionItemKind.Value; + break; + case SUGGESTION_TYPE.FUNCTIONS: + insertText = `${label}($0)`; + insertTextRules = monaco.languages.CompletionItemInsertTextRule.InsertAsSnippet; + if (typeof suggestion !== 'string') { + if ('text' in suggestion) break; + label = getFunctionSignatureLabel(suggestion.label, operationDefinitionMap); + const tinymathFunction = tinymathFunctions[suggestion.label]; + if (tinymathFunction) { + detail = 'TinyMath'; + kind = monaco.languages.CompletionItemKind.Method; + } else { + kind = monaco.languages.CompletionItemKind.Constant; + detail = 'Elasticsearch'; + // Always put ES functions first + sortText = `0${label}`; + command = TRIGGER_SUGGESTION_COMMAND; + } + } + break; + case SUGGESTION_TYPE.NAMED_ARGUMENT: + kind = monaco.languages.CompletionItemKind.Keyword; + if (label === 'kql' || label === 'lucene') { + command = TRIGGER_SUGGESTION_COMMAND; + insertText = `${label}='$0'`; + insertTextRules = monaco.languages.CompletionItemInsertTextRule.InsertAsSnippet; + sortText = `zzz${label}`; + } + label = `${label}=`; + detail = ''; + break; + case SUGGESTION_TYPE.KQL: + if (triggerChar === ':') { + insertText = `${triggerChar} ${label}`; + } else { + // concatenate KQL suggestion for faster query composition + command = TRIGGER_SUGGESTION_COMMAND; + } + if (label.includes(`'`)) { + insertText = (insertText || label).replaceAll(`'`, "\\'"); + } + break; + } + + return { + detail, + kind, + label, + insertText: insertText ?? label, + insertTextRules, + command, + additionalTextEdits: [], + // @ts-expect-error Monaco says this type is required, but provides a default value + range: undefined, + sortText, + filterText, + }; +} + +function getOperationTypeHelp( + name: string, + operationDefinitionMap: Record +) { + const { description: descriptionInMarkdown, examples } = getHelpTextContent( + name, + operationDefinitionMap + ); + const examplesInMarkdown = examples.length + ? `\n\n**${i18n.translate('xpack.lens.formulaExampleMarkdown', { + defaultMessage: 'Examples', + })}** + + ${examples.map((example) => `\`${example}\``).join('\n\n')}` + : ''; + return { + value: `${descriptionInMarkdown}${examplesInMarkdown}`, + }; +} + +function getSignaturesForFunction( + name: string, + operationDefinitionMap: Record +) { + if (tinymathFunctions[name]) { + const stringify = getFunctionSignatureLabel(name, operationDefinitionMap); + const documentation = tinymathFunctions[name].help.replace(/\n/g, '\n\n'); + return [ + { + label: stringify, + documentation: { value: documentation }, + parameters: tinymathFunctions[name].positionalArguments.map((arg) => ({ + label: arg.name, + documentation: arg.optional + ? i18n.translate('xpack.lens.formula.optionalArgument', { + defaultMessage: 'Optional. Default value is {defaultValue}', + values: { + defaultValue: arg.defaultValue, + }, + }) + : '', + })), + }, + ]; + } + if (operationDefinitionMap[name]) { + const def = operationDefinitionMap[name]; + + const firstParam: monaco.languages.ParameterInformation | null = hasFunctionFieldArgument(name) + ? { + label: def.input === 'field' ? 'field' : def.input === 'fullReference' ? 'function' : '', + } + : null; + + const functionLabel = getFunctionSignatureLabel(name, operationDefinitionMap, firstParam); + const documentation = getOperationTypeHelp(name, operationDefinitionMap); + if ('operationParams' in def && def.operationParams) { + return [ + { + label: functionLabel, + parameters: [ + ...(firstParam ? [firstParam] : []), + ...def.operationParams.map((arg) => ({ + label: `${arg.name}=${arg.type}`, + documentation: arg.required + ? i18n.translate('xpack.lens.formula.requiredArgument', { + defaultMessage: 'Required', + }) + : '', + })), + ], + documentation, + }, + ]; + } + return [ + { + label: functionLabel, + parameters: firstParam ? [firstParam] : [], + documentation, + }, + ]; + } + return []; +} + +export function getSignatureHelp( + expression: string, + position: number, + operationDefinitionMap: Record +): monaco.languages.SignatureHelpResult { + const text = expression.substr(0, position) + MARKER + expression.substr(position); + try { + const ast = parse(text); + + const tokenInfo = getInfoAtZeroIndexedPosition(ast, position); + + let signatures: ReturnType = []; + let index = 0; + if (tokenInfo?.parent) { + const name = tokenInfo.parent.name; + // reference equality is fine here because of the way the getInfo function works + index = tokenInfo.parent.args.findIndex((arg) => arg === tokenInfo.ast); + signatures = getSignaturesForFunction(name, operationDefinitionMap); + } else if (typeof tokenInfo?.ast === 'object' && tokenInfo.ast.type === 'function') { + const name = tokenInfo.ast.name; + signatures = getSignaturesForFunction(name, operationDefinitionMap); + } + if (signatures.length) { + return { + value: { + // remove the documentation + signatures: signatures.map(({ documentation, ...signature }) => ({ + ...signature, + // extract only the first section (usually few lines) + documentation: { value: documentation.value.split('\n\n')[0] }, + })), + activeParameter: index, + activeSignature: 0, + }, + dispose: () => {}, + }; + } + } catch (e) { + // do nothing + } + return { value: { signatures: [], activeParameter: 0, activeSignature: 0 }, dispose: () => {} }; +} + +export function getHover( + expression: string, + position: number, + operationDefinitionMap: Record +): monaco.languages.Hover { + try { + const ast = parse(expression); + + const tokenInfo = getInfoAtZeroIndexedPosition(ast, position); + + if (!tokenInfo || typeof tokenInfo.ast === 'number' || !('name' in tokenInfo.ast)) { + return { contents: [] }; + } + + const name = tokenInfo.ast.name; + const signatures = getSignaturesForFunction(name, operationDefinitionMap); + if (signatures.length) { + const { label } = signatures[0]; + + return { + contents: [{ value: label }], + }; + } + } catch (e) { + // do nothing + } + return { contents: [] }; +} + +export function getTokenInfo(expression: string, position: number) { + const text = expression.substr(0, position) + MARKER + expression.substr(position); + try { + const ast = parse(text); + + return getInfoAtZeroIndexedPosition(ast, position); + } catch (e) { + return; + } +} diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/math_tokenization.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/math_tokenization.tsx new file mode 100644 index 00000000000000..17394560f8031c --- /dev/null +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/editor/math_tokenization.tsx @@ -0,0 +1,66 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import { monaco } from '@kbn/monaco'; + +export const LANGUAGE_ID = 'lens_math'; +monaco.languages.register({ id: LANGUAGE_ID }); + +export const languageConfiguration: monaco.languages.LanguageConfiguration = { + wordPattern: /[^()'"\s]+/g, + brackets: [['(', ')']], + autoClosingPairs: [ + { open: '(', close: ')' }, + { open: `'`, close: `'` }, + { open: '"', close: '"' }, + ], + surroundingPairs: [ + { open: '(', close: ')' }, + { open: `'`, close: `'` }, + { open: '"', close: '"' }, + ], +}; + +export const lexerRules = { + defaultToken: 'invalid', + tokenPostfix: '', + ignoreCase: true, + brackets: [{ open: '(', close: ')', token: 'delimiter.parenthesis' }], + escapes: /\\(?:[\\"'])/, + tokenizer: { + root: [ + [/\s+/, 'whitespace'], + [/-?(\d*\.)?\d+([eE][+\-]?\d+)?/, 'number'], + [/[a-zA-Z0-9][a-zA-Z0-9_\-\.]*/, 'keyword'], + [/[,=:]/, 'delimiter'], + // strings double quoted + [/"([^"\\]|\\.)*$/, 'string.invalid'], // string without termination + [/"/, 'string', '@string_dq'], + // strings single quoted + [/'([^'\\]|\\.)*$/, 'string.invalid'], // string without termination + [/'/, 'string', '@string_sq'], + [/\+|\-|\*|\//, 'keyword.operator'], + [/[\(]/, 'delimiter'], + [/[\)]/, 'delimiter'], + ], + string_dq: [ + [/[^\\"]+/, 'string'], + [/@escapes/, 'string.escape'], + [/\\./, 'string.escape.invalid'], + [/"/, 'string', '@pop'], + ], + string_sq: [ + [/[^\\']+/, 'string'], + [/@escapes/, 'string.escape'], + [/\\./, 'string.escape.invalid'], + [/'/, 'string', '@pop'], + ], + }, +} as monaco.languages.IMonarchLanguage; + +monaco.languages.setMonarchTokensProvider(LANGUAGE_ID, lexerRules); +monaco.languages.setLanguageConfiguration(LANGUAGE_ID, languageConfiguration); diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/formula.test.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/formula.test.tsx index 4a511e14d59e00..e1c722fd9cb38e 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/formula.test.tsx +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/formula.test.tsx @@ -14,8 +14,10 @@ import { tinymathFunctions } from './util'; jest.mock('../../layer_helpers', () => { return { - getColumnOrder: ({ columns }: { columns: Record }) => - Object.keys(columns), + getColumnOrder: jest.fn(({ columns }: { columns: Record }) => + Object.keys(columns) + ), + getManagedColumnsFrom: jest.fn().mockReturnValue([]), }; }); @@ -142,7 +144,7 @@ describe('formula', () => { indexPattern, }) ).toEqual({ - label: 'Formula', + label: 'average(bytes)', dataType: 'number', operationType: 'formula', isBucketed: false, @@ -170,7 +172,7 @@ describe('formula', () => { indexPattern, }) ).toEqual({ - label: 'Formula', + label: 'average(bytes)', dataType: 'number', operationType: 'formula', isBucketed: false, @@ -204,7 +206,7 @@ describe('formula', () => { indexPattern, }) ).toEqual({ - label: 'Formula', + label: `average(bytes, kql='category.keyword: "Men\\'s Clothing" or category.keyword: "Men\\'s Shoes"')`, dataType: 'number', operationType: 'formula', isBucketed: false, @@ -233,7 +235,7 @@ describe('formula', () => { indexPattern, }) ).toEqual({ - label: 'Formula', + label: `count(lucene='*')`, dataType: 'number', operationType: 'formula', isBucketed: false, @@ -291,7 +293,7 @@ describe('formula', () => { operationDefinitionMap ) ).toEqual({ - label: 'Formula', + label: 'moving_average(average(bytes), window=3)', dataType: 'number', operationType: 'formula', isBucketed: false, @@ -375,6 +377,7 @@ describe('formula', () => { ...layer.columns, col1: { ...currentColumn, + label: formula, params: { ...currentColumn.params, formula, @@ -415,6 +418,7 @@ describe('formula', () => { ...layer.columns, col1: { ...currentColumn, + label: 'average(bytes)', references: ['col1X1'], params: { ...currentColumn.params, @@ -565,7 +569,7 @@ describe('formula', () => { ).toEqual({ col1X0: { min: 15, max: 29 }, col1X2: { min: 0, max: 41 }, - col1X3: { min: 43, max: 50 }, + col1X3: { min: 42, max: 50 }, }); }); }); @@ -787,6 +791,34 @@ invalid: " } }); + it('returns an error if formula or math operations are used', () => { + const formulaFormulas = ['formula()', 'formula(bytes)', 'formula(formula())']; + + for (const formula of formulaFormulas) { + expect( + formulaOperation.getErrorMessage!( + getNewLayerWithFormula(formula), + 'col1', + indexPattern, + operationDefinitionMap + ) + ).toEqual(['Operation formula not found']); + } + + const mathFormulas = ['math()', 'math(bytes)', 'math(math())']; + + for (const formula of mathFormulas) { + expect( + formulaOperation.getErrorMessage!( + getNewLayerWithFormula(formula), + 'col1', + indexPattern, + operationDefinitionMap + ) + ).toEqual(['Operation math not found']); + } + }); + it('returns an error if field operation in formula have the wrong first argument', () => { const formulas = [ 'average(7)', @@ -897,6 +929,150 @@ invalid: " ).toEqual(undefined); }); + it('returns no error for a query edge case', () => { + const formulas = [ + `count(kql='')`, + `count(lucene='')`, + `moving_average(count(kql=''), window=7)`, + `count(kql='bytes >= 4000')`, + `count(kql='bytes <= 4000')`, + `count(kql='bytes = 4000')`, + ]; + for (const formula of formulas) { + expect( + formulaOperation.getErrorMessage!( + getNewLayerWithFormula(formula), + 'col1', + indexPattern, + operationDefinitionMap + ) + ).toEqual(undefined); + } + }); + + it('returns an error for a query not wrapped in single quotes', () => { + const formulas = [ + `count(kql="")`, + `count(kql='")`, + `count(kql="')`, + `count(kql="category.keyword: *")`, + `count(kql='category.keyword: *")`, + `count(kql="category.keyword: *')`, + `count(kql='category.keyword: *)`, + `count(kql=category.keyword: *')`, + `count(kql=category.keyword: *)`, + `count(kql="category.keyword: "Men's Clothing" or category.keyword: "Men's Shoes"")`, + `count(lucene="category.keyword: *")`, + `count(lucene=category.keyword: *)`, + `count(lucene=category.keyword: *) + average(bytes)`, + `count(lucene='category.keyword: *') + count(kql=category.keyword: *)`, + `count(lucene='category.keyword: *") + count(kql='category.keyword: *")`, + `count(lucene='category.keyword: *') + count(kql=category.keyword: *, kql='category.keyword: *')`, + `count(lucene='category.keyword: *') + count(kql="category.keyword: *")`, + `moving_average(count(kql=category.keyword: *), window=7, kql=category.keywork: *)`, + `moving_average( + cumulative_sum( + 7 * clamp(sum(bytes), 0, last_value(memory) + max(memory)) + ), window=10, kql=category.keywork: * + )`, + ]; + for (const formula of formulas) { + expect( + formulaOperation.getErrorMessage!( + getNewLayerWithFormula(formula), + 'col1', + indexPattern, + operationDefinitionMap + ) + ).toEqual(expect.arrayContaining([expect.stringMatching(`Single quotes are required`)])); + } + }); + + it('it returns parse fail error rather than query message if the formula is only a query condition (false positive cases for query checks)', () => { + const formulas = [ + `kql="category.keyword: *"`, + `kql=category.keyword: *`, + `kql='category.keyword: *'`, + `(kql="category.keyword: *")`, + `(kql=category.keyword: *)`, + `(lucene="category.keyword: *")`, + `(lucene=category.keyword: *)`, + `(lucene='category.keyword: *') + (kql=category.keyword: *)`, + `(lucene='category.keyword: *') + (kql=category.keyword: *, kql='category.keyword: *')`, + `(lucene='category.keyword: *') + (kql="category.keyword: *")`, + `((kql=category.keyword: *), window=7, kql=category.keywork: *)`, + `(, window=10, kql=category.keywork: *)`, + `( + , window=10, kql=category.keywork: * + )`, + ]; + for (const formula of formulas) { + expect( + formulaOperation.getErrorMessage!( + getNewLayerWithFormula(formula), + 'col1', + indexPattern, + operationDefinitionMap + ) + ).toEqual([`The Formula ${formula} cannot be parsed`]); + } + }); + + it('returns no error for a query wrapped in single quotes but with some whitespaces', () => { + const formulas = [ + `count(kql ='category.keyword: *')`, + `count(kql = 'category.keyword: *')`, + `count(kql = 'category.keyword: *')`, + ]; + for (const formula of formulas) { + expect( + formulaOperation.getErrorMessage!( + getNewLayerWithFormula(formula), + 'col1', + indexPattern, + operationDefinitionMap + ) + ).toEqual(undefined); + } + }); + + it('returns an error for multiple queries submitted for the same function', () => { + expect( + formulaOperation.getErrorMessage!( + getNewLayerWithFormula(`count(kql='category.keyword: *', lucene='category.keyword: *')`), + 'col1', + indexPattern, + operationDefinitionMap + ) + ).toEqual(['Use only one of kql= or lucene=, not both']); + }); + + it("returns a clear error when there's a missing field for a function", () => { + for (const fn of ['average', 'terms', 'max', 'sum']) { + expect( + formulaOperation.getErrorMessage!( + getNewLayerWithFormula(`${fn}()`), + 'col1', + indexPattern, + operationDefinitionMap + ) + ).toEqual([`The first argument for ${fn} should be a field name. Found no field`]); + } + }); + + it("returns a clear error when there's a missing function for a fullReference operation", () => { + for (const fn of ['cumulative_sum', 'derivative']) { + expect( + formulaOperation.getErrorMessage!( + getNewLayerWithFormula(`${fn}()`), + 'col1', + indexPattern, + operationDefinitionMap + ) + ).toEqual([`The first argument for ${fn} should be a operation name. Found no operation`]); + } + }); + it('returns no error if a math operation is passed to fullReference operations', () => { const formulas = [ 'derivative(7+1)', diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/formula.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/formula.tsx index de7ecb4bc75da3..3ed50906908762 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/formula.tsx +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/formula.tsx @@ -10,8 +10,11 @@ import { OperationDefinition } from '../index'; import { ReferenceBasedIndexPatternColumn } from '../column_types'; import { IndexPattern } from '../../../types'; import { runASTValidation, tryToParse } from './validation'; +import { MemoizedFormulaEditor } from './editor'; import { regenerateLayerFromAst } from './parse'; import { generateFormula } from './generate'; +import { filterByVisibleOperation } from './util'; +import { getManagedColumnsFrom } from '../../layer_helpers'; const defaultLabel = i18n.translate('xpack.lens.indexPattern.formulaLabel', { defaultMessage: 'Formula', @@ -38,7 +41,7 @@ export const formulaOperation: OperationDefinition< > = { type: 'formula', displayName: defaultLabel, - getDefaultLabel: (column, indexPattern) => defaultLabel, + getDefaultLabel: (column, indexPattern) => column.params.formula ?? defaultLabel, input: 'managedReference', hidden: true, getDisabledStatus(indexPattern: IndexPattern) { @@ -49,13 +52,32 @@ export const formulaOperation: OperationDefinition< if (!column.params.formula || !operationDefinitionMap) { return; } - const { root, error } = tryToParse(column.params.formula); + + const visibleOperationsMap = filterByVisibleOperation(operationDefinitionMap); + const { root, error } = tryToParse(column.params.formula, visibleOperationsMap); if (error || !root) { return [error!.message]; } - const errors = runASTValidation(root, layer, indexPattern, operationDefinitionMap); - return errors.length ? errors.map(({ message }) => message) : undefined; + const errors = runASTValidation(root, layer, indexPattern, visibleOperationsMap); + + if (errors.length) { + return errors.map(({ message }) => message); + } + + const managedColumns = getManagedColumnsFrom(columnId, layer.columns); + const innerErrors = managedColumns + .flatMap(([id, col]) => { + const def = visibleOperationsMap[col.operationType]; + if (def?.getErrorMessage) { + const messages = def.getErrorMessage(layer, id, indexPattern, visibleOperationsMap); + return messages ? { message: messages.join(', ') } : []; + } + return []; + }) + .filter((marker) => marker); + + return innerErrors.length ? innerErrors.map(({ message }) => message) : undefined; }, getPossibleOperation() { return { @@ -72,8 +94,8 @@ export const formulaOperation: OperationDefinition< const label = !params?.isFormulaBroken ? useDisplayLabel ? currentColumn.label - : params?.formula - : ''; + : params?.formula ?? defaultLabel + : defaultLabel; return [ { @@ -81,21 +103,23 @@ export const formulaOperation: OperationDefinition< function: 'mapColumn', arguments: { id: [columnId], - name: [label || ''], + name: [label || defaultLabel], exp: [ { type: 'expression', - chain: [ - { - type: 'function', - function: 'math', - arguments: { - expression: [ - currentColumn.references.length ? `"${currentColumn.references[0]}"` : ``, - ], - }, - }, - ], + chain: currentColumn.references.length + ? [ + { + type: 'function', + function: 'math', + arguments: { + expression: [ + currentColumn.references.length ? `"${currentColumn.references[0]}"` : ``, + ], + }, + }, + ] + : [], }, ], }, @@ -119,7 +143,7 @@ export const formulaOperation: OperationDefinition< prevFormat = { format: previousColumn.params.format }; } return { - label: 'Formula', + label: previousFormula || defaultLabel, dataType: 'number', operationType: 'formula', isBucketed: false, @@ -152,4 +176,6 @@ export const formulaOperation: OperationDefinition< ); return newLayer; }, + + paramEditor: MemoizedFormulaEditor, }; diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/math_examples.md b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/math_examples.md new file mode 100644 index 00000000000000..ae244109ed53ee --- /dev/null +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/math_examples.md @@ -0,0 +1,28 @@ +Basic numeric functions that we already support in Lens: + +count() +count(normalize_unit='1s') +sum(field name) +avg(field name) +moving_average(sum(field name), window=5) +moving_average(sum(field name), window=5, normalize_unit='1s') +counter_rate(field name, normalize_unit='1s') +differences(count()) +differences(sum(bytes), normalize_unit='1s') +last_value(bytes, sort=timestamp) +percentile(bytes, percent=95) + +Adding features beyond what we already support. New features are: + +* Filtering +* Math across series +* Time offset + +count() * 100 +(count() / count(offset=-7d)) + min(field name) +sum(field name, filter='field.keyword: "KQL autocomplete inside math" AND field.value > 100') + +What about custom formatting using string manipulation? Probably not... + +(avg(bytes) / 1000) + 'kb' + \ No newline at end of file diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/parse.ts b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/parse.ts index 3bfc6fcbfc011e..517cf5f1bbf45c 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/parse.ts +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/parse.ts @@ -5,6 +5,7 @@ * 2.0. */ +import { i18n } from '@kbn/i18n'; import { isObject } from 'lodash'; import type { TinymathAST, TinymathVariable, TinymathLocation } from '@kbn/tinymath'; import { OperationDefinition, GenericOperationDefinition, IndexPatternColumn } from '../index'; @@ -12,7 +13,12 @@ import { IndexPattern, IndexPatternLayer } from '../../../types'; import { mathOperation } from './math'; import { documentField } from '../../../document_field'; import { runASTValidation, shouldHaveFieldArgument, tryToParse } from './validation'; -import { findVariables, getOperationParams, groupArgsByType } from './util'; +import { + filterByVisibleOperation, + findVariables, + getOperationParams, + groupArgsByType, +} from './util'; import { FormulaIndexPatternColumn } from './formula'; import { getColumnOrder } from '../../layer_helpers'; @@ -27,7 +33,7 @@ function parseAndExtract( indexPattern: IndexPattern, operationDefinitionMap: Record ) { - const { root, error } = tryToParse(text); + const { root, error } = tryToParse(text, operationDefinitionMap); if (error || !root) { return { extracted: [], isValid: false }; } @@ -61,9 +67,9 @@ function extractColumns( const nodeOperation = operations[node.name]; if (!nodeOperation) { // it's a regular math node - const consumedArgs = node.args.map(parseNode).filter(Boolean) as Array< - number | TinymathVariable - >; + const consumedArgs = node.args + .map(parseNode) + .filter((n) => typeof n !== 'undefined' && n !== null) as Array; return { ...node, args: consumedArgs, @@ -168,7 +174,7 @@ export function regenerateLayerFromAst( layer, columnId, indexPattern, - operationDefinitionMap + filterByVisibleOperation(operationDefinitionMap) ); const columns = { ...layer.columns }; @@ -188,6 +194,12 @@ export function regenerateLayerFromAst( columns[columnId] = { ...currentColumn, + label: !currentColumn.customLabel + ? text ?? + i18n.translate('xpack.lens.indexPattern.formulaLabel', { + defaultMessage: 'Formula', + }) + : currentColumn.label, params: { ...currentColumn.params, formula: text, diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/util.ts b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/util.ts index 5d9a8647eb7ab0..dd95ebdec5b8ab 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/util.ts +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/util.ts @@ -13,7 +13,7 @@ import type { TinymathNamedArgument, TinymathVariable, } from 'packages/kbn-tinymath'; -import type { OperationDefinition, IndexPatternColumn } from '../index'; +import type { OperationDefinition, IndexPatternColumn, GenericOperationDefinition } from '../index'; import type { GroupedNodes } from './types'; export function groupArgsByType(args: TinymathAST[]) { @@ -66,6 +66,16 @@ export function getOperationParams( }, {}); } +function getTypeI18n(type: string) { + if (type === 'number') { + return i18n.translate('xpack.lens.formula.number', { defaultMessage: 'number' }); + } + if (type === 'string') { + return i18n.translate('xpack.lens.formula.string', { defaultMessage: 'string' }); + } + return ''; +} + // Todo: i18n everything here export const tinymathFunctions: Record< string, @@ -73,145 +83,254 @@ export const tinymathFunctions: Record< positionalArguments: Array<{ name: string; optional?: boolean; + defaultValue?: string | number; + type?: string; }>; - // help: React.ReactElement; // Help is in Markdown format help: string; } > = { add: { positionalArguments: [ - { name: i18n.translate('xpack.lens.formula.left', { defaultMessage: 'left' }) }, - { name: i18n.translate('xpack.lens.formula.right', { defaultMessage: 'right' }) }, + { + name: i18n.translate('xpack.lens.formula.left', { defaultMessage: 'left' }), + type: getTypeI18n('number'), + }, + { + name: i18n.translate('xpack.lens.formula.right', { defaultMessage: 'right' }), + type: getTypeI18n('number'), + }, ], help: ` +Adds up two numbers. Also works with + symbol -Example: ${'`count() + sum(bytes)`'} -Example: ${'`add(count(), 5)`'} + +Example: Calculate the sum of two fields + +${'`sum(price) + sum(tax)`'} + +Example: Offset count by a static value + +${'`add(count(), 5)`'} `, }, subtract: { positionalArguments: [ - { name: i18n.translate('xpack.lens.formula.left', { defaultMessage: 'left' }) }, - { name: i18n.translate('xpack.lens.formula.right', { defaultMessage: 'right' }) }, + { + name: i18n.translate('xpack.lens.formula.left', { defaultMessage: 'left' }), + type: getTypeI18n('number'), + }, + { + name: i18n.translate('xpack.lens.formula.right', { defaultMessage: 'right' }), + type: getTypeI18n('number'), + }, ], help: ` +Subtracts the first number from the second number. Also works with ${'`-`'} symbol -Example: ${'`subtract(sum(bytes), avg(bytes))`'} + +Example: Calculate the range of a field +${'`subtract(max(bytes), min(bytes))`'} `, }, multiply: { positionalArguments: [ - { name: i18n.translate('xpack.lens.formula.left', { defaultMessage: 'left' }) }, - { name: i18n.translate('xpack.lens.formula.right', { defaultMessage: 'right' }) }, + { + name: i18n.translate('xpack.lens.formula.left', { defaultMessage: 'left' }), + type: getTypeI18n('number'), + }, + { + name: i18n.translate('xpack.lens.formula.right', { defaultMessage: 'right' }), + type: getTypeI18n('number'), + }, ], help: ` -Also works with ${'`*`'} symbol -Example: ${'`multiply(sum(bytes), 2)`'} +Multiplies two numbers. +Also works with ${'`*`'} symbol. + +Example: Calculate price after current tax rate +${'`sum(bytes) * last_value(tax_rate)`'} + +Example: Calculate price after constant tax rate +${'`multiply(sum(price), 1.2)`'} `, }, divide: { positionalArguments: [ - { name: i18n.translate('xpack.lens.formula.left', { defaultMessage: 'left' }) }, - { name: i18n.translate('xpack.lens.formula.right', { defaultMessage: 'right' }) }, + { + name: i18n.translate('xpack.lens.formula.left', { defaultMessage: 'left' }), + type: getTypeI18n('number'), + }, + { + name: i18n.translate('xpack.lens.formula.right', { defaultMessage: 'right' }), + type: getTypeI18n('number'), + }, ], help: ` +Divides the first number by the second number. Also works with ${'`/`'} symbol -Example: ${'`ceil(sum(bytes))`'} + +Example: Calculate profit margin +${'`sum(profit) / sum(revenue)`'} + +Example: ${'`divide(sum(bytes), 2)`'} `, }, abs: { positionalArguments: [ - { name: i18n.translate('xpack.lens.formula.expression', { defaultMessage: 'expression' }) }, + { + name: i18n.translate('xpack.lens.formula.value', { defaultMessage: 'value' }), + type: getTypeI18n('number'), + }, ], help: ` -Absolute value -Example: ${'`abs(sum(bytes))`'} +Calculates absolute value. A negative value is multiplied by -1, a positive value stays the same. + +Example: Calculate average distance to sea level ${'`abs(average(altitude))`'} `, }, cbrt: { positionalArguments: [ - { name: i18n.translate('xpack.lens.formula.expression', { defaultMessage: 'expression' }) }, + { + name: i18n.translate('xpack.lens.formula.value', { defaultMessage: 'value' }), + type: getTypeI18n('number'), + }, ], help: ` -Cube root of value -Example: ${'`cbrt(sum(bytes))`'} +Cube root of value. + +Example: Calculate side length from volume +${'`cbrt(last_value(volume))`'} `, }, ceil: { positionalArguments: [ - { name: i18n.translate('xpack.lens.formula.expression', { defaultMessage: 'expression' }) }, + { + name: i18n.translate('xpack.lens.formula.value', { defaultMessage: 'value' }), + type: getTypeI18n('number'), + }, ], + // signature: 'ceil(value: number)', help: ` -Ceiling of value, rounds up -Example: ${'`ceil(sum(bytes))`'} +Ceiling of value, rounds up. + +Example: Round up price to the next dollar +${'`ceil(sum(price))`'} `, }, clamp: { positionalArguments: [ - { name: i18n.translate('xpack.lens.formula.expression', { defaultMessage: 'expression' }) }, - { name: i18n.translate('xpack.lens.formula.min', { defaultMessage: 'min' }) }, - { name: i18n.translate('xpack.lens.formula.max', { defaultMessage: 'max' }) }, + { + name: i18n.translate('xpack.lens.formula.value', { defaultMessage: 'value' }), + type: getTypeI18n('number'), + }, + { + name: i18n.translate('xpack.lens.formula.min', { defaultMessage: 'min' }), + type: getTypeI18n('number'), + }, + { + name: i18n.translate('xpack.lens.formula.max', { defaultMessage: 'max' }), + type: getTypeI18n('number'), + }, ], + // signature: 'clamp(value: number, minimum: number, maximum: number)', help: ` -Limits the value from a minimum to maximum -Example: ${'`ceil(sum(bytes))`'} - `, +Limits the value from a minimum to maximum. + +Example: Make sure to catch outliers +\`\`\` +clamp( + average(bytes), + percentile(bytes, percentile=5), + percentile(bytes, percentile=95) +) +\`\`\` +`, }, cube: { positionalArguments: [ - { name: i18n.translate('xpack.lens.formula.expression', { defaultMessage: 'expression' }) }, + { + name: i18n.translate('xpack.lens.formula.value', { defaultMessage: 'value' }), + type: getTypeI18n('number'), + }, ], help: ` -Limits the value from a minimum to maximum -Example: ${'`ceil(sum(bytes))`'} +Calculates the cube of a number. + +Example: Calculate volume from side length +${'`cube(last_value(length))`'} `, }, exp: { positionalArguments: [ - { name: i18n.translate('xpack.lens.formula.expression', { defaultMessage: 'expression' }) }, + { + name: i18n.translate('xpack.lens.formula.value', { defaultMessage: 'value' }), + type: getTypeI18n('number'), + }, ], help: ` -Raises e to the nth power. -Example: ${'`exp(sum(bytes))`'} +Raises *e* to the nth power. + +Example: Calculate the natural exponential function + +${'`exp(last_value(duration))`'} `, }, fix: { positionalArguments: [ - { name: i18n.translate('xpack.lens.formula.expression', { defaultMessage: 'expression' }) }, + { + name: i18n.translate('xpack.lens.formula.value', { defaultMessage: 'value' }), + type: getTypeI18n('number'), + }, ], help: ` For positive values, takes the floor. For negative values, takes the ceiling. -Example: ${'`fix(sum(bytes))`'} + +Example: Rounding towards zero +${'`fix(sum(profit))`'} `, }, floor: { positionalArguments: [ - { name: i18n.translate('xpack.lens.formula.expression', { defaultMessage: 'expression' }) }, + { + name: i18n.translate('xpack.lens.formula.value', { defaultMessage: 'value' }), + type: getTypeI18n('number'), + }, ], help: ` Round down to nearest integer value -Example: ${'`floor(sum(bytes))`'} + +Example: Round down a price +${'`floor(sum(price))`'} `, }, log: { positionalArguments: [ - { name: i18n.translate('xpack.lens.formula.expression', { defaultMessage: 'expression' }) }, + { + name: i18n.translate('xpack.lens.formula.value', { defaultMessage: 'value' }), + type: getTypeI18n('number'), + }, { name: i18n.translate('xpack.lens.formula.base', { defaultMessage: 'base' }), optional: true, + defaultValue: 'e', + type: getTypeI18n('number'), }, ], help: ` -Logarithm with optional base. The natural base e is used as default. -Example: ${'`log(sum(bytes))`'} -Example: ${'`log(sum(bytes), 2)`'} +Logarithm with optional base. The natural base *e* is used as default. + +Example: Calculate number of bits required to store values +\`\`\` +log(sum(bytes)) +log(sum(bytes), 2) +\`\`\` `, }, // TODO: check if this is valid for Tinymath // log10: { // positionalArguments: [ - // { name: i18n.translate('xpack.lens.formula.expression', { defaultMessage: 'expression' }) }, + // { name: i18n.translate('xpack.lens.formula.value', { defaultMessage: 'value' }), type: getTypeI18n('number') }, // ], // help: ` // Base 10 logarithm. @@ -220,59 +339,89 @@ Example: ${'`log(sum(bytes), 2)`'} // }, mod: { positionalArguments: [ - { name: i18n.translate('xpack.lens.formula.expression', { defaultMessage: 'expression' }) }, + { + name: i18n.translate('xpack.lens.formula.value', { defaultMessage: 'value' }), + type: getTypeI18n('number'), + }, { name: i18n.translate('xpack.lens.formula.base', { defaultMessage: 'base' }), - optional: true, + type: getTypeI18n('number'), }, ], help: ` Remainder after dividing the function by a number -Example: ${'`mod(sum(bytes), 2)`'} + +Example: Calculate last three digits of a value +${'`mod(sum(price), 1000)`'} `, }, pow: { positionalArguments: [ - { name: i18n.translate('xpack.lens.formula.expression', { defaultMessage: 'expression' }) }, + { + name: i18n.translate('xpack.lens.formula.value', { defaultMessage: 'value' }), + type: getTypeI18n('number'), + }, { name: i18n.translate('xpack.lens.formula.base', { defaultMessage: 'base' }), + type: getTypeI18n('number'), }, ], help: ` Raises the value to a certain power. The second argument is required -Example: ${'`pow(sum(bytes), 3)`'} + +Example: Calculate volume based on side length +${'`pow(last_value(length), 3)`'} `, }, round: { positionalArguments: [ - { name: i18n.translate('xpack.lens.formula.expression', { defaultMessage: 'expression' }) }, + { + name: i18n.translate('xpack.lens.formula.value', { defaultMessage: 'value' }), + type: getTypeI18n('number'), + }, { name: i18n.translate('xpack.lens.formula.decimals', { defaultMessage: 'decimals' }), optional: true, + defaultValue: 0, + type: getTypeI18n('number'), }, ], help: ` Rounds to a specific number of decimal places, default of 0 -Example: ${'`round(sum(bytes))`'} -Example: ${'`round(sum(bytes), 2)`'} + +Examples: Round to the cent +\`\`\` +round(sum(bytes)) +round(sum(bytes), 2) +\`\`\` `, }, sqrt: { positionalArguments: [ - { name: i18n.translate('xpack.lens.formula.expression', { defaultMessage: 'expression' }) }, + { + name: i18n.translate('xpack.lens.formula.value', { defaultMessage: 'value' }), + type: getTypeI18n('number'), + }, ], help: ` Square root of a positive value only -Example: ${'`sqrt(sum(bytes))`'} + +Example: Calculate side length based on area +${'`sqrt(last_value(area))`'} `, }, square: { positionalArguments: [ - { name: i18n.translate('xpack.lens.formula.expression', { defaultMessage: 'expression' }) }, + { + name: i18n.translate('xpack.lens.formula.value', { defaultMessage: 'value' }), + type: getTypeI18n('number'), + }, ], help: ` Raise the value to the 2nd power -Example: ${'`square(sum(bytes))`'} + +Example: Calculate area based on side length +${'`square(last_value(length))`'} `, }, }; @@ -315,3 +464,11 @@ export function findVariables(node: TinymathAST | string): TinymathVariable[] { } return node.args.flatMap(findVariables); } + +export function filterByVisibleOperation( + operationDefinitionMap: Record +) { + return Object.fromEntries( + Object.entries(operationDefinitionMap).filter(([, operation]) => !operation.hidden) + ); +} diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/validation.ts b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/validation.ts index 5145c7959f1bb5..992b8ee2422e90 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/validation.ts +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/formula/validation.ts @@ -5,7 +5,7 @@ * 2.0. */ -import { isObject } from 'lodash'; +import { isObject, partition } from 'lodash'; import { i18n } from '@kbn/i18n'; import { parse, TinymathLocation } from '@kbn/tinymath'; import type { TinymathAST, TinymathFunction, TinymathNamedArgument } from '@kbn/tinymath'; @@ -58,6 +58,10 @@ interface ValidationErrors { message: string; type: { operation: string; count: number; params: string }; }; + tooManyQueries: { + message: string; + type: {}; + }; } type ErrorTypes = keyof ValidationErrors; type ErrorValues = ValidationErrors[K]['type']; @@ -90,15 +94,76 @@ export function hasInvalidOperations( return { // avoid duplicates names: Array.from(new Set(nodes.map(({ name }) => name))), - locations: nodes.map(({ location }) => location), + locations: nodes.map(({ location }) => location).filter((a) => a) as TinymathLocation[], }; } +export const getRawQueryValidationError = (text: string, operations: Record) => { + // try to extract the query context here + const singleLine = text.split('\n').join(''); + const allArgs = singleLine.split(',').filter((args) => /(kql|lucene)/.test(args)); + // check for the presence of a valid ES operation + const containsOneValidOperation = Object.keys(operations).some((operation) => + singleLine.includes(operation) + ); + // no args or no valid operation, no more work to do here + if (allArgs.length === 0 || !containsOneValidOperation) { + return; + } + // at this point each entry in allArgs may contain one or more + // in the worst case it would be a math chain of count operation + // For instance: count(kql=...) + count(lucene=...) - count(kql=...) + // therefore before partition them, split them by "count" keywork and filter only string with a length + const flattenArgs = allArgs.flatMap((arg) => + arg.split('count').filter((subArg) => /(kql|lucene)/.test(subArg)) + ); + const [kqlQueries, luceneQueries] = partition(flattenArgs, (arg) => /kql/.test(arg)); + const errors = []; + for (const kqlQuery of kqlQueries) { + const result = validateQueryQuotes(kqlQuery, 'kql'); + if (result) { + errors.push(result); + } + } + for (const luceneQuery of luceneQueries) { + const result = validateQueryQuotes(luceneQuery, 'lucene'); + if (result) { + errors.push(result); + } + } + return errors.length ? errors : undefined; +}; + +const validateQueryQuotes = (rawQuery: string, language: 'kql' | 'lucene') => { + // check if the raw argument has the minimal requirements + // use the rest operator here to handle cases where comparison operations are used in the query + const [, ...rawValue] = rawQuery.split('='); + const fullRawValue = (rawValue || ['']).join(''); + const cleanedRawValue = fullRawValue.trim(); + // it must start with a single quote, and quotes must have a closure + if ( + cleanedRawValue.length && + (cleanedRawValue[0] !== "'" || !/'\s*([^']+?)\s*'/.test(fullRawValue)) && + // there's a special case when it's valid as two single quote strings + cleanedRawValue !== "''" + ) { + return i18n.translate('xpack.lens.indexPattern.formulaOperationQueryError', { + defaultMessage: `Single quotes are required for {language}='' at {rawQuery}`, + values: { language, rawQuery }, + }); + } +}; + export const getQueryValidationError = ( - query: string, - language: 'kql' | 'lucene', + { value: query, name: language, text }: TinymathNamedArgument, indexPattern: IndexPattern ): string | undefined => { + // check if the raw argument has the minimal requirements + const result = validateQueryQuotes(text, language as 'kql' | 'lucene'); + // forward the error here is ok? + if (result) { + return result; + } try { if (language === 'kql') { esKuery.toElasticsearchQuery(esKuery.fromKueryExpression(query), indexPattern); @@ -113,7 +178,7 @@ export const getQueryValidationError = ( function getMessageFromId({ messageId, - values: { ...values }, + values, locations, }: { messageId: K; @@ -203,6 +268,11 @@ function getMessageFromId({ values: { operation: out.operation, count: out.count, params: out.params }, }); break; + case 'tooManyQueries': + message = i18n.translate('xpack.lens.indexPattern.formulaOperationDoubleQueryError', { + defaultMessage: 'Use only one of kql= or lucene=, not both', + }); + break; // case 'mathRequiresFunction': // message = i18n.translate('xpack.lens.indexPattern.formulaMathRequiresFunctionLabel', { // defaultMessage; 'The function {name} requires an Elasticsearch function', @@ -218,12 +288,22 @@ function getMessageFromId({ } export function tryToParse( - formula: string + formula: string, + operations: Record ): { root: TinymathAST; error: null } | { root: null; error: ErrorWrapper } { let root; try { root = parse(formula); } catch (e) { + // A tradeoff is required here, unless we want to reimplement a full parser + // Internally the function has the following logic: + // * if the formula contains no existing ES operation, assume it's a plain parse failure + // * if the formula contains at least one existing operation, check for query problems + const maybeQueryProblems = getRawQueryValidationError(formula, operations); + if (maybeQueryProblems) { + // need to emulate an error shape here + return { root: null, error: { message: maybeQueryProblems[0], locations: [] } }; + } return { root: null, error: getMessageFromId({ @@ -319,7 +399,10 @@ function getQueryValidationErrors( const errors: ErrorWrapper[] = []; (namedArguments ?? []).forEach((arg) => { if (arg.name === 'kql' || arg.name === 'lucene') { - const message = getQueryValidationError(arg.value, arg.name, indexPattern); + const message = getQueryValidationError( + arg as TinymathNamedArgument & { name: 'kql' | 'lucene' }, + indexPattern + ); if (message) { errors.push({ message, @@ -331,6 +414,12 @@ function getQueryValidationErrors( return errors; } +function checkSingleQuery(namedArguments: TinymathNamedArgument[] | undefined) { + return namedArguments + ? namedArguments.filter((arg) => arg.name === 'kql' || arg.name === 'lucene').length > 1 + : undefined; +} + function validateNameArguments( node: TinymathFunction, nodeOperation: @@ -349,7 +438,7 @@ function validateNameArguments( operation: node.name, params: missingParams.map(({ name }) => name).join(', '), }, - locations: [node.location], + locations: node.location ? [node.location] : [], }) ); } @@ -362,7 +451,7 @@ function validateNameArguments( operation: node.name, params: wrongTypeParams.map(({ name }) => name).join(', '), }, - locations: [node.location], + locations: node.location ? [node.location] : [], }) ); } @@ -375,7 +464,7 @@ function validateNameArguments( operation: node.name, params: duplicateParams.join(', '), }, - locations: [node.location], + locations: node.location ? [node.location] : [], }) ); } @@ -383,6 +472,16 @@ function validateNameArguments( if (queryValidationErrors.length) { errors.push(...queryValidationErrors); } + const hasTooManyQueries = checkSingleQuery(namedArguments); + if (hasTooManyQueries) { + errors.push( + getMessageFromId({ + messageId: 'tooManyQueries', + values: {}, + locations: node.location ? [node.location] : [], + }) + ); + } return errors; } @@ -426,7 +525,7 @@ function runFullASTValidation( type: 'field', argument: `math operation`, }, - locations: [node.location], + locations: node.location ? [node.location] : [], }) ); } else { @@ -436,9 +535,13 @@ function runFullASTValidation( values: { operation: node.name, type: 'field', - argument: getValueOrName(firstArg), + argument: + getValueOrName(firstArg) || + i18n.translate('xpack.lens.indexPattern.formulaNoFieldForOperation', { + defaultMessage: 'no field', + }), }, - locations: [node.location], + locations: node.location ? [node.location] : [], }) ); } @@ -452,7 +555,7 @@ function runFullASTValidation( values: { operation: node.name, }, - locations: [node.location], + locations: node.location ? [node.location] : [], }) ); } @@ -464,7 +567,7 @@ function runFullASTValidation( values: { operation: node.name, }, - locations: [node.location], + locations: node.location ? [node.location] : [], }) ); } else { @@ -493,9 +596,13 @@ function runFullASTValidation( values: { operation: node.name, type: 'operation', - argument: getValueOrName(firstArg), + argument: + getValueOrName(firstArg) || + i18n.translate('xpack.lens.indexPattern.formulaNoOperation', { + defaultMessage: 'no operation', + }), }, - locations: [node.location], + locations: node.location ? [node.location] : [], }) ); } @@ -506,7 +613,7 @@ function runFullASTValidation( values: { operation: node.name, }, - locations: [node.location], + locations: node.location ? [node.location] : [], }) ); } else { @@ -606,7 +713,11 @@ export function validateParams( } export function shouldHaveFieldArgument(node: TinymathFunction) { - return !['count'].includes(node.name); + return hasFunctionFieldArgument(node.name); +} + +export function hasFunctionFieldArgument(type: string) { + return !['count'].includes(type); } export function isFirstArgumentValidType(arg: TinymathAST, type: TinymathNodeTypes['type']) { @@ -628,7 +739,7 @@ export function validateMathNodes(root: TinymathAST, missingVariableSet: Set name).join(', '), }, - locations: [node.location], + locations: node.location ? [node.location] : [], }) ); } diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/index.ts b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/index.ts index 164415c1a1f6f3..a7bf415817797d 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/index.ts +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/index.ts @@ -153,6 +153,9 @@ export interface ParamEditorProps { updateLayer: ( setter: IndexPatternLayer | ((prevLayer: IndexPatternLayer) => IndexPatternLayer) ) => void; + toggleFullscreen: () => void; + setIsCloseable: (isCloseable: boolean) => void; + isFullscreen: boolean; columnId: string; indexPattern: IndexPattern; uiSettings: IUiSettingsClient; @@ -279,6 +282,11 @@ interface BaseOperationDefinitionProps { * Operations can be used as middleware for other operations, hence not shown in the panel UI */ hidden?: boolean; + documentation?: { + signature: string; + description: string; + section: 'elasticsearch' | 'calculation'; + }; } interface BaseBuildColumnArgs { @@ -290,6 +298,7 @@ interface OperationParam { name: string; type: string; required?: boolean; + defaultValue?: string | number; } interface FieldlessOperationDefinition { diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/last_value.test.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/last_value.test.tsx index 15ce3bdcd0b0f5..2ad91a7ba91a1e 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/last_value.test.tsx +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/last_value.test.tsx @@ -30,6 +30,9 @@ const defaultProps = { hasRestrictions: false, } as IndexPattern, operationDefinitionMap: {}, + isFullscreen: false, + toggleFullscreen: jest.fn(), + setIsCloseable: jest.fn(), }; describe('last_value', () => { diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/last_value.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/last_value.tsx index bde80accfbc676..bfc5ce39bc9390 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/last_value.tsx +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/last_value.tsx @@ -277,4 +277,20 @@ export const lastValueOperation: OperationDefinition ); }, + documentation: { + section: 'elasticsearch', + signature: i18n.translate('xpack.lens.indexPattern.lastValue.signature', { + defaultMessage: 'field: string', + }), + description: i18n.translate('xpack.lens.indexPattern.lastValue.documentation', { + defaultMessage: ` +Returns the value of a field from the last document, ordered by the default time field of the index pattern. + +This function is usefull the retrieve the latest state of an entity. + +Example: Get the current status of server A: +\`last_value(server.status, kql=\'server.name="A"\')\` + `, + }), + }, }; diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/metrics.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/metrics.tsx index 54a3ff0eb8bdbf..58fe91b23f2c7d 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/metrics.tsx +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/metrics.tsx @@ -42,6 +42,7 @@ const supportedTypes = ['number', 'histogram']; function buildMetricOperation>({ type, displayName, + description, ofName, priority, optionalTimeScaling, @@ -51,6 +52,7 @@ function buildMetricOperation>({ ofName: (name: string) => string; priority?: number; optionalTimeScaling?: boolean; + description?: string; }) { const labelLookup = (name: string, column?: BaseIndexPatternColumn) => { const label = ofName(name); @@ -67,6 +69,7 @@ function buildMetricOperation>({ type, priority, displayName, + description, input: 'field', timeScalingMode: optionalTimeScaling ? 'optional' : undefined, getPossibleOperationForField: ({ aggregationRestrictions, aggregatable, type: fieldType }) => { @@ -131,6 +134,26 @@ function buildMetricOperation>({ getErrorMessage: (layer, columnId, indexPattern) => getInvalidFieldMessage(layer.columns[columnId] as FieldBasedIndexPatternColumn, indexPattern), filterable: true, + documentation: { + section: 'elasticsearch', + signature: i18n.translate('xpack.lens.indexPattern.metric.signature', { + defaultMessage: 'field: string', + }), + description: i18n.translate('xpack.lens.indexPattern.metric.documentation', { + defaultMessage: ` +Returns the {metric} of a field. This function only works for number fields. + +Example: Get the {metric} of price: +\`{metric}(price)\` + +Example: Get the {metric} of price for orders from the UK: +\`{metric}(price, kql='location:UK')\` + `, + values: { + metric: type, + }, + }), + }, shiftable: true, } as OperationDefinition; } @@ -151,6 +174,10 @@ export const minOperation = buildMetricOperation({ defaultMessage: 'Minimum of {name}', values: { name }, }), + description: i18n.translate('xpack.lens.indexPattern.min.description', { + defaultMessage: + 'A single-value metrics aggregation that returns the minimum value among the numeric values extracted from the aggregated documents.', + }), }); export const maxOperation = buildMetricOperation({ @@ -163,6 +190,10 @@ export const maxOperation = buildMetricOperation({ defaultMessage: 'Maximum of {name}', values: { name }, }), + description: i18n.translate('xpack.lens.indexPattern.max.description', { + defaultMessage: + 'A single-value metrics aggregation that returns the maximum value among the numeric values extracted from the aggregated documents.', + }), }); export const averageOperation = buildMetricOperation({ @@ -176,6 +207,10 @@ export const averageOperation = buildMetricOperation({ defaultMessage: 'Average of {name}', values: { name }, }), + description: i18n.translate('xpack.lens.indexPattern.avg.description', { + defaultMessage: + 'A single-value metric aggregation that computes the average of numeric values that are extracted from the aggregated documents', + }), }); export const sumOperation = buildMetricOperation({ @@ -190,6 +225,10 @@ export const sumOperation = buildMetricOperation({ values: { name }, }), optionalTimeScaling: true, + description: i18n.translate('xpack.lens.indexPattern.sum.description', { + defaultMessage: + 'A single-value metrics aggregation that sums up numeric values that are extracted from the aggregated documents.', + }), }); export const medianOperation = buildMetricOperation({ @@ -203,4 +242,8 @@ export const medianOperation = buildMetricOperation({ defaultMessage: 'Median of {name}', values: { name }, }), + description: i18n.translate('xpack.lens.indexPattern.median.description', { + defaultMessage: + 'A single-value metrics aggregation that computes the median value that are extracted from the aggregated documents.', + }), }); diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/percentile.test.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/percentile.test.tsx index 2b7104112d63eb..0a3462ef20f3f7 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/percentile.test.tsx +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/percentile.test.tsx @@ -32,6 +32,9 @@ const defaultProps = { hasRestrictions: false, } as IndexPattern, operationDefinitionMap: {}, + isFullscreen: false, + toggleFullscreen: jest.fn(), + setIsCloseable: jest.fn(), }; describe('percentile', () => { diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/percentile.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/percentile.tsx index aa8f951d46b4f2..39b876050c2eae 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/percentile.tsx +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/percentile.tsx @@ -59,7 +59,9 @@ export const percentileOperation: OperationDefinition { @@ -213,4 +215,18 @@ export const percentileOperation: OperationDefinition ); }, + documentation: { + section: 'elasticsearch', + signature: i18n.translate('xpack.lens.indexPattern.percentile.signature', { + defaultMessage: 'field: string, [percentile]: number', + }), + description: i18n.translate('xpack.lens.indexPattern.percentile.documentation', { + defaultMessage: ` +Returns the specified percentile of the values of a field. This is the value n percent of the values occuring in documents are smaller. + +Example: Get the number of bytes larger than 95 % of values: +\`percentile(bytes, percentile=95)\` + `, + }), + }, }; diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/ranges/ranges.test.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/ranges/ranges.test.tsx index 295f988c6e3906..ca4e6d5df0a3c4 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/ranges/ranges.test.tsx +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/ranges/ranges.test.tsx @@ -102,6 +102,9 @@ const defaultOptions = { ]), }, operationDefinitionMap: {}, + isFullscreen: false, + toggleFullscreen: jest.fn(), + setIsCloseable: jest.fn(), }; describe('ranges', () => { diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/terms/terms.test.tsx b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/terms/terms.test.tsx index b272d1703377ce..32b7dfee828fce 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/terms/terms.test.tsx +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/terms/terms.test.tsx @@ -35,6 +35,9 @@ const defaultProps = { http: {} as HttpSetup, indexPattern: createMockedIndexPattern(), operationDefinitionMap: {}, + isFullscreen: false, + toggleFullscreen: jest.fn(), + setIsCloseable: jest.fn(), }; describe('terms', () => { diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/layer_helpers.test.ts b/x-pack/plugins/lens/public/indexpattern_datasource/operations/layer_helpers.test.ts index 38bc84ae9af357..ba3bee415f3f49 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/operations/layer_helpers.test.ts +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/layer_helpers.test.ts @@ -24,7 +24,7 @@ import type { IndexPattern, IndexPatternLayer } from '../types'; import { documentField } from '../document_field'; import { getFieldByNameFactory } from '../pure_helpers'; import { generateId } from '../../id_generator'; -import { createMockedFullReference } from './mocks'; +import { createMockedFullReference, createMockedManagedReference } from './mocks'; jest.mock('../operations'); jest.mock('../../id_generator'); @@ -91,10 +91,13 @@ describe('state_helpers', () => { // @ts-expect-error we are inserting an invalid type operationDefinitionMap.testReference = createMockedFullReference(); + // @ts-expect-error we are inserting an invalid type + operationDefinitionMap.managedReference = createMockedManagedReference(); }); afterEach(() => { delete operationDefinitionMap.testReference; + delete operationDefinitionMap.managedReference; }); describe('copyColumn', () => { @@ -102,19 +105,19 @@ describe('state_helpers', () => { const source = { dataType: 'number' as const, isBucketed: false, - label: 'Formula', + label: 'moving_average(sum(bytes), window=5)', operationType: 'formula' as const, params: { formula: 'moving_average(sum(bytes), window=5)', isFormulaBroken: false, }, - references: ['formulaX3'], + references: ['formulaX1'], }; const math = { customLabel: true, dataType: 'number' as const, isBucketed: false, - label: 'math', + label: 'formulaX2', operationType: 'math' as const, params: { tinymathAst: 'formulaX2' }, references: ['formulaX2'], @@ -135,7 +138,7 @@ describe('state_helpers', () => { label: 'formulaX2', operationType: 'moving_average' as const, params: { window: 5 }, - references: ['formulaX1'], + references: ['formulaX0'], }; expect( copyColumn({ @@ -387,6 +390,42 @@ describe('state_helpers', () => { ).toEqual(expect.objectContaining({ columnOrder: ['col1', 'col2', 'col3'] })); }); + it('should not change order of metrics and references on inserting new buckets', () => { + const layer: IndexPatternLayer = { + indexPatternId: '1', + columnOrder: ['col1', 'col2'], + columns: { + col1: { + label: 'Cumulative sum of count of records', + dataType: 'number', + isBucketed: false, + + // Private + operationType: 'cumulative_sum', + references: ['col2'], + }, + col2: { + label: 'Count of records', + dataType: 'document', + isBucketed: false, + + // Private + operationType: 'count', + sourceField: 'Records', + }, + }, + }; + expect( + insertNewColumn({ + layer, + indexPattern, + columnId: 'col3', + op: 'filters', + visualizationGroups: [], + }) + ).toEqual(expect.objectContaining({ columnOrder: ['col3', 'col1', 'col2'] })); + }); + it('should insert both incomplete states if the aggregation does not support the field', () => { expect( insertNewColumn({ @@ -2655,6 +2694,36 @@ describe('state_helpers', () => { expect(errors).toHaveLength(1); }); + it('should only collect the top level errors from managed references', () => { + const notCalledMock = jest.fn(); + const mock = jest.fn().mockReturnValue(['error 1']); + operationDefinitionMap.testReference.getErrorMessage = notCalledMock; + operationDefinitionMap.managedReference.getErrorMessage = mock; + const errors = getErrorMessages( + { + indexPatternId: '1', + columnOrder: [], + columns: { + col1: + // @ts-expect-error not statically analyzed + { operationType: 'managedReference', references: ['col2'] }, + col2: { + // @ts-expect-error not statically analyzed + operationType: 'testReference', + references: [], + }, + }, + }, + indexPattern, + {}, + '1', + {} + ); + expect(notCalledMock).not.toHaveBeenCalled(); + expect(mock).toHaveBeenCalledTimes(1); + expect(errors).toHaveLength(1); + }); + it('should ignore incompleteColumns when checking for errors', () => { const savedRef = jest.fn().mockReturnValue(['error 1']); const incompleteRef = jest.fn(); diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/layer_helpers.ts b/x-pack/plugins/lens/public/indexpattern_datasource/operations/layer_helpers.ts index 56fbb8edef5b43..b650a2818b2d48 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/operations/layer_helpers.ts +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/layer_helpers.ts @@ -169,6 +169,10 @@ export function insertNewColumn({ if (field) { throw new Error(`Can't create operation ${op} with the provided field ${field.name}`); } + if (operationDefinition.input === 'managedReference') { + // TODO: need to create on the fly the new columns for Formula, + // like we do for fullReferences to show a seamless transition + } const possibleOperation = operationDefinition.getPossibleOperation(); const isBucketed = Boolean(possibleOperation?.isBucketed); const addOperationFn = isBucketed ? addBucket : addMetric; @@ -358,9 +362,9 @@ export function replaceColumn({ tempLayer = resetIncomplete(tempLayer, columnId); if (previousDefinition.input === 'managedReference') { - // Every transition away from a managedReference resets it, we don't have a way to keep the state + // If the transition is incomplete, leave the managed state until it's finished. tempLayer = deleteColumn({ layer: tempLayer, columnId, indexPattern }); - return insertNewColumn({ + const hypotheticalLayer = insertNewColumn({ layer: tempLayer, columnId, indexPattern, @@ -368,6 +372,14 @@ export function replaceColumn({ field, visualizationGroups, }); + if (hypotheticalLayer.incompleteColumns && hypotheticalLayer.incompleteColumns[columnId]) { + return { + ...layer, + incompleteColumns: hypotheticalLayer.incompleteColumns, + }; + } else { + return hypotheticalLayer; + } } if (operationDefinition.input === 'fullReference') { @@ -859,7 +871,10 @@ function addBucket( visualizationGroups: VisualizationDimensionGroupConfig[], targetGroup?: string ): IndexPatternLayer { - const [buckets, metrics, references] = getExistingColumnGroups(layer); + const [buckets, metrics] = partition( + layer.columnOrder, + (colId) => layer.columns[colId].isBucketed + ); const oldDateHistogramIndex = layer.columnOrder.findIndex( (columnId) => layer.columns[columnId].operationType === 'date_histogram' @@ -873,12 +888,11 @@ function addBucket( addedColumnId, ...buckets.slice(oldDateHistogramIndex, buckets.length), ...metrics, - ...references, ]; } else { // Insert the new bucket after existing buckets. Users will see the same data // they already had, with an extra level of detail. - updatedColumnOrder = [...buckets, addedColumnId, ...metrics, ...references]; + updatedColumnOrder = [...buckets, addedColumnId, ...metrics]; } updatedColumnOrder = reorderByGroups( visualizationGroups, @@ -1169,8 +1183,20 @@ export function getErrorMessages( } > | undefined { - const errors = Object.entries(layer.columns) + const columns = Object.entries(layer.columns); + const visibleManagedReferences = columns.filter( + ([columnId, column]) => + !isReferenced(layer, columnId) && + operationDefinitionMap[column.operationType].input === 'managedReference' + ); + const skippedColumns = visibleManagedReferences.flatMap(([columnId]) => + getManagedColumnsFrom(columnId, layer.columns).map(([id]) => id) + ); + const errors = columns .flatMap(([columnId, column]) => { + if (skippedColumns.includes(columnId)) { + return; + } const def = operationDefinitionMap[column.operationType]; if (def.getErrorMessage) { return def.getErrorMessage(layer, columnId, indexPattern, operationDefinitionMap); @@ -1218,6 +1244,25 @@ export function isReferenced(layer: IndexPatternLayer, columnId: string): boolea return allReferences.includes(columnId); } +export function getReferencedColumnIds(layer: IndexPatternLayer, columnId: string): string[] { + const referencedIds: string[] = []; + function collect(id: string) { + const column = layer.columns[id]; + if (column && 'references' in column) { + const columnReferences = column.references; + // only record references which have created columns yet + const existingReferences = columnReferences.filter((reference) => + Boolean(layer.columns[reference]) + ); + referencedIds.push(...existingReferences); + existingReferences.forEach(collect); + } + } + collect(columnId); + + return referencedIds; +} + export function isOperationAllowedAsReference({ operationType, validation, diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/operations/mocks.ts b/x-pack/plugins/lens/public/indexpattern_datasource/operations/mocks.ts index 4a2e065269063a..2d7e70179fb3f7 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/operations/mocks.ts +++ b/x-pack/plugins/lens/public/indexpattern_datasource/operations/mocks.ts @@ -40,3 +40,28 @@ export const createMockedFullReference = () => { getErrorMessage: jest.fn(), }; }; + +export const createMockedManagedReference = () => { + return { + input: 'managedReference', + displayName: 'Managed reference test', + type: 'managedReference' as OperationType, + selectionStyle: 'full', + buildColumn: jest.fn((args) => { + return { + label: 'Test reference', + isBucketed: false, + dataType: 'number', + + operationType: 'testReference', + references: args.referenceIds, + }; + }), + filterable: true, + isTransferable: jest.fn(), + toExpression: jest.fn().mockReturnValue([]), + getPossibleOperation: jest.fn().mockReturnValue({ dataType: 'number', isBucketed: false }), + getDefaultLabel: jest.fn().mockReturnValue('Default label'), + getErrorMessage: jest.fn(), + }; +}; diff --git a/x-pack/plugins/lens/public/indexpattern_datasource/types.ts b/x-pack/plugins/lens/public/indexpattern_datasource/types.ts index 98dc767c44c7dd..f24c39f810b214 100644 --- a/x-pack/plugins/lens/public/indexpattern_datasource/types.ts +++ b/x-pack/plugins/lens/public/indexpattern_datasource/types.ts @@ -88,6 +88,8 @@ export interface IndexPatternPrivateState { isFirstExistenceFetch: boolean; existenceFetchFailed?: boolean; existenceFetchTimeout?: boolean; + + isDimensionClosePrevented?: boolean; } export interface IndexPatternRef { diff --git a/x-pack/plugins/lens/public/mocks.tsx b/x-pack/plugins/lens/public/mocks.tsx index 473c170aef2948..07935bb2f241b8 100644 --- a/x-pack/plugins/lens/public/mocks.tsx +++ b/x-pack/plugins/lens/public/mocks.tsx @@ -166,6 +166,9 @@ export function mockDataPlugin(sessionIdSubject = new Subject()) { nowProvider: { get: jest.fn(), }, + fieldFormats: { + deserialize: jest.fn(), + }, } as unknown) as DataPublicPluginStart; } diff --git a/x-pack/plugins/lens/public/types.ts b/x-pack/plugins/lens/public/types.ts index 33946166186722..b421d57dae6e1f 100644 --- a/x-pack/plugins/lens/public/types.ts +++ b/x-pack/plugins/lens/public/types.ts @@ -198,6 +198,11 @@ export interface Datasource { } ) => { dropTypes: DropType[]; nextLabel?: string } | undefined; onDrop: (props: DatasourceDimensionDropHandlerProps) => false | true | { deleted: string }; + /** + * The datasource is allowed to cancel a close event on the dimension editor, + * mainly used for formulas + */ + canCloseDimensionEditor?: (state: T) => boolean; getCustomWorkspaceRenderer?: ( state: T, dragging: DraggingIdentifier @@ -300,11 +305,15 @@ export type DatasourceDimensionEditorProps = DatasourceDimensionPro // Not a StateSetter because we have this unique use case of determining valid columns setState: ( newState: Parameters>[0], - publishToVisualization?: { shouldReplaceDimension?: boolean; shouldRemoveDimension?: boolean } + publishToVisualization?: { + isDimensionComplete?: boolean; + } ) => void; core: Pick; dateRange: DateRange; dimensionGroups: VisualizationDimensionGroupConfig[]; + toggleFullscreen: () => void; + isFullscreen: boolean; }; export type DatasourceDimensionTriggerProps = DatasourceDimensionProps; diff --git a/x-pack/plugins/lens/server/usage/schema.ts b/x-pack/plugins/lens/server/usage/schema.ts index ab3945a0162a68..c3608176717c56 100644 --- a/x-pack/plugins/lens/server/usage/schema.ts +++ b/x-pack/plugins/lens/server/usage/schema.ts @@ -14,6 +14,12 @@ const eventsSchema: MakeSchemaFrom = { type: 'long', _meta: { description: 'Number of times the user opened one of the in-product help popovers.' }, }, + toggle_fullscreen_formula: { + type: 'long', + _meta: { + description: 'Number of times the user toggled fullscreen mode on formula.', + }, + }, indexpattern_field_info_click: { type: 'long' }, loaded: { type: 'long' }, app_filters_updated: { type: 'long' }, @@ -162,6 +168,10 @@ const eventsSchema: MakeSchemaFrom = { type: 'long', _meta: { description: 'Number of times the moving average function was selected' }, }, + indexpattern_dimension_operation_formula: { + type: 'long', + _meta: { description: 'Number of times the formula function was selected' }, + }, }; const suggestionEventsSchema: MakeSchemaFrom = { @@ -183,6 +193,12 @@ const savedSchema: MakeSchemaFrom = { lnsDatatable: { type: 'long' }, lnsPie: { type: 'long' }, lnsMetric: { type: 'long' }, + formula: { + type: 'long', + _meta: { + description: 'Number of saved lens visualizations which are using at least one formula', + }, + }, }; export const lensUsageSchema: MakeSchemaFrom = { diff --git a/x-pack/plugins/lens/server/usage/visualization_counts.ts b/x-pack/plugins/lens/server/usage/visualization_counts.ts index 3b9bb99caf5b81..f0c48fb1152e81 100644 --- a/x-pack/plugins/lens/server/usage/visualization_counts.ts +++ b/x-pack/plugins/lens/server/usage/visualization_counts.ts @@ -43,6 +43,31 @@ export async function getVisualizationCounts( size: 100, }, }, + usesFormula: { + filter: { + match: { + operation_type: 'formula', + }, + }, + }, + }, + }, + }, + runtime_mappings: { + operation_type: { + type: 'keyword', + script: { + lang: 'painless', + source: `try { + if(doc['lens.state'].size() == 0) return; + HashMap layers = params['_source'].get('lens').get('state').get('datasourceStates').get('indexpattern').get('layers'); + for(layerId in layers.keySet()) { + HashMap columns = layers.get(layerId).get('columns'); + for(columnId in columns.keySet()) { + emit(columns.get(columnId).get('operationType')) + } + } + } catch(Exception e) {}`, }, }, }, @@ -56,16 +81,19 @@ export async function getVisualizationCounts( // eslint-disable-next-line @typescript-eslint/no-explicit-any function bucketsToObject(arg: any) { const obj: Record = {}; - arg.buckets.forEach((bucket: { key: string; doc_count: number }) => { + arg.byType.buckets.forEach((bucket: { key: string; doc_count: number }) => { obj[bucket.key] = bucket.doc_count + (obj[bucket.key] ?? 0); }); + if (arg.usesFormula.doc_count > 0) { + obj.formula = arg.usesFormula.doc_count; + } return obj; } return { - saved_overall: bucketsToObject(buckets.overall.byType), - saved_30_days: bucketsToObject(buckets.last30.byType), - saved_90_days: bucketsToObject(buckets.last90.byType), + saved_overall: bucketsToObject(buckets.overall), + saved_30_days: bucketsToObject(buckets.last30), + saved_90_days: bucketsToObject(buckets.last90), saved_overall_total: buckets.overall.doc_count, saved_30_days_total: buckets.last30.doc_count, saved_90_days_total: buckets.last90.doc_count, diff --git a/x-pack/plugins/telemetry_collection_xpack/schema/xpack_plugins.json b/x-pack/plugins/telemetry_collection_xpack/schema/xpack_plugins.json index 7c96dce3fac7f5..8e52450d393b03 100644 --- a/x-pack/plugins/telemetry_collection_xpack/schema/xpack_plugins.json +++ b/x-pack/plugins/telemetry_collection_xpack/schema/xpack_plugins.json @@ -2162,6 +2162,12 @@ "description": "Number of times the user opened one of the in-product help popovers." } }, + "toggle_fullscreen_formula": { + "type": "long", + "_meta": { + "description": "Number of times the user toggled fullscreen mode on formula." + } + }, "indexpattern_field_info_click": { "type": "long" }, @@ -2371,6 +2377,12 @@ "_meta": { "description": "Number of times the moving average function was selected" } + }, + "indexpattern_dimension_operation_formula": { + "type": "long", + "_meta": { + "description": "Number of times the formula function was selected" + } } } }, @@ -2385,6 +2397,12 @@ "description": "Number of times the user opened one of the in-product help popovers." } }, + "toggle_fullscreen_formula": { + "type": "long", + "_meta": { + "description": "Number of times the user toggled fullscreen mode on formula." + } + }, "indexpattern_field_info_click": { "type": "long" }, @@ -2594,6 +2612,12 @@ "_meta": { "description": "Number of times the moving average function was selected" } + }, + "indexpattern_dimension_operation_formula": { + "type": "long", + "_meta": { + "description": "Number of times the formula function was selected" + } } } }, @@ -2666,6 +2690,12 @@ }, "lnsMetric": { "type": "long" + }, + "formula": { + "type": "long", + "_meta": { + "description": "Number of saved lens visualizations which are using at least one formula" + } } } }, @@ -2709,6 +2739,12 @@ }, "lnsMetric": { "type": "long" + }, + "formula": { + "type": "long", + "_meta": { + "description": "Number of saved lens visualizations which are using at least one formula" + } } } }, @@ -2752,6 +2788,12 @@ }, "lnsMetric": { "type": "long" + }, + "formula": { + "type": "long", + "_meta": { + "description": "Number of saved lens visualizations which are using at least one formula" + } } } } diff --git a/x-pack/test/functional/apps/lens/formula.ts b/x-pack/test/functional/apps/lens/formula.ts new file mode 100644 index 00000000000000..e9e5051c006f02 --- /dev/null +++ b/x-pack/test/functional/apps/lens/formula.ts @@ -0,0 +1,198 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import expect from '@kbn/expect'; +import { FtrProviderContext } from '../../ftr_provider_context'; + +export default function ({ getService, getPageObjects }: FtrProviderContext) { + const PageObjects = getPageObjects(['visualize', 'lens', 'common', 'header']); + const find = getService('find'); + const listingTable = getService('listingTable'); + const browser = getService('browser'); + const testSubjects = getService('testSubjects'); + + describe('lens formula', () => { + it('should transition from count to formula', async () => { + await PageObjects.visualize.gotoVisualizationLandingPage(); + await listingTable.searchForItemWithName('lnsXYvis'); + await PageObjects.lens.clickVisualizeListItemTitle('lnsXYvis'); + await PageObjects.lens.goToTimeRange(); + + await PageObjects.lens.configureDimension({ + dimension: 'lnsXY_yDimensionPanel > lns-dimensionTrigger', + operation: 'average', + field: 'bytes', + keepOpen: true, + }); + + await PageObjects.lens.switchToFormula(); + await PageObjects.header.waitUntilLoadingHasFinished(); + // .echLegendItem__title is the only viable way of getting the xy chart's + // legend item(s), so we're using a class selector here. + // 4th item is the other bucket + expect(await find.allByCssSelector('.echLegendItem')).to.have.length(3); + }); + + it('should update and delete a formula', async () => { + await PageObjects.visualize.navigateToNewVisualization(); + await PageObjects.visualize.clickVisType('lens'); + await PageObjects.lens.goToTimeRange(); + await PageObjects.lens.switchToVisualization('lnsDatatable'); + + await PageObjects.lens.configureDimension({ + dimension: 'lnsDatatable_metrics > lns-empty-dimension', + operation: 'formula', + formula: `count(kql=`, + keepOpen: true, + }); + + const input = await find.activeElement(); + await input.type('*'); + + await PageObjects.header.waitUntilLoadingHasFinished(); + expect(await PageObjects.lens.getDatatableCellText(0, 0)).to.eql('14,005'); + }); + + it('should insert single quotes and escape when needed to create valid KQL', async () => { + await PageObjects.visualize.navigateToNewVisualization(); + await PageObjects.visualize.clickVisType('lens'); + await PageObjects.lens.goToTimeRange(); + await PageObjects.lens.switchToVisualization('lnsDatatable'); + + await PageObjects.lens.configureDimension({ + dimension: 'lnsDatatable_metrics > lns-empty-dimension', + operation: 'formula', + formula: `count(kql=`, + keepOpen: true, + }); + + let input = await find.activeElement(); + await input.type(' '); + await input.pressKeys(browser.keys.ARROW_LEFT); + await input.type(`Men's Clothing`); + + await PageObjects.common.sleep(100); + + let element = await find.byCssSelector('.monaco-editor'); + expect(await element.getVisibleText()).to.equal(`count(kql='Men\\'s Clothing ')`); + + await PageObjects.lens.typeFormula('count(kql='); + input = await find.activeElement(); + await input.type(`Men\'s Clothing`); + + element = await find.byCssSelector('.monaco-editor'); + expect(await element.getVisibleText()).to.equal(`count(kql='Men\\'s Clothing')`); + }); + + it('should persist a broken formula on close', async () => { + await PageObjects.visualize.navigateToNewVisualization(); + await PageObjects.visualize.clickVisType('lens'); + await PageObjects.lens.goToTimeRange(); + await PageObjects.lens.switchToVisualization('lnsDatatable'); + + // Close immediately + await PageObjects.lens.configureDimension({ + dimension: 'lnsDatatable_metrics > lns-empty-dimension', + operation: 'formula', + formula: `asdf`, + }); + + expect(await PageObjects.lens.getErrorCount()).to.eql(1); + }); + + it('should keep the formula when entering expanded mode', async () => { + await PageObjects.visualize.navigateToNewVisualization(); + await PageObjects.visualize.clickVisType('lens'); + await PageObjects.lens.goToTimeRange(); + await PageObjects.lens.switchToVisualization('lnsDatatable'); + + // Close immediately + await PageObjects.lens.configureDimension({ + dimension: 'lnsDatatable_metrics > lns-empty-dimension', + operation: 'formula', + formula: `count()`, + keepOpen: true, + }); + + await PageObjects.lens.toggleFullscreen(); + + const element = await find.byCssSelector('.monaco-editor'); + expect(await element.getVisibleText()).to.equal('count()'); + }); + + it('should allow an empty formula combined with a valid formula', async () => { + await PageObjects.visualize.navigateToNewVisualization(); + await PageObjects.visualize.clickVisType('lens'); + await PageObjects.lens.goToTimeRange(); + await PageObjects.lens.switchToVisualization('lnsDatatable'); + + await PageObjects.lens.configureDimension({ + dimension: 'lnsDatatable_metrics > lns-empty-dimension', + operation: 'formula', + formula: `count()`, + }); + await PageObjects.lens.configureDimension({ + dimension: 'lnsDatatable_metrics > lns-empty-dimension', + operation: 'formula', + }); + + await PageObjects.header.waitUntilLoadingHasFinished(); + expect(await PageObjects.lens.getErrorCount()).to.eql(0); + }); + + it('should duplicate a moving average formula and be a valid table', async () => { + await PageObjects.visualize.navigateToNewVisualization(); + await PageObjects.visualize.clickVisType('lens'); + await PageObjects.lens.goToTimeRange(); + await PageObjects.lens.switchToVisualization('lnsDatatable'); + + await PageObjects.lens.configureDimension({ + dimension: 'lnsDatatable_rows > lns-empty-dimension', + operation: 'date_histogram', + field: '@timestamp', + }); + + await PageObjects.lens.configureDimension({ + dimension: 'lnsDatatable_metrics > lns-empty-dimension', + operation: 'formula', + formula: `moving_average(sum(bytes), window=5`, + keepOpen: true, + }); + await PageObjects.lens.closeDimensionEditor(); + + await PageObjects.lens.dragDimensionToDimension( + 'lnsDatatable_metrics > lns-dimensionTrigger', + 'lnsDatatable_metrics > lns-empty-dimension' + ); + expect(await PageObjects.lens.getDatatableCellText(1, 1)).to.eql('222420'); + expect(await PageObjects.lens.getDatatableCellText(1, 2)).to.eql('222420'); + }); + + it('should keep the formula if the user does not fully transition to a quick function', async () => { + await PageObjects.visualize.navigateToNewVisualization(); + await PageObjects.visualize.clickVisType('lens'); + await PageObjects.lens.goToTimeRange(); + await PageObjects.lens.switchToVisualization('lnsDatatable'); + + await PageObjects.lens.configureDimension({ + dimension: 'lnsDatatable_metrics > lns-empty-dimension', + operation: 'formula', + formula: `count()`, + keepOpen: true, + }); + + await PageObjects.lens.switchToQuickFunctions(); + await testSubjects.click(`lns-indexPatternDimension-min incompatible`); + await PageObjects.common.sleep(1000); + await PageObjects.lens.closeDimensionEditor(); + + expect(await PageObjects.lens.getDimensionTriggerText('lnsDatatable_metrics', 0)).to.eql( + 'count()' + ); + }); + }); +} diff --git a/x-pack/test/functional/apps/lens/index.ts b/x-pack/test/functional/apps/lens/index.ts index 1efceace8b1676..99b75bdabe6c4a 100644 --- a/x-pack/test/functional/apps/lens/index.ts +++ b/x-pack/test/functional/apps/lens/index.ts @@ -41,6 +41,7 @@ export default function ({ getService, loadTestFile }: FtrProviderContext) { loadTestFile(require.resolve('./geo_field')); loadTestFile(require.resolve('./lens_reporting')); loadTestFile(require.resolve('./lens_tagging')); + loadTestFile(require.resolve('./formula')); // has to be last one in the suite because it overrides saved objects loadTestFile(require.resolve('./rollup')); diff --git a/x-pack/test/functional/apps/lens/smokescreen.ts b/x-pack/test/functional/apps/lens/smokescreen.ts index 2a4d56bbea791a..5d775f154c9430 100644 --- a/x-pack/test/functional/apps/lens/smokescreen.ts +++ b/x-pack/test/functional/apps/lens/smokescreen.ts @@ -172,10 +172,11 @@ export default function ({ getService, getPageObjects }: FtrProviderContext) { await testSubjects.existOrFail('indexPattern-dimension-formatDecimals'); + await PageObjects.lens.closeDimensionEditor(); + expect(await PageObjects.lens.getDimensionTriggerText('lnsXY_yDimensionPanel')).to.eql( 'Test of label' ); - await PageObjects.lens.closeDimensionEditor(); }); it('should be able to add very long labels and still be able to remove a dimension', async () => { @@ -587,6 +588,57 @@ export default function ({ getService, getPageObjects }: FtrProviderContext) { ); }); + it('should not leave an incomplete column in the visualization config with field-based operation', async () => { + await PageObjects.visualize.navigateToNewVisualization(); + await PageObjects.visualize.clickVisType('lens'); + await PageObjects.lens.goToTimeRange(); + + await PageObjects.lens.configureDimension({ + dimension: 'lnsXY_yDimensionPanel > lns-empty-dimension', + operation: 'min', + }); + + expect(await PageObjects.lens.getDimensionTriggerText('lnsXY_yDimensionPanel')).to.eql( + undefined + ); + }); + + it('should not leave an incomplete column in the visualization config with reference-based operations', async () => { + await PageObjects.visualize.navigateToNewVisualization(); + await PageObjects.visualize.clickVisType('lens'); + await PageObjects.lens.goToTimeRange(); + + await PageObjects.lens.configureDimension({ + dimension: 'lnsXY_xDimensionPanel > lns-empty-dimension', + operation: 'date_histogram', + field: '@timestamp', + }); + await PageObjects.lens.configureDimension({ + dimension: 'lnsXY_yDimensionPanel > lns-empty-dimension', + operation: 'moving_average', + field: 'Records', + }); + + expect(await PageObjects.lens.getDimensionTriggerText('lnsXY_yDimensionPanel')).to.eql( + 'Moving average of Count of records' + ); + + await PageObjects.lens.configureDimension({ + dimension: 'lnsXY_yDimensionPanel > lns-dimensionTrigger', + operation: 'median', + isPreviousIncompatible: true, + keepOpen: true, + }); + + expect(await PageObjects.lens.isDimensionEditorOpen()).to.eql(true); + + await PageObjects.lens.closeDimensionEditor(); + + expect(await PageObjects.lens.getDimensionTriggerText('lnsXY_yDimensionPanel')).to.eql( + undefined + ); + }); + it('should transition from unique count to last value', async () => { await PageObjects.visualize.navigateToNewVisualization(); await PageObjects.visualize.clickVisType('lens'); diff --git a/x-pack/test/functional/page_objects/lens_page.ts b/x-pack/test/functional/page_objects/lens_page.ts index c0111afad28932..44aebed17925d3 100644 --- a/x-pack/test/functional/page_objects/lens_page.ts +++ b/x-pack/test/functional/page_objects/lens_page.ts @@ -107,6 +107,7 @@ export function LensPageProvider({ getService, getPageObjects }: FtrProviderCont isPreviousIncompatible?: boolean; keepOpen?: boolean; palette?: string; + formula?: string; }, layerIndex = 0 ) { @@ -114,10 +115,15 @@ export function LensPageProvider({ getService, getPageObjects }: FtrProviderCont await testSubjects.click(`lns-layerPanel-${layerIndex} > ${opts.dimension}`); await testSubjects.exists(`lns-indexPatternDimension-${opts.operation}`); }); - const operationSelector = opts.isPreviousIncompatible - ? `lns-indexPatternDimension-${opts.operation} incompatible` - : `lns-indexPatternDimension-${opts.operation}`; - await testSubjects.click(operationSelector); + + if (opts.operation === 'formula') { + await this.switchToFormula(); + } else { + const operationSelector = opts.isPreviousIncompatible + ? `lns-indexPatternDimension-${opts.operation} incompatible` + : `lns-indexPatternDimension-${opts.operation}`; + await testSubjects.click(operationSelector); + } if (opts.field) { const target = await testSubjects.find('indexPattern-dimension-field'); @@ -125,6 +131,10 @@ export function LensPageProvider({ getService, getPageObjects }: FtrProviderCont await comboBox.setElement(target, opts.field); } + if (opts.formula) { + await this.typeFormula(opts.formula); + } + if (opts.palette) { await this.setPalette(opts.palette); } @@ -357,7 +367,7 @@ export function LensPageProvider({ getService, getPageObjects }: FtrProviderCont await retry.try(async () => { await testSubjects.click('lns-palettePicker'); const currentPalette = await ( - await find.byCssSelector('[aria-selected=true]') + await find.byCssSelector('[role=option][aria-selected=true]') ).getAttribute('id'); expect(currentPalette).to.equal(palette); }); @@ -379,6 +389,18 @@ export function LensPageProvider({ getService, getPageObjects }: FtrProviderCont }); }, + async isDimensionEditorOpen() { + return await testSubjects.exists('lns-indexPattern-dimensionContainerBack'); + }, + + // closes the dimension editor flyout + async closeDimensionEditor() { + await retry.try(async () => { + await testSubjects.click('lns-indexPattern-dimensionContainerBack'); + await testSubjects.missingOrFail('lns-indexPattern-dimensionContainerBack'); + }); + }, + async enableTimeShift() { await testSubjects.click('indexPattern-advanced-popover'); await retry.try(async () => { @@ -398,14 +420,6 @@ export function LensPageProvider({ getService, getPageObjects }: FtrProviderCont await testSubjects.click('errorFixAction'); }, - // closes the dimension editor flyout - async closeDimensionEditor() { - await retry.try(async () => { - await testSubjects.click('lns-indexPattern-dimensionContainerBack'); - await testSubjects.missingOrFail('lns-indexPattern-dimensionContainerBack'); - }); - }, - async isTopLevelAggregation() { return await testSubjects.isEuiSwitchChecked('indexPattern-nesting-switch'); }, @@ -549,7 +563,8 @@ export function LensPageProvider({ getService, getPageObjects }: FtrProviderCont }); } const errors = await testSubjects.findAll('configuration-failure-error'); - return errors?.length ?? 0; + const expressionErrors = await testSubjects.findAll('expression-failure'); + return (errors?.length ?? 0) + (expressionErrors?.length ?? 0); }, async searchOnChartSwitch(subVisualizationId: string, searchTerm?: string) { @@ -1025,5 +1040,27 @@ export function LensPageProvider({ getService, getPageObjects }: FtrProviderCont ); await PageObjects.header.waitUntilLoadingHasFinished(); }, + + async switchToQuickFunctions() { + await testSubjects.click('lens-dimensionTabs-quickFunctions'); + }, + + async switchToFormula() { + await testSubjects.click('lens-dimensionTabs-formula'); + }, + + async toggleFullscreen() { + await testSubjects.click('lnsFormula-fullscreen'); + }, + + async typeFormula(formula: string) { + // Formula takes time to open + await PageObjects.common.sleep(500); + await find.byCssSelector('.monaco-editor'); + await find.clickByCssSelectorWhenNotDisabled('.monaco-editor'); + const input = await find.activeElement(); + await input.clearValueWithKeyboard(); + await input.type(formula); + }, }); }