Let users know data is sampled

[ccschmitz]

In some instances we hit the sampling tables for logs and traces. This can result in some discrepancies between what we show in the UI and what we have in our DB. We also have some issues where the stats we show for totals don't match the count of results. We want to show some UI explaining that the counts might be off a little.

We would also like to update our fetching logic to hit the main table if the sampling table doesn't return any results.

[linear]

HIG-4646 Let users know data is sampled

[greptile-apps]

To address the issue of data sampling discrepancies, follow these steps:

1. Backend (logs.go): Modify the ReadLogs and ReadLogsHistogram functions to check the main logs table if the sampling table returns no results. Implement a fallback mechanism within these functions.

   Example change for ReadLogs:

   if len(results) == 0 {
       // Fallback to main logs table
       // Re-run the query against LogsTableConfig instead of logsSamplingTableConfig
   }

2. Frontend (TraceLogs.tsx): Add a UI element (e.g., a banner or tooltip) to inform users about potential count discrepancies due to data sampling. This could be conditionally rendered based on a flag set by the backend indicating whether sampling data was used.

   Example UI update:

   {isSampled && <Alert message="Data is sampled and counts may not accurately reflect total records." type="warning" />}

3. Documentation/Comments: Update code comments and documentation to clearly explain the logic for handling sampled data and the potential for discrepancies in counts. This will help maintain clarity for future development and debugging.

References

/backend/clickhouse/logs.go
/frontend/src/pages/Traces/TraceLogs.tsx
/backend/clickhouse/migrations/000024_create_logs_sampling_table.up.sql
/backend/clickhouse/migrations/000043_create_traces_sampling_table.up.sql
/docs-content/general/6_product-features/1_session-replay/filtering-sessions.md
/docs-content/general/6_product-features/2_error-monitoring/filtering-errors.md

Ask Greptile