Missing features and known differences

[xvello]

Missing features before rolling-out to team2

• kafka message key for {token}:{distinct_id} locality
• implement utf16 escaping, see safe_clickhouse_string invocations in capture.py. The plugin-server equivalent is easier to read. [slack thread] -> serde rejects such payloads, see this documentation
• fill in data, use django_compat test for confirmation -> maybe skip some of the data massaging if the tests pass without, list in "other sdks compat" section
• CORS support, see django code for it
• tuning via envvars, easy to add new options

Missing features before rolling-out to customers - posthog-js only

• Overflow detection (local for now, reuse same algorithm) and LIKELY_ANONYMOUS_IDS
• how do we handle custom proxies that might not forward /i/?
• Billing limits (needs a redis client, update the values out-of-band, fail open)
• Kafka writes timeouts and error handling, maybe implement limited retries? -> rdkafka handles retries for us, up to 5 minutes by default. We'll timeout at the nginx level for now, and keep the messages in the rdkafka produce queue

Missing features for compat with other SDKs

For all these (and the ones we'll add), let's instrument the django code path to check whether it's actually active, and how many teams would be impacted

• Source sent_at from the event body if present (used by some sdks + custom clients)
• Source sent_at from the body on x-www-form-urlencoded requests: old posthog-js versions?
• Source events from toplevel batch field if present
• Check whether we indeed silently drop events with missing fields as documented, instead of returning an error -> if we keep dropping, let's implement an ingestion warning for this!

Known differences with django capture

These won't be fixed unless we aim at being compatible with the long tail of posthog-js versions:

• the raw kafka message does not hold a site_url anymore, it looks unused now -> confirm it's the case
• no support for lz64 compression, was removed from posthog-js
• no support for the /engage endpoint, we can leave it routed to django
• events bigger than the maxkafka message size trigger an INVALID_REQUEST status instead of a INTERNAL_ERROR
• dates written to kafka are in RFC3339 format, a subset of ISO8601 that plugin-server should accept OK. Let's make sure CH does (partition_stats consumer)
• sent_at timestamp in second not supported. Will be ignored, and event timestamp used without correction

[xvello]

Scratchpad

• Overflow
• Alerting - https://github.com/PostHog/charts/pull/514
• Tracing deploy on prod
• Rollout to free tier
• More trace spans
• Re-organize code to prepare for replay endpoint
• Get list of custom proxy domains to confirm /i/ is properly forwarded
• Delete EventSink::send (always use send_batch)