search-logger

This is a tool which logs user searches to our "reporting" Elasticsearch cluster. These logs include:

• what search term somebody typed in
• how many results we returned
• which results (if any) that they clicked on.

We use this to analyse search behaviour and improve our queries. For example, we can look at searches that return 0 results and discuss whether there really are no results, or whether we should change the results we return.

How search events get logged

flowchart LR
    W[website] --> S[Segment]
    S[Segment] --> K[Kinesis stream]
    K --> L[search logger Lambda]
    L --> R[(reporting<br/>cluster)]

    classDef externalNode fill:#e8e8e8,stroke:#8f8f8f
    class S,R,W externalNode

    classDef repoNode fill:#c8ecee,stroke:#298187,stroke-width:2px
    class K,L repoNode

Our website sends tracking events to Segment. Those segments are forwarded to a Kinesis data stream, which triggers a Lambda function. That Lambda function writes the search logs into the reporting cluster.

This repo contains:

• the source code for the Lambda function
• the Terraform definitions for the Kinesis stream and the Lambda function

Developer info

• To deploy a new version the function:

  $ cd lambda
  $ yarn deploy
  $ cd ../terraform
  $ ./run_terraform.sh apply

• You can see the Lambda logs in the logging cluster.