OWASP Secure Headers Project statistics

Statistics about HTTP response security headers usage mentioned by the OWASP Secure Headers Project (OSHP).

See this issue for details.

Data source

MAJESTIC was used instead of the CISCO Top 1 million sites CSV file because it contain less malware domains.

• MAJESTIC Top 1 million sites list.
• CISCO Top 1 million sites CSV file.

# Download the MAJESTIC Top 1 million sites CSV file
$ wget http://downloads.majestic.com/majestic_million.csv
# Transform the downloaded file to an input source that use the same format 
# than the CISCO Top 1 million sites CSV file
$ cat majestic_million.csv | awk -F  "," 'NR>1 {print $1 "," $3}' > data/input.csv
$ rm majestic_million.csv

Scripts

They are all stored in the scripts folder and they are Python 3.x based.

The script oshp_headers contains the HTTP headers suggested by the OSHP project. It is a configuration material.

Visual Studio Code is used for the scripts development. A Visual Studio Code workspace file is provided for the project with recommended extensions.

Files:

• gather_data: Script gathering the information about HTTP security headers usage in a SQLITE database based on the "MAJESTIC Top 1 million sites CSV file" data source.
• generate_stats_md_file: Script using the gathered data to generate/update the markdown file stats, with mermaid pie charts with differents statistics about HTTP security headers usage.

Data

They are all stored in the data folder.

Files:

• input.csv: MAJESTIC Top 1 million sites list formated as one entry ranking,domain by line.
• data.db: SQLITE database with information about HTTP security headers usage.

Data and statistics update

ℹ️ Only the first 200000 entries of the CSV datasource are used to fit the processing timeframe allowed for a github action workfows using the free tiers.

The update is scheduled in the following way via two dedicated github actions workfows:

• The first day of every month the data database is updated.
• The second day of every month the statistic file is updated:
  • The files input.csv, data.db and stats.md are attached to the build as artefacts. Therefore, it is possible to access to the statistics and its corresponding input data for dates in the past.

Note

• If other type of charts are need then the quickchart.io python API will be used (free part).
• DB Browser for SQLite can be used to access to the raw data of the SQLITE DB.