Scrape FIPS algorithm data

[J08nY]

Initial description by @J08nY

Data from the FIPS algorithm dataset is not utilized and mined fully. We can follow the links to the algorithm page and get more data that will help us. This can help in cert id cleanup to get rid of the algo references.

Details

Currently, the FIPSAlgorithm object is built from rows of a pandas DataFrame constructed merely from the list of Algorithms, see below

sec-certs/src/sec_certs/dataset/fips_algorithm.py

Line 98 in f41d077

df = pd.read_html(html_path)[0]

This table does not include valuable attributes found on the individual pages of the algorithm. The proposed enhancement should:

• Track the URL for each of the algorithms, e.g., https://csrc.nist.gov/projects/Cryptographic-Algorithm-Validation-Program/details?product=3989
• Scrape the contents of the corresponding html and extract the values for the following attributes:
  • algorithm type
  • description
  • version
  • algorithm capabilities
• The FIPSAlgorithm object (see below) should be enriched with the attributes mentioned above.

sec-certs/src/sec_certs/sample/fips_algorithm.py

Line 13 in f41d077

class FIPSAlgorithm(PandasSerializableType, ComplexSerializableType):

Further guidance

One can isolate the pipeline stage that processes the algorithm dataset simply by

from sec_certs.dataset.fips_algorithm import FIPSAlgorithmDataset

alg_dset = FIPSAlgorithmDataset.from_web()
alg_dset.to_json("/path/to/some/file.json")

The PR implementing this enhancement should modify the parse_algorithms_from_html method.

[adamjanovsky]

Just FYI, the current state of #275 is that I've refactored building of a dataset of FIPS Algorithms. The certificate only store strings of the algorithm identifiers and are nowhere connected to the respecitve objects. So once we improve algorithm scraping, we could connect these two datasets as well.

[adamjanovsky]

@Julik24 this is the task that we've discussed today. Before attempting to contribute, please be sure to go through https://sec-certs.org/docs/contributing.html, especially the Quality Assurance section. The typical development workflow is described at https://docs.github.com/en/get-started/using-github/github-flow.

Please, assign yourself to the issue once you accept the invitation.