Google Summer of Code 2022 application

[GMishx]

Hello all!

Update: The wiki for 2022 is available at https://github.com/fossology/fossology/wiki/Google-Summer-of-Code-Proposals-2022

FOSSology as an org is planning to apply for Google Summer of Code this year as well. In preparation to do so, I am creating this discussion to gather ideas for the GSoC projects.

This year, there are some changes made by Google in the program. You can read more about it on their website. But as a summary,

• Two sized projects are acceptable by program:
  • Medium sized projects (~175 hours)
  • Large projects (~350 hours)
• The program is open to 18 years and older people.
  • No longer limited to college students only.
• Flexible time lines between 12 and 22 weeks.

Please feel free to drop any questions here (or start a new discussion) you have regarding the program, you want to submit a proposal idea, you want to be mentor in the program, etc.

While submitting a project idea, please tag it to be "Medium" or "Large" sized and who you'll prefer working on it "student" or "professional".
Some resources from Google:

• https://google.github.io/gsocguides/mentor/defining-a-project-ideas-list
• https://google.github.io/gsocguides/mentor/making-your-ideas-page

[GMishx]

Idea!

Title: SPDX naming updates and reporting

Goal: Update SPDX license names and support for report formats

1. Updating SPDX license names to new naming convention. Resource
2. Fixing issues with SPDX RDF (see notes bellow).
3. Other SPDX format reports, five file formats possible:

• Tag/value (.spdx)
  • Exists
• JSON(.spdx.json)
  • Can be implemented
• YAML(.spdx.yml)
  • Can be implemented
• RDF/xml(spdx.rdf)
  • Exists
• Spreadsheets (.xls)
  • Need to understand if helpful
  • Generate a report SPDX->XLS #1379

(See #2133 for sample implementation of simple formats)
(See https://github.com/spdx/spdx-spec/tree/development/v2.2.2/examples)

Category	Rating
Low Hanging Fruit	***
Risk/Exploratory	**
Fun/Periphial	**
Core Development	***
Project Infrastructure	***
Project size	Medium
Preferred contributor	Student

Notes from SPDX DocFest:

• File types are sometime OTHER vs SOURCE
• Originator/Supplier missing
• Download Location is NOASSERTION
• Package verification code is wrong. Should be all the source files
  • https://spdx.github.io/spdx-spec/package-information/#79-package-verification-code-field
• Filename should start with './' https://spdx.github.io/spdx-spec/package-information/#74-package-file-name-field
• Should rename file name as per https://spdx.github.io/spdx-spec/conformance/#44-standard-data-format-requirements

[Toniprni]

As a suggestion, I would add to this the following task: SPDX RDF should export concluded licenses as conjunctive licenses not disjunctive licenses.

https://spdx.github.io/spdx-spec/SPDX-license-expressions/#d46-license-expressions-in-rdf

Currently, in SPDX RDF exports, all license conclusions are generated as disjunctive license sets. Disjunctive licenses refer to license sets with the OR operator. It would be better for Fossology to generate license conclusions in the form of conjunctive licenses, i.e. with the AND operator because Fossology’s results are more of a list of licenses per file and are not really an organized SPDX expression. Fossology does not currently process licenses with an OR operator other than by adding a Dual-license finding to the file.

Also if the licenses would be exported as conjunctive licenses, the SPDX RDF could be exported from one Fossology and imported into another without making any changes to the SPDX RDF itself (if licenses are imported as disjunctive into Fossology, it will automatically add the Dual-license tag even to licenses which are not dual-licensed). This would be great as it would enable to distribute one's own work to several Fossology instances via SPDX.

[GMishx]

Idea to solve SPDX license name issue:
Since there can be different variants of a license in a FOSSology server to encompass different license text, replacing the existing license name with SPDX license name will cause collisions there. For example, one might have a license which looks like BSD-3-Clause but with different license text like NumPy license.

However, if we add a new field called something like "SPDX license name" and this name then can be used while generating reports and the old name still used in the UI to make it easy to identify different licenses.

Related issues:

• SPDX coverage from FOSSology project #1309
• SPDX RDF export writes conjunctive licenses under <spdx:DisjunctiveLicenseSet> #1693
• SPDX import does not conclude licenses based on hash values but on file name #1868

[GMishx]

Idea!

Title: REST API and UI improvements

Goal: Bringing new FOSSologyUI towards completion

1. Continue work on React repo.

• Bring in new features.
• New design patterns.

2. Improve REST API and expose more endpoints

• https://github.com/fossology/fossology/issues?q=is%3Aissue+is%3Aopen+REST
• https://github.com/Shruti3004/FOSSology-REST-API/issues

Category	Rating
Low Hanging Fruit	-
Risk/Exploratory	*
Fun/Periphial	***
Core Development	**
Project Infrastructure	***
Project size	Large
Preferred contributor	Student/Professional

[GMishx]

Idea!

Title: Integrating Open Source Review Toolkit

Goal: Using ORT to fetch dependencies and generate SBOM

Build systems fetch the required dependencies (library/artifact) for a project while building the project. Its important to get an insight of these dependencies for license compliance check.

The OSS Review Toolkit is an open source project helps to find dependencies in a project.

The goal of this project is to render the project dependencies created by ort and display those in the fossology-UI. Dependencies can be scheduled directly from the UI and scan with fossology.

Category	Rating
Low Hanging Fruit	-
Risk/Exploratory	-
Fun/Periphial	**
Core Development	***
Project Infrastructure	*
Project size	Large
Preferred contributor	Student/Professional

[sameer1046]

oss-review-toolkit/ort#2694
This is the other way I guess

[GMishx]

Idea!

Title: Adopting REUSE standards in FOSSology

Goal: Adopting REUSE.software specs in FOSSology codebase

Copyright and licensing is difficult, especially when reusing software from different projects that are released under various different licenses. REUSE make it easier for you to declare the licenses under which your works are released, but they also make it easier for a computer to understand how your project is licensed. This specification defines a standardized method for declaring copyright and licensing for software projects. REUSE also helps in creating a bill of materials with just one simple command.

FOSSology currently uses old methods of defining licenses on source files which can lead to some ambiguity. Following the REUSE specs, the code base of FOSSology should be updated with new licensing format.

Note: On its own, the project is very small for being a GSoC project. We can club it with other tasks, like working on issue #1592

Category	Rating
Low Hanging Fruit	**
Risk/Exploratory	*
Fun/Periphial	*
Core Development	*
Project Infrastructure	***
Project size	Small
Preferred contributor	Student

[GMishx]

Idea!

Title: Improving FOSSology CI scanner image

Goal: Enhancing current scanner image with new reports and features

As a fun project, FOSSology started combining scanners in a simple and small Docker image which can be run on CI providers. The image is currently capable of understanding build environment (GitLab/GitHub Actions/Travis) and use their API's to fetch diff of a branch or scan the complete repo. The capabilities of image include license scanning with Nomos and ojo scanners, copyright and keyword scanning with respective scanners. The image makes use of a Python script to perform all the tasks.

1. The script, however, currently prints report in text format on console and generate the same as an artifact. It can be improved by generating reports in other formats like SPDX reports. See their Python API.
2. The integration with GitHub Actions can be improved by reporting line number where a license violation is found.
3. Allowing user to provide a different list of Keywords for scanning (currently stored at /usr/local/share/fossology/keyword/agent/keyword.conf).
4. Improving on whitelist format with feature to provide it from other sources, currently it is read from a file which is expected to be in the root of repo being scanned.

Category	Rating
Low Hanging Fruit	-
Risk/Exploratory	**
Fun/Periphial	***
Core Development	**
Project Infrastructure	***
Project size	Large
Preferred contributor	Student/Professional

[GMishx]

More things to do:

1. Introduce concept of project in FOSSology.

• See Create a notion of "Projects" #1738

2. Create a server backed image

• There will be a server running on a remote machine.
• The CI script will push the package to the server for scan with auto conclusion.
• The package will become a version of a project.
• Downloads the report and make it available as an artifact.
• If there are unidentified licenses left, provide a link for manual review.
• This can be extended to integrate with ticket management systems like Jira and redmine.

[NicolasToussaint]

Thanks @GMishx ; turned point 1 into an "Idea!" below

[GMishx]

Idea!

Title: Enhancement with ClearlyDefined.io (spasht)

Goal: Contribute compliance metadata back to ClearlyDefined.io and community

Clearly defined is a project for collecting metadata about published software. This metadata shall help, among other things, for achieving OSS license compliance. More info can be found at:

• Docs -- https://docs.clearlydefined.io
• GitHub -- https://github.com/clearlydefined/clearlydefined
• API Docs -- https://api.clearlydefined.io/api-docs/#/curations/patch_curations
• Discord -- http://discord.gg/wEzHJku
• Twitter -- http://twitter.com/clearlydefd

The spasht agent is already pulling the data from ClearlyDefined, following enhancements are required:

• Fetch the main license of the package.
• What to do if package does not exist on ClearlyDefined.io
• Push the curated data back to ClearlyDefined.io
  • Current understanding: Send a patch request to PATCH /curations and in response, there will be a GitHub prNumber and url

  • {
      "contributionInfo": {
        "summary": "title (100 char)",
        "details": "What's the problem",
        "resolution": "What's fixed and how",
        "type": "missing/incorrect/incomplete/ambiguous/other",
        "removeDefinitions": false
      },
      "patches": [
        {
          "coordinates": {"type": "composer", "provider": "packagist",
            "namespace": "athome", "name": "odtphp", "revision": "1.5"},
          "revisions": {
            "1.5": {
              "files": [
                {
                  "path": "Listeur-odtphp-9f31202/library/zip/pclzip/pclzip.lib.php",
                  "license": "LGPL-2.1-or-later",
                  "attributions": [
                    "123123"
                  ]
                }
              ]
            }
          }
        }
      ]
    }

Category	Rating
Low Hanging Fruit	**
Risk/Exploratory	*
Fun/Periphial	**
Core Development	**
Project Infrastructure	*
Project size	Medium
Preferred contributor	Student/Professional

[Kaushl2208]

Idea!

Title: Compatibility for PHP-8

Goal: FOSSology should be compatible with the PHP-8 version

1. Syntax compatibility for PHP-8
2. Backward compatibility with PHP-7.2.24
3. Omit all the deprecated features.
4. Migrating views from plain PHP to Twig.
5. Introduction to OPcache or other compatible caching mechanism
6. FOSSology should run on top of PHP8

References

(See #1925 for sample implementation)
(See #2107 for new dependencies)

Docs:

PHP Migration Docs: https://www.php.net/manual/en/migration80.php
Twig Docs: https://twig.symfony.com/doc/3.x/

Category	Rating
Low Hanging Fruit	*
Risk/Exploratory	**
Fun/Periphial	*
Core Development	***
Project Infrastructure	***
Project size	Medium
Preferred contributor	Student

Good to have: Improving the MVC architecture of Fossology using Symfony

Symfony as of now is used purely for dependency injection but it can do a lot more. Introducing the capabilities and making the architecture more stable is a good extension to the project.

[NicolasToussaint]

Idea!

Title: Introduce concept of project in FOSSology.

Originally described in #1738

Goal: Gather uploads under identified projects

A project would be defined by:

• a project ID (as primary key)
• a project name
• a root folder in Fossology
• one or more Group IDs and associated access.
• a URL / ID / Free field to link the project to an external tool

This could be used, for example:

• to gather multiple uploads under the same umbrella,
  • and make bulk changes changes to all uploads / sub-directories that belong to a given project
  • reuse (cleared) findings from previous scan of same Project (or same component)
• to associate a link to a ticketing system or project management solution
• to produce metrics with Work-In-Progress dashboard

In terms of UI integration:

• Add an optional selection of project for new uploads
• Add pages to list, edit, delete projects (in Organize or Admin menu)
• Display the project name in the "Folder Navigation" window, and in the yellow band

Other extensions:

• inside Projects, create Components
  • have a proper name
  • are versioned
• leverage the use of Fossology tags

Practical use example:

• I have a new project FooBar
• This project is composed of 3 components: Front-end, Back-end and Mobile application
• I want uploads to be stored in folders projects/FooBar/Front-end, ...
• I want to be able to give permissions for all Folders and Uploads for that Project
• I want to automatically reuse findings from previous scans of same components

Category	Rating
Low Hanging Fruit	*
Risk/Exploratory	**
Fun/Periphial	*
Core Development	***
Project Infrastructure	*
Project size	Medium
Preferred contributor	Student

[hastagAB]

Idea!

Title: Improve Minerva OSS Dataset and implement models for Atarashi

OSS Dataset Repository: Minerva Dataset Generation
Atarashi Repository: FOSSology/Atarashi

Goal: To implement a semantic text model for finding out OSS license similarity with best accuracy

In GSoC 2021 @FOSSology we have created our initial OSS License Dataset "Minerva Dataset Generation" which can be used to build Machine Learning models for license detection in Atarashi. We are now planning to try out few machine learning models to be trained on our dataset for the use case of Atarashi (OSS License Detection). As the open source license texts differ by very few tokens only which changes the meaning of the text. Semantic Similarity models might be the good place to try but we are open to discuss more about any other models that can fit our use case perfectly.

Tasks:

• Improve Minerva Dataset Generation for more accurate license files
• Research and suggest various ML/DL models to be implemented for Atarashi (open for discussion)
• Implement the best model discussed for Atarashi
• Improve the performance

Category	Rating
Low	Hanging Fruit
Risk/Exploratory	*
Fun/Periphial	***
Core Development	**
Project Infrastructure	*
Project size	Medium
Preferred contributor	Student/Professional

[GMishx]

Idea!

Title: Overhauling scheduler design

Goal: Improving FOSSology scheduler or replacing with OTS solution

The existing scheduler design is causing new issues which need to be addressed. Moreover, existing scheduler design is not touched in years.

Concerning points

1. The scheduler is written in C which makes it next to impossible to find cause of a failure.
2. The C language does not support exception handling out of the box. It makes code less readable and prone to errors.
3. The linear queue design causes issue when there should be only one instance of an agent running for an upload, but overall the agent is not mutually exclusive.

   For example, if the monkbulk has a limit set to 1, it should be implied for only single upload. But with linear queue, this monkbulk job will block all other agents from executing even when they are not effected by the results of monkbulk.
   This essentially makes the agent mutually exclusive even though, there is a special flag EXCLUSIVE for the very same purpose: https://github.com/fossology/fossology/wiki/Job-Scheduler#agentconfs

• One idea on redesigning the queue, it can be broken into buckets per upload each maintaining its own priority queue. There can be another queue for global operations like maintenance, delagent, etc.
• Doing so, each bucket can be traversed in round-robin and pick first pending job and check against host limit. This will eliminate the scenario mentioned in point 3. Also, exclusive agents can be sent to global queue.

    upload specific queue
  |-<upload_2> -> nomos, copyright, ojo, keyword
  |-<upload_3> -> monkbulk, decider, monkbulk, decider
  |-<upload_4> -> reuser, decider
  
  global queue
  -> delagent,
  

4. Since the FOSSology is released, there can be number of new scheduling libraries being released which needs to be explored. They can be a nice addition to the project.

Category	Rating
Low Hanging Fruit	-
Risk/Exploratory	**
Fun/Periphial	***
Core Development	***
Project Infrastructure	*
Project size	Large
Preferred contributor	Professional

[GMishx]

Idea!

Title: Debian packaging for Debian repository

Goal: Improve Debian packaging and make it acceptable for APT

The existing effort to put FOSSology under Debian packaging list needs to be taken forward. A repository under Debian Salsa was setup initially but not maintained any more: https://salsa.debian.org/fossology-team/fossology
It is configured to use gbp.

Blockers

1. The Debian building mechanism does not allow installation from sources other than apt. The Composer packages need to be packed as Debian packages and shipped with FOSSology.
2. Packaging and shipping other tools needs to satisfy their licensing terms.
3. The versions of packages in APT and actual versions used are different.
4. APT also provides JS libraries like JQuery and DataTables but RHL does not.

See also

• feat(buildsystem): add CMake Buildsytem #2075
• https://wiki.debian.org/PackagingWithGit
• https://wiki.debian.org/SimplePackagingTutorial
• https://wiki.debian.org/Diagrams
• https://wiki.debian.org/PHP
• https://peertube.debian.social/videos/watch/0fb2dbc4-f43d-477e-8b14-20c426f970de

Category	Rating
Low Hanging Fruit	*
Risk/Exploratory	**
Fun/Periphial	***
Core Development	*
Project Infrastructure	***
Project size	Small
Preferred contributor	Student/Professional