External IdP validation feature for API Backends #401

anmunoz · 2018-03-06T10:56:30Z

I am Andrés Muñoz, from Technical University of Madrid, working with @aalonsog, I have been working on the idea of implementing an external validation service in API-Umbrella in order to verify if a user token is valid or not. For achieving that, I followed the flow showed in the diagram #349 by @aalonsog and the guidelines stated by you in the previous comments.

This new feature allows to the API-umbrella users the possibility of make request to a registered API backend, using an API key or an OAuth2 token. If the user uses a token in the request, this token is verified using an external IdP, once the token is validated the user information is retrieved and redirect to the API backend following the usual workflow. The list of IdP's included in this development are:

FIWARE
Google
Facebook
GitHub

Hence, after the inclusion of this feature, the gatekeeper architecture is modified as is showed in the next figure:

This work was made in collaboration with @aalonsog @agordillo and @jsalvachua.

If you have any coment or need more information about it, please let me know.

…inside of the gatekeeper configuration for managing token validation with FIWARE Wilma PEP Poxy, this fields allow to define the PEP proxy host and port to establish the connection. The caching option was changed to false because the token validation option doesn't allow caching yet. Also, the error messages defined in the api settings were changed for including token as other validation method.

…the word "token" in the url and the header "X-Auth-Token" in the requests made to the platform. With this change is possible to manage and differentiate if the user made a request using an API_Key or OAuth Token

…s allowed to pass the token are by param and header. The variable api_key was changed by a key, value table, this table has two values, fist the key_value that stores the value of the api_key or the token, and the second value called key_type which stores the type of key provided by the user, it can take two values token or api_key All the variables that previously stored the api_key value, was changed to store api_key ["key_value"], that represents exactly the same value using the new api_key table.

…odified to allow and parse a roles tables that have more than one value by rol, for example: if the function receives the roles in this format Roles":[{"name":"rol1","id":"16"}]. With the modification the function checks if the roles value has more than one value and takes only the value for the key "name" ant after that invert the table.

…ng if the token is valid and retrieve the user information. The function takes the PEP Proxy host and port as parameters and sends a request with the header X-Auth-Token and the value of the token provided by the user. If the token is valid, PEP proxy sends a response with the user information asociated to the token, otherwise, it sends a message indicating the result of the validation process with his status, 404 , 402, etc.

…on, for achieving this, an extra validation through the key_type value of the api_key variable was used. If the key_value is equal to "api_key", the key_value is checked in the Mongo database. In case of the value of key_values is equal to "token", the token is sent to the PEP Proxy for validation, if the token is valid, the user information is returned. If the user information is retrieved from the database this information is stored in the variables of the platform as usually did. In case that the user information was retrieved using token validation, the variables the Nick_name is assigned to the user_id and the Roles is assigned to the roles of the platform. Moreover, if the user information coming from of the token validation with PEP Proxy doesn't have the value of "email", this field is stored in the platform using the Nick_name value, this value is needed for displaying, the user associated with each request in the Analytics option of the web app . The rest of the changes are related to the reassignment of the api_key variable using api_key ["key_value"]

New feature for OAuth2 Token Validation using Fiware Wilma PEP-Proxy, v0.1 OAuth2 authentication with FIWARE Pep Proxy only, next version need to allow OAuth2 token validation with any authentication service provider

…P-Proxy"

Revert "New feature for OAuth2 Token Validation using Fiware Wilma PEP-Proxy"

…inside of the gatekeeper configuration for managing token validation with externals Identity Providers (Fiware, Google, Github, Facebook). The caching option was changed to false because the token validation option doesn't allow caching yet. Also, the error messages defined in the api settings were changed for including token as other validation method.

… the word "token" in the url and the header "X-Auth-Token" in the requests made to the platform. With this change is possible to manage and differentiate if the user made a request using an API_Key or OAuth2 Token

…modified to allow and parse a roles tables that have more than one value by rol, for example: if the function receives the roles in this format Roles":[{"name":"rol1","id":"16"}]. With the modification the function checks if the roles value has more than one value and takes only the value for the key "name" and after that, the function inverts the table.

…, Fiware, Github) for checking if a token is valid and retrieve the user properties. The function takes the token provided by the user and the IdP provider registered in the api-backend for checking if the token is valid making a validation request to the corresponding IdP. If the token is valid, the user information stored in the IdP is retrieved.

…s allowed to pass the token are by param and header. The variable api_key was changed by a key, value table, this table has three elements, first the key_value that stores the value of the api_key or the token, the second value called key_type which stores the type of key provided by the user, it can take two values token or api_key and, Finally, the las element named "idp" has a value that indicates if and which external IdP has the associated the requested api-backend. All the variables that previously stored the api_key value, was changed to store api_key ["key_value"], that represents exactly the same value using the new api_key table.

…ion, for achieving this, an extra validation through the key_type value of the api_key variable was used. If the key_value is equal to "api_key", the key_value is checked in the Mongo database. In case of the value of key_values is equal to "token", the token is sent to the IdP validation with the corresponding api_key["idp"] value for sending a validation request directly to the IdP associated to the api-backend, if the token is valid, the user information is returned. If the user information is retrieved from the database this information is stored in the variables of the platform as usually did. In case that the user information was retrieved using token validation and the IdP was Fiware, the variables the id is assigned to the user_id and the Roles is assigned to the roles of the platform. In case of the other externals IdP, the name and the email of the user are retrieved only. The rest of the changes are related to the reassignment of the api_key variable using api_key ["key_value"]

…using api_key or token even when an IdP has been associated to that API backend

@agordillo

…ackends by @agordillo

… web interface

GUI · 2018-03-19T17:39:11Z

@anmunoz: Hi there! Sorry for the delay in getting back to you. We've been busy with other things, but I'll try to take a look at this in the coming weeks. Thanks for the contribution and getting this started!

…n mode authentication and authorization, for these reasons the request to the IDM are different. For authentication the url needs to include the token and the app-ID and for authorization is needed to include, token, action, resource and app-ID

Urtza2 · 2018-11-27T12:30:26Z

Hi,
I'm Urtza Iturraspe, works in Tecnalia and I am using Ckan, biz ecosystem, Fiware at general in an European Project. I am trying to make offer using BIZ from Ckan I can do it but from BIZ I have some problems,
I have modified all of these files in api-umbrella folder. Now I am trying to create a product Specificacion using BIZ Ecosystem and CKAN Dataset API as asset type and it returns an error because app_id returned is "None".
I have two API Backend defined in API-Umbrella admin window.
In Postman I make a GET call

https://localhost:9222/api-umbrella/v1/apis.json&start=0&length=100
{'headers': {u'X-Api-Key': u'U7B9ysawwNN0hBjEvBDeoJEyVEbEODDbjx2b85dX', u'X-Admin-Auth-Token': u'nx4HYGgPM8SbCTqRsmlXxPS7EPTTPbmkalgQQqyB'}, 'verify': False}

And It returns:

{
    "draw": 0,
    "recordsTotal": 2,
    "recordsFiltered": 2,
    "data": [
        {
            "backend_host": "www.ckanreplicate.com",
            "backend_protocol": "https",
            "balance_algorithm": "least_conn",
            "created_at": "2018-10-31T07:22:58Z",
            "created_by": "be817f88-e2d2-4b91-85ad-23a695921a04",
            "deleted_at": null,
            "frontend_host": "www.ckanreplicate.com",
            "name": "ckan",
            "servers": [
                {
                    "host": "www.ckanreplicate.com",
                    "port": 443,
                    "id": "545d9ad3-9406-4523-92d6-c91642e146a1"
                }
            ],
            "settings": {
                "allowed_ips": null,
                "allowed_referers": null,
                "anonymous_rate_limit_behavior": null,
                "api_key_verification_level": null,
                "api_key_verification_transition_start_at": null,
                "append_query_string": "",
                "authenticated_rate_limit_behavior": null,
                "disable_api_key": null,
                "error_data": {},
                "error_templates": {},
                "http_basic_auth": null,
                "pass_api_key_header": false,
                "pass_api_key_query_param": false,
                "rate_limit_mode": null,
                "require_https": null,
                "require_https_transition_start_at": null,
                "require_idp": null,
                "required_roles": [
                    "api-umbrella-contact-form",
                    "api-umbrella-key-creator",
                    "provider",
                    "issue_customer"
                ],
                "required_roles_override": null,
                "id": "3455d01d-0892-499f-a453-76e9b8a59f4e",
                "default_response_headers": null,
                "headers": null,
                "override_response_headers": null,
                "rate_limits": null
            },
            "sort_order": 0,
            "updated_at": "2018-11-27T11:44:51Z",
            "updated_by": "be817f88-e2d2-4b91-85ad-23a695921a04",
            "url_matches": [
                {
                    "backend_prefix": "/",
                    "frontend_prefix": "/",
                    "id": "8bfabbf6-16cc-465b-bc7d-9ed25fcfbfcf"
                }
            ],
            "version": 2,
            "id": "d2a306b0-2203-4d69-8613-0dd17d416375",
            "rewrites": null,
            "sub_settings": null,
            "frontend_prefixes": "/"
        },
        {
            "backend_host": "www.bizreplicate.com",
            "backend_protocol": "http",
            "balance_algorithm": "least_conn",
            "created_at": "2018-11-27T11:47:16Z",
            "created_by": "be817f88-e2d2-4b91-85ad-23a695921a04",
            "deleted_at": null,
            "frontend_host": "www.bizreplicate.com",
            "name": "biz",
            "servers": [
                {
                    "host": "www.bizreplicate.com",
                    "port": 8443,
                    "id": "b908f0f3-bf86-463b-99af-58e8165a9f2d"
                }
            ],
            "settings": {
                "allowed_ips": null,
                "allowed_referers": null,
                "anonymous_rate_limit_behavior": null,
                "api_key_verification_level": null,
                "api_key_verification_transition_start_at": null,
                "append_query_string": null,
                "authenticated_rate_limit_behavior": null,
                "disable_api_key": null,
                "error_data": {},
                "error_templates": {},
                "http_basic_auth": null,
                "pass_api_key_header": null,
                "pass_api_key_query_param": null,
                "rate_limit_mode": null,
                "require_https": null,
                "require_https_transition_start_at": null,
                "require_idp": null,
                "required_roles": [
                    "api-umbrella-contact-form",
                    "provider",
                    "issue_customer",
                    "api-umbrella-key-creator"
                ],
                "required_roles_override": null,
                "id": "8b8fbf95-90c9-4385-b2cb-124dc5176b69",
                "default_response_headers": null,
                "headers": null,
                "override_response_headers": null,
                "rate_limits": null
            },
            "sort_order": 10000,
            "updated_at": "2018-11-27T11:47:32Z",
            "updated_by": "be817f88-e2d2-4b91-85ad-23a695921a04",
            "url_matches": [
                {
                    "backend_prefix": "/",
                    "frontend_prefix": "/",
                    "id": "8b899aeb-2ddd-4e95-bc48-115a70a2e8b1"
                }
            ],
            "version": 2,
            "id": "d77196d6-c2e5-4159-90e0-bc067697d63b",
            "rewrites": null,
            "sub_settings": null,
            "frontend_prefixes": "/"
        }
    ]
}
``
If I take a look umbrella_client.py file (ckan_api_dataset plugin)  validate_service method:

def validate_service(self, path):
err_msg = 'The provided asset is not supported. '
'Only services protected by API Umbrella are supported'

print ("Umbrella_client.py##validte_service INI")     
print ("Umbrella_client.py##validte_service##path:"+str(path))   
# Split the path of the service 
    paths = [p for p in path.split('/') if p != '']
    if not len(paths):
        # API umbrella resources include a path for matching the service
        raise PluginError(err_msg)

    # Make paginated requests to API umbrella looking for the provided paths
    url = '/api-umbrella/v1/apis.json'
    def page_processor(api):
        front_path = [p for p in api['frontend_prefixes'].split('/') if p != '']
        return len(front_path) == 0 or (len(front_path) <= len(paths) and front_path == paths[:len(front_path)])

    matching_elem = self._paginate_data(url, err_msg, page_processor)
    data_string = json.dumps(matching_elem)

    # If the API is configured to accept access tokens from an external IDP save its external id
    **app_id = None
    if 'idp_app_id' in matching_elem['settings'] and len(matching_elem['settings']['idp_app_id']):
        app_id = matching_elem['settings']['idp_app_id']**

    return app_id


I have not matching_elem['settings']['idp_app_id']  ???


Please someone can help me with this?

Thanks a lot
Urtza

anmunoz · 2018-11-27T12:47:34Z

Hi @Urtza2 have you tried with the github repo of apinf (https://github.com/apinf/api-umbrella) forked from api-umbrella?, that repo have a stable release that includes external idp validation. On the other hand this repo have more features for authorization that I think that is not your case of use

Urtza2 · 2018-11-27T12:53:48Z

No I have install using these instructions ...
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 379CE192D401AB61
$ echo "deb https://dl.bintray.com/nrel/api-umbrella-ubuntu xenial main" | sudo tee /etc/apt/sources.list.d/api-umbrella.list
$ sudo apt-get update
$ sudo apt-get install api-umbrella

And after I make the changes that appear in this issue.
I try using it.
Thanks

Urtza2 · 2018-11-29T12:46:21Z

How can I install API-Umbrella of github repo of apinf (https://github.com/apinf/api-umbrella)?.
Docker file contain information of NREL repository.
I have download as zip from this repo an try doing this:

shell> ./configure and I get some errors like:

fatal: Not a git repository (or any of the parent directories): .git
date: invalid date ‘-128-NOTFOUND’
-- Checking for one of the modules 'uuid'
CMake Error at build/work/cmake/share/cmake-3.8/Modules/FindPkgConfig.cmake:641 (message):
  None of the required 'uuid' found
Call Stack (most recent call first):
  CMakeLists.txt:83 (pkg_search_module)


CMake Error at CMakeLists.txt:65 (MESSAGE):
  Could not find autoconf
Call Stack (most recent call first):
  build/cmake/rsyslog.cmake:4 (require_program)
  CMakeLists.txt:96 (include)


-- Configuring incomplete, errors occurred!
See also "/home/user1/Downloads/api-umbrella-master_APINF/CMakeFiles/CMakeOutput.log".

I'll try doing the same with NREL repo:
1.- ./configure -> get Makefile
2.- make -> get a lot of error.

What is the correct way to install API-Umbrella of this repo (https://github.com/apinf/api-umbrella) ?

Thanks in advance
Urtza

anmunoz and others added 29 commits October 26, 2017 12:14

Merge pull request #1 from ging/token_validation

f0839eb

New feature for OAuth2 Token Validation using Fiware Wilma PEP-Proxy, v0.1 OAuth2 authentication with FIWARE Pep Proxy only, next version need to allow OAuth2 token validation with any authentication service provider

Revert "New feature for OAuth2 Token Validation using Fiware Wilma PE…

0e86c30

…P-Proxy"

Merge pull request #2 from ging/revert-1-token_validation

8f57775

Revert "New feature for OAuth2 Token Validation using Fiware Wilma PEP-Proxy"

Correcting validation mistake for allowing access to the api backend …

739c99e

…using api_key or token even when an IdP has been associated to that API backend

Fix on Vagrantfile for synchronizing files in admin-ui

80bb4fb

Add external idps field to apis form

58cc570

Add require_idp field to apis (client and server sides)

79c4a43

Adding the web interface fields for external idp validation for api b…

cecb00d

…ackends by @agordillo

Fixing some code for integrating the external IdP validation with the…

1e561c7

… web interface

Pre-merge

0574328

Merge branch 'master' of https://github.com/ging/api-umbrella

93fa85c

Adding documentation for using external IdP validation

0ee3b3d

Correcting the documentation

fb48514

documentation update

a1d583c

Merge branch 'master' into master

d52a594

Pre-merge for validating trusted applications

678b1d5

Merge branch 'master' of https://github.com/ging/api-umbrella

1f108d9

aalonsog mentioned this pull request Mar 21, 2018

Oauth support? #38

Open

anmunoz added 3 commits October 8, 2018 10:43

working with default IDP declared in he config.yml

ee9b3b8

correction config.yml

b58c352

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

External IdP validation feature for API Backends #401

External IdP validation feature for API Backends #401

anmunoz commented Mar 6, 2018

GUI commented Mar 19, 2018

Urtza2 commented Nov 27, 2018

anmunoz commented Nov 27, 2018

Urtza2 commented Nov 27, 2018

Urtza2 commented Nov 29, 2018

External IdP validation feature for API Backends #401

Are you sure you want to change the base?

External IdP validation feature for API Backends #401

Conversation

anmunoz commented Mar 6, 2018

GUI commented Mar 19, 2018

Urtza2 commented Nov 27, 2018

anmunoz commented Nov 27, 2018

Urtza2 commented Nov 27, 2018

Urtza2 commented Nov 29, 2018