Fetch Data from External API

[luismts]

Summary

This feature proposes the integration of a "Fetch Data from External API" functionality within Directus. It aims to empower users to seamlessly connect with external APIs, retrieve data, and synchronize it with Directus for enhanced flexibility and real-time updates.

You can see a full list of Enhancements and Desired Features for Directus here.

Basic Example

// Example Configuration
{
  "apiEndpoint": "https://example-api.com/data",
  "authentication": {
    "apiKey": "your_api_key",
    "username": "your_username",
    "password": "your_password"
  },
  "parameters": {
    "param1": "value1",
    "param2": "value2"
  },
  "storageOption": "id"
}

Motivation

The motivation behind this feature is to streamline data management within Directus by providing a native mechanism for fetching and integrating external API data. Use cases include dynamic content updates, real-time dashboards, and centralized data management. This feature aligns with Directus' commitment to flexibility and adaptability.

Detailed Design

External API Integration

• Allow users to configure Directus to connect with external APIs seamlessly.
• Provide an intuitive interface for setting up API endpoints, authentication, and parameters.
• Allow users to choose the storage option for the fetched data (reference value, e.g., ID, or the actual value, e.g., Name).

Data Mapping and Transformation

• Enable users to map and transform data retrieved from external APIs to align with Directus database structures.
• Implement customizable data transformation options to accommodate various API response formats.

Scheduled Data Sync

• Introduce the option to schedule automatic data synchronization with external APIs at predefined intervals.
• Allow users to specify the update frequency, ensuring real-time or periodic data updates.

Error Handling and Logging

• Implement a robust error-handling mechanism to manage issues during data fetching.
• Provide detailed logging features to track API integration activities for troubleshooting and auditing purposes.

Requirements List

Must Have:

• Seamless configuration for external API integration.
• Intuitive interface for setting up API endpoints and authentication.
• Data mapping and transformation options.

Should Have:

• Ability to choose storage options for fetched data.
• Scheduled data synchronization with external APIs.
• Robust error-handling mechanism.

Could Have:

• Additional advanced customization options for API integration.

Won't Have:

• Direct integration with every possible external API (initially focusing on common use cases).

Drawbacks

• Increased implementation complexity due to the variety of external APIs.
• Potential learning curve for users new to API integration concepts.
• Consideration needed for backward compatibility and impact on existing Directus applications.

Alternatives

• Users can continue to use custom solutions for external API integration.
• Development of external plugins or extensions to handle specific API integrations.

Adoption Strategy

• Provide comprehensive documentation and tutorials for users to adopt the new feature.
• Minimize breaking changes by offering backward compatibility where possible.
• Implement a migration path for existing Directus applications.

Unresolved Questions

• Detailed design considerations for specific API response formats.
• Impact on performance with frequent scheduled data synchronization.
• Exploration of additional security measures for API integration.

[luismts]

Enhancements and Desired Features for Directus

I have created a thread with a list of features and enhancements that could further drive the utility and versatility of Directus.

You can see it here.

[davidtrattnig]

Thanks for the great request @luismts!

I'd like to do something similar. I'm wondering if one of the following is already possible with the extension feature? Any hints?

Basic Integration

1. In my collection, store the ID of some record available via some external API.
2. When displaying the collection, fetch the actual data from the external API.

Advanced Integration

1. Pre-Fetch relevant records from the external API.
2. Instead of entering the number in my collection's ID field, I'd like to get a dropdown, already visualizing the external data in some way.
3. When selecting an item, the ID of the record is stored to the local collection.
4. When listing the collection, the records are shown, as if they where local data. Optionally with some caching.