Designing AI Plugins for Any Provider: Overcoming Hidden Engineering Hurdles

The Fragmented AI Landscape: Before Universal Compatibility

Imagine you’re a developer tasked with building an AI plugin for a knowledge management app. Initially, you integrate with a single AI provider—everything works fine. But soon, users request support for other providers, each with different APIs, rate limits, input formats, and response structures. Suddenly, your plugin is a patchwork of provider-specific code, brittle and difficult to maintain.

Before universal compatibility, plugin developers often face these challenges:

• Divergent APIs and protocols: Each AI provider has its own endpoint structure, authentication methods, and request/response formats.
• Inconsistent feature sets: Some providers offer chat models, others only completion APIs. Some support fine-tuning or embeddings, others don’t.
• Variable rate limits and pricing plans: Handling throttling gracefully requires provider-specific logic.
• Different error handling and retry semantics: A one-size-fits-all approach to retries can backfire.

Users suffer too. They want flexibility to switch providers or use multiple ones simultaneously, but the plugin either locks them into one or requires complex manual configuration.

After: A Unified Architecture for Provider-Agnostic AI Plugins

Now, imagine a redesigned plugin architecture built from the ground up for provider-agnosticism. This design abstracts away provider differences behind a clean, consistent interface. The plugin exposes a unified API to the rest of the app, so users can seamlessly choose or combine AI providers without hiccups.

Key elements of this transformation include:

1. Abstraction Layer: Provider-Agnostic Interfaces

Instead of calling provider APIs directly, the plugin implements an abstraction layer that defines common operations like generateText(), createEmbedding(), or chat(). Each provider module implements these operations according to its own specifications, but the rest of the plugin only deals with the abstraction.

This decouples core logic from provider specifics and makes it easier to add new providers without rewriting the main plugin code.

2. Feature Detection and Capability Negotiation

Since not all providers support the same features, the plugin includes runtime capability detection. When a user selects a provider, the plugin queries its capabilities—are chat models supported? Are embeddings available? Is streaming possible?

The interface then adapts dynamically, enabling or disabling features based on provider support, preventing user confusion and errors.

3. Centralized Rate Limiting and Retry Management

Managing rate limits and retries is complex but crucial for reliability. The plugin implements a centralized system that respects each provider’s limits and implements backoff strategies tailored per provider.

This prevents flooding APIs and provides a smooth user experience even when limits are approached.

4. Unified Error Handling and Reporting

Errors from different providers can vary wildly in format and severity. The plugin normalizes error responses into a standard set of error types and messages, making debugging and user feedback consistent.

5. Flexible Authentication and Configuration

Each provider requires different authentication—API keys, OAuth tokens, or other credentials. A secure, flexible config system allows users to manage multiple credentials and switch providers on the fly.

Real-World Example: Switching AI Providers Seamlessly

Consider a researcher using a note-taking app with an AI plugin. Initially, the plugin supports Provider A only. The researcher likes Provider A’s chat capabilities but wants to try Provider B’s embedding quality for knowledge graph generation.

Before the redesign, the researcher would have to manually switch plugin settings, possibly restart the app, or even deal with different plugin versions.

After the redesign, the plugin offers a simple dropdown to select providers. The researcher switches to Provider B for embeddings, while still using Provider A for chat. The plugin manages the different API calls internally, presenting a seamless experience.

The Workflow

• User selects Provider A for chat and Provider B for embeddings in plugin settings.
• The plugin checks capabilities of both providers.
• When requesting chat completions, it routes calls to Provider A; for embeddings, to Provider B.
• Rate limits and errors are handled independently.
• The user gets the best of both worlds without complex setup or confusion.

Checklist for Designing Provider-Agnostic AI Plugins

• Define a clear abstraction layer covering common AI operations
• Implement per-provider adapters conforming to the abstraction
• Include feature detection to adjust UI and functionality dynamically
• Centralize rate limiting and retry logic with provider-specific rules
• Normalize error responses for consistent handling
• Design flexible configuration to manage multiple provider credentials
• Ensure secure storage and handling of sensitive credentials
• Test with multiple providers to identify edge cases and inconsistencies

Challenges That Remain

Despite these improvements, engineering such a plugin is not without hurdles:

• Evolving APIs: Providers frequently update their APIs. Maintaining adapters requires ongoing effort.
• Latency and performance variability: Different providers respond at different speeds, complicating UX expectations.
• Feature gaps: Some AI capabilities are unique to certain providers, forcing compromises.
• Cost management: Different pricing models can affect user behavior and plugin design.

Addressing these requires a design mindset focused on modularity, maintainability, and user-centered flexibility.

Final Thoughts

Building AI plugins that work with any provider transforms both developer experience and user flexibility. The before scenario shows a fragmented, brittle system locked to a single provider. The after scenario reveals a modular, adaptable architecture that gracefully handles differences and gives users freedom to choose.

This transformation demands careful abstraction, dynamic capability handling, and robust error and rate management. But the payoff is a more resilient, future-proof plugin that can evolve alongside the fast-moving AI ecosystem.

If you want to bring this workflow into Obsidian, Note Companion is one option to explore.