Edit

Add HTTP source to an eventstream (preview)

This article shows you how to add an HTTP source to an eventstream.

The HTTP connector provides a no-code, configurable way to stream data from any REST API directly into Eventstream for real-time processing. It allows you to continuously pull data from SaaS platforms and public data feeds and automatically parse JSON responses into structured events. It also offers example public feeds to help you get started quickly, select an example API, enter your API key, and let Eventstream prefill the required headers and parameters.

Prerequisites

  • A workspace with Fabric capacity or Fabric Trial workspace type.
  • Access to the workspace with Contributor or higher workspace roles.
  • An HTTP endpoint that is publicly accessible. If it resides in a protected network, connect to it by using Eventstream connector virtual network injection.

Add HTTP as a source

If you haven't added any source to your eventstream yet, select the Connect data sources tile. You can also select Add source > Connect data sources on the ribbon.

Screenshot that shows the selection of the tile for using an external source.

If you're adding the source to an already published eventstream, switch to Edit mode. On the ribbon, select Add source > Connect data sources.

Screenshot that shows selections for adding external sources.

Search for http, and select Connect to add the HTTP connector to your eventstream.

Screenshot that shows the selection of HTTP as the source type in the Select a data source wizard.

Configure and connect to HTTP

You can select one of the Example public APIs to get started quickly with predefined headers and parameters, or select Exit to configure your own custom HTTP connector.

Use example public APIs

  1. Select an example public API to quickly set up the HTTP connector: Screenshot that shows example API page.

    • Binance. Get the latest Bitcoin price
    • CoinGecko. Get the current Bitcoin price in USD
    • Transport for London. Get live arrival predictions for a London station
    • Alpha Vantage. Get the real-time Microsoft (MSFT) stock price
    • OpenWeather. Get the weather forecast for a location
    • NewsAPI. Get the latest business headlines in the US

  2. For APIs that require an API key, use the provided link to apply for the API key and enter it in the input box. The connector then automatically creates the Fabric cloud connection and prefills the required HTTP headers and parameters.

    Important

    The Binance example API doesn't require an API key, but you're responsible for complying with its Terms of Use.

  3. If you choose Use existing connection, select Apply example to prefill the required headers and parameters for the example API without entering an API key.

  4. Alternatively, select Exit to close the window and configure your own HTTP source from scratch.

Configure your own HTTP source

Follow the steps to configure your own HTTP source without an example API.

  1. Create a new connection. Select New connection and enter the required properties for your HTTP source:

    Screenshot that shows where to select the new connection page.

    • Base Url: The base URL of the HTTP endpoint.
    • Token Audience Uri (optional): The resource that the token is intended for.
    • Connection name: Enter a name for the connection.
    • Authentication kind: Currently, the HTTP connector only supports Anonymous, Basic, and API Key authentication.

    Screenshot that shows the new connection page.

  2. Request method: Select GET or POST.

  3. Headers and parameters: If you selected API Key authentication when you created the cloud connection, use the dynamic parameter ${apiKey} in your headers or parameters to reference the API Key.

    For example, to connect to CoinGecko, use the header: x_cg_demo_api_key = ${apiKey}.

    Important

    Do NOT enter your API key or other credentials in the headers or parameters.

  4. Request interval (s): Specifies the time in seconds to wait between consecutive requests; valid range is [1, 3600].

  5. Maximum retries： The maximum number of times the connector retries a request when an error occurs; valid range is [10, 100].

  6. Retry backoff (s): The time in seconds to wait following an error before the connector retries; valid range is [1, 3600].

  7. Retry status code: The HTTP error codes returned that prompt the connector to retry the request. Enter a comma-separated list of codes or range of codes. Ranges are specified with a start and optional end code. For example:

    • 400- includes all codes greater than or equal to 400.
    • 400-500 includes codes from 400 to 500, including 500.
    • 404,408,500- prompts the connector to retry on 404 NOT FOUND, 408 REQUEST TIMEOUT, and all 5xx error codes.

    Some status codes are always retried, such as unauthorized, timeouts, and too many requests.

    Screenshot that shows the configured settings.

Handle paginated APIs

Some REST APIs return data in multiple pages instead of a single response. For example, an API might return 100 records in each response and require additional requests to retrieve the remaining records. The HTTP connector supports pagination so that it can retrieve data across multiple pages in a polling cycle.

Supported pagination types

The HTTP connector supports these pagination patterns:

  • Page-based pagination: The request includes a page parameter, such as page=1 or page=2. The connector increments the page number to request the next page.
  • Cursor-based pagination: The API response includes a cursor or continuation token, such as continuationToken or nextPageToken. The connector uses this value to request the next page.

Configure pagination

To use pagination, configure these settings in the HTTP connector:

  • Pagination enabled: Turns on pagination handling.
  • Pagination mode: Specifies the pagination type. Supported values are Page and Cursor.
  • Response data JSON pointer: Specifies the location of the data array in the API response, such as /value, /items, or /articles.

Page-based pagination example

For a page-based API, the first request might be GET /api/data?page=1.

Example configuration:

  • Pagination enabled: true.
  • Pagination mode: Page.

The connector sends requests for subsequent pages, such as page=2 and page=3. Pagination stops when the API response indicates that no more data is available, such as by returning an empty array.

Cursor-based pagination example

For a cursor-based API, the response might include a continuation token, such as "continuationToken": "ABC123".

The connector extracts the continuation token and uses it in the next request, such as GET /api/data?continuationToken=ABC123. Pagination stops when the response doesn't include another token or the token value is null.

When pagination stops

The connector stops requesting additional pages when one of the following conditions is met:

  • The API returns an empty response.
  • The response doesn't include a next page token.
  • The API indicates that no more pages are available.

Considerations

  • Each polling cycle starts from the first page.
  • Confirm that the API supports pagination before you enable this feature.
  • Incorrect pagination configuration or JSON pointer settings might cause missing or duplicated data.

Use pagination when an API limits the number of records returned in each request.

Stream or source details

  1. On the Connect page, follow one of these steps based on whether you're using Eventstream or Real-Time hub.

    • Eventstream:

      In the Source details pane to the right, follow these steps:

      1. For Source name, select the Pencil button to change the name.

      2. Notice that Eventstream name and Stream name are read-only.

    • Real-Time hub:

      In the Stream details section to the right, follow these steps:

      1. Select the Fabric workspace where you want to create the eventstream.

      2. For Eventstream name, select the Pencil button, and enter a name for the eventstream.

      3. The Stream name value is automatically generated for you by appending -stream to the name of the eventstream. This stream appears on the real-time hub's All data streams page when the wizard finishes.

  2. Select Next at the bottom of the Configure page.

Review and connect

On the Review + connect screen, review the summary, and select Add (Eventstream) or Connect (Real-Time hub).

You can see the HTTP source added to your eventstream in Edit mode.

A screenshot of the added HTTP source in Edit mode with the Publish button highlighted.

Select Publish to publish the changes and begin streaming data from HTTP source to the eventstream.

A screenshot of the added HTTP source in Live view with the Publish button highlighted.

Limitations

  • The HTTP connector currently supports only JSON API responses.
  • OAuth authentication isn't supported.
  • The HTTP source doesn't support CI/CD features, including Git Integration and Deployment Pipeline. Exporting or importing an Eventstream item that includes this source through Git might result in errors.

Related content

Other connectors:

  • Last updated on