Bringing data in
Fetch data on a schedule
A Fetch connector reads a web address on an interval you choose and writes what it finds into tables. Use it when another system already publishes its data at an address.
What the address must be
- It starts with
https://. Plainhttp://is refused with “must start with https://”. - It names no port. Mestra uses the standard https port.
- It carries no user name or password. Send those as a header instead.
- It must be reachable from the internet. An address that isn’t allowed is refused when Mestra fetches it, with “address not allowed”.
- Mestra does not follow redirects. Give the final address.
- It must answer within 10 seconds, with at most 1 MiB, and with the right content type:
application/jsonfor JSON, ortext/csvorapplication/csvfor CSV.
Add a fetch connector
An owner does this on the Ingest page.
- Select Add a connector, Fetch, then REST.
- Configure. Enter the address. Add headers if the address needs them (see below). Choose the Format: Work it out, JSON or CSV. Select Pull the data. Mestra fetches the address once, shows what it holds and fetches nothing else. Nothing is saved. Or open Paste a sample instead and paste what the address answers, then select Read the sample.
- Shape the table: name each column, choose its type, tick Key and Required, and remove columns you don’t want. See Routes, modes and columns. If the answer holds several lists, choose the list under Where the records are. Choosing another list starts the columns again and fetches the address again.
- Test. Select Fetch now to see the rows Mestra would make and the ones it would refuse. Nothing is saved.
- Save and schedule. Name the connector and set Every, in minutes or hours. Select Save connector.
If Mestra says “No records were found in the data”, check the address or the sample, or choose another list.
How often
Every must be at least 1 minute. A smaller value is refused with “Every must be at least 1 minute.” The Ingest page shows the schedule as Every 15 minutes, Every hour and so on. Hover over it to see when the next run is due.
Mestra runs a fetch connector on its interval, one run at a time. Run now on the connector’s page fetches at once, whether or not it is due. If a fetch is already running, Run now says it is busy.
An answer that is exactly the same as the last one is not routed again. The run shows Nothing new. Runs of “nothing new” in a row are shown as one line, such as “Nothing new, 1,000 checks, 9:00 am to 4:40 pm”.
Headers and secrets
Some addresses need a key or a token in a header. In the Headers part of the form, add a header name and its value.
- The value is sealed when you save. It is never shown again, never put in the connector’s document and never logged. Only the header name is kept in the document.
- Saving new values replaces all the saved ones. If you change one header, enter every header’s value again. Mestra says “Enter the value again. Saving new values replaces all the saved ones.”
- Values are tied to the address. If you change the address’s scheme or host, Mestra says “Enter the header values again: the address changed.”
- Some headers are never sent:
Host,Content-Length,Transfer-Encoding,Connection,Upgrade,Accept-Encoding,Te,Trailer, and any that start withProxy-. A header name must be a valid header name and can be listed once. - If a header has no saved value when the fetch runs, the run fails with “no value saved for header” and the name.
When a fetch fails
A failed run shows on the connector’s page under Past runs, with a short reason. The reason never holds a header value or any text of the answer. The connector shows Failing, and the tables keep the data from the last good run. The next run tries again.
| Reason | What it means and what to do |
|---|---|
status 404 (or another number) |
The address answered with that HTTP status. Check the address and the headers. |
answer too large |
The answer is over 1 MiB. Ask for less, for example a page or a date range. |
answer was not the expected type |
The Content-Type isn’t application/json (JSON) or text/csv or application/csv (CSV). |
address not allowed |
Mestra won’t fetch that address. Use a public https:// address. |
couldn't reach it |
The address didn’t answer in time or couldn’t be reached. |
no value saved for header … |
A listed header has no value. Edit the connector and enter it. |
answer could not be read |
The answer doesn’t read as the format. The answer is still kept. |
a route failed on the answer |
A route couldn’t write its table. The answer is kept and can be replayed. |
After you change the connector
Saving a change makes a new version. The next fetch uses it. To read again what already arrived with the new version, select Replay. See Runs, history and replay.
Limits
| Limit | Value |
|---|---|
| Shortest interval | 1 minute |
| Time for the address to answer | 10 seconds |
| Size of the answer | 1 MiB |
| Connector document | 64 KiB |
| Path into a record | 32 parts |