Bringing data in
Connectors
A connector brings data into Topos and builds your tables. Each connector has one way of getting data and one or more routes. A route picks records out of what arrived and writes them into one table. See Routes, modes and columns.
The three types
| Type | How data arrives | When it runs | Page |
|---|---|---|---|
| Receive | Your system sends data to an address Mestra gives you, with a key. | As it arrives. | Send data to Mestra |
| Fetch | Mestra reads JSON or CSV from a web address on an interval you choose. | On the interval, or when you select Run now. | Fetch data on a schedule |
| Upload | You upload a spreadsheet or CSV file and it becomes a table. | When the file is uploaded. | Upload a file |
The connectors you can add are:
| Connector | Type | What it is |
|---|---|---|
| Custom | Receive | Your system sends data to an address Mestra gives you, with a key. You choose the tables. |
| Point of sale | Receive | Your till sends its menu and sales to an address Mestra gives you, into two ready-made tables. See Templates. |
| REST | Fetch | Mestra reads JSON or CSV from a web address on an interval you choose. |
| Spreadsheet or CSV | Upload | You upload a spreadsheet or CSV file and choose its columns. |
If you need a connector that isn’t listed, the add flow links to Support.
The Ingest page
Ingest lists your connectors in one table.
| Column | What it shows |
|---|---|
| Connector | Its name. Select it to open it. |
| Type | Receive, Fetch or Upload. |
| Schedule | Every 15 minutes and so on for a fetch (hover to see when the next run is due), As it arrives for a receive connector, By hand for an upload. |
| Health | Healthy, Failing, Overdue or Never run. A failing connector shows its error below. |
| Last run | When it last ran and what it did, such as “120 rows” or Nothing new or Failed. |
| Builds | The tables it fills, each a link. |
An owner sees Add a connector. Everyone else sees the list.
How health is judged
| Health | Meaning |
|---|---|
| Failing | The last run failed. The label says for how long, such as “Failing for 3 hours”, and the error is shown. Tables keep the data from the last good run. |
| Overdue | A fetch connector is more than one of its own intervals past the time its next run was due, and it isn’t running now. |
| Never run | It has never run, landed anything or failed. |
| Healthy | None of the above. |
The home page counts failing and overdue connectors under Ingest and lists each under needs a look.
Add a connector
An owner selects Add a connector. There are five steps, and nothing is saved until the last.
- Type. Choose Receive, Fetch or Upload.
- Connector. Choose one. A search box shows when there are six or more of a type.
- Configure.
- For Custom and REST, Mestra shows the data as a table you can shape. For a push connector, paste a sample of what your system sends. For a fetch connector, enter the address and select Pull the data, or paste a sample instead. Name each column, choose its type, tick Key and Required, and remove columns you don’t want. If the data holds several lists of records, choose which list under Where the records are. For several tables from one payload, or to read only some records, select Set it up by hand.
- For Point of sale, the tables are ready made. There is nothing to map.
- For Spreadsheet or CSV, choose a file and shape its columns.
- Test. Try the connector once. A fetch connector fetches the address and shows the rows it would make and the ones it would refuse. A receive connector tests a pasted sample. Nothing is saved. You can go on without testing.
- Save and schedule. Name it. A fetch connector asks for Every, in minutes or hours. A receive connector makes a first key. Save connector finishes.
If you leave with something entered, Mestra asks “Leave without saving?”. Choosing a different connector after entering something asks “Start again?”.
The steps are at /ingest?add=1, and Back keeps what you entered.
A connector’s page
Select a connector to open it. The page has these parts.
- Where it goes: Send to (the address, for a receive connector), Fetches (the address and its interval, for a fetch connector), Writes (its tables) and Version.
- Downstream: a picture of what depends on the connector: its tables, the mappings that read them, the model types those fill, and the decisions that read them. If nothing depends on it yet, the page says so.
- Past runs and What has landed: see Runs, history and replay.
- Keys (receive connectors, owners only): see Send data to Mestra.
- Versions: see Runs, history and replay.
If the last run failed, a note says “Last run failed:” with the reason, and when the tables last held good data.
An owner also has these buttons.
| Button | What it does |
|---|---|
| Run now | Fetch connectors only. Fetches the address now, whether or not it is due, and says “Fetched.” |
| Replay | Routes everything that has landed again with the current version. See Runs, history and replay. |
| Edit | Opens the connector to change it. Saving makes a new version. Upload connectors can’t be edited. |
| Delete | Forgets the connector’s address, keys, settings and saved versions. Its tables are kept if they hold data, and so is what has landed. |
Connector keys
A connector has a key, the short name in its address, such as pos. It uses lower-case letters, digits and underscores, starts with a letter, and can’t be changed after the connector is made. A connector’s name is up to 80 characters and can change.
Saving a connector sends a document of at most 64 KiB.