Endpoints Specification Reference

The endpoint specification allows for a great deal of configuration of how the endpoint behaves. You can easily:

  • Remap the method you use to call an endpoint
  • Model the data returned from an endpoint
  • Configure parameters, headers, and/or the body that are sent with each request to that endpoint
  • Recast error response codes (4xx, 5xx) to successes (2xx)
  • Cache responses from services that allow it

Defining Methods

You can specify behaviors for each different HTTP method used to call the BitScoop Data API. In the following example, there are different behaviors for calling the 'Example' endpoint using 'GET', 'POST', and 'LINK', while the 'Example2' endpoint has a specification for calling it with 'PUT':

Example:

{
    "endpoints": {
        "Example": {
            "description": "Different methods for calling Example"
            "GET": {
                <endpoint spec>
            },
            "POST": {
                <endpoint spec>
            },
            "LINK": {
                <endpoint spec>
            }
        },
        "Example2": {
            "PUT": {
                <endpoint spec>
            }
        }
    }
}

Note that you can put a description field alongside the methods to describe the endpoint in general, but this description is just text, not a configurable endpoint. The methods than can be specified are 'ACL', 'BIND', 'CHECKOUT', 'CONNECT', 'COPY', 'DELETE', 'GET', 'HEAD', 'LINK', 'LOCK', 'M-SEARCH', 'MERGE', 'MKACTIVITY', 'MKCALENDAR', 'MKCOL', 'MOVE', 'NOTIFY', 'PATCH', 'POST', 'PROPFIND', 'PROPPATCH', 'PURGE', 'PUT', 'REBIND', 'REPORT', 'SEARCH', 'SUBSCRIBE', 'TRACE', 'UNBIND', 'UNLINK', 'UNLOCK', and 'UNSUBSCRIBE'. 'OPTIONS' cannot be used.

You do not have to specify any methods for an endpoint, but can just use a method spec directly:

{
    "endpoints": {
        "Example": {
            <endpoint spec>
        }
    }
}

If you do this, then that endpoint is called via a 'GET'.

Escape sequences

Many fields in the endpoint spec allow for dynamic substitution within values. Within a string value, you begin and end the escape sequences with braces like this: ''. You can only substitute from certain sets of values:

  • parameters
  • headers
  • body
  • model
  • identifier
  • connection
  • environment

For all of these except for 'identifier', you will need to specify sub-fields using dot notation:

{
    "url": "https://example.com",
    "endpoints": {
        "User": {
            "route": {
                "path": "/users/{{ parameters.user_id }}"
            }
        }
    }
}

In this example, whatever is passed as the query parameter 'user_id' as part of the call to the Data API will be substituted into the path when calling the route for the User endpoint.

Endpoint specification

Each endpoint spec has a number of configurable fields:

method (optional) <String>

The method to use when BitScoop calls the specified route. If none is specified, then the method used to call the Data API is used. The allowable methods are 'ACL', 'BIND', 'CHECKOUT', 'CONNECT', 'COPY', 'DELETE', 'GET', 'HEAD', 'LINK', 'LOCK', 'M-SEARCH', 'MERGE', 'MKACTIVITY', 'MKCALENDAR', 'MKCOL', 'MOVE', 'NOTIFY', 'PATCH', 'POST', 'PROPFIND', 'PROPPATCH', 'PURGE', 'PUT', 'REBIND', 'REPORT', 'SEARCH', 'SUBSCRIBE', 'TRACE', 'UNBIND', 'UNLINK', 'UNLOCK', and 'UNSUBSCRIBE'.

You may be confused as to why an endpoint spec has a method field when you specified one or more methods already. This is done intentionally so that your calls to BitScoop are not tied to how the mapped API is called. Take the following example of a confusingly-implemented API that is made a lot cleaner using BitScoop:

Example:

{
    "version": "1.0",
    "url": "https://example.com",
    "endpoints": {
        "Users": {
            "GET: {
                "method": "POST",
                "route": "/people/retreive"
            },
            "SEARCH": {
                "method": "GET",
                "route": "/people/search,
                "parameters": {
                    "given_name": "{{ parameters.name }}
                }
            }
        }
    }
}

The API being mapped requires you to make 'POST' requests to retrieve a list of users, and a 'GET' request to perform a search of users. This map turns the retrieval into a 'GET' request and the searching into a 'SEARCH' request. To use this map to perform the user retrieval, you would make a 'GET' request to 'https://data.api.bitscoop.com/<map_id>/Users', and it would make a 'POST' request to 'https://example.com/people/retrieve'. Similarly, to search for users named John, you would make a 'SEARCH' request to 'https://data.api.bitscoop.com/<map_id>/Users?name=John' and it would make a 'GET' request to 'https://example.com/people/search?given_name=John'. This hopefully makes your code more intuitive, as no one looking at it will have to understand that a 'POST' request is really doing what they would expect to do with a 'GET'.

If that's still confusing, then you can ignore it all and just map the methods as they are on the service's API:

{
    "version": "1.0",
    "url": "https://example.com",
    "endpoints": {
        "Users": {
            "POST: {
                "route": "/people/retreive"
            },
            "GET": {
                "route": "/people/search,
                "parameters": {
                    "given_name": "{{ parameters.name }}
                }
            }
        }
    }
}

The above example is for the same API as used in the previous one, but without any method remapping. You're following the original API's spec of doing a POST to retrieve users and a GET to search through users.

description (optional) <String>

A text description of the method being mapped, for reference purposes only.

Example:

{
    "description": "Retrieve a list of users"
}

route (semi-optional) <Object>

The route to call when calling this endpoint spec. If you are not using the 'collection' and/or 'single' fields for this endpoint, then this is required. If one or both of those are in use, then this is optional, as their routes will override what is specified here.

route has two fields:

path (required) <String>

The path to call when this endpoint spec is called. This can either be just a path or a full URL:

{
    "url": "https://example.com",
    "endpoints": {
        "Users: {
            route: {
                "path": "/users"
            }
        },
        "Comments": {
            route: {
                "path": "https://test.com/comments"
            }
        }
    }
}

If you just specify a path, then it will be appended to the URL specified at the top level of the map. In this example, calling Users will make a 'GET' request to 'https://example.com/users'. If you specify a full URL, then exactly that URL will be used, and the top-level URL will be ignored. In this example, calling Comments will make a 'GET' request to 'https://test.com/comments'.

data (optional) <String>

The field of the returned data to select and pass forward for modeling. Use dot notation to select fields multiple levels deep, e.g. 'results.count'.

Sometimes services return data structures that contain more information than you require. You can use this field to strip out the fields you don't need.

Let's say an endpoint updates a database and returns the status of that update in the body like this:

{
    "status": "OK",
    "results": {
        "updated": {
            "count": 43,
            "list": [<objects>]
        },
        "not_updated": {
            "count": 182,
            "list": [<objects>]
        }
    }
}

All you care about is the list of items that were updated. You could write the route like this:

{
    "Users": {
        "route": {
            "path": "/users",
            "data": "results.updated.list"
        }
    }
}

All that would be returned is the list of updated users. If a model is specified, then this selected data is passed through that model.

parameters (optional) <Object>

Key/value pairs that define HTTP query parameters that will be sent with every outbound request for this endpoint. This will override any globally-defined parameters of the same name.

Example:

{
    "parameters": {
        "foo": "bar"
    }
}

headers (optional) <Object>

Key/value pairs that define HTTP headers that will be sent with every outbound request for this endpoint. This will override any globally-defined headers of the same name.

Example:

{
    "headers": {
        "bar": "foo"
    }
}

body (optional) <Object>

JSON body that will be sent with every outbound request for this endpoint. This will override a globally-defined body.

Example:

{
    "body": {
        "yub": "yub"
    }
}

model (optional) <Object or String>

The Model to use to parse the data that is returned from calls made via this endpoint. Can either be a Model object, or the name of a Model specified in the top-level 'models' field.

Example:

{
    "url": "https://example.com",
    "endpoints": {
        "Users": {
            "route": {
                "path": "/Users"
            },
            "model": {
                "id": "string",
                "name": "string",
                "age": "integer"
            }
        },
        "Comments": {
            "route: {
                "path": "/comments"
            },
            "model": "Comments"
        }
    },
    "models": {
        "Comments": {
            "id": "string",
            "text": "string
        }
    }
}

recast (optional) <Integer>

An HTTP status code that you wish to recast successful calls to.

Example:

{
    "recast": 302
}

This is generally not needed right here, but rather is used in the 'responses' section to recast errors.

responses (optional) <Object>

A set of key/value pairs defining how different responses from this endpoint should be handled. The key is a string with the response code and the value is an object containing a model and/or 'recast'. Neither is required, but having a response with neither specified is no more useful than not specifying it at all.

Example:

{
    "responses": {
        "200": {
            "model": {
                "key": "id",
                "fields": {
                    "id": "string",
                    "text": "string",
                    "likes": 27
                }
            }
        },
        "404": {
            "model": {
                "key": "code",
                "fields": {
                    "code": "integer",
                    "status": "string"
                }
            },
            "recast": 204
        }
    }
}

In this example, responses with a status code of 200 are parsed with a specific model. 404's are turned into 204's and parsed with a different model. Note that recast responses are only parsed using the model for the status code they originally had; they are not additionally parsed by a model for the code they are recast as. If 404's were recast as 200's, they would not be parsed by the 200 model. If a status code is returned that is not specified, its data is passed through untouched; 2xx series codes will be returned as successes, while 3xx, 4xx, and 5xx will be returned as errors.

You may be wondering when this would be useful. If you are parsing a list of objects, and those objects have related fields that call other endpoints, not all of those related calls may succeed. If even one of them fails, the entire overall call to BitScoop will be returned as an error, even if that 'failed' call is not a problem to you.

Let's say that you're getting a list of items that have OEmbed links to images, and you want to automatically populate those images from the links. Some of those OEmbed links may be bad for reasons beyond your control. You can recast those failed OEmbed calls as a 2xx call so that they don't error out the call chain, while still getting the images whose links are valid.

If you do not specify the 'responses' field at all, then whatever model and recast are specified at the same level as 'resposnes' are used for all responses.

scopes (optional) <Array of Strings>

The scopes that this endpoint needs to function. Only useful for a map that has an auth spec that has scopes, such as OAuth2.

Example:

{
    "scopes": ['users:read', 'users:write']
}

cache (optional) <Object>

If present, calls to this endpoint will be cached for the specified amount of time in seconds. Caching should only be enabled if the API being mapped permits it.

Example:

{
    "cache": {
        "expires": 300
    }
}

single / collection (optional) <Object or false>

Both of these fields, if present and not false, have as their value an endpoint spec object that can contain any of the above fields, apart from 'single' and 'collection'. These fields are used to more precisely specify endpoints that either do or do not have an identifier in the Data API call.

The fields inside either of these endpoint specs will override the higher-level fields if they are also present. Consider the following example:

Example:

{
    "Users": {
        "GET": {
            "route": {
                "path": "/users"
            },
            "headers": {
                "Authorization": "Bearer <token>"
            },
            "single": {
                "route": {
                    "path": "/getUsers"
                },
                "parameters": {
                    "id": "{{ identifier }}"
                }
            },
            "collection": {
                "parameters": {
                    "skip": "{{ parameters.skip }}",
                    "limit": "{{ parameters.limit }}"
                },
                "headers": {
                    "foo": "bar"
                }
            }
            "model": {
                "key": "id",
                "fields": {
                    "id": "string",
                    "name": "string",
                    "age": "integer"
                }
            }
        }
    }
}

The API being mapped has two endpoints, one that retrieves a list of users and the other that retrieves a single user. Both use the same authorization header for authentication. Let's say you want to use the same model in both cases since the user object is the same and you don't want to repeat large portions of the map making a second endpoint.

With this map, calling 'https://data.api.bitscoop.com/<map_id>/Users?skip=0&limit=20' will call the 'collection' endpoint. In this case, there's no 'route' specified in 'collection', so it uses the default 'route' of '/users'. It will add on the query parameters 'skip' and 'limit' with the values of 0 and 20 based on what was used to call the Data API. Since no model was specified in 'collection', it will use the model specified at the top of the endpoint spec. The header "foo": "bar" will be sent along with the authorization header; top-level headers and parameters are only overridden if the same one is specified in 'single' or 'collection', but non-conflicting ones are merged together.

Alternatively, you call 'https://data.api.bitscoop.com/<map_id>/Users/123'. The value after the slash following the endpoint name is called the 'identifier', and can be referenced in the path, parameters, headers, etc. The presence of an identifier indicates that the single route should be used, which in this case will evaluate to '/getUsers?id=123' since we passed an identifier of 123. The identifier is used to fill in the query parameter that will be sent to the endpoint we're mapping. The only header that's sent is the authorization header. As with 'collection', there's no model specified, so 'single' will use the model specified at the top of the endpoint spec.

If you specify 'single' or 'collection' as the boolean 'false', then the Data API will throw an error if you try to call that single or collection endpoint. This is not required, but is useful if you want to make sure the endpoint is not called inappropriately.

populate (optional) <String, Object, or Array>

Indicates which related fields in the model (and related fields of those related fields) should always be populated. If this is a string and its value is '*', then all related fields and related fields of those related fields will be populated. If this is a string that is not '*', then it is specifying only one related field:

Example:

{
    "Users": {
        "route": {
            "path": "/users"
        },
        "model": {
            "key": "id",
            "fields": {
                "id": "string",
                "cc_number": "string",
                "credit_card": {
                    "type": "related",
                    "ref": "CreditCards",
                    "reverse": {
                        "identifier": "{{ model.cc_number }}"
                    }
                }
            }
        },
        "populate": "credit_card"
    },
    "CreditCards" {
        "collection": false,
        "single: {
            "route": {
                "path": "/credit_cards/{{ identifier }}"
            },
            "model": {
                "key": "id",
                "fields": {
                    "id": "string",
                    "first_name": "string",
                    "last_name": "string",
                    "number": "string"
                }
            }
        }
    }
}

In this situation, the populate field of users says to populate the 'credit_card' field from the 'CreditCards' single route on every call.

This field can also be an array, which means populating multiple related fields:

Example:

{
    "Users": {
        "route": {
            "path": "/users"
        },
        "model": {
            "key": "id",
            "fields": {
                "id": "string",
                "cc_number": "string",
                "postal_code": "number"
                "credit_card": {
                    "type": "related",
                    "ref": "CreditCards",
                    "reverse": {
                        "identifier": "{{ model.cc_number }}"
                    }
                },
                "home_city": {
                    "type": "related",
                    "ref": "Cities",
                    "reverse": {
                        "paramters": {
                            "zip": "{{ model.postal_code }}"
                        }
                    }
                }
            }
        },
        "populate": ["credit_card", "home_city"]
    },
    "CreditCards" {
        "collection": false,
        "single: {
            "route": {
                "path": "/credit_cards/{{ identifier }}"
            },
            "model": {
                "key": "id",
                "fields": {
                    "id": "string",
                    "first_name": "string",
                    "last_name": "string",
                    "number": "string"
                }
            }
        }
    },
    "Cities": {
        "route": {
            "path": "/cities"
        },
        "parameters": {
            "zip": "{{ parameters.zip }}"
        },
        "model": {
            "key": "id",
            "fields": {
                "id": "string",
                "name": "string",
                "population": "integer",
                "square_miles": "number"
            }
        }
    }
}

In this example, the populate field of users says to always populate both the 'credit_card' and 'home_city' related fields on every call.

If you want to populate multiple levels of a related field chain, you need to create an object where the keys are the first-level fields and the values are the second-level fields:

Example:

{
    "Users": {
        "route": {
            "path": "/users"
        },
        "model": {
            "key": "id",
            "fields": {
                "id": "string",
                "cc_number": "string",
                "postal_code": "number"
                "credit_card": {
                    "type": "related",
                    "ref": "CreditCards",
                    "reverse": {
                        "identifier": "{{ model.cc_number }}"
                    }
                }
            }
        },
        "populate": {
            "credit_card": "bank"
        }
    },
    "CreditCards" {
        "collection": false,
        "single: {
            "route": {
                "path": "/credit_cards/{{ identifier }}"
            },
            "model": {
                "key": "id",
                "fields": {
                    "id": "string",
                    "first_name": "string",
                    "last_name": "string",
                    "number": "string",
                    "bank_id": "string",
                    "bank": {
                        "type": "related",
                        "ref": "Banks",
                        "reverse": {
                            "identifier": "{{ model.bank_id }}"
                        }
                    }
                }
            }
        }
    },
    "Banks": <endpoints spec>
}

The populate field in the above example specifies populating the 'credit_card' related field of 'Users', then additionally populating the 'bank' related field of 'CreditCards'.

If you have a very complicated related field chain, you may end up needing to combine all of the above:

    "populate": {
        "credit_card": [
            {
                "bank": "home_city"
            },
            "cc_issuer"
        ]
    }

In this example, the 'credit_card' related field in the model will be populated. The model on the endpoint that 'credit_card' calls populates two related fields, 'bank' and 'cc_issuer'. 'cc-issuer' is the end of its chain, but 'bank' will populate its related field 'home_city'.

You can always pass a population header when calling the Data API, and that population directive will override any directive specified in the map.