Automated data exchange and API
Blackmoon API
Real-time loan purchase offers generation Daily portfolio updates Collateral markup collectingLoan purchasing process
Interaction diagramWorking with API
Sending requests Authentication and tokens Common parameters Response ExampleMethods
Offers offers.create offers.accept offers.decline offers.cancel Data sending data.sendApplications data.sendRepayments Collateral collateral.getRelevantPortfolio collateral.getRetainedPortfolio collateral.getLeftPortfolio collateral.getLastUpdateDate Portfolio portfolio.reconcileDirect... portfolio.getReverseCess... Auth refreshTokens Misc ping Error and warning codes Error codes Warning codesIntegration using DB replica
API limitations DB interactionCompilance documents
Automated data exchange and API
There are several ways in which a Loan Originator can integrate with Blackmoon platform, depending on preferred data exchange method. The final goal of integration is to setup an automated data exchange protocol, with Blackmoon receiving the information on the loan portfolio of the Originator, and Originator receiving and processing Blackmoon’s offers of purchase and/or collateral portfolio markups. The most common ways to setup data exchange are:
- Blackmoon API
- Originator’s own API
- Data exchange/grabbing using VPN connection to Originator’s restricted read-only Database replica (any DB vendor is ok) containing limited set of tables and fields
- File exchange using sFTP servers / Amazon buckets / other file storage providers
- File exchange by email sending
Blackmoon API
The API allows Originator to integrate with the Blackmoon platform. To fully integrate two major parts need to be implemented - real-time purchase offers generation and sending daily loan portfolio updates.Real-time loan purchase offers generation
This part of the API allows Originator to ask Blackmoon system to generate purchase offers for new loans as soon as they are issued, so Investors can participate in purchasing in real-time. Blackmoon evaluates the loan, calculates the fees and replies with a generated purchase offer.
This part is implemented using offers.create and offers.accept/decline/cancel methods.
Daily portfolio updates
This part of the API allows Originator to send loan and repayments data that Blackmoon uses to track Originator’s overall portfolio performance. For that two methods should be implemented: data.sendApplications and data.sendRepayments methods.
data.sendApplications call schedule:
- every day, as the new day begins, except 1st day of month: all currently open loans + loans closed in last 7 days + all loans ever purchased by Blackmoon investors (even those which were sold back, i.e. loans for which one or more cession transaction exist) regardless of status
- every 1st day of month, as the new day begins: all currently open loans + loans closed in last 32 days + all loans ever purchased by Blackmoon investors (even those which were sold back, i.e. loans for which one or more cession transaction exist) regardless of status
- every day, as the new day begins, except 1st day of month: all repayments received during last 7 days + all loans ever purchased by Blackmoon investors (even those which were sold back, i.e. loans for which one or more cession transaction exist) regardless of status
- every 1st day of month, as the new day begins: all repayments received during last 32 days + all loans ever purchased by Blackmoon investors (even those which were sold back, i.e. loans for which one or more cession transaction exist) regardless of status
Notes:
- repayments batch should use not "repayment date", but "date you have found out about this payment" - so if yesterday you find out about some repayment made 3 months ago (and as such having repayment date = -3 months), this repayment should appear in yesterday's repayments batch
- you will need to send data for all your loans, including those you have already sent to offers.create - these two methods work independently of each other
- data for all loans should be sent, including those that were not bought by Blackmoon
Collateral markup collecting
This part of the API allows you to collect Blackmoon-generated loan collateral “markup”, i.e. set of loans from active Originator’s portfolio that were marked as pledged to a particular investor in accordance to collateral funding agreements.
There is one main method collateral.getRelevantPortfolio that will return currently pledged set of loans and three methods for cross-checks and validation purposes: collateral.getRetainedPortfolio, collateral.getLeftPortfolio, collateral.getLastUpdateDate.
Important Notice
No changes must be made to the internal automated procedures of servicing and collection of transferred loans.
According to the cooperation agreement with Blackmoon the originator is engaged as the servicer of the transferred loans and the originator undertakes to treat such loans in all ways similar to the loans held on its balance sheet or sold to other investors.
Business process overview of loan purchasing
Interaction diagram
Working with API
Sending requests
All requests are sent via HTTPS using GET or POST by API url (found in your account integration parameters). All parameters are passed as GET or POST parameters. Default method is GET.
Authentication and tokens
To authenticate with API, an access token should be passed with each query (found in integration parameters).
Access token expires by default after 2 weeks, so you will need to receive a new one when it does. To automate this, you can use auth.refreshTokens method to receive new access/refresh token pair.
Common parameters
Response
All API responses are json encoded. A response is an object with two fields, one of each is always 'success' or 'error', and the second one, if present, is 'warning'.
Success fields are described in method specification. Warning field contains two fields, 'warningMessage' and 'warningCode'.
Please note that, although warning is not an error, it is a sign that something could be wrong with the query, so you should always check for warningMessage and warningCode in reply, log it and take appropriate actions.
"success": {
"offerId": "8"
},
"warning": {
"warningMessage": "This offer has already been accepted",
"warningCode": "100"
}
}
"error": {
"errorMessage": "Bad access token",
"errorCode": 2,
"requestParameters": {
"organizationId": "14",
"accessToken": "bad access token"
}
}
}
Example
"success": {
"message": "pong",
"date": "2015-01-30T14:47:39+0000",
"requestParameters": {
"organizationId": "organizationId",
"accessToken": "accessToken"
}
}
}
Methods
Offers
offers.create
Create an offer for a new loan. Please note that organizationall offers you have received
as a result of invoking this method should be either accepted or declined in 1 hour
or less, or they will be declined automatically by system.
Offers.create returns
updatedDatetime which you should always record as it was returned, since you will use it
later during reconciliation.
Sandbox version of this method does not use production calculations and instead
performs some random (but repeatable) calculation on your applicationFields. It means
that decision and fee are meaningless and are always the same for the same loan. If you
want to test both willParticipate values, just change any input value within
allowed ranges (i.e. income) in applicationFields until result changes as you like.
The nature of this calculation also results in wildly differing results even when
introducing most subtle changes. This is to be expected.
fieldName: fieldValidationRule // All loan fields needed for us to assess your loan will be added after examining your data in documentation in your personal API account
"success": {
"updatedDatetime": ISO-formatted date and time when this offer was last updated,
"willParticipate": boolean value states whether or not we are interested in participating in this loan,
"offerId": unique offer id for loan, zero if willParticipate is false,
"originationFee": origination fee in loan currency, zero if willParticipate is false. Origination fee is absolute, so originationFee = 100 for loan amount = €1000 means that origination fee is €100,
"servicingFee": servicing fee in percentage, zero if willParticipate is false. Servicing fee is a percentage of cashflow receivable, which would be paid to originator upon payment date. If payment from customer = €100 and servicingFee = 0.1525, then originator receives €15.25 as a servicing fee upon that payment,
"bidAmount": bid amount that Blackmoon is ready to make, zero if willParticipate is false (used only for originators which can sell partial loans, in other cases bidAmount will be always equal to loanAmount),
"loanAmount": loan amount which was read from applicationFields and for which fee was calculated (in loan currency), so you can double-check it was interpreted correctly by us,
"investorLegalEntityId": ID of an investor that bought the loan (integer). You will use this ID to differentiate between different legal entities buying loans from you.
}
}
offers.accept
Accept previously received offer.
Offers.accept returns updatedDatetime which you should always
record as it was returned, since you will use it later during reconciliation.
"success": {
"updatedDatetime": ISO-formatted date and time when this offer was last updated,
"offerId": offer id that was passed
}
}
offers.decline
Decline previously received offer.
Offers.decline returns updatedDatetime which you should always
record as it was returned, since you will use it later during reconciliation.
"success": {
"updatedDatetime": ISO-formatted date and time when this offer was last updated,
"offerId": offer id that was passed
}
}
offers.cancel
Cancel previously accepted offer. Please note that offer cancellation should only be used in special
cases, e.g. when user cancels an approved loan, so underlying accepted offer should be cancelled.
Offers.cancel returns updatedDatetime which you should always record as it was returned, since you will
use it later during reconciliation.
"updatedDatetime": ISO-formatted date and time when this offer was last updated,
"offerId": offer id that was passed
}
Data sending
data.sendApplications
Send loans data. See schedule in Bulk loan data export section.
Loans file fields:
applicationId: unique loan id
fieldName: fieldValidationRule // All loan fields needed for us to assess your loan will be added after examining your data in documentation in your personal API account
"success": []
}
data.sendRepayments
Send repayments data. See schedule in Bulk loan data export section.
Required fields in header are:
repaymentId - numeric, unique repayment id
applicationId - numeric, unique loan id
datetime - repayment date and time in ISO8601 format, example: 2015-05-21T10:20:38+03:00
amount - numeric, repayment amount
"success": []
}
Collateral
collateral.getRelevantPortfolio
Returns latest (if it was successfully generated during current week) collateral markup i.e. updated set of loans that are pledged to investors in accordance to collateral agreements.
{
"success": {
"ready": false
}
}
{ "success": { "ready": true, "relevant": [ {"loanId": "12345", "flici": "12345", "investorId": "24"}, {"loanId": "22345", "flici": "22345", "investorId": "25"}, {"loanId": "32345", "flici": "22345", "investorId": "25"}, ... ] } }
collateral.getRetainedPortfolio
Returns a part of latest collateral markup containing only loans that were present in previous markup and retained their pledge status.
{
"success": {
"ready": false
}
}
{ "success": { "ready": true, "retained": [ {"loanId": "12345", "flici": "12345", "investorId": "24"}, {"loanId": "22345", "flici": "22345", "investorId": "25"}, {"loanId": "32345", "flici": "22345", "investorId": "25"}, ... ] } }
collateral.getLeftPortfolio
Returns only loans that were present in previous collateral markup update but left current markup, so that they are not pledged anymore.
{
"success": {
"ready": false
}
}
{ "success": { "ready": true, "left": [ {"loanId": "12345", "flici": "12345", "investorId": "24"}, {"loanId": "22345", "flici": "22345", "investorId": "25"}, {"loanId": "32345", "flici": "22345", "investorId": "25"}, ... ] } }
collateral.getLastUpdateDate
Returns date of latest markup update in Y-m-d format. Useful for automated checks whether a new markup has been generated.
{ "success": "2017-07-03" }
Portfolio
portfolio.reconcileDirectCessionTransactions
Reconcile cession transaction on loans bought by Blackmoon investors, for all time. If any discrepancies
detected, Blackmoon tech support should be contacted and discrepancies removed.
Please note that
we need some time after receiving your data to process it, so the cession transactions might not be ready
to reconcile just yet. If that is the case, the API will reply that it is not ready yet and you should
retry this request 1 hour later, and again until you receive ready = true in reply.
[
{
"investorLegalEntityId": ID of investor legal entity who owns this loan and with which cession transaction is performed, returned in offers.create,
"loanId": ID of the loan participating in this cession transaction,
"type": cession transaction type, "direct" or "topup",
"day": YYYY-MM-DD-formatted date of cession transaction,
"price": cession transaction price,
"originationFee": cession transaction origination fee, returned in offers.create,
"servicingFee": cession transaction servicing fee, returned in offers.create
},
... // for every cession transaction made for all time
]
{
"success": {
"ready": false
}
}
{
"success": {
"ready": true,
"totalCessionTransactionsCount": 23,
"correctCessionTransactionsCount": 23,
"platformOnly": [],
"platformMissing": []
}
}
{
"success": {
"ready": true,
"totalDirectCessionTransactionsCount": 23,
"correctDirectCessionTransactionsCount": 21,
"platformOnly": [{ // These cession transactions were not found in request and exist only on Blackmoon platform
"investorLegalEntityId": 24,
"loanId": 89,
"type": "direct"
"day": "2016-05-30"
"price": 2450.31
"originationFee": 99.17,
"servicingFee": 0.01
}],
"platformMissing": [{// These cession transactions were found in request but they do not exist on Blackmoon platform
"investorLegalEntityId": 24,
"loanId": 43,
"type": "direct"
"day": "2016-06-10"
"price": 1.99
"originationFee": 10.00,
"servicingFee": 1858.39
},
}]
}
portfolio.getReverseCessionTransactions
Get full list of loans Blackmoon investor sell back to originator, for all time.
{
"success": {
"ready": false
}
}
{
"success": {
"ready": true,
"reverseCessionTransactions": [{
"loanId": 3917867,
"price": "199.3",
"type": "UpperThreshold", // Possible values: Overdue, UpperThreshold, LowerThreshold, Cancelled
"date": "2016-01-01",
},
{ // An entry for every reverse cession transaction
...
}]
}
}
Auth
auth.refreshTokens
Refresh both access and refresh tokens.
"success": {
"accessToken": new access token,
"refreshToken": new refresh token,
"accessTokenExpirationTime": new access token expiration time
}
}
Misc
ping
Helper method which replies with a message 'pong', current server date and all parameters that you have sent. Useful to perform connectivity checks and debug tricky parameters.
"success": {
"message": pong,
"date": current server time,
"parameters": [ ...array of all passed parameters ]
}
}
Error and warning codes
Error codes
1 Validation failed2 Bad access token
3 Bad refresh token
5 Access token expired
6 This is a sandbox-only method
7 File not valid
8 Error while reading file
9 Offer not found
10 Internal API error
11 Access denied
12 File was not uploaded, possibly too big
100 Unable to decode applicationFields
101 Unable to calculate fee
102 A resolved offer already exists for this loan
103 An offer already exists for this loan
110 Cannot accept, bad offer status
120 Cannot decline, bad offer status
130 No loans file passed
140 No repayments file passed
150 Unable to calculate limits
160 Unable to reconcile offer
170 Unable to cancel offer, bad offer status
Warning codes
100 This offer has already been accepted101 This offer has already been declined
Integration using DB replica
API limitations
API integration has some natural limitations that affect daily batch data sending.
“Pushing” information instead of “pulling”
To perform information update, Originator must explicitly invoke Blackmoon API method, so Blackmoon cannot perform manual data checks without asking Originator to resend some of the data. This complicates integration process and requires additional resources to be provided for maintenance afterward.
Additional layer of entities and services meaning additional points of failure
API services, cronjobs, internal middlewares for temporary information storage have to be maintained and monitored both for being functional and for storing consistent amounts of data. Many times we faced the issues with incomplete batches generation, for example, daily loans batch arrives in a consistent state while repayments arrive only partially due to some middleware memory limit being reached.
Changes to DB information storage must be reconciled against API requests generation
To create API request Originator must “prepare” DB-stored information to be sent to Blackmoon. This means that all changes in information storage must be “reconciled” and checked against compatibility with such preparations. Often there are situations where subtle changes in DB are not being translated to API request generation mechanisms and that in turn causes API requests to change unpredictably. In such cases, this gets noticed on the analytical level after some time.
Shared responsibility in case of errors
If something is wrong with the data exchange, both sides spend resources for investigations, meaning 50% loss of effort in general. If the process is majorly owned by one party that leaves very high probability that only this party should investigate the issue.
DB interaction
The factors above led us to create a new form of integration - using a restricted DB replica that Blackmoon can access and pull data from. Such replica should:
- be accessible through secure VPN connection, without Internet-facing ports
- be restricted to a read-only user for Blackmoon
- contain only agreed set of tables and fields that in the case of API integration would be sent in data.send-methods
- be regularly updated/replicated from production DB of the Originator. Replication mechanisms of modern DBs are very robust and tend to “just work”, and require near-zero maintenance
- occupy a separate set of resources (VM/VPC/physical host) so in a case of high load it would not affect production DB performance
- contain general-purpose indexes in tables for quick querying
In general, this DB should contain two tables “loans” and “repayments” connected through loanId entity.
If such DB is accessible for Blackmoon, all interactions for data gathering are simplified and fall onto Blackmoon team. Blackmoon creates and maintains data grabbing services, handles mapping, monitors changes. That eliminates the need for implementation of data.send-API methods by Originator and consequently speeds up integration process.