Methods provided by the Data Exchange library

The following functions you can use within your model, are exposed by the Data Exchange library.

Methods for reading and writing data

dex::AddMapping(mappingName, mappingFile)

Parses mappingFile to create a mapping called mappingName. The function will return 1 on success, or 0 on failure.

Arguments

  • mappingName – the name of the mapping to be created

  • mappingFile – the relative path to the mapping file to be parsed.

dex:DeleteMapping(mappingName)

Deletes an existing mapping mappingName. The function will return 1 on success, or 0 on failure. Mappings referring to runtime identifiers, must be deleted prior to deleting the corresponding runtime library.

Arguments

  • mappingName – the name of the mapping to be created

dex::ReadFromFile(dataFile, mappingName, emptyIdentifiers, emptySets, resetCounters)

Reads data from file dataFile into model identifiers using mapping mappingName. Note that the identifiers used in the included-mapping and write-filter will not be emptied, regardless of the emptyIdentifiers argument. When the mapping contains an the included-mapping or the iterative-existing attributes, emptying sets is likely to cause problems, unless the domain sets referred in these attributes are defined. The function will return 1 on success, or 0 on failure.

Arguments

  • dataFile – the relative path to the data file to be read

  • mappingName – the name of the mapping to be used

  • emptyIdentifiers – indicates whether all identifiers referred in the mapping should be emptied prior to reading the file

  • emptySets – indicates whether all domain and range sets referred in the mapping should be emptied prior to reading the file

  • resetCounters – indicates whether to reset all counters for iterative-binds-to indices prior to reading the file

dex::WriteToFile(dataFile, mappingName, pretty)

Writes file dataFile from data in model identifiers using mapping mappingName. The function will return 1 on success, or 0 on failure.

Arguments

  • dataFile – the relative path to the data file to write to

  • mappingName – the name of the mapping to be used for writing

  • pretty – indicates whether to use a pretty writer (enhances readibility at the cost of bigger file size)

dex::ReadAllMappings()

Read all mappings contained in the folder Mappings and store all successfully read mappings in the set dex::Mappings. The function will return 1 on success, or 0 on failure.

dex::ReadMappings(startPath, recursive)

Read all mappings contained in the folder Mappings contained in startPath, and store all successfully read mappings in the set dex::Mappings. The function will return 1 on success, or 0 on failure.

Arguments

  • startPath – optional string argument indicating the folder containing mapping folder (default “”, indicating the project folder)

  • recursive – optional argument indicating whether or not to search for mapping files recursively in the Mappings folder

dex::ReadAnnotations()

Read all dex::Dataset, dex::TableName, and dex::ColumnName annotations specified in the model, and use these annotations to fill the identifiers

  • dex::Datasets

  • dex::Tables

  • dex::Columns

  • dex::DatasetTableMapping

  • dex::ColumnName

  • dex::DatasetTableColumnName

  • dex::DatasetTableColumnIndex

  • dex::DatasetTableColumnIdentifier

When every table can needs to be included in just a single dataset, you can uniquely specify the dataset-table mapping using annotations only. If tables need to be included in multiple datasets, you can manually modify the identifier dex::DatasetTableMapping to add any table to the datasets you wish to include them in.

dex::GenerateDatasetMappings()

Generate standardized table and Excel sheet mappings based on the dex::Dataset, dex::TableName, and dex::ColumnName annotations. The generated mappings will be stored in the Mappings/Generated subfolder of the project folder. All generated mappings will automatically be added to the set of available mappings, and can be directly used to read and write the standardized JSON, XML, CSV or Excel data sources based on the data exchange annotations. The function will return 1 on success, or 0 on failure.

You can influence how mappings will be generated through:

  • dex::DatasetDenseMappings: when a row will be written, determines wither all columns in that row will be written (default), or only the columns containing a non-default value

  • dex::DatasetXMLAttributeMappings: determines whether the generated XML format will write all values as XML attribute values (default) or as element values. Indices will always be written as XML attributes.

You can use the generated mappings directly with the functions dex::WriteToFile() and dex::ReadFromFile() as with any manually created mapping.

Changing encodings and normalizations

The Data Exchange library only accepts UTF-8 JSON, XML and CSV files. Through the following functions you can change the encoding of a file prior to reading or after writing its contents. The library also contains a number of functions to normalize composed Unicode characters in strings and sets in your model to either the NFC or NFD normalization.

dex::ConvertFileToEncoding(inputFile, inputEncoding, ouputFile, outputEncoding, noBOM)

Converts file inputFile with encoding inputEncoding to file outputFile with outputEncoding, optionally with a BOM.

Arguments

  • inputFile – file path of the input file

  • inputEncoding – encoding of the input file from the predefined set AllCharacterEncodings

  • outputFile – file path of the output file

  • outputEncoding – encoding of the output file from the predefined set AllCharacterEncodings

  • noBOM – optional argument indicating whether or not the output file should start with a BOM (default 1)

dex::NormalizeString(inStr, outStr, normalization)

Normalize inStr to outStr using the normalization procedure indicated by normalization.

Arguments

  • inStr – scalar input string parameter to hold the string value to normalize

  • outStr – scalar output string parameter that will hold the normalized string

  • normalization – optional element parameter into dex::Normalizations indicating the normalization to apply (either nfc (default) or nfd)

dex::NormalizeSet(aSet, normalization)

Normalize all elements in the set aSet using the normalization procedure indicated by normalization. All elements that changed by the selected normalization will be renamed in the set.

Arguments

  • aSet – set argument indicating the set for which to normalize all elements

  • normalization – optional element parameter into dex::Normalizations indicating the normalization to apply (either nfc (default) or nfd)

HTTP Client methods

The Data Exchange library contains collection of functions implemented using libCurl (see the libCurl documentation). The following methods are exposed by the Data Exchange library to send HTTP client requests and to handle their responses.

dex::client::NewRequest()

Create a new HTTP request with (unique) identification theRequest to the URL url, with method httpMethod (optional, default GET). Upon response from the web server, the callback method callback will be called. The prototype of callback should be the same as the function dex::client::EmptyCallback. For POST and PUT methods, you can specify the file requestFile from which to take the request body of the request. If you specify the optional responseFile argument, the response body will be captured in the specified file. If ommitted the response body will be silently discarded. The function will return 1 on success, or 0 on failure. If a traceFile is being specified, tracing for the request will be enabled, and the detail trace output from libcurl will be stored in the specified file.

Arguments

  • theRequest – string parameter holding the unique identification of the request.

  • url – string parameter holding the URL of the request, including any query parameters you want to add to the request.

  • callback – element parameter into AllProcedures, holding the callback to be called asynchronously after the response to the HTTP request has been received

  • httpMethod – (optional) element parameter into dex::client::HTTPMethods, specifying the HTTP method to use for the request (default GET)

  • requestFile – (optional) string parameter holding the filename from which to take the request body

  • responseFile – (optional) string parameter holding the filename in which to store the response body :param traceFile: (optional) string parameter holding the filename in which all trace information about the request is being stored

dex::client::CloseRequest()

Close the request theRequest and all resources held by the Data Exchange library for the request. If the request has been executed, but Data Exchange library is still listening for a response to the request, it will stop doing so. By default, the Data Exchange library will close the request directly after its callback method has been called to free its resources as soon as possible (e.g. when a large number of request is being executed). Notice that closing a request will not remove any request or response files specified in dex::client::NewRequest. The function will return 1 on success, or 0 on failure.

Arguments

  • theRequest – string parameter holding the unique identification of the request to close

dex::client::CloseAllRequests()

Close any outstanding HTTP requests, that have been created and may still be executing. The function will return 1 on success, or 0 on failure.

dex::client::PerformRequest()

Execute a previously created HTTP request theRequest. Upon response, the Data Exchange library will call the specified callback function asynchronously, as soon as the request has been completed and the AIMMS engine is idle. To force :token:callback`to be called synchronously within a procedure of your model, you can use the method dex::client::WaitForResponses. The function will return 1 on success, or 0 on failure.

Arguments

  • theRequest – string parameter holding the unique identification of the request to execute

dex::client::SetDefaultOptions()

Using the function dex::client::SetDefaultOptions you can specify multiple string and integer-valued Curl options that will be applied to all requests, to modify the behavior of libCurl. All available Curl options can be found in the sets dex::client::StringOptions and dex::client::IntOptions. For the interpretation of these options please refer to the Curl options documentation. The function will return 1 on success, or 0 on failure.

Arguments

  • intOptions – integer parameter over the set dex::client::intOptions holding the default integer Curl options to set

  • stringOptions – string parameter over the set dex::client::StringOptions holding the default string Curl options to set

dex::client::AddRequestOptions()

Using the function dex::client::AddRequestOptions you can specify multiple string and integer-valued Curl options to request theRequest, to modify the behavior of libCurl. All available Curl options can be found in the sets dex::client::StringOptions and dex::client::IntOptions. For the interpretation of these options please refer to the Curl options documentation. The function will return 1 on success, or 0 on failure.

Arguments

  • theRequest – string parameter holding the unique identification of the request to add request options to.

  • intOptions – integer parameter over the set dex::client::intOptions holding the integer Curl options to set

  • stringOptions – string parameter over the set dex::client::StringOptions holding the string Curl options to set

dex::client::AddStringOption()

Low-level method to set a single string-valued Curl option for request theRequest. The argument stringOptionId should be the id corresponding to the option taken from the parameter dex::client:CurlOptionId. The function will return 1 on success, or 0 on failure.

Arguments

  • theRequest – string parameter holding the unique identification of the request to add the string-valued request option to.

  • stringOptionId – parameter holding the Curl id for the option (taken from dex::client:CurlOptionId).

  • optionValue – string parameter holding the option value.

dex::client::AddIntOption()

Low-level method to set a single integer-valued Curl option for request theRequest. The argument intOptionId should be the id corresponding to the option taken from the parameter dex::client:CurlOptionId. The function will return 1 on success, or 0 on failure.

Arguments

  • theRequest – string parameter holding the unique identification of the request to add the integer-valued request option to.

  • intOptionId – parameter holding the Curl id for the option (taken from dex::client:CurlOptionId).

  • optionValue – parameter holding the option value.

dex::client::SetDefaultHeaders()

Using the function dex::client::AddRequestHeaders you can specify any HTTP headers you want to add to subsequent request. Notice that some Curl options will also result in the addition of HTTP headers to the request. The function will return 1 on success, or 0 on failure.

Arguments

  • headers – string parameter over a (user-defined) set of headernames holding the corresponding header values to add to all subsequent requests.

dex::client::AddRequestHeaders()

Using the function dex::client::AddRequestHeaders you can specify any HTTP headers you want to add to request theRequest. Notice that some Curl options will also result in the addition of HTTP headers to the request. The function will return 1 on success, or 0 on failure.

Arguments

  • theRequest – string parameter holding the unique identification of the request to add request headers to.

  • headers – string parameter over a (user-defined) set of headernames holding the corresponding header values to add

dex::client::AddRequestHeader()

Using the function dex::client::AddRequestHeader you can specify a single HTTP header you want to add to request theRequest. The function will return 1 on success, or 0 on failure.

Arguments

  • theRequest – string parameter holding the unique identification of the request to add a request header to.

  • headers – string parameter holding the header name to add

  • headerValue – string parameter holding the header value to add

dex::client::AddMimePart()

Using the function dex::client::AddMimePart you can create a multi-part MIME body for a POST request. The function will return 1 on success, or 0 on failure.

Arguments

  • theRequest – string parameter holding the unique identification of the request for which to create a multi-part MIME body.

  • partname – string parameter holding the name of the part

  • partfile – string parameter holding the name of the file containing the contents of the part.

  • asfile – parameter indicating whether part is to be treated as a file part, in which case the base name of partfile is transferred as the remote file name.

dex::client::EmptyCallback()

Prototype function for any callback to be added as the callback parameter of the function dex::client::NewRequest. Inside the callback you can retrieve info items provided by libCurl and any response headers regarding the executed request, or handle the response file associated with the request. To free resources, the Data Exchange library will delete a request directly after its callback has been called. At such point, you will not be able to retrieve any info items for the request any longer, but, you as a caller will remain responsible for deleting any request and response files you may have specified.

Arguments

  • theRequest – string parameter holding the unique identification of the request for which the callback is called.

  • statusCode – HTTP status code of the response.

  • errorCode – Curl error code for the response in case the request was not successful.

dex::client::GetInfoItems()

Using the function dex::client::GetInfoItems you can retrieve string- and integer-valued info items provided by libCurl regarding the executed request inside the callback function specified in the dex::client::NewRequest method. For the interpretation of the available info items, see the Curl info documentation. The function will return 1 on success, or 0 on failure.

Arguments

  • theRequest – string parameter holding the unique identification of the request for you want to retrieve info items

  • infoItems – subset of dex::client::CurlInfoItems holding the collection of string- or integer-valued info items you want to retrieve.

  • intInfoItems – output parameter holding the integer-valued info item values.

  • stringInfoItems – output string parameter holding the string-value info item values.

dex::client::GetStringInfoItem()

Using the function dex::client::GetStringInfoItem you can retrieve a single string-valued info item provided by libCurl regarding the executed request inside the callback function specified in the dex::client::NewRequest method. The parameter stringinfoId should hold the id corresponding to the info item taken from the parameter dex::client:CurlInfoId. The function will return 1 on success, or 0 on failure.

Arguments

  • theRequest – string parameter holding the unique identification of the request for you want to retrieve info items

  • stringinfoId – parameter holding the id of the string-valued info item

  • infoValue – output string parameter holding the value of the requested string info item.

dex::client::GetIntInfoItem()

Using the function dex::client::GetStringInfoItem you can retrieve a single integer-valued info item provided by libCurl regarding the executed request inside the callback function specified in the dex::client::NewRequest method. The parameter intinfoId should hold the id corresponding to the info item taken from the parameter dex::client:CurlInfoId. The function will return 1 on success, or 0 on failure.

Arguments

  • theRequest – string parameter holding the unique identification of the request for you want to retrieve info items

  • intinfoId – parameter holding the id of the integer-valued info item

  • infoValue – output parameter holding the value of the requested integer info item.

dex::client::GetResponseHeaders()

Using the function you can retrieve the HTTP headers of the response of theRequest. The function will return 1 on success, or 0 on failure.

Arguments

  • theRequest – string parameter holding the unique identification of the request for you want to retrieve the response headers

  • headers – output string parameter over a (user-defined) header set, holding the values of all headers in response, which will be added to the header set.

dex::client::GetErrorMessage()

With this function you can retrieve the Curl error message associated with the error code passed back via a request calbback. The function will return 1 on success, or 0 on failure.

Arguments

  • errorCode – parameter holding the error code passed back via a request callback

  • errorMessage – output string parameter holding the associated error message

dex::client::WaitForResponses()

Using this function you can block the execution of the calling procedure for a maximum of timeout milliseconds to wait for incoming responses of any outstanding HTTP requests. As soon as a first response is available for any of the outstanding requests within the given timeout, its associated callback will be called, as well as for any other available responses. If there are no further responses, the function will return. The function will return 1 if one or more responses came in within the given timeout, or 0 on timeout.

Arguments

  • timeout – the maximum time in milliseconds to wait for any incoming responses.

dex::client::SetParallelConnections()

With this function you can set the maximum number of client connections that will be used concurrently. Any HTTP request submitted using dex::client::PerformRequest will be executed using one of these concurrent connections. If the number of non-processed requests exceeds the maximum number of concurrent connections, the request will be queued until a connection becomes available.

Arguments

  • nrconn – the desired maximum number of concurrent client connections allowed (default 16).

dex::client::QueryMapEncode()

Using this function you can construct a URL-encoded list of query parameters that you want to add to a URL. All query parameters are separated by an &, and you can add it to a URL by appending it with a ? token to the URL.

Arguments

  • queryMap – an indexed string parameter over a set of query parameters, holding the associated query parameter values

  • queryString – a scalar output string parameter holding the URL-encoded query parameter string that you can append to the URL.

dex::client::StopClient()

This function close all outstanding requests, and uninitialize libCurl to handle any incoming responses. The function will return 1 on success, or 0 on failure.

dex::client::ProxyResolve()

Use the OS proxy configuration to discover a proxy for the given URL. Whenever a proxy is found it can be added to a HTTP request via the CURLOP_PROXY option. This function is only implemented for the Windows OS.

Arguments

  • url – the URL for which to determine a proxy

  • proxyUrl – output string argument to hold the proxy URL for the given URL.

dex::client::Poll()

Convenience function to poll for certain events by executing a procedure at a given interval. This can for instance by used to regularly check the status of a long-running REST call. Only one function can poll at any given moment.

Arguments

  • pollingProcedure – element parameter into AllProcedures holding the procedure to be executed regularly. The procedure should have no arguments. Polling will be stopped whenever the procedure returns a value of 0, in all other cases polling will continue.

  • interval – fixed interval in milliseconds in between calls to the polling procedure.

dex::client::StopPolling()

Alterative method to externally stop the sequence of calls to a polling procedure added via dex::client::Poll().

HTTP Server methods

The Data Exchange library supports exposing procedures in your model as endpoints of an HTTP REST service. You can configure and use this service via the methods below.

dex::api::StartAPIService()

This function will collect all procedures with a dex::ServiceName annotation, and will start the HTTP service listener, to listen to, and handle incoming service requests. Prior to calling dex::api::StartAPIService, you can configure the listen port and maximum accepted request size in MB, through the configuration parameters:

  • dex::api::ListenerPort (default 8080)

  • dex::api::MaxRequestSize (default 128 MB)

dex::api::StopAPIService()

This function will stop the HTTP service listener waiting for incoming requests.

dex::api::Yield()

You can use this function yield control for a maximum of timeout milliseconds to the HTTP server component of the Data Exchange library to handle incoming requests synchronously. The function will return 1 if one or more requests were handled within the given timeout, or 0 on timeout.

Arguments

  • timeout – the maximum time in milliseconds to wait for, and handle, any incoming requests.

Memory streams

Any file

can also be a memory stream. If the file name starts with a #, the Data Exchange library will assume that the specified file name is to be interpreted as a memory stream. Memory streams for the output file of the function dex::WriteToFile() and the response file of the function dex::client::NewRequest() will create a memory stream with the given file name as its key, while the input file of the function dex::ReadFromFile() and the request file of the function dex::client::NewRequest() will assume an existing memory stream with the given key.

Memory streams with keys starting with ## used as request or response files will be automatically deleted when the corresponding dex::client request is closed.

The following functions are available for management of the memory streams.

dex::DeleteStream()

Delete the memory stream corresponding to key streamName.

Arguments

  • streamName – name of the stream key to delete (including the #)

dex::DeleteAllStreams()

This function will delete all streams created via dex::WriteToFile() and dex::client::NewRequest().

dex::SetDefaultStreamSize()

Every stream created will hold space for streamSize bytes. When more bytes are written to a memory stream it will automatically double the available amount of memory but at the expense of copying the existing content. The initial default stream size is 64 KB.

Arguments

  • streamSize – the default stream size (in bytes) to use.

dex::ImportStreamContent()

Import the content of a string parameter into a new memory stream. The name of the stream should start with a #, to allow the stream to be used by other functions of the Data Exchange library. This function supports string parameters up to 8 KB of content.

Arguments

  • streamName – name of memory stream to import content into

  • content – input string parameter holding the string to import into the memory stream

dex::ExportStreamContent()

Export the content of an existing memory stream into a string parameter. This function supports exporting memory streams up to 8KB.

Arguments

  • streamName – name of memory stream to export content from

  • content – output string parameter to hold the content (up to 8KB) exported from the memory stream

dex::WriteStreamToFile()

Write the content of an existing memory stream to a file.

Arguments

  • streamName – name of memory stream to write content from

  • fileName – name of the to which the content of the stream needs to be written.