Data Exchange Library

The AIMMS Data Exchange library is your one-stop resource for consuming and providing REST APIs, and map data formats regularly used in such APIs onto identifiers in your model.

The library allows mapping multi-dimensional AIMMS data onto tree-based data formats such as JSON, YAML, XML and table-based formats such as CSV/TSV, Excel, Parquet and databases (as a trivial tree-based format). It does so by letting you describe the repetitive structure of a given JSON, YAML, XML, CSV, Excel, Parquet or database format in a mapping file that you can subsequently use to read data of a given format into multi-dimensional identifiers in your model, or write multi-dimensional data in your model to a given format. If you have an example JSON, YAML, XML, CSV, Excel or Parquet file, creating a mapping file with corresponding identifiers in the model to map the data onto is a fairly straightforward process.

The library has similarities with the existing ReadXML and WriteXML functions in AIMMS, extending the existing capabilities of those functions in a common code base that supports JSON, YAML, XML, CSV/TSV, Excel and Parquet formats alike. There is also an overlap in functionality with the DataLink library which solely focuses on exchanging tabular data with different types of tabular data sources, and has a provider for CSV and Excel files.

Next to hand-crafting mappings, the Data Exchange library also can generate tabular mapping for JSON, XML, CSV/TSV, Excel, Parquet and databases based on easy-to-use model annotations. This greatly simplifies the effort to create and use tabular data exchange with other applications. In addition, the library contains support for reading/writing arbitrary JSON files without the need to define a mapping through the functions dex::json::ReadInstance() and dex::json::WriteInstance().

Besides the ability to map identifier data onto various data formats, the Data Exchange library also provides an libCurl-based HTTP client, which allows you to interact with REST APIs using the DataExchange-supported formats for passing request bodies and/or retrieving API results. Obviously, you can also use the library to read and write JSON, YAML, XML, CSV/TSV, Excel and Parquet files in a standalone fashion.

The Data Exchange library also allows you to expose procedures in your model through a REST API, based on simple and easy-to-use model annotations to specify the desired service names. Data Exchange mappings can be used for parsing the request bodies, and generating the responses. This capability will be the basis for future model-based REST API capabilities of the AIMMS Cloud Platform. The service can also be used in AIMMS Developer for testing and debugging model-based API services before deployment, or for allowing AIMMS models to be integrated in with, for instance, a Python session on your desktop.

Every AIMMS Cloud account is equipped with an Azure Data Lake Storage account. The Data Exchange library offers a collection of functions to create file systems in this Data Lake Storage account, and to effortlessly upload and download files to it. The Data Exchange library also supports conveniently storing the data of one or more complete dataset instances as a collection of Parquet files in a container of a Data Lake Storage account.

Finally, the Data Exchange library support data transfer with various commonly used data warehouses. At this point the Data Exchange library supports only the Snowflake data warehouse.

The following sections explain the Data Exchange library in more detail.

• Using the Data Exchange library for communicating data
  • Example: JSON mapping
  • Example: YAML mapping
  • Example: XML Mapping
  • Example: CSV mapping
  • Example: Parquet mapping
  • Example: Excel mapping
  • Example: Database mapping
• Data Exchange Mappings
  • JSON Mapping elements
  • YAML Mapping elements
  • XML Mapping elements
  • Row-based Table mapping elements
  • Mapping attributes
  • Unicode normalization
  • How does the mapping work for reading and writing?
• Standard Data Exchange formats
  • The row-based format
  • A document-based JSON format
  • Creating your own annotation-based formats
• Application Database
  • Example: Database mapping
  • Database tables
  • Reading and Writing
  • Create Or Modify
  • Supported Databases
• Consuming REST APIs
  • Asynchronous HTTP client
  • Using OAuth2 for API authorization
  • Debugging client requests
• Generating API client code from an OpenAPI specification
  • Auto-generating runtime libraries
  • Generating support for reading from and writing to a JSON schema
  • Generating API clients from an OpenAPI specification
  • Replacing the runtime library by a permanent library
  • Data-driven generic API client
• Providing REST APIs
  • Specifying REST task types in your model
  • Service end-points exposed
  • Standard REST Task Service
  • Activating the REST service
  • Using the echo service
  • Yielding time to the API service to handle requests
• Azure Data Lake Storage
  • File systems
  • Authorization
  • Managing file systems
  • Transferring files
  • Support for storing data of generated datasets in DLS as a collection of Parquet files
• Data warehouses
  • Configuring the Snowflake data warehouse
  • Using the Snowflake library
• Methods provided by the Data Exchange library
  • Methods for reading and writing data
  • Changing encodings and normalizations
  • HTTP Client methods
  • Support for OAuth2 authorization
  • Support functions for hashing and encoding
  • HTTP Server methods
  • Memory streams
  • Generators
  • File transfer functions
  • Managing JWT Tokens
  • Creating SAS URL query strings
  • Data Lake Storage file systems
  • Data Lake Storage file transfer
  • Snowflake functions
  • Reading, writing and iterating arbitrary JSON or YAML documents
• DEX Troubleshooting
  • Errors and Warnings
• DataExchange Library Release Notes
  • New Features and Bug Fixes