Using the Pyrolyzer-8894 API for CAMEO

Pyrolyzer-8894 has made available an HTTP API enabling the use of its carbon calculations infrastructure in the context of CAMEO. We chose to expose the functionality as an HTTP API (not a command-line program) so that we can make updates immediately available.

Prerequisites

In order to use this API, you will need:

1. An authentication token provided by CAMEO, for example my-auth-token-890DFGHjfkdlsfJHE45457UHGE76jdfhksfY.
2. A valid user-id.

Usage

The API is available at https://api-cameo.sig-gis.com/cameo/v1.

Step 1: Uploading the Tree Data file

The first step as with doing an analysis on a scenario from the CAMEO site consists of uploading the Tree Data, by calling /upload-file/<filename-suffix>:

import os.path
import requests
import json
import time

# Base API URL
api_url = "https://api-cameo.sig-gis.com/cameo/v1/relay-server"
cameo_auth_token = 'YOUR-CAMEO-AUTH-TOKEN'
user_id = '<YOUR-USER-ID>'
fvs_variant = '<TWO-LETTER-VARIANT>'

def upload_file(file_path: str):
    filename = os.path.basename(file_path)
    with open(file_path, 'rb') as input_file:
        http_resp = requests.post(
            f'{api_url}/upload-file/{filename}',
            headers={'Authorization': f'Bearer {cameo_auth_token}'},
            files={'file': (file_path, input_file)},
            timeout=10
        )
        http_resp.raise_for_status()
        logical_path = http_resp.json()
        return logical_path

tree_data_logical_path = upload_file("/path/to/tree_data.xls")
cameo_to_fvs_keyfile = upload_file("/path/to/first_keyfile.key")
fvs_keyfile = upload_file("/path/to/second_keyfile.key")

The value returned by the /relay-server/upload-file/:filename_suffix endpoint is a 'logical path' - something that the API will now how to translate to a concrete file location on the CAMEO cluster. Clients should consider it an opaque value and not try to understand its structure.

Step 2: Submit job to do CAMEO Analysis

To submit the job to help you do the all the process involved in the carbon calculations. We are now ready to actually trigger simulations, by calling /relay-server/submit-job:

# Only these arguments are technically required arguments - but you are free to change any of them.
def submit_job(
        user_id,
        fvs_variant,
        scenario_name,
        fvs_keyfile,
        cameo_to_fvs_keyfile,
        tree_data_path
):
    body = json.dumps(
        {
            "sig_relay_arguments": {
                "scenario-name": "testing scenario",
                "fvs-variant": fvs_variant,
                "user-id": user_id,
                "cameo_to_fvs": {
                    "keyfile": cameo_to_fvs_keyfile,
                    "input-spreadsheet": tree_data_path,
                    "max-dbh": 48,
                    "max-height": 145,
                    "model-length": 100,
                    "num-dbh-breaks": 1,
                    "num-plot-size": 2,
                    "num-sub-plot-size": 1,
                    "scenario-desc": scenario_name,
                    "top-volume-id": 1,
                    "version-id": 1,
                    "broken-top-id": 1,
                    "phase-id": 1,
                    "defect-criteria-id": 1,
                    "edit-perm-id": 1,
                    "view-perm-id": 2
                },
                "fvs": {
                    "keyfile": fvs_keyfile
                },
                "fvs_to_carbon": {
                    "analysis-type-id": 1,
                    "growth-model-id": 1,
                    "tree-years-id": 1,
                    "fall-down-percent": 0
                }
            }
        })
    print("Sending:", body)
    http_resp = requests.post(
        f'{api_url}/submit-job',
        headers={
            'Authorization': f'Bearer {cameo_auth_token}',
            'Content-Type': 'application/json',
            'Accept': 'application/json'
        },
        data=body,
        timeout=10
    )
    http_resp.raise_for_status()
    return http_resp.json()['sig_relay_deferred_uid']

my_uid_for_polling = submit_job(user_id, fvs_variant, "test scenario", fvs_keyfile, cameo_to_fvs_keyfile, tree_data_logical_path)

The above queues a request for running simulations in CAMEO's computing infrastructure. We do not immediately get a result: instead we get my_uid_for_polling, with which we will be able to poll the API to monitor completion status (see next section).

To submit a partial job workflow, the request should provide the agency_arguments map with the optionals agency/exclude-nodes, agency/target and agency/constants keys.

Usually when excluding a node (agency/exclude-nodes), the request should still provide the needed arguments for all remaining nodes. For that, the request should also provide the agency/constants argument.

2.1 Partial workflow: calling all steps except fvs

If the request body is like the example below, the fvs step will be bypassed. To make this possible, the agency/constants values must be set accordingly.

    body = json.dumps(
        {
            "sig_relay_arguments": {
                "scenario-name": "testing_scenario",
                "fvs-variant": fvs_variant,
                "user-id": user_id,
                "cameo_to_fvs": {
                    "keyfile": cameo_to_fvs_keyfile,
                    "input-spreadsheet": tree_data_path,
                    "max-dbh": 48,
                    "max-height": 145,
                    "model-length": 100,
                    "num-dbh-breaks": 1,
                    "num-plot-size": 2,
                    "num-sub-plot-size": 1,
                    "scenario-desc": scenario_name,
                    "top-volume-id": 1,
                    "version-id": 1,
                    "broken-top-id": 1,
                    "phase-id": 1,
                    "defect-criteria-id": 1,
                    "edit-perm-id": 1,
                    "view-perm-id": 2
                },
                "fvs_to_carbon": {
                    "analysis-type-id": 1,
                    "growth-model-id": 1,
                    "tree-years-id": 1,
                    "fall-down-percent": 0
                }
            },
            "agency_arguments": {
                "agency/constants": {
                    "fvs-file": "/path/to/testing_scenario.db"
                },
                "agency/exclude-nodes": ["cameo-services/fvs"],
            }
        }
    )

2.2 Partial workflow: calling only fvs step

If the request body is like the example below, only the fvs step will be run. To make this possible, the agency/constants values must be set accordingly.

    body = json.dumps(
        {
            "sig_relay_arguments": {
                "scenario-name": "testing_scenario",
                "fvs-variant": fvs_variant,
                "user-id": user_id,
                "fvs": {"keyfile": fvs_keyfile},
            },
            "agency_arguments": {
                "agency/constants": {
                    "scenario-id": 99,
                    "input-database": "/path/to/testing_scenario_yyyy-mm-dd_cameo-to-fvs_hash.db",
                },
                "agency/target": "cameo-services/fvs",
                "agency/exclude-nodes": [
                    "cameo-services/cameo-to-fvs",
                    "cameo-services/fvs-to-carbon",
                ],
            },
        }
    )

Argument documentation

Here is a summary of the option keys available to pass.

Top-Level

The upper-most level of the sig_relay_arguments map has these keys. Each service has its own child object with their own parameters.

The upper-most level of the agency_arguments map has these keys. They can be combined to change the regular flow (cameo-to-fvs -> fvs -> fvs-to-carbon):

• This options is especially useful in combination with agency/exclude-nodes in order to provide inputs for the flows that depends off the excluded node(s)

Cameo To FVS

The Cameo To FVS microservice captures the functionality of importing tree data and keywords into Cameo to produce FVS input data.

FVS

The FVS Service runs the output from Cameo To FVS through FVS with a specified keyfile, uploaded by the user.

FVS To Carbon

The FVS To Carbon microservice utilizes the results from the FVS microservice to generate a carbon report, downloadable by users in the form of a 7z archive.

Step 3: Polling for results

The following code snippet shows how to await the results of simulations, by polling the /relay-server/check-job-status/:sig_relay_deferred_uid endpoint every 5 seconds:

def await_cameo_results(sig_relay_deferred_uid: str, polling_interval_s=1):
    dfr_status = 'sig_relay_deferred_pending'
    results = None
    while dfr_status == 'sig_relay_deferred_pending':
        http_resp = requests.get(
            f'{api_url}/check-job-status/' + sig_relay_deferred_uid,
            headers={'Authorization': f'Bearer {cameo_auth_token}'},
            timeout=10
        )
        http_resp.raise_for_status()
        body = http_resp.json()
        dfr_status = body['sig_relay_deferred_status']
        if dfr_status == 'sig_relay_deferred_success':
            results = body['sig_relay_deferred_result']
        elif dfr_status == 'sig_relay_deferred_error':
            raise Exception("Simulations failed!")
        else:
            time.sleep(polling_interval_s)
    return results

my_results = await_cameo_results(my_uid_for_polling, 5)
print(my_results)

# To download output files
def download_file(url, file_name):
    with open(file_name, "wb") as downloaded:
        http_resp = requests.get(
            url,
            headers={'Authorization': f'Bearer {cameo_auth_token}'},
            timeout=10
        )
        downloaded.write(http_resp.content)


for key in ["carbon-report", "fvs-file", "input-database"]:
    output_base_path = "path/path/path"
    key_file_url = my_results[key]
    key_file_name = key_file_url.rsplit('/', 1)[-1]
    download_file(key_file_url, f"{output_base_path}/{key_file_name}")