SONM Task manager guide

SONM Task manager (Taskman) is a bot for customers. Its primary goal is to automate deal and task managing making it as easy as possible.

Written with Python 3.7., open-sourced and available for changes with no limitations.

Concept:

  • describe hardware needs for your task(s), including preferred price and number of instances;
  • define your task(s);
  • run the bot (in user session or in the background);
  • relax and watch how the bot will take deals, start your task and perform task tracking.

Features:

  • create multiple orders (constant quantity) to run given task(s);
  • handle deal opening and run the task;
  • track task start, task status;
  • close deals:
    • if the task failed to start (and add a worker to blacklist);
    • if the task has finished with success;
  • create an order for a new deal if the previous deal has been closed, and task has not finished yet;
  • able to handle multiple different task types (configs must differ by tag);
  • provide password-protected HTML page with stats;
  • easy apply of config changes - config reloads once per miniute, and this allow a user to:
    • increase of reduce the number of nodes for a particular task;
    • add or remove tasks from executor.

Preparations

You should have SONM installed (see how to install SONM for customer).

  1. Install Python 3.7 and pip 3.7.
  2. Clone GitHub repository (or download the source code and unzip to preferred folder).
  3. Run pip3.7 install -r requirements.txt to install dependencies.

Configuration files

All configs are stored at ./conf folder.

Main configuration file

Location ./conf/config.yaml

  • Set sonm-node API endpoint and keystore settings in '#node config' section.
  • Set web-server settings (provide HTML page with statistics).
  • Set the list of task configs you want to run in SONM.

Example:

# HTTP Server preferences
# statistics page available on http://localhost:8081 when bot is running.
http_server:
  run: true
  port: 8081
  user: "sonm"
  password: "sonm"

#SONM Node preferences
# default endpoint for SONM Node REST API is 'http://127.0.0.1:15031'
node_address: 'http://127.0.0.1:15031'

# keystore location and password (default keystore location in Linux is "/etc/sonm/sonm-keystore/")
ethereum:
  key_path: "/etc/sonm/sonm-keystore/"
  password: "changeMe"

#default timeout for all requests to Node API is 60 seconds, you may customize this (optional).
#timeout: 60

#time since last heartbeat (in seconds) - drops the deal and restart particular node if its status stuck
restart_timeout: 600

# LIST OF TASKS to run
tasks:
  - claymore_config.yaml
  - cs16_config.yaml

Resource cluster requirements for a task

You should have a separated configuration file for each type of task you want to run. Task type is described by two config files:

  • resource cluster definition (current section);
  • task definition (see next section).

The bot will create orders for this config and run the task, if its name mentioned in main configuration file.

Location ./conf/<any_name>.yaml

Describe:

  • cluster size (number of nodes to be launched);
  • tag (if you're planning to run several different tasks, they should differ by tag);
  • price preferences:
    • the bot will ask for a predicted price, if price predictor is running, this is a SONM Node feature, see SONM Node config for details;
    • you may set a price limit in 'max_price' setting, and the bot will set this price for orders if price predictor is unavailable or returns higher price;
  • counterparty (optional) - if you want to have a deal with particular supplier, set his master address in this preference.
  • hardware needs for your task;
  • 'template_file' - a name of a file with task definition (see 'Task config file' section below).

Example ('cs16_config.yaml' mentioned in main config above):

# cluster size
numberofnodes: 1
#cluster name, bid's and tasks will use it
tag: "CS16"

# final price = predicted price * (1 + price_coefficient/100)
price_coefficient: 10

# $ per hour
max_price: "0.002"

# Estimated time of successful start of task (blacklist worker if task failed before ets)
ets: 180

task_start_timeout: 600

# Template for task yaml
template_file: "cs16_task.yaml"

# duration ("0h" for spot deals; "4h" (for example) for forward deals)
duration: 24h

# optional, sets counterparty for orders to take, must be HEX(40) string. Will be removed automatically in case of error/not set
counterparty: ""
# Identity level of the counterparty. Can be "anonymous", "registered", "identified" and "professional".
identity: "anonymous"

# MB, integers only
ramsize: 512
# GB, integers only
storagesize: 1
#number of cores, integers only
cpucores: 1
sysbenchsingle: 500
sysbenchmulti: 1000
# Mbits, integers only
netdownload: 10
# Mbits, integers only
netupload: 10

#network settings
# Indicates whether overlay networking is required, boolean only
overlay: false
# Indicates whether inbound connections are required and public IP should be present on worker, boolean only
incoming: true

#gpu config (optional). If not required, set gpucount: "0"
#number of units
gpucount: 0
# GPU unit RAM,MB, integers only
gpumem: 0
# MH/s, integers only
ethhashrate: 0

Task config file

Location ./conf/<any_name>.yaml

This is a standard task definition for a task you want to run in SONM. It describes Docker image name and launch properties for a Docker container.

You may use any platform features in this definition (environment variables, port exposing, SSH key to getting access to your container running in SONM, volumes and overlay settings, etc.). See task definition on our GitHub.

Also, you may use node tag in this task definition. Node tag is your task tag with node index (for example 'Claymore_15'). You may see node tag on HTML stats page when the bot is running.

Example ('cs16_task.yaml' mentioned in task cluster config above):

container:
  image: cs16ds/server:latest
  ssh_key: "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAA--censored--"
  env:
    START_MAP: de_aztec
    SERVER_NAME: "CS 1.6 on SONM"
    SERVER_PASSWORD: sonmsonm
    ADMIN_STEAM: "STEAM_0:--censored--"
  restart_policy:
    name: always
  expose:
    - 26900:26900/udp
    - 27020:27020/udp
    - 27015:27015/udp
    - 27015:27015

Running the bot

  • ./taskman.py - if you want to run the bot in user session;
  • nohup ./taskman.py & - if you want to nrun the bot in background

Bot's log location: ./out/logs/monitor.log

Log rotation - once per day, at 00:00, previous logs are available at the same folder, filename contains a timestamp.

Log output example:

$ ./taskman.py
Press Ctrl+C to interrupt script
2018-11-09 17:11:58,519 - INFO - Sonm api instance created:
    Eth key location: .../sonm-io/core/target/key_my/key.json
    Eth address: --censored--
    Sonm node endpoint: http://127.0.0.1:15031
    Default timeout: 60 sec
2018-11-09 17:12:01,632 - INFO - Creating task file for Node TEST_2
2018-11-09 17:12:01,650 - INFO - Found deal, id 47182 (Node TEST_2)
2018-11-09 17:12:03,950 - INFO - Creating task file for Node CS16_1
2018-11-09 17:12:03,982 - INFO - Found deal, id 47181 (Node CS16_1)
2018-11-09 17:12:04,192 - INFO - Creating task file for Node TEST_1
2018-11-09 17:12:04,261 - INFO - Starting HTTP server...
2018-11-09 17:12:04,263 - INFO - Starting task on node CS16_1 ...
2018-11-09 17:12:04,266 - INFO - Agent started on port: 8081
 * Running on http://0.0.0.0:8081/ (Press CTRL+C to quit)
2018-11-09 17:12:05,270 - INFO - Creating order file for Node TEST_1
2018-11-09 17:12:05,270 - INFO - Predicted price for Node TEST_1 is 0.0322 USD/h, with coefficient 0.0355 USD/h, order price is 0.0200 USD/h
2018-11-09 17:12:05,274 - INFO - Create order for Node TEST_1
2018-11-09 17:12:06,272 - INFO - Starting task on node TEST_2 ...
2018-11-09 17:12:11,680 - INFO - Task (Node TEST_2) started: deal 47182 with task_id 255dc193-63cf-445a-8119-15204b296b1f
2018-11-09 17:12:27,191 - INFO - Order for Node TEST_1 is 686373

When bot is running, you may access HTML page with its stats on http://localhost:8081

You may see your current token balance, status for each node, task uptime (in seconds), and other stats.

If the deal has closed before the task has successfully completed, the bot will create a new order and wait for a new deal to run the task.

Changing the configs

You may change the task list (remove or add tasks), reduce or increase the cluster of nodes for each task, and you don't need to restart bot to apply these changes. Bot reloads configs once per minute and implements changes.

If you change hardware requirements (CPU, GPU, RAM, storage, network), changes will apply when a new order has to be created.

You may use any SONM CLI commands as well.

For example, if you change hardware requirements (CPU, GPU, RAM, storage, network) for active orders (when you don't have the deal for the order yet), you may run command sonmcli order purge. This will cancel all active orders, and the bot will recreate orders with new hardware requirements.

Stopping the bot

  • If all tasks has finished successfully, the bot will stop automatically.
  • If you run the bot in user session, you may press Crtl-C to stop it.
  • If you run the bot in background: kill -15 $(ps ax | grep "taskman.py" | grep -v "grep" | awk '{print $1}')

IMPORTANT NOTE:

Stopping the bot does not affect active orders or deals. Be carefull ! Check active deals and orders after stopping the bot.

You may check:

  • current active deals with sonmcli deal list command (and sonmcli deal purge to close all deals),
  • active orders with sonmcli order list command (and sonmcli order purge to cancel all active orders).

See SONM CLI Guide for additional info.