Downloading logs

When you just want a text file

Web-based systems for browsing and searching logs are great, but sometimes you just want a straight forward .txt file that you can open in your favorite editor… or sharpen your awk sed and grep skills on.

So we provide an API for reading/downloading your node logs (Geth, Constellation, etc.) that can be used to stream the original content down into a local file.

Take a look at the API 101 tutorial if you are new to our REST API.

The /logs API

By default the logs API will query backwards from the end of the log, so if you want the last 50 lines of the geth log:

/api/v1/consortia/<$CONSORTIUM>/environments/<$ENVIRONMENT>/nodes/<$NODE>/logs/geth?maxlines=50

This will return a JSON array, so if you’re using curl on the command line you can use the handy jq tool to process the output:

curl ... | jq -r '.[]'

If you want the beginning of the file, then use frompos:

/api/v1/consortia/<$CONSORTIUM>/environments/<$ENVIRONMENT>/nodes/<$NODE>/logs/geth?maxlines=50?frompos=0

You can increase the number of lines you return with maxlines, but eventually you will request more data than it makes sense to query on a single API call.

So each request will return either x-kaleido-startpos (for queries from the end of the file) or x-kaleido-endpos (for queries from the start of the start).

A little tool to make life easy

So you don’t need to do lots of scripting, here is a handy python script to output the logs from a node.

On Mac install Python with brew install python3 On Ubuntu/Debian Linux install Python with apt-get install python3

Run it like so to interactively choose the node you want to get logs from:

# first export the variables; be sure to replace the placeholder value with
# your API key

export APIURL="https://console.kaleido.io/api/v1"
export APIKEY="XXXXciyyv77k-DdYYpRPBEa/aeONTn3najyMpoxncqfFSwuYF+nDquWc="

# now kick off the script

python logviewer.py

You will iteratively pass values to the Consortium, Environment, Member, Node and Log Type fields in order to generate the corresponding values.

Example output:

0) api101 [yu6ugcni]
Consortium: 0
Using yu6ugcni
0) Auto-generated Environment [mflg1hd7]
Environment: 0
Using mflg1hd7
0) Default Organization [ddnye6il]
1) Jenkins [ctt686fq]
2) Concourse [bfnm0cby]
Member: 1
Using ctt686fq
0) JenkinsNode1 [yjt76yyd]
Node: 0
Using yjt76yyd
0) geth
1) constellation
Log type: 0
DEBUG[03-18|17:32:38] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
DEBUG[03-18|17:32:58] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
...

Once you’ve run it interactively to see the IDs, you can supply these as arguments:

python logviewer.py -c yu6ugcni -e mflg1hd7 -m ctt686fq -n yjt76yyd -t geth -l 50

If you want to download the whole of a file, try:

python logviewer.py -c yu6ugcni -e mflg1hd7 -m ctt686fq -n yjt76yyd -t constellation -l 0 > /tmp/constellation.log

The script itself

#!/usr/local/bin/python3

import urllib.request, json, os, re, argparse

log_types = ['geth','constellation']
parser = argparse.ArgumentParser(description='Get logs.')
parser.add_argument('--apikey', '-k', type=str,
                    required=(not 'APIKEY' in os.environ),
                    help='the API Key (or APIKEY env var)')
parser.add_argument('--apiurl', '-u', type=str,
                    required=(not 'APIURL' in os.environ),
                    help='the API URL (or APIURL env var)')
parser.add_argument('--consortium', '-c', type=str, nargs='?',
                    help='the consortium ID')
parser.add_argument('--environment', '-e', type=str, nargs='?',
                    help='the environment ID')
parser.add_argument('--member', '-m', type=str, nargs='?',
                    help='the membership ID')
parser.add_argument('--node', '-n', type=str, nargs='?',
                    help='the node ID')
parser.add_argument('--type', '-t', type=str, nargs='?',
                    help='the log type', choices=log_types)
parser.add_argument('--lines', '-l', type=str, nargs='?',
                    help='the number of lines, or 0 for whole file',
                    default=50)
args = parser.parse_args()

if (args.apikey is None):
  args.apikey = os.environ['APIKEY']
if (args.apiurl is None):
  args.apiurl = os.environ['APIURL']

# Get the auth token
url = '{0}/authtoken'.format(args.apiurl)
req = urllib.request.Request(url, \
  json.dumps({ 'apikey': args.apikey }).encode(), \
  {'Content-Type': 'application/json'})
res = urllib.request.urlopen(req)
token = json.load(res)[u'token']
res.close()
headers =  {'Content-Type': 'application/json', 'Authorization': 'Bearer {0}'.format(token) }

def pick_from_list(arr, descriptions, prompt):
  idx = ''
  for (i, desc) in enumerate(descriptions):
    print('{0}) {1}'.format(i, desc))
  while not (re.match("^\d+$", idx) and int(idx) < len(arr)):
    idx = input('{0}: '.format(prompt))
  return arr[int(idx)]

def choose_from_api_json(path, prompt, name_prop=u'name', id_prop=u'_id'):
  url = '{0}/{1}'.format(args.apiurl, path)
  res = urllib.request.urlopen(urllib.request.Request(url, headers=headers))
  json_arr = json.load(res)
  res.close()
  options = []
  for json_entry in json_arr:
    options.append('{0} [{1}]'.format(json_entry[name_prop], json_entry[id_prop]))
  chosen_id = pick_from_list(json_arr, options, prompt)[u'_id']
  print('Using {0}'.format(chosen_id))
  return chosen_id

if (args.consortium is None):
  args.consortium = choose_from_api_json('/consortia', 'Consorium')
if (args.environment is None):
  args.environment = choose_from_api_json('/consortia/{0}/environments'\
    .format(args.consortium), 'Environment')
if (args.member is None):
  args.member = choose_from_api_json('/consortia/{0}/memberships'\
    .format(args.consortium), 'Member', name_prop=u'org_name')
if (args.node is None):
  args.node = choose_from_api_json('/consortia/{0}/environments/{1}/nodes?membership_id={2}'\
    .format(args.consortium, args.environment, args.member), 'Node')
if (args.type is None):
  args.type = pick_from_list(log_types, log_types, 'Log type')

if (args.lines == '0'):
  morelines = True
  frompos = 0
  while (morelines):
    url = '{0}/consortia/{1}/environments/{2}/nodes/{3}/logs/{4}?frompos={5}&maxlines=50'\
      .format(args.apiurl, args.consortium, args.environment, args.node, args.type, frompos)
    res = urllib.request.urlopen(urllib.request.Request(url, headers=headers))
    jres = json.load(res)
    res.close()
    for line in jres:
      print(line)
    frompos = res.info().getheader('x-kaleido-endpos')
    morelines = (len(jres) > 0)
else:
  url = '{0}/consortia/{1}/environments/{2}/nodes/{3}/logs/{4}?maxlines={5}'\
    .format(args.apiurl, args.consortium, args.environment, args.node, args.type, args.lines)
  res = urllib.request.urlopen(urllib.request.Request(url, headers=headers))
  jres = json.load(res)
  res.close()
  for line in jres:
    print(line)