Categories
Geen categorie

Update

En toen was het ineens februari 2021. De tijd vliegt. Te snel om een persoonlijke blog bij te houden met periodieke mentale boeren. Ergo, ik schrijf nog steeds. Veel en vaak. Boeken en software. En doe nog veel meer dingen. Druk baasje van nature maar zonder een testosteron-kanon te zijn (denk ik dan). Anyway, er komt vanzelf wel weer een volgende post. Of niet. En anders is er altijd www.i2rs.nl nog, of Twitter. Ciao!

Categories
Geen categorie

Een visie op de toekomst van overheidssoftware

Zie bijgaande pdf (open_doorkijk_visiedocument_v1.1) voor mijn visie op de ontwikkeling van algemeen nut beogende software.

Categories
Geen categorie

My talk for Tecnoteca’s CMDBuild day 2016

You can see it here: https://www.youtube.com/watch?v=vY_LZQ_8udU

Categories
Geen categorie

Communicating with REST to CMDBuild

Intro

Hi. Today I like to talk about a nice open source tool called CMDBuild. CMDBuild is an excellent open source project that you can use at the heart of your IT structure. It offers you a web-interface that allows you to define objects and their relations. Once done you can use CMDbuild to enter information for these objects.

Example: you define an object “server” and and object “IP-address”. You define a 1-to-n relation from “server” to “IP-address”. Once done you can add servers to a list of server-objects and add IP-objects to server objects.

This is all fine and dandy but why would you like to do this? Well, after setting up the way you organize your infrastructure (eg: you could define on top of the server-objects “application”-objects to which servers belong, or locations where servers are located, etc.) you can use this information to hook up other systems.

As an example: knowing the IP-address of a server and it’s application you can instruct the monitoring application Zabbix to apply a certain template to automatically setup monitoring for that server.

Now the next challenge is how to get information out of CMDBuild. For this the application offers a SOAP and a REST interface.

For IT-newbies: This is the preferred way to communicate with an application. Yes, you can write a slick SQL Select statement to get the info out of the database but who guarantees you that after the next update the database model isn’t seriously altered?

There is only one challenge: there is a lot of documentation for the application (user manual, administration manual, web services manual) and once you get it working it’s all downhill from there. The catch is to get a small example running using only the current documentation.

Side note: allthough I have no ties to the CMDBuild organization I urge all readers who are using CMDBuild professionally to also buy themselves a support contract with the CMDBuild programmers firm Tecnoteca . Why? Well, if you place such a thing in the core of your infrastructure would you accept the risk that they go belly up because everybody uses the software thinking it’s a free ride? No. For a small amount of money you buy yourself peace of mind that you’ve done the right thing. And the fact that it provides you with excellent support is an added bonus.

Techno-speak

After a lot of trial and error on my part I got an example curl script from Tecnoteca that succesfully talks to the REST interface. It looks like this:

curl -v -X POST 'http://yourcmdbuildurl.nl/cmdbuild/services/rest/v1/sessions' -H "Content-Type: application/json" --data-binary '{"username": "admin", "password": "yourpassword"}' | json_pp

After checking that this script works and it gives you some meaningfull json output I got back to work.

When using the REST interface of CMDBuild you have first have to authenticate to it and get a token that identifies your session in subsequent REST calls. So the authentication token is important to have. Without it, you’re stuck.

First challenge was to make the same functionality work in Python. In the end this is what I wrote:

#!/usr/bin/python

import requests, json, pprint
import os,sys
from pprint import pprint
from prettytable import PrettyTable
from requests.auth import HTTPDigestAuth
import logging


## The next lines enable debugging at httplib level (requests->urllib3->http.client)
## You will see the REQUEST, including HEADERS and DATA, and RESPONSE with HEADERS but without DATA.
## The only thing missing will be the response.body which is not logged.
#try:
#    import http.client as http_client
#except ImportError:
#    # Python 2
#    import httplib as http_client
#http_client.HTTPConnection.debuglevel = 1
#
## You must initialize logging, otherwise you'll not see debug output.
#logging.basicConfig()
#logging.getLogger().setLevel(logging.DEBUG)
#requests_log = logging.getLogger("requests.packages.urllib3")
#requests_log.setLevel(logging.DEBUG)
#requests_log.propagate = True

print "***************************************************************"
print "*** Login and get authentication token "
print "***************************************************************"

cmdbuild_url = "http://server:8080/cmdbuild/services/rest/v2/sessions/"
data = {'username': 'soap', 'password': 'secret'}
headers = {'Content-type': 'application/json', 'Accept': '*/*'}
r = requests.post(cmdbuild_url, data=json.dumps(data), headers=headers)

print r.json()
r1=r.json()
sessionid=r1["data"]["_id"]
print "***************************************************************"
print " Authentication token is : " + sessionid
print "***************************************************************"


print "***************************************************************"
print "*** Session info"
print "***************************************************************"
#curl -v --trace-ascii - -X GET "http://server:8080/cmdbuild/services/rest/v1/sessions/${id}" -H "Content-Type: application/json"

cmdbuild_url = "http://server:8080/cmdbuild/services/rest/v2/sessions/"+sessionid
headers = {'Content-type': 'application/json', 'Accept': '*/*', 'CMDBuild-Authorization': sessionid }
r = requests.get(cmdbuild_url, data=json.dumps(data), headers=headers)
print r.json()
pprint(r.json())

There is a lot more to say about this, but basically, when you made this work you can read the manual, get the WSDL, find the REST endpoints you can call and download the WADL that also lists the available endpoints for your version of the software.

Advanced stuff

Now that you’ve got your hands dirty on some easy REST stuff let’s look at something more complex. The CMDBuild system is filled with objects called “classes”. Let’s first get a list of classes:

print "***************************************************************"
print "*** Classes "
print "***************************************************************"
#curl -v --trace-ascii - -X GET 'http://server:8080/cmdbuild/services/rest/v1/classes' -H "Content-Type: application
/json" -H "CMDBuild-Authorization: ${id}"

cmdbuild_url = "http://server:8080/cmdbuild/services/rest/v2/classes"
headers = {'Content-type': 'application/json', 'Accept': '*/*', 'CMDBuild-Authorization': sessionid }
r = requests.get(cmdbuild_url, data=json.dumps(data), headers=headers)
print "There are " + str(r.json()["meta"]["total"]) + " results"
#print r.json()
pprint(r.json())

Now that we have our list of classes in the system we can loop through them and get the individual details of every class:

for value in r.json()["data"]:
 print "\nTrying to get cards for : " + value["_id"] 
 id=value["_id"]
 #for id in value["_id"]:
 #pprint(id)
 print "Getting id: " + id

 print "***************************************************************"
 print "*** Class '"+id+"'"
 print "***************************************************************"
 #Asset 
 cmdbuild_url = "http://server:8080/cmdbuild/services/rest/v2/classes/" + id 
 headers = {'Content-type': 'application/json', 'Accept': '*/*', 'CMDBuild-Authorization': sessionid }
 r = requests.get(cmdbuild_url, data=json.dumps(data), headers=headers)
 #print "There are " + str(r.json()["meta"]["total"]) + " results for class " + id + "?"
 #print r.json()
 pprint(r.json())

 print "***************************************************************"
 print "*** Class '"+id+"' attributes"
 print "***************************************************************"
 # GET .../classes/Asset/attributes 
 cmdbuild_url = "http://server:8080/cmdbuild/services/rest/v2/classes/" + id + "/attributes"
 headers = {'Content-type': 'application/json', 'Accept': '*/*', 'CMDBuild-Authorization': sessionid }
 r = requests.get(cmdbuild_url, data=json.dumps(data), headers=headers)
 print "There are " + str(r.json()["meta"]["total"]) + " results for class " + id + " attributes "
 #print r.json()
 pprint(r.json())

 print "***************************************************************"
 print "*** Class '"+id+"' cards"
 print "***************************************************************"
 #GET .../classes/Asset/cards
 cmdbuild_url = "http://server:8080/cmdbuild/services/rest/v2/classes/" + id + "/cards"
 headers = {'Content-type': 'application/json', 'Accept': '*/*', 'CMDBuild-Authorization': sessionid }
 r = requests.get(cmdbuild_url, data=json.dumps(data), headers=headers)
 print "There are " + str(r.json()["meta"]["total"]) + " results for class " + id + " cards "
 #print r.json()
 pprint(r.json())

 

Happy programming!

Update 1:CMDBuild builders Tecnoteca tweet about this posting:

Update 2: Follow-up testimonial publised on the website of the CMDBuild builders Tecnoteca: article

Update 3: I got permission from Deltares to publish the sourcecode to a small CMDBuild python library module.

Categories
Geen categorie

Upgrading Zabbix server from Ubuntu 12.04 to 14.04 (using a MySQL database)

So, I like Zabbix. Sue me 🙂

But upgrading is a whole different ballgame. Zabbix on Ubuntu 12.04 is version 1.8.1 and Zabbix on Ubuntu 14.04 is version 2.2.2. And Zabbix only does an automatic database upgrade from version 2.0 onwards. So, basically, you are on your own and Google does not seem to help either.

So, let me tell you what I did and maybe it works for you too. If not, I am sorry.

In the end I had to checkout the Zabbix sources to find a database patchfile.

So I did:

svn co svn://svn.zabbix.com/trunk

And I found the following directory:  trunk/upgrades/dbpatches/2.0

And it has a “upgrade” script. Don’t bother, it does not work as is. But there is also a “patch” subdirectory with a shitload of sql scripts. They turn out to be just the ticket. So I listed them al in a file, put a mysql command in from of them and, for safety, a “read” between every line and run the script.

So, my logging output in /var/log/zabbix-server/zabbix_server.log changed from:

6033:20140701:133019.785 WEB monitoring: YES
6033:20140701:133019.785 VMware monitoring: YES
6033:20140701:133019.785 Jabber notifications: YES
6033:20140701:133019.785 Ez Texting notifications: YES
6033:20140701:133019.785 ODBC: YES
6033:20140701:133019.785 SSH2 support: YES
6033:20140701:133019.785 IPv6 support: YES
6033:20140701:133019.785 ******************************
6033:20140701:133019.785 using configuration file: /etc/zabbix/zabbix_server.conf
6033:20140701:133019.789 Cannot upgrade database: the database must correspond to  version 2.0 or later. Exiting ...

to:

9446:20140701:151423.638 Starting Zabbix Server. Zabbix 2.2.2 (revision 42525).
9446:20140701:151423.638 ****** Enabled features ******
9446:20140701:151423.638 SNMP monitoring: YES
9446:20140701:151423.638 IPMI monitoring: YES
9446:20140701:151423.638 WEB monitoring: YES
9446:20140701:151423.638 VMware monitoring: YES
9446:20140701:151423.638 Jabber notifications: YES
9446:20140701:151423.638 Ez Texting notifications: YES
9446:20140701:151423.638 ODBC: YES
9446:20140701:151423.638 SSH2 support: YES
9446:20140701:151423.638 IPv6 support: YES
9446:20140701:151423.638 ******************************
9446:20140701:151423.638 using configuration file: /etc/zabbix/zabbix_server.conf
9446:20140701:151423.797 current database version (mandatory/optional):  02010000/02010000
9446:20140701:151423.797 required mandatory version: 02020000
9446:20140701:151423.797 starting automatic database upgrade
9446:20140701:151423.831 completed 0% of database upgrade
9446:20140701:151423.863 completed 1% of database upgrade
9446:20140701:151423.955 completed 2% of database upgrade
9446:20140701:151424.055 completed 3% of database upgrade
9446:20140701:151424.371 completed 4% of database upgrade
9446:20140701:151424.916 completed 5% of database upgrade
9446:20140701:151425.232 completed 6% of database upgrade
9446:20140701:151425.701 completed 7% of database upgrade

etc.
So, before you begin, please make a backup of your database first!

The script I used in the end was this one:

mysql -h mysqlhost -p -D zabbix -pmysqlpassword < acknowledges.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < actions.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < alerts.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < applications.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < auditlog_details.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < auditlog.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < autoreg_host.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < conditions.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < config.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < dchecks.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < dhosts.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < drules.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < dservices.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < escalations.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < events.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < expressions.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < functions.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < globalmacro.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < globalvars.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < graph_discovery.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < graphs_items.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < graphs.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < graph_theme.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < groups.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < help_items.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < history_log.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < history.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < history_str.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < history_str_sync.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < history_sync.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < history_text.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < history_uint.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < history_uint_sync.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < host_inventory.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < hostmacro.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < hosts_groups.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < hosts_profiles_ext.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < hosts_profiles.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < hosts.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < hosts_templates.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < housekeeper.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < httpstepitem.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < httpstep.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < httptestitem.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < httptest.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < icon_mapping.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < icon_map.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < ids.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < images.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < interface.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < item_discovery.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < items_applications.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < items.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < maintenances_groups.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < maintenances_hosts.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < maintenances.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < maintenances_windows.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < mappings.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < media.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < media_type.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < node_cksum.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < nodes.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < opcommand_grp.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < opcommand_hst.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < opconditions.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < operations.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < opgroup.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < opmediatypes.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < opmessage_grp.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < opmessage.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < opmessage_usr.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < optemplate.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < profiles.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < proxy_autoreg_host.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < proxy_dhistory.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < proxy_history.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < regexps.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < rights.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < screens_items.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < screens.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < scripts.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < service_alarms.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < services_links.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < services.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < services_times.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < sessions.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < slideshows.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < slides.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < sysmap_element_url.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < sysmaps_elements.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < sysmaps_links.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < sysmaps_link_triggers.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < sysmaps.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < sysmap_url.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < timeperiods.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < trends.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < trends_uint.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < trigger_depends.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < trigger_discovery.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < triggers.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < user_history.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < users_groups.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < users.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < usrgrp.sql
read dummy
mysql -h mysqlhost -p -D zabbix -pmysqlpassword < valuemaps.sql

After the update you are (almost) ready to go. When you get the url to /zabbix working in Apache he hits you with a configurator with lot of questions because he wants a /etc/zabbix/zabbix.conf.php file. Simply answer the questions and install the config file and you are good to go.

Enjoy!