Creating large-scale high-resolution Maps from various Sources

Everyone knows services such as Google Earth or Microsoft Maps. They are great for exploring but they are not so great when it comes to creating large-scale high-resolution images, e.g. for a wall poster or alike. Also, they have their shortcomings when it is about comparing maps or images from different sources to “pick the best”.

So I went out searching… and here is a little process to share. The task is: to create a single-image map of Mallorca with a high-level map resolution. So let’s see…

The software I am using for this is called SAS.Planet – and it seems to be a Russian-built piece of software (so my first installation came up all Cyrillic…). I ended up downloading a copy from “somewhere” and then used the internal Help -> Check for Updates feature to grab the latest release.

So this is all of Mallorca (using a Google Maps layer) but not the level of detail I want… which is more like this:

So in general, I have a choice of seeing “all of it at lower level” or “seeing more detail but lesser area”. Let’s change that: zoom back out until you see the whole of the island. Then use the Rectangular Selection tool to draw a map around it.

A window will open that allows for a couple of options. In general, I have had the best experience with downloading the required tiles first, then stitching them together. So on the Download tab, all I do is to download all tiles of Zoom Level 15 and click Start.

If you want to monitor the download process, you may switch on Tile Shading using View – Cached Tiles Maps and select the level you are downloading – z15.

This will display each tile downloaded for that Zoom Level to appear with a gray shading. In addition, you can monitor the progress in the Download Window. Now, you need some patience and wait…some tile servers are faster, some are slower.

When all tiles have been downloaded, go back to your current selection via Operations -> Selection Manager -> Last Selection and switch to the Stitch tab.

Enter a file name and location in Save to and select the appropriate Zoom Level (15 in our example). Then click Start.

The image that is now created is roughly 10 MB in size – the dimensions are 15000 x 11000 px – so pretty large.

Of course, one cannot see much at this magnification but zooming in anywhere onto this picture shows the true level of detail:

Another nice feature: if I do not like the map that has been produced, SAS.Planet allows me to switch to a different tile server but maintain the current selections. So let’s use Google’s Landscape Tiles instead to see if we can get a better result on the mountain region of the island.

You can see: the selection box is still present so I go back to the Selection Manager and download the tiles for Level 15 of Google’s Landscape Layer – same procedure as above.

When I go back to create the stitched image from the downloaded tiles, the result again is a file of 19 MB, same dimensions as the first one.

The level of detail, again, will only reveal itself when zooming in but the overall representation is more what I think I would like to have on the wall… the hills are showing as a well defined relief…

There are plenty of other layers available in SAS.Planet – using the OpenTopo Tile Server provides yet a different view for the map:

And of course – if I do not like a map-type of poster on my wall, I can always switch to the various satellite image providers – mainly Google Earth and Microsoft Bing. Here is a comparison between the image quality of the two – Microsoft on the left, Google on the right.

It comes down to a matter of taste at this level – on a higher level of detail, the Microsoft tiles are more crisp but the Google tiles are more up-to-date.

 

Posted in Allgemein | Leave a comment

FHEM – Temperature Sensors in multiple locations

One of the things that I want to use FHEM for is monitoring the temperature and humidity on a variety of locations (different rooms, patio, etc.)

I am using a set of TX 35 DHT sensors, the connection to FHEM is implemented using a JeeLink v3c USB device. I will describe that in a different post, also how you would pair the sensors with FHEM – for now, let’s just pretend that was done.

I use to define my FHEM Objects “manually” in the fhem.cfg file – so here is the definition for the TX 35 DHT on the patio.

Image 01 - The Definition for the Patio SensorAs you can see, the sensor itself is a LaCrosse sensor, connected to my JeeLink device (called myJeeLink). The sensor is called pt.Temp – the prefix pt standing for patio. If you do not only want to display the current values but also would like to gather historic data (e.g. for graphs), you will have to log the readings somehow.

The easiest way is shown above – using a FileLog device. However, if you work with multiple sensors and record the data over a long time (which is the intention) than working with filelogs is not really the brightest thing to do: they are slow and they will use up your diskspace…quickly!

So let’s try and get this right stright away: instead of logging to a file, let’s log to a database. Sounds complicated? Maybe – let’s find out.

Connecting FHEM to a PostGreSQL Database

The connection requires two steps – the first one is to update the db.conf file to match your database definitions. A sample file for each database supported can be found in the /opt/fhem/contrib/dblog directory of the FHEM installation. For PostGreSQL, the following is needed:

%dbconfig= (
        connection => "Pg:database=fhem;host=localhost",
        user => "fhemuser",
        password => "fhempassword"
);

And of course, you will have to update this with our own database name, host, user, and password information!

And there is a really important thing to do – probably only if your PostgreSQL Server is on a remote machine but hey, it would not hurt either way: you need to install an additional Perl library:

sudo apt-get install libdbd-pg-perl

Otherwise, you will have your system continously “waiting for connection”…

Now, once you have the db.conf file set up and restarted your FHEM Server (shutdown restart) and you have created the database schema in the logging database accordingly (see /contrib/dblog/db_create*.sql scripts in the FHEM home directory!) you can create a global logging device:

define logdb DbLog ./db.conf .*:.*

This defined a Device logdb of type DbLog with the configuration file db.conf in the FHEM home directory and it logs… everything.

Image 02 - The PostgreSQL Database connectedMind the state property: it needs to say connected… using any sort of database maintenance tool, you should not be able to monitor data flowing in:

Image 03 - Data written to Table CURRENTNow that we have connected the PostgreSQL Database sucessfully, it is time to return to the creation of the individual sensors.

Pairing new Tx 35 DHT Sensors

The TX 35 DHT sensor is actually a LaCrosse Sensor. Pairing – at least as far as I know so far – can only be done in one way:

  1. Remove the batteries from the TX 35 DHT if there are any installed.
  2. Go to FHEM and select the JeeLink Device (from the Everything tab)
  3. It has a set command for LaCrossePairForSec – enter 60 for 60 seconds and click the set button.
  4. Insert the batteries into the TX 35 DHT and wait for the temperature to show again…

Once you refresh the FHEM Interface, you will see a LaCrosse tab on the left. Select the Sensor and note the DEF value – that’s your Device ID!

Image 04 - Freshly paired LaCrosse SensorIf you want to edit your fhem.conf file manually (like I do), this is the ID to work with for that particular sensor!

Image 05 - The LaCrosse in the StudyI am actually repeating this for a total of seven sensors – one of them, the one on the patio, is an older model that does not do humidity. But the TX 35 DHT are marked as indoor use so I will keep the outside one for the moment.

Image 06 - All LaCrosse Sensors installedSo above is the list of all sensors in FHEM with their respective readings… and they are busily dumping their values into the PostgreSQL Database 🙂 – so far, so good.

For now, one last thing remains to be done: the constant logging of each and every value reported (as an event) quickly fills the database with garbage. I am only interested in those events that indicate a change to the previous status. And luckily, FHEM supports this out-of-the-box: by setting the object’s event-on-change-reading attribute to .* we will only see those value changes come up.. the “clutter” in the Event Monitor dies down dramatically… and that’s it for the moment, let’s gather some data in the logs.

Posted in Allgemein | Leave a comment

FHEM – Working with Objects

Now that we have installed the FHEM Server onto a Raspberry Pi, it is about time to do something with it. We have seen the result of the “naked install” at the end of the last post:

Picture 04 - FHEM Web PageThe web page itself is available but of course, nothing is configured into the system yet. Before we start working with FHEM, here is a brief discussion about how FHEM is organized.

It’s all about one file

In FHEM, it’s all about one file: fhem.cfg. This is the central configuration file which contains all – all! – definitions for the system. You can edit it directly or you can interact with the system using the Command Line and the configuration file will be updated in the background.

After the initial installation, the file contains some configuration already – how you edit it, is up to you. I have chosen to use WinSCP and got the file local onto my PC, then used Notepad++ to work with it.

Picture 03 - The Default Config FileEverything is an “Object”

In FHEM, everything you may want to work with, is considered “an Object”. It does not matter what exactly it is – a device, a log file, a room, a setting – everything is an object. With the default installation done, you can already see some objects in FHEM’s inventory – just select Everything on the left to see all objects that are currently defined – and their status.

Picture 04 - Default Object DefinitionsComparing the list of the fhem.cfg file, you can easily see that the definitions correspond. In other words: everything that is defined in the configuration file is accessible through the web interface and everything you configure through the web interface is saved to the configuration file. So how you work, is really entirely up to you – but editing the configuration file results in a “cleaner” an better structured file than having the system just dump the information there…

Our first “own” Object

To demonstrate how object definition works in FHEM, I have decided to use a very basic example that does not even need hardware. FHEM knows something called a “Dummy Object”. Perfect for getting to learn the system without having to waste money on actual hardware devices. Try this in the Command Line:

define mySwitch dummy

Through the Command Line, it results in an immediate creation of the Dummy Object – but only “in memory” – mind the “Save config” entry on top of the menu bar!

Picture 05 - A Dummy SwitchThis updated the config file – actually, the file received a couple more lines as it itself has been updated from an older version but the one line we are interested in is at the bottom:

Picture 06 - The updated Config FileYou can see that the object has a Name (“mySwitch”) and a Type (“dummy”). Those are the two parameters we have provided in the define command above. It also has a couple of other elements – sucha s a Number (or “Nr”) and a State (currently “???”) but these we did not define – they come from the system and are defaulting to initial values or are auto-calculated.

At the bottom, you can see that an Object has Attributes – in the screenshot above, one is defaulted (“room”) but others are possible. An Attribute is used to provide additional information to an Object. It typically describes the object in more detail.

Let’s take the Room attribute for the moment. Given that we may have a Home Automation System in mind that works in more than one room, it is logical to provide any device we might implement with a Room attribute to structure our devices better. Otherwise, we would have to remember which switch we have in which room…

Picture 07 - Configuration File UpdateThis time, I have chosen to update the fhem.cfg file directly and bring it back to my Raspberry Pi. Just copying the file back, however, does not help – FHEM does not “read” the updated configuration – you need to restart the FHEM Server.

shutdown restart

After the service has restarted, you can see the result of the configuration update right away:

Picture 08 - The Updated ConfigExamining the Dummy Switch more closely, you can now see that the Room attribute has been added – and that FHEM is using it in the menu to separate the devices by rooms.

Picture 09 - Device SpecificsNow, what do we do with that dummy switch? Usually, you would use a switch to… switch on a light? OK – but we need a “light” first:

Picture 10 - Lamp Device and Switch CommandAs you can see, the new dummy “myLight” is defined (Line 44) and placed in the Living Room (Line 45). At the same time, I have extended my existing switch “MySwitch” to include a command (can be “on” or “off”) to actually “switch” it (Line 41).

Picture 11 - The Switch is onIn the Web Interface, FHEM now allows me to switch “mySwitch” on or off (it is now “on”) and the system visualizes the state correctly. Last thing to do is to connect the switch to the light…

Another type of Object – an “Event”

Chaning the state of the switch is actually an “Event” – something that “happens”. And objects are defined in the already familiar way:

Picture 12 - The Notify EventThis is called a “Notify” Object because the eventy of the switch notifies the light to so something. Or in other words: if mySwitch enters the state of “on”, change the state of myLight to “on”. Needless to say that this would be a very unidirectional handling – there is currently no way to switch the light off again 😉

Picture 13 - The Notify ObjectObviously, I could just repeat the step and create another Notify Object that is triggered when mySwitch if entering the state of “off” – but that is tedious. Instead, we are “tricking” our way:

Picture 14 - The Notify on all EventsInstead of “hardcoding” the values of the event triggered by mySwitch (“on”, “off”), we are reacting to simply every event – every time mySwitch changes state (and invokes the Notify Object) we are doing something to myLight.

The trick is using $EVENT (mind the uppercase style!) – that is a system variable that takes the value of the initiating object’s event (e.g. “on”) and by setting it on myLight, we can get around with one line of code. But that only works if the values are the same on both sides!

Not bad for a first attempt – we now can edit the configuration file, create devices, place them in rooms and have them interact based upon events. Next up is some cool stuff I always wanted to do… building my own weather station & indoor clima monitor.

Posted in Home Automation, Raspberry Pi | Leave a comment

FHEM – Updating the System

Like any other piece of software, FHEM needs updates. New devices may be supported or bugs may be fixed or new functionality may be added. So it is wise to keep the system up to date but at the same time, do so with care and the ability to roll back in case something goes wrong or requires more work to be done than you might have at the moment.

Rule #1: Always have a Backup

The first rule before any update is to have a backup of the system. For FHEM on a Raspberry, there are two ways of doing that: the probably most complete one is to backup the entire SD Card using a tool such as the USB Image Tool. That simply backs up the entire system and you can restore the environment as a whole should something go wrong.

Of course, if you have other environments or need to transfer the configuration between say a “Test System” and a “Live System”, you might want to choose a different approach.

FHEM has an integrated backup method. To use it, simply type the following in the FHEM Command line:

backup

Then hit Enter.

Picture 01 - Backup the FHEM EnvironmentTypically, the file resides on the Raspberry Pi’s File System in the location /opt/fhem/Backup.

Like the backup, the update is invoked via FHEM’s Command Line – guess what command?

update

The update loads a complete file list first so no worries if stuff starts to scoll through. Theoretically, the last message should inform you that the update has been performed, in my case, you really had to read the output as this was not the last line.

Picture 02 - Backup doneNow, it is time to restart FHEM – either by rebooting the Raspberry Pi or by issuing the following FHEM Command:

shutdown restart

And that should be it…

Posted in Allgemein | Leave a comment

Getting started with Home Automation

Home Automation is getting more and more attention – and devices become more and more available for prices that are affordable. So time for me to start looning into it. In combination with a Raspberry Pi, this is a neat way of adding some geek stuff to your home that looks impressive and might even help you save some money and/or increase your comfort level.

Using a Raspberry Pi for Home Automation

Considering “things to do” with a Raspberry Pi almost automatically sparks the idea of using it for some kind of home automation. It is small, it is cheap, it is extensible, it has a huge community and it can do the job.

I have an “old” Raspberry (a Raspberry PI Modell B Rev 2.0, 512MB model) sitting around that I can use as a test environment – that is trying stuff without messing up my actual “live” Home Automation Server. If things work, I can then transfer them to the live environment.

Now, before anything can be done, a clean install of the Raspberry Pi is required – and I have described this in a post way back when – so check it out here. At the time of writing this post, the current version is Raspbian Jessie with a Kernel Version 4.4, released on May 27, 2016. Of course, I am also applying the regular update and upgrade to be on the lastest versions of everything:

sudo apt-get update
sudo apt-get upgrade

Of course, there are a couple of other things you would want to consider for a “real” Home Automation Server but I am going to skip them for my test environment:

  • Provisioning of a fixed IP Address,
  • Changing the default user and password,
  • securing the Raspberry Pi against unauthorized access

And maybe some more that I forgot to mention… but since this is about playing with a Home Automation Server, I guess I just leave it at that and install one.

FHEM as Home Automation Server

Let get around this quickly: I have just chosen FHEM – there may be other solutions, there may be better solutions – this is the one I am going to play with.

like with any other solution, it needs to be installed first. But first, the Raspberry itself needs two more little bits sorted out.

First, we need to worry about the time and timezone. Let’s see where are are at with the Raspberry Pi – the command simply is

date

And the result is equally short:

Picture 01 - Date and Time on Raspberry PiTo reconfigure the time zone, you got multiple possibilities, one is the following:

sudo dpkg-reconfigure tzdata

You will receive a graphical interface to select the region first, then the city. Upon confirmation, you will see the updated configuration in the shell.

Picure 02 - Updated Timezone InformationFinally, we need some additional software packages on the Raspberry Pi – mainly Perl libraries:

sudo apt-get -f install && sudo apt-get install perl libdevice-serialport-perl libio-socket-ssl-perl libwww-perl libxml-simple-perl -y

With that done, we are ready to install FHEM – which does not come via the usual package manager but via a direct download from the fhem.de website:

sudo wget http://fhem.de/fhem-5.6.deb && sudo dpkg -i fhem-5.6.deb

This might take a moment – at least my Raspberry Pi is not the fastes anymore.

Picture 03 - FHEM installedThis is it – the software a such is now installed. A little bit on permissions needs to be adapted – change to the /opt director and do the following:

cd /opt
sudo chmod -R a+w fhem && sudo usermod -a -G tty pi && sudo usermod -a -G tty fhem

Now reboot the Raspberry Pi using the sudo reboot now command and then wait for it to come back. Then open any Internet Browser on your local machine and go to the Raspberry Pi’s FHEM Web Page:

Picture 04 - FHEM Web PageNote the port: it is 8083! If you see the above page, you should be all set to continue configuring the FHEM Server.

Posted in Home Automation, Raspberry Pi | Leave a comment