FHEM Home Automation – Cleaning up the Server – Part I

My FHEM Server has now been working without any issues for quite some time, definitely more than a year, probably closer to two years. Time to clean up some parts, especially since the latest updates did not install successfully and I had to revert back to the installed version.

Over the last year(s), things “worked” and while they still do, it is time to also come up with a new configuration – things, that I have learnt in the meantime. Hardware that has been bought in the meantime. New ideas that have developed in the meantime. To get all these “into a new system”, I have decided to build a playground on a Virtual Machine. The operating system of choice is Debian 9, because it is so similar to the Raspberry Pi that I eventually want to move the installation to.

So this one should be fast enough for some playing. And maybe I will regret the settings later, the Raspberry Pi is much slower.

I have decided for the “Desktop installation – that is: a graphical desktop environment (which is not the same than the Raspberry Pi will be). And a Terminal window for “command line to the bitter end”…

After the installation has finished, there are some things that should be done – the first one is to update the system to the latest patch level (the two commands can be executed from a Terminal window in superuser mode):

apt-get update
apt-get upgrade
apt-get install net-tools

Next, I need to install VMWare Tools (which, of course, is only necessary if you are running VMWare Player or VMWare Workstation). That, by the way, is the reason why in the command sequence above the net-tools package is installed…

Getting FHEM onto the Box

There are a few things that need to be done now – the first one obviously being the installation of FHEM on the box. This is best performed as “manual installation” – the most current instructions are here: debian.fhem.de

The first three steps of the instruction for the manual installation (ending with the command dpkg -i fhem-5.8.deb are responsible for installing the required libraries, fetching the FHEM Package and installing it. This includes the creation of the fhem system user:

grep fhem / etc/passwd

shows the user’s definition. It also shows that the user’s home directory is /opt/fhem (where the installation resides) and there is no shell assigned – /bin/false – which makes the user unable to login.

Technically, the last statement might not be quite correct: the user can login but it would not get any shell and therefore no command line (plus: no password unless we would set one…)

Now this presents an interesting problem: if you do the following:

cd /opt/fhem
ls -l

to display the files and their permissions in the FHEM installation directory, you will notice that all files are owned by the user fhem and the default group is set to dialout.

Catch-22 with the Configuration Files, Users, and Permissions

To me, that poses a small problem: the configuration files that we will need to create for FHEM. Of course: you can edit the fhem.cfg file straight through the web interface but that is a tedious and rather stupid way of doing it (imho). Just two reasons: there are far better editors than the built-in one (I, for example, prefer Notepad++ on a Windows Box) and it is hard to do version control on your files straight on the Unix File system.

Of course, there are plenty of ways of getting files to and from a Linux system (including direct drag & drop into my virtual machine) but considering a headless (no user-interface) environment on my Raspberry Pi, the most logical way seems to be a FTP Server on the Unix box.

Installing ProFTPD into the environment

An easy-to-install and powerfull FTP Server is ProFTPD. You can get it installed via

apt-get install proftpd-basic

Once installed and started, the default is listening on TCP/IP v6 – which is the first thing I need to change. At the same time, we can ensure that a user that is allowed to log in using FTP does not require a Shell assignment (like the fhem user discussed above) and last but not least we can control the default home directory for the FTP Users.

Most of this already plays also into the Unix Permission Concept so it will be a mixed discussion here.

Fist of all, the additional configuration for ProFTPD is applied using a custom config file located in /etc/proftpd/conf.d – that directory will be maintained even if ProFTPD is updated. Let me call my file custom.conf – it is a simple XML File:

Note: Please understand that this is a most basic security setting and is sufficient for FHEM Boxes that are in a secure environment (e.g. not accessible over the Internet, only your and yourself on the network, etc.)

ProFTPD can do much more in terms of security – secured connections using SSL/TLS Encryption, non-standard FTP Ports, etc. – if you want to find out more, please make sure that you use the search engine of your choice to locate the appropriate information on the Internet. And never use the above configuration in anything else but a test environment!

Once the configuration file is placed, restart the FTP Service

systemctl restart prodtfd.service

You can then check of the service is actually running

service proftpd status

and if it is listening on Port 21, the default FTP Port.

netstat -tlp | grep proftp

With that said, you should be ready to access the server via FTP – at least, after you have added your user of choice to the ftpuser group…

groupadd ftpuser (if it does not exist yet)
usermod --append --groups ftpuser [UserName]
grep ftpuser /etc/group

The last command must return one line with the definition of users in the group.

To be or not to be…

The next question – hotly discussed, I am sure – is the user to work with. Of course, you can use ProFTPD with a more elaborate configuration to allow any user to connect and still set those users’ root directory to /opt/fhem (where you want to copy the configuration files to). You can even go as far to make sure that any file copied is placed with the appropriate file permissions, having fhem as the owner and dialout as the group (which is what the FHEM installer sets by default when you install a fresh copy).

But there are alternatives: you can work directly as the fhem user (which requires you to give the user a password and place him in the ftpuser group).

passwd fhem

I am sure there are plenty of people around that will object to that approach because fhem is considered a service account and should not be used for login…

Alternatively, you can work “as your own user” and adjust the file permissions afterwards (each file you will copy into the directory using FTP will by default have your user as owner (instead of fhem) and your primary group as group (instead of dialout).

To change the owner of all files in /opt/fhem use the following command:

chown -R fhem:dialout /opt/fhem

which you need to execute as superuser. Bottom line: there are so many ways of doing it – and each one has its own rights and wrongs. Pick your own one but make sure you understand the implications with respect to the security of your FHEM Installation!

One more thing…

One more thing about security: by default, the fhem user ends up in the group dialout. It is important that fhem also is member of the group tty which controls access to serial and USB Ports (in case you have a USB Device attached to your FHEM Server…)

usermod --append --groups tty fhem

takes care of that.

Is it working or not?

By now you might have a working FHEM Server – which you can check on the command line via

/etc/init.d/fhem status

and it might be accessible via HTTP under http://[Your server]:8083/fhem

If the server is not working or is not reachable, it is time to take a first look at the logs (well, even if it is working, it might be a good idea…)

The FHEM Logs

The FHEM Logs are located in /opt/fhem/log and depending on your way of access to your server, you can either view them straight on the box (e.g. using the tail command) or access them through the FTP Server and view them in an editor like Notepad++:

This one is a log file of the first successful start – it shows the config file that is read (fhem.cfg – oh wonder!), the status of the outbound ports (8083, 8084, 8085), a security message, the server’s FeatureLevel (5.8) and the fact that “9 entities” have been defined…

If there are errors, this is your first place to go – as we will see in a moment.

Updating FHEM

I want to provoke a situation that has caused me a lot of headache because I did not see the issue – only with the help of the FHEM Forum (suggestion to every FHEM user!) I got it solved (in minutes!).

My FHEM Installation is still the one that was installed with the package above – but FHEM is constantly updated. So in the Web Interface, go to the command line at the top and type the following:

update check

This will (likely) create a long list of files that can be updated – and should be updated (especially since we are just starting with the installation). But keep in mind: an update can cause nasty surprises as we will see…

To update FHEM, issue the command

update

and wait. After a short while, you should see more or less the following:

The important line is “update finished, ‘shutdown restart’ is needed to activate the changes”. So issue the following command in the Web Interface:

shutdown restart

The nasty surprise in this one: the Web Interface ceases to work after the update! No connection is possible – and at the first glance, you have not changed anything! I have to admit: this pissed me off for quite some time.

I understand that if you want to run an environment like FHEM, you need to be familiar with what you are doing (and I appreciate this fact). On the other hand, if a working environment is turned “non-working” with an update that immediately renders the system inaccessible… that is not necessarily the way I would prefer it to be.

But back to the problem (and as I said: the solution was straight forward, thanks to the FHEM Forum!). Check the FHEM Log File again:

There are some significant changes compared to the previous log (and we did not change any of the configuration, just performed an update!)

There now is a PERL Warning that seems to be associated with the USB Checks. If you search for the error message, it becomes obvious that is related to the USB Check defined in the default fhem.cfg file:

define initialUsbCheck notify global:INITIALIZED usb create

It appears that the system’s behavior has been changed but I cannot see any reference to that change in change log /opt/fhem/CHANGED. In any case: you can remove or comment out the respective line and the error should be gone.

That, however, does not bring back the Web Interface. Again, a look at the CHANGED file does not bring up anything obvious. Except for:

 - feature: allowed module added. allowedCommands, basicAuth, password,
 globalpassword attributes moved to this module.

Well, look at the error message in the log again:

Connection refused from the non-local address 192.XXX.XXX.XXX:53262, 
as there is no working allowed instance defined for it

At this point, it becomes a language issue: because of the wording, I interpreted the “allowed” part as part of the sentence – if you know what to look for, it tells you that a connection is refused because there is no defined instance of the allowed device.

If you would have understood, you could have looked it up in the FHEM Reference. And it could have been corrected earlier.

Please note that I have defined a clear text username and password in this example – which obviously is inappropriate for anything but a test environment!

The better way of doing it is to create a base64-encoded password string from the Unix Command Line and use this instead:

echo -n fhemuser:secret | base64

Creates ZmhlbXVzZXI6c2VjcmV0 as password phrase which then can be used in the config file:

So now: copy the fhem.cfg file back to the server (unless you have edited it there directly) and restart FHEM:

/etc/init.d/fhem stop (if it is not stopped anyway)
/etc/init.d/fhem start

And now try the Web Interface again – you should see it working now. With all of that done, we are through with Part I of creating a brand-new FHEM Environment. The initial configuration parts will follow in Part II.

Posted in Home Automation, Raspberry Pi | Leave a comment

Creating large-scale high-resolution Maps from various Sources

Everyone knows services such as Google Earth or Microsoft Maps. They are great for exploring but they are not so great when it comes to creating large-scale high-resolution images, e.g. for a wall poster or alike. Also, they have their shortcomings when it is about comparing maps or images from different sources to “pick the best”.

So I went out searching… and here is a little process to share. The task is: to create a single-image map of Mallorca with a high-level map resolution. So let’s see…

The software I am using for this is called SAS.Planet – and it seems to be a Russian-built piece of software (so my first installation came up all Cyrillic…). I ended up downloading a copy from “somewhere” and then used the internal Help -> Check for Updates feature to grab the latest release.

So this is all of Mallorca (using a Google Maps layer) but not the level of detail I want… which is more like this:

So in general, I have a choice of seeing “all of it at lower level” or “seeing more detail but lesser area”. Let’s change that: zoom back out until you see the whole of the island. Then use the Rectangular Selection tool to draw a map around it.

A window will open that allows for a couple of options. In general, I have had the best experience with downloading the required tiles first, then stitching them together. So on the Download tab, all I do is to download all tiles of Zoom Level 15 and click Start.

If you want to monitor the download process, you may switch on Tile Shading using View – Cached Tiles Maps and select the level you are downloading – z15.

This will display each tile downloaded for that Zoom Level to appear with a gray shading. In addition, you can monitor the progress in the Download Window. Now, you need some patience and wait…some tile servers are faster, some are slower.

When all tiles have been downloaded, go back to your current selection via Operations -> Selection Manager -> Last Selection and switch to the Stitch tab.

Enter a file name and location in Save to and select the appropriate Zoom Level (15 in our example). Then click Start.

The image that is now created is roughly 10 MB in size – the dimensions are 15000 x 11000 px – so pretty large.

Of course, one cannot see much at this magnification but zooming in anywhere onto this picture shows the true level of detail:

Another nice feature: if I do not like the map that has been produced, SAS.Planet allows me to switch to a different tile server but maintain the current selections. So let’s use Google’s Landscape Tiles instead to see if we can get a better result on the mountain region of the island.

You can see: the selection box is still present so I go back to the Selection Manager and download the tiles for Level 15 of Google’s Landscape Layer – same procedure as above.

When I go back to create the stitched image from the downloaded tiles, the result again is a file of 19 MB, same dimensions as the first one.

The level of detail, again, will only reveal itself when zooming in but the overall representation is more what I think I would like to have on the wall… the hills are showing as a well defined relief…

There are plenty of other layers available in SAS.Planet – using the OpenTopo Tile Server provides yet a different view for the map:

And of course – if I do not like a map-type of poster on my wall, I can always switch to the various satellite image providers – mainly Google Earth and Microsoft Bing. Here is a comparison between the image quality of the two – Microsoft on the left, Google on the right.

It comes down to a matter of taste at this level – on a higher level of detail, the Microsoft tiles are more crisp but the Google tiles are more up-to-date.

 

Posted in Allgemein | Leave a comment

FHEM – Temperature Sensors in multiple locations

One of the things that I want to use FHEM for is monitoring the temperature and humidity on a variety of locations (different rooms, patio, etc.)

I am using a set of TX 35 DHT sensors, the connection to FHEM is implemented using a JeeLink v3c USB device. I will describe that in a different post, also how you would pair the sensors with FHEM – for now, let’s just pretend that was done.

I use to define my FHEM Objects “manually” in the fhem.cfg file – so here is the definition for the TX 35 DHT on the patio.

Image 01 - The Definition for the Patio SensorAs you can see, the sensor itself is a LaCrosse sensor, connected to my JeeLink device (called myJeeLink). The sensor is called pt.Temp – the prefix pt standing for patio. If you do not only want to display the current values but also would like to gather historic data (e.g. for graphs), you will have to log the readings somehow.

The easiest way is shown above – using a FileLog device. However, if you work with multiple sensors and record the data over a long time (which is the intention) than working with filelogs is not really the brightest thing to do: they are slow and they will use up your diskspace…quickly!

So let’s try and get this right stright away: instead of logging to a file, let’s log to a database. Sounds complicated? Maybe – let’s find out.

Connecting FHEM to a PostGreSQL Database

The connection requires two steps – the first one is to update the db.conf file to match your database definitions. A sample file for each database supported can be found in the /opt/fhem/contrib/dblog directory of the FHEM installation. For PostGreSQL, the following is needed:

%dbconfig= (
        connection => "Pg:database=fhem;host=localhost",
        user => "fhemuser",
        password => "fhempassword"
);

And of course, you will have to update this with our own database name, host, user, and password information!

And there is a really important thing to do – probably only if your PostgreSQL Server is on a remote machine but hey, it would not hurt either way: you need to install an additional Perl library:

sudo apt-get install libdbd-pg-perl

Otherwise, you will have your system continously “waiting for connection”…

Now, once you have the db.conf file set up and restarted your FHEM Server (shutdown restart) and you have created the database schema in the logging database accordingly (see /contrib/dblog/db_create*.sql scripts in the FHEM home directory!) you can create a global logging device:

define logdb DbLog ./db.conf .*:.*

This defined a Device logdb of type DbLog with the configuration file db.conf in the FHEM home directory and it logs… everything.

Image 02 - The PostgreSQL Database connectedMind the state property: it needs to say connected… using any sort of database maintenance tool, you should not be able to monitor data flowing in:

Image 03 - Data written to Table CURRENTNow that we have connected the PostgreSQL Database sucessfully, it is time to return to the creation of the individual sensors.

Pairing new Tx 35 DHT Sensors

The TX 35 DHT sensor is actually a LaCrosse Sensor. Pairing – at least as far as I know so far – can only be done in one way:

  1. Remove the batteries from the TX 35 DHT if there are any installed.
  2. Go to FHEM and select the JeeLink Device (from the Everything tab)
  3. It has a set command for LaCrossePairForSec – enter 60 for 60 seconds and click the set button.
  4. Insert the batteries into the TX 35 DHT and wait for the temperature to show again…

Once you refresh the FHEM Interface, you will see a LaCrosse tab on the left. Select the Sensor and note the DEF value – that’s your Device ID!

Image 04 - Freshly paired LaCrosse SensorIf you want to edit your fhem.conf file manually (like I do), this is the ID to work with for that particular sensor!

Image 05 - The LaCrosse in the StudyI am actually repeating this for a total of seven sensors – one of them, the one on the patio, is an older model that does not do humidity. But the TX 35 DHT are marked as indoor use so I will keep the outside one for the moment.

Image 06 - All LaCrosse Sensors installedSo above is the list of all sensors in FHEM with their respective readings… and they are busily dumping their values into the PostgreSQL Database 🙂 – so far, so good.

For now, one last thing remains to be done: the constant logging of each and every value reported (as an event) quickly fills the database with garbage. I am only interested in those events that indicate a change to the previous status. And luckily, FHEM supports this out-of-the-box: by setting the object’s event-on-change-reading attribute to .* we will only see those value changes come up.. the “clutter” in the Event Monitor dies down dramatically… and that’s it for the moment, let’s gather some data in the logs.

Posted in Allgemein | Leave a comment

FHEM – Working with Objects

Now that we have installed the FHEM Server onto a Raspberry Pi, it is about time to do something with it. We have seen the result of the “naked install” at the end of the last post:

Picture 04 - FHEM Web PageThe web page itself is available but of course, nothing is configured into the system yet. Before we start working with FHEM, here is a brief discussion about how FHEM is organized.

It’s all about one file

In FHEM, it’s all about one file: fhem.cfg. This is the central configuration file which contains all – all! – definitions for the system. You can edit it directly or you can interact with the system using the Command Line and the configuration file will be updated in the background.

After the initial installation, the file contains some configuration already – how you edit it, is up to you. I have chosen to use WinSCP and got the file local onto my PC, then used Notepad++ to work with it.

Picture 03 - The Default Config FileEverything is an “Object”

In FHEM, everything you may want to work with, is considered “an Object”. It does not matter what exactly it is – a device, a log file, a room, a setting – everything is an object. With the default installation done, you can already see some objects in FHEM’s inventory – just select Everything on the left to see all objects that are currently defined – and their status.

Picture 04 - Default Object DefinitionsComparing the list of the fhem.cfg file, you can easily see that the definitions correspond. In other words: everything that is defined in the configuration file is accessible through the web interface and everything you configure through the web interface is saved to the configuration file. So how you work, is really entirely up to you – but editing the configuration file results in a “cleaner” an better structured file than having the system just dump the information there…

Our first “own” Object

To demonstrate how object definition works in FHEM, I have decided to use a very basic example that does not even need hardware. FHEM knows something called a “Dummy Object”. Perfect for getting to learn the system without having to waste money on actual hardware devices. Try this in the Command Line:

define mySwitch dummy

Through the Command Line, it results in an immediate creation of the Dummy Object – but only “in memory” – mind the “Save config” entry on top of the menu bar!

Picture 05 - A Dummy SwitchThis updated the config file – actually, the file received a couple more lines as it itself has been updated from an older version but the one line we are interested in is at the bottom:

Picture 06 - The updated Config FileYou can see that the object has a Name (“mySwitch”) and a Type (“dummy”). Those are the two parameters we have provided in the define command above. It also has a couple of other elements – sucha s a Number (or “Nr”) and a State (currently “???”) but these we did not define – they come from the system and are defaulting to initial values or are auto-calculated.

At the bottom, you can see that an Object has Attributes – in the screenshot above, one is defaulted (“room”) but others are possible. An Attribute is used to provide additional information to an Object. It typically describes the object in more detail.

Let’s take the Room attribute for the moment. Given that we may have a Home Automation System in mind that works in more than one room, it is logical to provide any device we might implement with a Room attribute to structure our devices better. Otherwise, we would have to remember which switch we have in which room…

Picture 07 - Configuration File UpdateThis time, I have chosen to update the fhem.cfg file directly and bring it back to my Raspberry Pi. Just copying the file back, however, does not help – FHEM does not “read” the updated configuration – you need to restart the FHEM Server.

shutdown restart

After the service has restarted, you can see the result of the configuration update right away:

Picture 08 - The Updated ConfigExamining the Dummy Switch more closely, you can now see that the Room attribute has been added – and that FHEM is using it in the menu to separate the devices by rooms.

Picture 09 - Device SpecificsNow, what do we do with that dummy switch? Usually, you would use a switch to… switch on a light? OK – but we need a “light” first:

Picture 10 - Lamp Device and Switch CommandAs you can see, the new dummy “myLight” is defined (Line 44) and placed in the Living Room (Line 45). At the same time, I have extended my existing switch “MySwitch” to include a command (can be “on” or “off”) to actually “switch” it (Line 41).

Picture 11 - The Switch is onIn the Web Interface, FHEM now allows me to switch “mySwitch” on or off (it is now “on”) and the system visualizes the state correctly. Last thing to do is to connect the switch to the light…

Another type of Object – an “Event”

Chaning the state of the switch is actually an “Event” – something that “happens”. And objects are defined in the already familiar way:

Picture 12 - The Notify EventThis is called a “Notify” Object because the eventy of the switch notifies the light to so something. Or in other words: if mySwitch enters the state of “on”, change the state of myLight to “on”. Needless to say that this would be a very unidirectional handling – there is currently no way to switch the light off again 😉

Picture 13 - The Notify ObjectObviously, I could just repeat the step and create another Notify Object that is triggered when mySwitch if entering the state of “off” – but that is tedious. Instead, we are “tricking” our way:

Picture 14 - The Notify on all EventsInstead of “hardcoding” the values of the event triggered by mySwitch (“on”, “off”), we are reacting to simply every event – every time mySwitch changes state (and invokes the Notify Object) we are doing something to myLight.

The trick is using $EVENT (mind the uppercase style!) – that is a system variable that takes the value of the initiating object’s event (e.g. “on”) and by setting it on myLight, we can get around with one line of code. But that only works if the values are the same on both sides!

Not bad for a first attempt – we now can edit the configuration file, create devices, place them in rooms and have them interact based upon events. Next up is some cool stuff I always wanted to do… building my own weather station & indoor clima monitor.

Posted in Home Automation, Raspberry Pi | Leave a comment

FHEM – Updating the System

Like any other piece of software, FHEM needs updates. New devices may be supported or bugs may be fixed or new functionality may be added. So it is wise to keep the system up to date but at the same time, do so with care and the ability to roll back in case something goes wrong or requires more work to be done than you might have at the moment.

Rule #1: Always have a Backup

The first rule before any update is to have a backup of the system. For FHEM on a Raspberry, there are two ways of doing that: the probably most complete one is to backup the entire SD Card using a tool such as the USB Image Tool. That simply backs up the entire system and you can restore the environment as a whole should something go wrong.

Of course, if you have other environments or need to transfer the configuration between say a “Test System” and a “Live System”, you might want to choose a different approach.

FHEM has an integrated backup method. To use it, simply type the following in the FHEM Command line:

backup

Then hit Enter.

Picture 01 - Backup the FHEM EnvironmentTypically, the file resides on the Raspberry Pi’s File System in the location /opt/fhem/Backup.

Like the backup, the update is invoked via FHEM’s Command Line – guess what command?

update

The update loads a complete file list first so no worries if stuff starts to scoll through. Theoretically, the last message should inform you that the update has been performed, in my case, you really had to read the output as this was not the last line.

Picture 02 - Backup doneNow, it is time to restart FHEM – either by rebooting the Raspberry Pi or by issuing the following FHEM Command:

shutdown restart

And that should be it…

Posted in Allgemein | Leave a comment