DS-GVO – “Datenschutz-Grundverordnung”: a German tounge-breaker for “General Data Protection Regulation”

Posted on March 24, 2018 by azapf1972

The EU has a new law (OK, it is not “new” but it comes into effect May 2018). And that has stirred up some mud because it has an impact on almost any website, blog, forum, etc. I am talking about the EU General Data Protection Regulation (GDPR) in general and the German Datenschutz-Grundverordnung (DS-GVO) in particular.

Without going into the details of the regulation, I still wanted to share some changes I applied to my website operated under WordPress. I am not claiming these are “complete” or all “necessary” or “correct” but this is what I have done.

  • First of all, I have disabled the ability to register as a user (Settings > General > Membership > Disable “Anyone can register”).
  • Then I have turned off all ability to comment on my posts (Setting > Discussion > Disable “Allow link notifications”, “Allow people to post comments on new articles”, “Comment author must fill out name and eMail”)

The ability to comment is vital on some blogs and this is a case-by-case decision that might hurt. For me, most of my comments where nonsense or of no general interest. It just makes my life easier. And people can still contact me via my eMail which is posted anyway.

  • I then went in and deleted all registered users (6.000 of them where spam accounts anyway…)
  • I also deleted all existing comments.
  • I set the comment settings for all existing posts to “Not able to comment”) – you can do that from Posts > All Posts, then select all articles and chose “Edit” as Bulk Action, then click “Apply”. Set the “Comments” to “Do not allow”).

Last but not least, I am running WP Statistics so I went to the Statistics Settings and enabled “Hash IP Addresses”. With the result that from that point forward, all IP Addresses are hashed (but still counted properly).

If you are running some external, even free plugin to track visitors (like a visitor map) you might have to check what your plug-ins do…and maybe disable them if they are sending data elsewhere…

Last but not least, you have to update your Data Privacy Statement to reflect what you are doing, what you are not doing and for the things you are doing, why you are doing it. Plus grant the rights to your users to send you an inquiry about their stored data (if you store any data) and report/remove/correct at their requests.

Posted in Allgemein | Comments Off on DS-GVO – “Datenschutz-Grundverordnung”: a German tounge-breaker for “General Data Protection Regulation”

Data Privacy Statement

Posted on March 24, 2018 by azapf1972

In accordance with the EU Legislation on Data Privacy and Data Security, the Data Privacy Statement of this web site has been updated. By continuing to use this web site, you are accepting the statements provided as adequate and acceptable to you.

Posted in Allgemein | Comments Off on Data Privacy Statement

FHEM Home Automation – Cleaning up the Server – Part I

Posted on November 18, 2017 by azapf1972

My FHEM Server has now been working without any issues for quite some time, definitely more than a year, probably closer to two years. Time to clean up some parts, especially since the latest updates did not install successfully and I had to revert back to the installed version.

Over the last year(s), things “worked” and while they still do, it is time to also come up with a new configuration – things, that I have learnt in the meantime. Hardware that has been bought in the meantime. New ideas that have developed in the meantime. To get all these “into a new system”, I have decided to build a playground on a Virtual Machine. The operating system of choice is Debian 9, because it is so similar to the Raspberry Pi that I eventually want to move the installation to.

So this one should be fast enough for some playing. And maybe I will regret the settings later, the Raspberry Pi is much slower.

I have decided for the “Desktop installation – that is: a graphical desktop environment (which is not the same than the Raspberry Pi will be). And a Terminal window for “command line to the bitter end”…

After the installation has finished, there are some things that should be done – the first one is to update the system to the latest patch level (the two commands can be executed from a Terminal window in superuser mode):

apt-get update
apt-get upgrade
apt-get install net-tools

Next, I need to install VMWare Tools (which, of course, is only necessary if you are running VMWare Player or VMWare Workstation). That, by the way, is the reason why in the command sequence above the net-tools package is installed…

Getting FHEM onto the Box

There are a few things that need to be done now – the first one obviously being the installation of FHEM on the box. This is best performed as “manual installation” – the most current instructions are here: debian.fhem.de

The first three steps of the instruction for the manual installation (ending with the command dpkg -i fhem-5.8.deb are responsible for installing the required libraries, fetching the FHEM Package and installing it. This includes the creation of the fhem system user:

grep fhem / etc/passwd

shows the user’s definition. It also shows that the user’s home directory is /opt/fhem (where the installation resides) and there is no shell assigned – /bin/false – which makes the user unable to login.

Technically, the last statement might not be quite correct: the user can login but it would not get any shell and therefore no command line (plus: no password unless we would set one…)

Now this presents an interesting problem: if you do the following:

cd /opt/fhem
ls -l

to display the files and their permissions in the FHEM installation directory, you will notice that all files are owned by the user fhem and the default group is set to dialout.

Catch-22 with the Configuration Files, Users, and Permissions

To me, that poses a small problem: the configuration files that we will need to create for FHEM. Of course: you can edit the fhem.cfg file straight through the web interface but that is a tedious and rather stupid way of doing it (imho). Just two reasons: there are far better editors than the built-in one (I, for example, prefer Notepad++ on a Windows Box) and it is hard to do version control on your files straight on the Unix File system.

Of course, there are plenty of ways of getting files to and from a Linux system (including direct drag & drop into my virtual machine) but considering a headless (no user-interface) environment on my Raspberry Pi, the most logical way seems to be a FTP Server on the Unix box.

Installing ProFTPD into the environment

An easy-to-install and powerfull FTP Server is ProFTPD. You can get it installed via

apt-get install proftpd-basic

Once installed and started, the default is listening on TCP/IP v6 – which is the first thing I need to change. At the same time, we can ensure that a user that is allowed to log in using FTP does not require a Shell assignment (like the fhem user discussed above) and last but not least we can control the default home directory for the FTP Users.

Most of this already plays also into the Unix Permission Concept so it will be a mixed discussion here.

Fist of all, the additional configuration for ProFTPD is applied using a custom config file located in /etc/proftpd/conf.d – that directory will be maintained even if ProFTPD is updated. Let me call my file custom.conf – it is a simple XML File:

Note: Please understand that this is a most basic security setting and is sufficient for FHEM Boxes that are in a secure environment (e.g. not accessible over the Internet, only your and yourself on the network, etc.)

ProFTPD can do much more in terms of security – secured connections using SSL/TLS Encryption, non-standard FTP Ports, etc. – if you want to find out more, please make sure that you use the search engine of your choice to locate the appropriate information on the Internet. And never use the above configuration in anything else but a test environment!

Once the configuration file is placed, restart the FTP Service

systemctl restart prodtfd.service

You can then check of the service is actually running

service proftpd status

and if it is listening on Port 21, the default FTP Port.

netstat -tlp | grep proftp

With that said, you should be ready to access the server via FTP – at least, after you have added your user of choice to the ftpuser group…

groupadd ftpuser (if it does not exist yet)
usermod --append --groups ftpuser [UserName]
grep ftpuser /etc/group

The last command must return one line with the definition of users in the group.

To be or not to be…

The next question – hotly discussed, I am sure – is the user to work with. Of course, you can use ProFTPD with a more elaborate configuration to allow any user to connect and still set those users’ root directory to /opt/fhem (where you want to copy the configuration files to). You can even go as far to make sure that any file copied is placed with the appropriate file permissions, having fhem as the owner and dialout as the group (which is what the FHEM installer sets by default when you install a fresh copy).

But there are alternatives: you can work directly as the fhem user (which requires you to give the user a password and place him in the ftpuser group).

passwd fhem

I am sure there are plenty of people around that will object to that approach because fhem is considered a service account and should not be used for login…

Alternatively, you can work “as your own user” and adjust the file permissions afterwards (each file you will copy into the directory using FTP will by default have your user as owner (instead of fhem) and your primary group as group (instead of dialout).

To change the owner of all files in /opt/fhem use the following command:

chown -R fhem:dialout /opt/fhem

which you need to execute as superuser. Bottom line: there are so many ways of doing it – and each one has its own rights and wrongs. Pick your own one but make sure you understand the implications with respect to the security of your FHEM Installation!

One more thing…

One more thing about security: by default, the fhem user ends up in the group dialout. It is important that fhem also is member of the group tty which controls access to serial and USB Ports (in case you have a USB Device attached to your FHEM Server…)

usermod --append --groups tty fhem

takes care of that.

Is it working or not?

By now you might have a working FHEM Server – which you can check on the command line via

/etc/init.d/fhem status

and it might be accessible via HTTP under http://[Your server]:8083/fhem

If the server is not working or is not reachable, it is time to take a first look at the logs (well, even if it is working, it might be a good idea…)

The FHEM Logs

The FHEM Logs are located in /opt/fhem/log and depending on your way of access to your server, you can either view them straight on the box (e.g. using the tail command) or access them through the FTP Server and view them in an editor like Notepad++:

This one is a log file of the first successful start – it shows the config file that is read (fhem.cfg – oh wonder!), the status of the outbound ports (8083, 8084, 8085), a security message, the server’s FeatureLevel (5.8) and the fact that “9 entities” have been defined…

If there are errors, this is your first place to go – as we will see in a moment.

Updating FHEM

I want to provoke a situation that has caused me a lot of headache because I did not see the issue – only with the help of the FHEM Forum (suggestion to every FHEM user!) I got it solved (in minutes!).

My FHEM Installation is still the one that was installed with the package above – but FHEM is constantly updated. So in the Web Interface, go to the command line at the top and type the following:

update check

This will (likely) create a long list of files that can be updated – and should be updated (especially since we are just starting with the installation). But keep in mind: an update can cause nasty surprises as we will see…

To update FHEM, issue the command

update

and wait. After a short while, you should see more or less the following:

The important line is “update finished, ‘shutdown restart’ is needed to activate the changes”. So issue the following command in the Web Interface:

shutdown restart

The nasty surprise in this one: the Web Interface ceases to work after the update! No connection is possible – and at the first glance, you have not changed anything! I have to admit: this pissed me off for quite some time.

I understand that if you want to run an environment like FHEM, you need to be familiar with what you are doing (and I appreciate this fact). On the other hand, if a working environment is turned “non-working” with an update that immediately renders the system inaccessible… that is not necessarily the way I would prefer it to be.

But back to the problem (and as I said: the solution was straight forward, thanks to the FHEM Forum!). Check the FHEM Log File again:

There are some significant changes compared to the previous log (and we did not change any of the configuration, just performed an update!)

There now is a PERL Warning that seems to be associated with the USB Checks. If you search for the error message, it becomes obvious that is related to the USB Check defined in the default fhem.cfg file:

define initialUsbCheck notify global:INITIALIZED usb create

It appears that the system’s behavior has been changed but I cannot see any reference to that change in change log /opt/fhem/CHANGED. In any case: you can remove or comment out the respective line and the error should be gone.

That, however, does not bring back the Web Interface. Again, a look at the CHANGED file does not bring up anything obvious. Except for:

 - feature: allowed module added. allowedCommands, basicAuth, password,
 globalpassword attributes moved to this module.

Well, look at the error message in the log again:

Connection refused from the non-local address 192.XXX.XXX.XXX:53262, 
as there is no working allowed instance defined for it

At this point, it becomes a language issue: because of the wording, I interpreted the “allowed” part as part of the sentence – if you know what to look for, it tells you that a connection is refused because there is no defined instance of the allowed device.

If you would have understood, you could have looked it up in the FHEM Reference. And it could have been corrected earlier.

Please note that I have defined a clear text username and password in this example – which obviously is inappropriate for anything but a test environment!

The better way of doing it is to create a base64-encoded password string from the Unix Command Line and use this instead:

echo -n fhemuser:secret | base64

Creates ZmhlbXVzZXI6c2VjcmV0 as password phrase which then can be used in the config file:

So now: copy the fhem.cfg file back to the server (unless you have edited it there directly) and restart FHEM:

/etc/init.d/fhem stop (if it is not stopped anyway)
/etc/init.d/fhem start

And now try the Web Interface again – you should see it working now. With all of that done, we are through with Part I of creating a brand-new FHEM Environment. The initial configuration parts will follow in Part II.

Posted in Home Automation, Raspberry Pi | Leave a comment

Creating large-scale high-resolution Maps from various Sources

Posted on March 29, 2017 by azapf1972

Everyone knows services such as Google Earth or Microsoft Maps. They are great for exploring but they are not so great when it comes to creating large-scale high-resolution images, e.g. for a wall poster or alike. Also, they have their shortcomings when it is about comparing maps or images from different sources to “pick the best”.

So I went out searching… and here is a little process to share. The task is: to create a single-image map of Mallorca with a high-level map resolution. So let’s see…

The software I am using for this is called SAS.Planet – and it seems to be a Russian-built piece of software (so my first installation came up all Cyrillic…). I ended up downloading a copy from “somewhere” and then used the internal Help -> Check for Updates feature to grab the latest release.

So this is all of Mallorca (using a Google Maps layer) but not the level of detail I want… which is more like this:

So in general, I have a choice of seeing “all of it at lower level” or “seeing more detail but lesser area”. Let’s change that: zoom back out until you see the whole of the island. Then use the Rectangular Selection tool to draw a map around it.

A window will open that allows for a couple of options. In general, I have had the best experience with downloading the required tiles first, then stitching them together. So on the Download tab, all I do is to download all tiles of Zoom Level 15 and click Start.

If you want to monitor the download process, you may switch on Tile Shading using View – Cached Tiles Maps and select the level you are downloading – z15.

This will display each tile downloaded for that Zoom Level to appear with a gray shading. In addition, you can monitor the progress in the Download Window. Now, you need some patience and wait…some tile servers are faster, some are slower.

When all tiles have been downloaded, go back to your current selection via Operations -> Selection Manager -> Last Selection and switch to the Stitch tab.

Enter a file name and location in Save to and select the appropriate Zoom Level (15 in our example). Then click Start.

The image that is now created is roughly 10 MB in size – the dimensions are 15000 x 11000 px – so pretty large.

Of course, one cannot see much at this magnification but zooming in anywhere onto this picture shows the true level of detail:

Another nice feature: if I do not like the map that has been produced, SAS.Planet allows me to switch to a different tile server but maintain the current selections. So let’s use Google’s Landscape Tiles instead to see if we can get a better result on the mountain region of the island.

You can see: the selection box is still present so I go back to the Selection Manager and download the tiles for Level 15 of Google’s Landscape Layer – same procedure as above.

When I go back to create the stitched image from the downloaded tiles, the result again is a file of 19 MB, same dimensions as the first one.

The level of detail, again, will only reveal itself when zooming in but the overall representation is more what I think I would like to have on the wall… the hills are showing as a well defined relief…

There are plenty of other layers available in SAS.Planet – using the OpenTopo Tile Server provides yet a different view for the map:

And of course – if I do not like a map-type of poster on my wall, I can always switch to the various satellite image providers – mainly Google Earth and Microsoft Bing. Here is a comparison between the image quality of the two – Microsoft on the left, Google on the right.

It comes down to a matter of taste at this level – on a higher level of detail, the Microsoft tiles are more crisp but the Google tiles are more up-to-date.

 

Posted in Allgemein | Leave a comment

FHEM – Temperature Sensors in multiple locations

Posted on July 9, 2016 by azapf1972

One of the things that I want to use FHEM for is monitoring the temperature and humidity on a variety of locations (different rooms, patio, etc.)

I am using a set of TX 35 DHT sensors, the connection to FHEM is implemented using a JeeLink v3c USB device. I will describe that in a different post, also how you would pair the sensors with FHEM – for now, let’s just pretend that was done.

I use to define my FHEM Objects “manually” in the fhem.cfg file – so here is the definition for the TX 35 DHT on the patio.

Image 01 - The Definition for the Patio SensorAs you can see, the sensor itself is a LaCrosse sensor, connected to my JeeLink device (called myJeeLink). The sensor is called pt.Temp – the prefix pt standing for patio. If you do not only want to display the current values but also would like to gather historic data (e.g. for graphs), you will have to log the readings somehow.

The easiest way is shown above – using a FileLog device. However, if you work with multiple sensors and record the data over a long time (which is the intention) than working with filelogs is not really the brightest thing to do: they are slow and they will use up your diskspace…quickly!

So let’s try and get this right stright away: instead of logging to a file, let’s log to a database. Sounds complicated? Maybe – let’s find out.

Connecting FHEM to a PostGreSQL Database

The connection requires two steps – the first one is to update the db.conf file to match your database definitions. A sample file for each database supported can be found in the /opt/fhem/contrib/dblog directory of the FHEM installation. For PostGreSQL, the following is needed:

%dbconfig= (
        connection => "Pg:database=fhem;host=localhost",
        user => "fhemuser",
        password => "fhempassword"
);

And of course, you will have to update this with our own database name, host, user, and password information!

And there is a really important thing to do – probably only if your PostgreSQL Server is on a remote machine but hey, it would not hurt either way: you need to install an additional Perl library:

sudo apt-get install libdbd-pg-perl

Otherwise, you will have your system continously “waiting for connection”…

Now, once you have the db.conf file set up and restarted your FHEM Server (shutdown restart) and you have created the database schema in the logging database accordingly (see /contrib/dblog/db_create*.sql scripts in the FHEM home directory!) you can create a global logging device:

define logdb DbLog ./db.conf .*:.*

This defined a Device logdb of type DbLog with the configuration file db.conf in the FHEM home directory and it logs… everything.

Image 02 - The PostgreSQL Database connectedMind the state property: it needs to say connected… using any sort of database maintenance tool, you should not be able to monitor data flowing in:

Image 03 - Data written to Table CURRENTNow that we have connected the PostgreSQL Database sucessfully, it is time to return to the creation of the individual sensors.

Pairing new Tx 35 DHT Sensors

The TX 35 DHT sensor is actually a LaCrosse Sensor. Pairing – at least as far as I know so far – can only be done in one way:

  1. Remove the batteries from the TX 35 DHT if there are any installed.
  2. Go to FHEM and select the JeeLink Device (from the Everything tab)
  3. It has a set command for LaCrossePairForSec – enter 60 for 60 seconds and click the set button.
  4. Insert the batteries into the TX 35 DHT and wait for the temperature to show again…

Once you refresh the FHEM Interface, you will see a LaCrosse tab on the left. Select the Sensor and note the DEF value – that’s your Device ID!

Image 04 - Freshly paired LaCrosse SensorIf you want to edit your fhem.conf file manually (like I do), this is the ID to work with for that particular sensor!

Image 05 - The LaCrosse in the StudyI am actually repeating this for a total of seven sensors – one of them, the one on the patio, is an older model that does not do humidity. But the TX 35 DHT are marked as indoor use so I will keep the outside one for the moment.

Image 06 - All LaCrosse Sensors installedSo above is the list of all sensors in FHEM with their respective readings… and they are busily dumping their values into the PostgreSQL Database 🙂 – so far, so good.

For now, one last thing remains to be done: the constant logging of each and every value reported (as an event) quickly fills the database with garbage. I am only interested in those events that indicate a change to the previous status. And luckily, FHEM supports this out-of-the-box: by setting the object’s event-on-change-reading attribute to .* we will only see those value changes come up.. the “clutter” in the Event Monitor dies down dramatically… and that’s it for the moment, let’s gather some data in the logs.

Posted in Allgemein | Leave a comment