QNAP LUN Recovery – Preparing to recover VMWare ESX Datastores

When all things break, you are sometimes left with just your latest backup (hopefully) and the need to get to some files.

In this case, the problem was a broken QNAP NAS where we recovered some LUN RAW-Device files from. To add to our challenge, the LUNs recovered did not contain any mountable file system such as NTFS or so – they essentially had been used as VMWare ESX Datastores – so the files in there are VMWare Definition Files & Virtual Machines plus their file system is VMFS – the VMWare File System (which is not directly mountable/readable under Ubuntu)

In preparation for a recovery, I installed an Ubuntu 13.04 System and then added the following components via apt-get install.

  • iscsitarget: installs the iscsi target components – on some Ubuntu versions, this is sufficient to then host the LUNs, the 13.04 version (and maybe others) will throw an error “FATAL: Module iscsi_trgt not found.” which means I also have to install
  • iscsitarget-dkms: this one will compile and install – afterwards, the iscsi target service can be started (configuration assumed).
  • vmfs-tools: allows mounting and accessing partitions formatted in VMFS.
  • open-iscsi: installs the iscsi initiator (client) bits and pieces you need in order to mount an iscsi LUN locally.

Obviously, you might want to consider an apt-get update to update the dependcy and package libraries if you have not done so in a bit… even if you don’t have a system like that available, you can install one on any PC you can spare for the job in 30-60 minutes.

Note on the PC: use a physical PC if you have large amounts of data or many LUNs to recover. Make sure your disks – if you have to use external disks – are USB 3 for performance and your PC also has a USB 3.0 bus available…

Posted in Ubuntu | Tagged | Leave a comment

The Beauty of PI

“Pi, the ratio of the circumference of a circle to its diameter, and this is just the beginning; it keeps on going, forever, without ever repeating. Which means that contained within this string of decimals, is every single other number. Your birthdate, combination to your locker, your social security number, it’s all in there, somewhere. And if you convert these decimals into letters, you would have every word that ever existed in every possible combination; the first syllable you spoke as a baby, the name of your latest crush, your entire life story from beginning to end, everything we ever say or do; all of the world’s infinite possibilities rest within this one simple circle. Now what you do with that information; what it’s good for, well that would be up to you.”

– from “Person of Interest, Season 2, Episode 11

I was watching this episode casually – but this statement did intrigue me. If true, I would have wished for my math lessons back in school to let me explore things like this and provide information beyond the pure presentation of facts… spark my imagination! Make me want to challange and discover! Let me experiment!…

So let’s see how much truth the statement up there holds: yes, Pi is an irrational number which means it cannot be expressed as a/b. Pi never ends… and it never infinitely repeats… so far, the statement above is correct. Wether it contains all possible combinations of digits somewhere within is beyond my knowledge to say… but one thing to keep in mind is that no matter how many digits we compute for Pi, the chance of finding a specific complex pattern in the digits known today is unbelievably small – but if Pi never ends and never repeats itself, somewhere in there…

So let’s try and find my birthday in Pi – 14.07.1972 – and I am looking for the representation in the German date scheme: 14071972. And hey, it is in there… starting at the 10.266.360 decimal digit. (see http://www.subidiom.com/pi/)

Let’s try something else – let’s search the string “Pi” itself. Obviously, Pi only contains decimals and a string is not a decimal. But we can find a way to convert a string into a decimal (actually, we can find multiple ways to do so).

We could simply fall back to the ASCII Code where each character is represented by a number: “P” is 80 (decimal), “i” is 105 (decimal). So looking for the ASCII representation, we could be searching for “80105” (and would find it at the 18.098th digit).

So how about my name, Andreas. You can use an online ASCII Calculator to make your life easier (http://www.branah.com/ascii-converter) and so, my name would be represented as “65 110 100 114 101 97 115” decimal. Now, this already is a complex pattern and I am not lucky enough to find “myself” in the first 2 billion digits of Pi. But shorter names like “Tom” (at decimal 16.524.559) and “Sam” (at decimal 4.246.411) and “Max” (at decimal 5.245.782) can be found.

However, if I take a different coding scheme (counting a letter’s position in the alphabet where A=1, B=2 and so on), “Andreas” comes out as “1 14 4 18 5 1 19” and this sequence is present at decimal 1.593.228.194…

Germany does not have a social security number but we have a tax number – I don’t tell you mine but it is an 11-digit number. Again, no luck – pattern too complex for just the first 2 billion digits. So let’s try at which point I get lucky… and that happened faster than I though: I only had to take off the last 2 digits (make it a 9-digit number) to find the beginning of my tax number in Pi.

And so it goes on:

  • all my (4-digit) PIN numbers for credit cards are in there (na, I am not going to tell you where!),
  • my bank account number is in there (send me money to the digits starting around decimal 548.684.250),
  • my wife’s birthday is in there (somewhere around digit 108.242.600), actually all of the dirthdays of all of my family members I tried are in there.
  • my telephone number (without area code) is in there, and
  • my cell phone number (without provider code) is in there

As you can see – as long as the pattern is relatively small (less than 10 digits), your chances of finding it within the first 2 Billion digits is relatively good. And given that Pi goes on forever, we would just need more digits (actually, many more digits!) to find almost anything you are looking for… somewhere.

By the way, Pi is also nice to create passwords and PINs: just come up with a number (say your birthday) and take the numbers of that particular digit as password or PIN: taking 10 digits starting at position 14071972 gives me “1109437165”. You could look it up almost anywhere you are (given you have a Smartphone) and you would of couse NOT use your birthday but some other “secret” number as starting point for your sequence…

So even if the discussed scene is more poetic and dramatic than maybe mathematically correct it still sparks imagination – it makes people curios. And maybe that is exactly the initiator someone needs to start thinking and exploring…

Posted in Allgemein | Leave a comment

Linux Discovery – using Ubuntu as iSCSI Target Server

For reasons I’ll explain later I had to dive a bit into iSCSI and Ubuntu – the background is a desaster recovery for a crashed QNAP Device but the only significant relation is that that crashed box had been used as an iSCSI Target Server and we needed to recover data.

A 10.000-Ft.-View of iSCSI

I am not going to dive into the depths of iSCSI – you can read the definition here. But it is important to know that iSCSI knows two “sides” – the side that provides the service (and the data) and the side that consumes it (the client).

The iSCSI Target generally is the “box” providing the service. This can be a dedicated NAS or a Computer running a software iSCSI solution such as Open-iSCSI or StarWind. This box can “publish” multiple iSCSI Nodes, a “node” essentially being an emulated “device” such as a harddisk or a tape device.

The iSCSI Initiator is mostly used for two things: discovering the “published” iSCSI Nodes on a given iSCSI Target and connecting to a specific iSCSI Node or several Nodes.

Ubuntu & Open-iSCSI

For my little discovery, I have chosen Open-iSCSI as it easily installs with Ubuntu and my crashed QNAP Device is also using it… choice made for me. The installation into a test environment running Ubuntu 13.10 is easy – get a terminal window, gain root-level access and issue install the iSCSI Target Service.

apt-get install iscsitarget

This installs the iSCSITarget Service into the system – and that is it (for the server). Now, the iSCSI Initiator is installed as easily:

apt-get install open-iscsi

Starting the iSCSI Target Service

Before we can start the newly installed service, we need to make at least a minimal configuration change – we need to allow the service to be enabled.

nano /etc/default/iscsitarget

You will find a very short configuration file – and you only need to change one line (at this point).

01 - ISCSI Configurationchange the first line to

ISCSITARGET_ENABLE=true

and save the file. Then start the iSCSITargetService…

service iscsitarget start

You should see something like this – and not more…

02 - Service StartNow, this is not much yet – there is a service, well done, but we do not have anything to serve just yet… what we need is an iSCSI LUN.

The iSCSI LUN

LUN stands for Logical Unit Number – a rather stupid term for something that eventually is nothing more (or less) than a file that – through the iSCSI Interface – appears to the iSCSI Initiator like a “physical disk” available somewhere… we are basically creating and using a “virtual” harddisk.

You can store your LUNs (which are “raw device” or “image files”) wherever you want (and have access to) – to keep it simple for now, I am placing the LUNs in a directory /LUN which I have to create (as root) first:

mkdir /LUN

Next, I am going to create an image file – say we want a “virtual” harddisk of 4 GB (my virtual machine currently has a rather small boot volume and we have not made preparations to move the LUNs anywhere else…

dd if=/dev/zero of=/LUN/MyLUN0 count=0 obs=1 seek=4G

This creates a file called MyLUN0 in /LUN with /dev/zero (nothing) as input file (if) and the said file as output file (of). The Output Block Size is 1 (obs) and we jump 4 GB ahead (seek) which basically will create a matching but empty file.

03 - The LUN Image FileOK, so now we have an image file – next is to turn that image file into a valid iSCSI Target Node to serve…

Defining the iSCSI Target Node

Edit the file /etc/iet/ietd.conf – this is where the definitions what to server are made…

nano /etc/iet/ietd.conf

Scrolt to the end and add the following:

Target iqn.2014-01.com.andreaszapf:MyISCSINode
       Lun 0 Path=/LUN/MyLUN0

Then save the file and exit Nano. Finally, restart the iSCSI Target Service:

service iscsitarget restart

You can use dmesg to check if there are any errors – and next, we need an iSCSI Initiator…

Adding the iSCSI Initiator

We have already installed open-iscsi which gives us access to iscsiadm – the core utility to manage the iSCSI Initiator. Use iscsiadm to discover the published iSCSI Target Nodes on a particular machine (portal):

iscsiadm --mode discovery --type sendtargets --portal 127.0.0.1

As you can guess, we are looking at our own machine using the local TCP/IP Loop-back, 127.0.0.1…

It will – if everything went OK – tell you that you have a target available to connect to by the name of iqn.2014-01.com.andreaszapf:MyISCSINode. Exactly the name you have given your target above. So let’s connect:

iscsiadm --mode node --target iqn.2014-01.com.andreaszapf:MyISCSINode
     --portal 127.0.0.1 --login

This, hopefully, gives you a little bit of satisfying feedback:

05 - iSCSI Node ConnectNow, keep in mind that we have said an iSCSI Target Node, once connected, is like a virtual harddisk – so let’s examine this, the tool of choice this time is a graphical tool in Ubuntu: Disks.

06 - Disks OutputSo we got ourselves a 4.3 GB harddisk as /dev/sdb with currently no contents. We can now create partitions in it and use it just like any other disk…

07 - Resulting DiskFor the moment, I would like to leave it as this – but in the next step, I will need to extend this disk to more than 4GB, hopefully without destroying data already placed within…

Posted in Linux, Ubuntu | Tagged | Leave a comment

Linux Discovery – Unity Desktop

Once Ubuntu is installed, I can start the system, log in and find myself on a Desktop.

01 - LauncherThe times when Linux was a shell-oriented system are long gone – like Windows is no longer living on a DOS Prompt. Although in both system, the Terminal Window – for those that need it – is still available.

Ubuntu’s Desktop is called “Unity”. And if you are used to Windows, don’t worry – it behaves much like the Windows Desktop does – you learning curve will be minimal.

On the left of the screen, you will find the Launcher. The Launcher is the equivalent to the Taskbar in Windows. Here, you will find Quicklist, a set of icons “pinned” to the Launcher – much like items pinned to the Taskbar in Windows. If you don’t want an icon there, just right-click it and select “Unlock from launcher” and it will be gone.

At the top of the Launcher, you will find the Dash – represented by the Ubuntu logo. The Dash is harder to compare to Windows – it is a mixture of the Start button, recent items and a Search… I will have to gain some experience using it but it seems to be pretty handy at this time.

The Dash will search your computer as well as online resources if you provide a search term – that way you can find local applications as well as a direct reference to website, e.g. Wikipedia.

So let’s search for the term “harddisk” via Dash – as my screen size is limited (for screenshot reasons) I also use the Dash Filters in the upper right corner.

02 - Dask in ActionThe result, following the filter criteria on the right – points me to a local application for disk management as well as to online reference sources such as Wikipedia.

The Dash comes in various flavours which can be switched using the symbols at the bottom.

03 - Dash Symbol BarThese symbols are called “lenses” and by using the different lens, you can display recently used and installed applications (2nd smybol), files & folders (3rd symbol), videos (4th symbol) and so on. It should not go unnoticed that the inclusion of external sources such as Amazon, eBay, Google, etc. is harshly disputed as a concern to loss of privacy.

The top of the screen is occupied by the Global Menu Bar (aka Top Menu Bar) and the Indicators on the right side. The Indicators pretty much compare to the Windows System Tray, the Menu Bar has no Windows equivalent.

To understand the Top Menu Bar, it is best to launch an application – let’s start FireFox with the symbol from the Quicklist.

04 - Top Menu Bar in ActionOnce FireFox is started, the Top Menu Bar is updated with menu command of the application – as opposed to Windows, the menu is not “within the application window” but assimilated into the Top Menu Bar. With two applications open, the Top Menu Bar always shows the menu of the active window.

This allows for another very cool feature of the Unity Desktop: the Head-up Display (HUD). By pressing and releasing the Alt-Key, you can bring up a search window that allows you to search for menu commands of the active application.

FireFox has a couple of Menu items related to page zooming – you can find them in the View menu in the Global Menu Bar.

06 - FireFox MenusYou can see Zoom, Zoom in, Zoom Out and Zoom Text Only. If you now press and release the Alt-Key and then type zoom into the search bar, you will see the following:

05 - HUD in ActionAnd you can select the menu item right from there – a very cool feature, I have to admit, for applications with large menu structures…

Finally, Unity supports a variety of keyboard shortcuts – you can display the most important ones by holding down the Superkey, on most (windows-oriented) keyboards the “Windows”-Key… ironically…

Now that should be sufficient to navigate the Unity Desktop and get things moving…

Posted in Linux, Ubuntu | Tagged | Leave a comment

Linux Discovery – Ubuntu

I have been a Windows guy for years (actually, I have been an MS-DOS guy before Windows was around) – and I never really looked in to Linux so far, mostly because I could not see a reason to exchange a “standard” operating system for another one and live in two worlds… but it is time to look beyond the horizon and discover the “other world” as well – partially because some things I own (Raspberry Pi, QNAP) operate on Linux-based environments, partially learning never stops.

I am lucky enough to have sufficient IT Know-How and gear around to make this discovery at no cost – or very little. I will mostly work on Virtual Machines (VMWare) but there is a physical PC available to load with Ubuntu as well (some things cannot be virtualized).

So let’s get started with sizing a system – Linux requires little with respect to hardware but I would like to define a “standard machine” for my testing, just to make sure results are comparable. Here is what I am thinking of (the Virtual Machine receiving the same dimensions as my physical box where possible):

  • Processor: the physical box has an Intel Core2Extreme X9650 running at 3 GHz. That is a four-core CPU so my virtual box will receive 1 Processor with four Cores as well – the host has an Intel i7-4770 with with four Cores that give 8 logical processors.
  • Memory: the physical box has 8 GB installed, the virtual box will receive 8 GB as well.
  • Harddisk: I will provide a first disk with 100GB to the Ubuntu Virtual Machine but I will add additional disks later to work with a multi-disk environment.

Other than that, the virtual machine will have one network card (set to NAT), USB 3.0 Controller and Sound Card.

Ubuntu Installation

I have already described the installation of an earlier version of Ubuntu – Version 12.04 LTS – in another post. Though this one was a server installation, my new box will be using the Ubuntu 13.10 Desktop Distribution – with the sizing of the machine, it will have to be the 64bit distribution.

Starting the virtual machine with the mounted ISO Image of the Ubuntu distribution brings up the Welcome screen after a few seconds.

01 - Welcome ScreenNote the option to try Ubuntu – certainly neat if you don’t have a virtual environment and don’t want to tamper with your PC just now. Me, I don’t mind… I just want to Install Ubuntu.

The next screen shows me the minimum requirements for the installation and allows me to automatically receive updates during the install process as well as download some 3rd Party software not included in the distribution – I activate both options.

02- Preparing to InstallMy options on the next screen are around disk management – I simply stick to the defaults for now which means that my disk will not be encrypted and I don’t look at LVM now (mostly because I want to discover it later…)

03 - Disk ManagementErasing the data on my disk is not an issue – it is a virtual environment anyway but if you do this on your default PC, be aware of the impact!

Before the installation actually starts, I additionally need to tell (or rather confirm) my time zone to the installer.

04 - Time Zone SelectionAnother important feature is the keyboard layout – although I am installing the english version, I need a German keyboard layout, otherwise I will have a bit of a fiddeling with special characters.

The option to detect the keyboard layout is pretty smart – by being asked to press specific keys, the installer figures out the keyboard layout itself – now that is user friendly!

05 - Keyboard LayoutFinally, before the installation can start, I also need to provide some information about my intended primary user which is created during the process.

06 - User AccountOnce more, the installer is pretty smart – it not only defaults certain values, it also provides imemdiate feedback on the correctness and quality of information.

Obviously, “the cloud” is something that Ubuntu cannot resist to support – like Microsoft Windows 8, I am asked to create an “Ubuntu One” account to centrally store certain information… I think, I will not make it that easy to certain institutions to get my data… so I skip this step.

07 - Ubuntu OneAfter clicking the “Log in later” option, my system actually installs.

08 - Installation ProcessIt only takes a few minutes to install – finally, I need to restart my virtual machine to boot the newly installed Ubuntu system.

Note: the automatic reboot would not work with my installation here – there is a chance that this is either caused by the virtual machine or me, tryint to take out the ISO image prematurely. A “reset” of the machine then booted Ubuntu properly.

09 - Login ScreenSo much for the installation – that was easy & straight forward. A very user friendly installer which makes me hope that using the system is as user friendly as this…

Posted in Linux, Ubuntu | Tagged | Leave a comment