Error: Failed to clone disk: The system cannot find the file specified (25) when using vmkfstools in Vmware 5.1

I have been working on setting up Apache VCL 2.3 with VMWare 5.1 and getting continual errors about not being able to find the file when VCL attempts to clone the disk.  I ran through a series of tests ensuring I could read and write to source and destination, but would get this error when trying the vmkfstools command by itself.  I finally found the solution on this blog.  A module needs to be enabled within VMWare to allow this type of clone to occur.  The command is:

esxcli system module load -m multiextent

3D Printing CT Scans on a Makerbot

Converting a DICOM CT scan to a 3D Stereolithography (.stl) file is a simple process that produces interesting results. These files can then be printed on a 3D printer such as the Makerbot Replicator 2x.  The DICOM files used in this demonstration were used with permission from Dr. L. M. Witmer.

To do the conversion you will need two pieces of software.  The first is InVesalius (http://svn.softwarepublico.gov.br/trac/invesalius) and the second is FreeCad.  Both are freely available under their respective licenses.
Begin by opening InVesalius and click "Import Medical Images".

Then browse to the folder containing your DICOM CT scan files and click OK


InVesalius will import the DICOM Files and display the scan when done.

Now select "Import" in the lower left hand corner.

You will now see a screen similar to the following:

Select "Create Surface" from the left hand task pane and wait a few minutes while it processes.

Once complete you will see a three dimensional rendering of the CT scans in the lower right panel.  You can spin and rotate this model to verify it rendered properly.

Finally, in the left hand task pane expand "4. Export Data" and choose Export 3D surface. Choose a filetype and filename and save.  Note for Makerbot's Makerware program, use the .STL file type.

This particular scan has some artifacts at the bottom that  cause a lot of wasted support structure to be printed on the Makerbot.  To remove these, launch FreeCad and open your file.
Select "Mesh Design" (1) from the drop down at the top.  Orient the model so that you can see the artifacts you would like to remove by using the various view buttons at the top (2).  Now click on your model and the "Cut with Picked Polygon" tool (3) will become available.



Select it and click the points around the object you want to remove.   Now, right click and choose "Inner" and the objects inside your polygon will be removed.

Finally select the "Export a mesh to a file" tool (4) and export your model back to an .STL file and you are ready to load your model into Makerware and print it.

Raspberry Pi work Picked up by the Raspberry Pi Foundation

The Raspberry Pi Foundation has picked up Dave Soriano's Chemical Ecology work.  The article and video can be seen here.  I have setup a Pi for him that connects to our wireless network and streams his work to Bambuser.com using some of the information I have made available here on my blog.  The Raspberry Pi is great for this work because Pitt's wireless infrastructure requires 802.1x authentication, which most web cams do not provide.  A wireless card and a little configuration with the wpa supplicant were a great solution.  Great work Dave!

Flipping Coins: A demonstration of the PERL MPI Libraries

The Parallel::MPI module for PERL allows a programmer to take advantage of the Message Passing Interface (MPI) to perform parallel programming operations.  While studying this library, I wrote the following simple routine to both udnerstand and demonstrate MPI functions within perl.  I present it here as a brief tutorial in the use of the Parallel::MPI module.  This script was run on a cluster of 2 to 32 Raspberry Pi's.

The basic premise is flipping a coin a million times.  Except instead of flipping a coin, I generate a random value between 0 and 1.  If the value is less than or equal to .5 is is a head, if larger it is a tail.  As a serial operation, this could take quite a while.  However, if we parallelize this process, each member will take a subset of the one million flips (one million devided by the number of nodes) and report back the its results when done.  The master node then tallies up the number of heads and tails reported by each node to produce a final result.  In a perfect world, this should be 50% heads and 50% tails, but as the random function is not completely random, these values will vary and we calculate an error based on that.


First we begin by loading the Parallel::MPI model and Sys::Hostname.  The latter allows us to retrieve the hostname of the node executing the code, which can be beneficial for debugging purposes.

#!/usr/bin/perl
# Initialize the Libraries
use Parallel::MPI qw(:all);
use Sys::Hostname;

 Next we initialize the MPI Library.

MPI_Init();

Now we initialize some variables.  MPI_COMM_WORLD is a communication string that all nodes use for communicating with one another.  This can be called whatever you would like, but MPI_COMM_WORLD is what is referenced in the documentation.  These variables are xecuted on each node and hold node specific information. 

• $numprocs holds the total number of nodes in the cluster.  
• $myid holds the id within the cluster starting with 0.  The last value of $myid will be 1 less than the number of nodes.  
• $name stores the node name.

# Get the number of nodes in the cluster
$numprocs = MPI_Comm_size(MPI_COMM_WORLD);
# Determine each members ID
$myid =     MPI_Comm_rank(MPI_COMM_WORLD);
# Determine each members hostname
$name = hostname;
print "Process $myid of $numprocs on $name\n";

Node 1 starts the timer using the MPI_Wtime function and is used later in combination with the end time to determine the total time required to process the problem.  This is helpful in demonstrating the benefits of adding a node to the cluster.  

if ($myid==0) {$startwtime = MPI_Wtime();}

Now we need to initialize some variables.

• $iterations holds the number of times we will flip the coin.
• $heads stores the number of heads.
• $tails stores the number of tails.

$iterations=1000000;
$heads=0;
$tails=0;

We now generate a counter ($i) that increments by the number of nodes ($numprocs) and increases until the number of iterations ($iterations) is reached.  What this means is that for a cluster of four computers, node 0 will start at 0 and at its next iteration $i will be 4, then 8, and so on.  Similarly node 1 will start at 1 and $i will be 5 at the second iteration, then 9 and so on.  Each node will continue this way until the value of $iterations (one million) is reached.  For each iteration, $x is assigned a random value between 0 and 1.  If $x is less then .5 the number of heads on this node is incremented by one.  If it is larger than or equal to .5 the number of tails is incremented by 1.

for ($i=$myid;$i<$iterations;$i+=$numprocs) {
 $x=rand();
 if ($x<=.5) {
  $heads=$heads+1;
 } else {$tails=$tails+1;}

}

For debugging purposes I print the Node id and number of heads and tails it computed.

print "Heads on $myid: $heads Tails on $myid: $tails\n\n";

Now check the ID of the node.  If this is not Node 0, we will populate an array (@info) with the total number of heads calculated by this nodes ($heads) as the first element of the array and the total number of tails ($tails) as the second.  The MPI_Send function is the utilized to send this information to Node 0.  The syntax is as follows: MPI_SEND(VALUE,Number of values,Data_type,Node to send to,Communication Tag, Communication label). So in this code I am sending @info as the value, @info contains 2 values, they are of MPI_INT data type, are being sent to Node 0, with a tag of 1, on the MPI_COMM_WORLD communication label.

if ($myid!=0) {
  @info[0]=$heads;
  @info[1]=$tails;
  MPI_Send(\@info,2,MPI_INT,0,1,MPI_COMM_WORLD);
} 

Now we set $headcnt equal to the number of heads calculated by Node 0 and $tailcnt equal to the number of tails calculated by Node 0.  These values will be added to the totals calculated by other nodes shortly. For Node 0 we set $x at 1 and increase $x by 1 until we reach the number of nodes in the cluster ($numprocs).  For each value of $x, Node 0 issues an MPI_Recv, to receive the data sent from the section of code above.  The syntax here is MPI_Recv(value, number of values, data type,node number,tag, communication label).  So below we are storing the received values in @info, are receiving 2 values, data type is MPI_INT, receiving from the node number currently stored in $x, tag is 1, and communication label is MPI_COMM_WORLD.  Once the values are received we add them to the current totals for $headcnt and $tailcnt respectively.

else {

    $headcnt=$heads;
    $tailcnt=$tails;
    for($x=1;$x<$numprocs;$x++) {
    MPI_Recv(\@info,2,MPI_INT,$x,1,MPI_COMM_WORLD);
    $headcnt=$headcnt+@info[0];
    $tailcnt=$tailcnt+@info[1];
  }

}

Now that the calculations have completed MPI_Wtime() is used to get a current timestamp.  This will be used to calculated elapsed time later on.  MPI_Barrier is called to halt processing until all nodes complete communication, and MPI_Finalize ends the MPI session.

$endwtime = MPI_Wtime();
MPI_Barrier(MPI_COMM_WORLD);
MPI_Finalize();

Now on Node 0, we calculate the elapsed time and store it in $time, calculate the percent error for heads and tails and the print the results.  As additional members are added to the cluster, the overall time to complete the processing will be seen to decrease.

if ($myid==0) {
  $time=$endwtime-$startwtime;
  $headerror=abs(($headcnt-$iterations/2)/$headcnt*100);
  $tailerror=abs(($tailcnt-$iterations/2)/$tailcnt*100);
  print "Heads Count: $headcnt Tails Count: $tailcnt\n";
  print "Head Error: $headerror% Tail Error: $tailerror%\n";
  print "Time: $time\n\n";}

The above script is calle dusing the mpiexec program via a command similar to the following:

mpiexec -f ~/mpi_testing/machinefile -np 2 ~/scripts/flip.pl 1

• ~/mpi_testing/machinefile is a list of ip addresses of all participating nodes.  One address per line.
• -np 2 states to use two nodes for the calculation
• ~/scripts/flip.pl is the path to the program to be run. 

Sample output can be seen below:

I hope this script helps to demonstrate and clarify some of the MPI functions and uses within the Parallel::MPI PERL module.

Solomon's words for the wise: PITT-BRADFORD STUDENTS CREATE SUPER COMPUTER OUT OF RASPBERRY PI

Below is an article about the Pi Super Computer I have been working on.

Solomon's words for the wise: PITT-BRADFORD STUDENTS CREATE SUPER COMPUTER OUT OF RASPBERRY PI

Raspberry Pi Thin Client for Apache VCL (Virtual Computing Lab)

Apache VCL is a broker for remote access to virtual computing resources.  The system allows a user to reserve a virtual computer and remotely access it via the RDP protocol.  When a reservation is made, the VCL server connects to a VMware server, configures a virtual machine based on the image selected by the user and assigns it to the user for the requested period.

Currently, students in the CIST program at Pitt-Bradford utilize their personal computers to access this virtual lab under a pilot program developed by Aaron Bryant.  With my current research into the Raspberry Pi and its uses, I wanted to see if it could be utilized as a thin client for the Apache VCL system.  I was not surprised to find that a thin client image for the Raspberry Pi was already underway at the Raspberry Pi Thin Client Project.  I have taken their image and modified it to produce a thin client kiosk allowing a student to log into the VCL web interface, reserve a machine, and access it all in about two minutes (including the minute it takes the VCL system to provision the virtual machine).  Once the remote desktop session to the reserved virtual machine is initiated, the experience is the same as if the session was started from a standard desktop computer.

I currently have USB redirection working and am working on printer redirection over the RDP protocol.  The major challenge I faced was getting the browser to automatically open the RDP file to initiate the connection.  I was unable to find a Remote Desktop application for linux that could be associated with and automatically open the file (if you know of one, please let me know).  Instead I wrote a PERL script to parse the RDP file and feed the necessary information to rdesktop and its various switches.  I ran into a bit of difficulty with the username and password, but finally after opening the file in a hex editor, I discovered there were non-printable characters at the end of each line and was able to strip them from the string using a regular expression for the hex values of these characters.

Once the image is finalized,  I will post full instructions along with PERL script to parse the RDP file.

Using Bash scripting to automate Network Administrative Tasks

Automating repetitive tasks is an essential skill for any Systems Administrator or Network Administrator.  The following script is a trick I have used for years to push configuration commands to network switches or other network devices that have a telnet interface.  (Yes, there is an assumed risk of the plain text telnet password.)  If you have multiple devices that need the same configuration, you can push these changes to them via a loop in a bash script.  This script accepts input from a text file and then echoes a series of commands to the network device.  The following is an example in which we need to push an SNMP community and access control list to a series of Cisco switches.

#!/bin/sh
cat switches.txt | while read SWITCH
do(
        sleep 1
        echo myusername   
        sleep 1
        echo P@55w0rd
        sleep 1
        echo en
        sleep 1
        echo P@ssword
        sleep 1
        echo config t
        sleep 1
        echo access-list 41 permit 10.10.10.1
        sleep 1
        echo access-list 41 permit 192.168.0.1
        sleep 1
        echo snmp-server community myCommunity RW 41
        sleep 1
        echo wr
) | telnet $SWITCH
done

First the script reads the switch IP address from a text file (the password could be read in too).  The script then uses the "echo" command to send the configuration to the telnet command.  You will see the first several commands initiate the log in and then invoke enable mode (en).  The sleep statements in between each command give the switch time to process the previous command before receiving the subsequent one.  Finally, the script then enters into the configuration terminal (config t) creates an access list with entries, and then assigns it to a read/write SNMP community.  The switches.txt file would have one IP address per line for each switch that would need this configuration.  If you wanted to read in other information like a password, just tab separate the data and another variable after SWITCH in the cat switches.txt... line.  I have also used this method to periodically reboot Blackboard Vending units as well as push configurations to them and to tftp switch configurations back to a storage server.

The above script should work on any Linux machine including a Raspberry Pi, which would make a very inexpensive server to run this script and other similar scripts.

Running a script as a service on Raspbian Linux

If you followed the instructions on Setting up a webcam on the Raspberry Pi, you may also want your feed to launch automatically.  The following process can also be used to run any script at boot after all other services have launched.

These instructions assume you have created broadcast.sh as instructed here.  To launch another script, just use appropriate path and script name.

rc.local script modified to launch broadcast.sh on boot

1. Open a linux terminal
2. sudo nano /etc/rc.local
3. Arrow down to the "fi" of the if statement that prints the address.
4. Hit enter and then the up arrow.
5. type "/home/pi/broadcast.sh&" without the quotes-- Note the Ampersand "&" is important.  This allows the rc.local script to launch your script and continue to booting without hanging and waiting for your script to complete.
6. press ctrl+x and hit "Y" to save.

Now your broadcast will start automatically when the Pi reboots.

Dr. Soriano has the live feed up and running.  The Pi webcam stream is monitoring his Chemical Ecology experiments with the cockroaches and they can be viewed here.

PERL and Mandelbrot Fractals

Mandelbrot Fractal created on a single Raspberry Pi

As I continue to research parallel computing methods on the raspberry pi cluster, I have been investigating fractal generation, specifically Mandelbrot fractals.  In the process, I found an example PERL script that generated a Mandelbrot fractal.  I wish I had the original link, but I have lost it.  I have rewritten it several times over in an attempt to both understand and simplify the process before I attempt parallelizing it.  You will notice in the image above that I chose to add random colors to the interior instead of making it a solid color.  The current phase of the code can be found here.  I will be working on adding the parallel components to it this week and will add a post with the code when it has been completed.

Migrating File Shares to a New Server

I recently needed to perform a physical to virtual migration (P2V) of  a file server.  I tried using my P2V software to accomplish this, but the virtual server would not boot after migration.  So the question became what do you do when its time to upgrade to a new file server or want to migrate to a virtual server without expensive P2V software.  I have used this method in the past when upgrading a physical file server and it served me well for this process as well.

If your file server is on a domain and all you need are the files, shares, and permissions this can be accomplished rather easily. This process should of course be performed during a scheduled downtime or maintenance windows and the new server needs to be configured with the same drive letters and file paths as the old server.  First, simply backup your files and restore them to the corresponding drive(s) on the new server.  I like to start this process a day or two before the actual migration.  I perform a full image backup of the shares and restore it to the new server ahead of time and follow up with a differential backup and restore at the time of migration. This reduces the time required to complete the migration dramatically.  When performing your backups, include file permissions and disk quotas where applicable.

Next you export two registry keys from the old server and import them into the new server. (This process can be done prior to migration as well as long as no new shares or permissions will be adjusted prior to migration). These keys are SYSTEM\CurrentControlSet\Services\LanmanServer\Shares and SYSTEM\CurrentControlSet\Services\LanmanServer\Shares\Security (more information here). 

LanmanServer\Shares Registry Key

Reboot the new server and voila your shares are up and running on the new server.

Finally during your scheduled maintenance window, shutdown the old server.  Then rename and readdress the new server to use the old servers information and you are done.  Alternatively if the the new server needs to have a different name and/or IP address, update your log in scripts to point to the new server.

Pi Cubed: Calculating Pi on 32 Pi's on Pi Day

Pi Calculated using the Monte Carlo Method and Visualized using GD

In celebration of Pi day I am posting my code for calculating Pi utilizing the MPI libraries in PERL.  This script calculates Pi utilizing the Monte Carlo method (further information here).  I have taken this common calculation and incorporated the GD module to visualize the results for the student. Each member performs a set of calculations and then creates a transparent image plotting the points it calculated.  Points within the circle are blue, those outside are red.  Each member then copies its image to the master where they are merged.  Examples of each members image can be seen below and the final image is posted above.  Code for this project can be found here.

Points plotted on Node27

Points Plotted on Node 30

Raspberry Pi Cam

 

Here is another Raspberry Pi project that I have been working on. Dave Soriano will be using it to conduct Chemical Ecology experiments testing attractants on cockroaches. The pi is in the red case on the left and connected to a webcam overtop of the aquarium. An attractant will be placed in the base and holes have been drilled in the bottom to allow vapors to pass through to the cockroach.

The Pi is setup to stream to bambuser.com using a webcam and ffmpeg. For anyone interested in streaming from a Pi, instructions can be found here.