Thursday, November 25, 2010

A Crumb for Later: ffmpeg runes to generate an animation

I found a faded post-it note from a friend who gave me the following incantation to convert a set of still images into an MPEG animation using ffmpeg. This note is a crumb for later investigation.

ffmpeg -r1 -f image2 -i <input> -vcodec mpg2video \
 -r 30000/1001 -b 25000k <outMovie.mpg>

If memory serves, the 30000/1001 may need to be changed to the decimal value, but I'll see if ffmpeg will parse and calculate on its own ;-)

Wednesday, November 24, 2010

Easily moving pictures from camera to hard drive

Getting files from the camera to hard drive is sometimes accompanied by confusion and consternation when custom actions are needed. Given powerful tools and a bit of facility with scripting, retrieving the files can be as simple as

movePics.sh <fromDir> <toDir>

A special love note for my better half. For the old camera disk use

/DATA01/bin/movePics.sh /media/disk/DCIM/101MSDCF /DATA01/Pictures 

For the new camera disk 1 use

/DATA01/bin/movePics.sh /media/1A48-D026/DCIM/100D7000 /DATA01/Pictures

Love, your system administrator.

The heart of this solution is the perl exiftool (http://owl.phy.queensu.ca/~phil/exiftool/) that can interrogate, modify, and otherwise act on tags in image files. This article describes an application of the renaming functionality (http://owl.phy.queensu.ca/~phil/exiftool/filename.html). Specifically, when moving the files from the camera media (e.g. /media/1A48-D026/DCIM/100D7000) to the root of an images directory (e.g. /DATA01/Pictures) the requirements are:

  1. Separate pictures into directories by date
  2. Format of the directories are year/month/year-month-day where the date is given in digits only
  3. Create directories if they do not already exist
  4. Ensure that the extensions are lower case
    • The need for this was driven by the use of some web applications that do not display files with uppercase extensions
  5. The date information shall come from the original date and time in the image header

All these requirements are met in a single invocation

exiftool '-FileName<DateTimeOriginal' \
 -d /DATA01/Pictures/%Y/%m/%Y-%m-%d/%%f.%%le \
 /media/1A48-D026/DCIM/100D7000

Two potentially tricky bits are the double percent (%%) for the file and extension formats (%%f and %%e respectively), and the lowercase modifier for the extension (%%le). The need for quotes around the first argument is because the exiftool syntax < would otherwise be interpreted by the shell instead of being sent to the script.

While the invocation is not terribly difficult, there is enough typing to make it worthwhile for a simple script (e.g. movePics.sh). A minimal implementation that takes two arguments — a from and to directory — is written here in bash.

#!/bin/sh
srcdir=$1
tgtdir=$2
exiftool '-FileName<DateTimeOriginal' \
 -d $tgtdir/%Y/%m/%Y-%m-%d/%%f.%%le $srcdir

Wednesday, September 15, 2010

Could Richard Riley have solved my problem?

2010-09-08 Wed 18:55

Today on the orgmode mailing list Richard Riley posted a function that works with the Google command line tools (http://code.google.com/p/googlecl/) to create Blogger entries directly from my favorite emacs mode, orgmode. I have been looking for just such a solution to make blogging more transparent, and lo it is here. This post is the first test of the method, and it works. (Of course, I'm writing this before actually testing it, but since the new function evaluated without error then it must be right! ;-)

To be sure, there were quite a few steps to get to this point. Here was the recipe.

  1. Sign up for a gmail account at http://gmail.google.com
    • While I already had a Google account for Blogger, the username (email address for my local service provider) seemed to be causing problems while authenticating (OAUTH) with googlecl.
  2. Install python-gdata version 1.2.4 or higher
    sudo apt-get install python-gdata
    
  3. Download googlecl Debian package (googlecl_0.9.10-1_all.deb) from http://code.google.com/p/googlecl/downloads/list
  4. Install googlecl
    sudo dpkg --install googlecl_0.9.10-1_all.deb 
    
  5. Call the blogger service to initiate authentication
    google blogger list
    
    • Enter the gmail account (e.g. myname@gmail.com) at the prompt "Please specify user"
    • Grant authorization in a browser
      • The browser may automatically launch with the URL
      • If not, then the link is provided in the terminal
    • Press return in the terminal
      • If there are error messages, then something probably went wrong :-)
      • If a list of blog entries appears then authentication is accomplished
  6. Send this blog post from orgmode

At present org-googlecl is a project at http://github.com/rileyrg/org-googlecl. Thank you Richard.

Tuesday, September 14, 2010

Getting org-googlecl

2010-09-14 Tue 21:14

A few days ago I wrote about how to set up googlecl in order to use Richard Riley's wonderful addition to emacs called org-googlecl. Tragically I did not even hint at how to get it. This post gives that hint, but assumes that git is already installed on the system and that the user knows how to configure emacs to use code in non-standard locations.

  1. Change to the directory for the installation (e.g. ~/prj)
    cd ~/prj
    
  2. Clone the repository
    git clone http://github.com/rileyrg/org-googlecl.git
    
  3. Add the new directory (~/prj/org-googlecl) to the emacs path by putting something like the following in a configuration file
    (add-to-list 'load-path "~/prj/org-googlecl")
    
  4. Load the library in emacs
    (load-library "org-googlecl")
    
  5. Set the configuration items for the blog
    (setq googlecl-blogname "Natural Log of X")
    (setq googlecl-username "<GoogleUserName@gmail.com>")
    
  6. After writing an entry in orgmode, publish the entry (defined by the outline level) to the blog
    M-x org-googlecl-blog RET
    
  7. Keep current with org-googlecl by updating the source
    cd ~/prj/org-googlecl
    git pull
    

Sunday, September 12, 2010

Rearranging output file from a PGM image conversion.

2010-09-12 Sun 11:44

convert ex-e_g.jpg[32x32+8+5] \
 -colorspace gray \
 -transpose \
 -compress none \
 -depth 8 PGM:- |\
  tail -n +4 |\
  tr -s ' \012' '\012' |\
  pr -T -s -32 -l32 \
  > ex-e_g_crop32x32.txt 

The convert command is from ImageMagick. The [32x32+0+0] crops as the image is read. The -colorspace Gray ensures that the image will be grayscale. The -compress none signals to use plain ASCII rather than a binary format. PGM output (to STDOUT) from 0 to 255 is ensured with -depth 8 PGM:-. The PGM header is removed with tail -n +4 and a single column of numbers is generated with the tr command.

The workhorse for the reformat is the pr tool which prepares file for printing. The -a puts columns across which is necessary for this flow to obtain the proper orientation. (This could have also been done with the -transpose switch to convert.) The -T ensures that no headers or footers are used, and that no page breaks are inserted. The -s switch puts tabs between values. The number of columns is specified with -32 and the number of lines with -l32.

Take note that initial tests without the -s command yielded files where the numbers were incorrect. This seems to have been due to something related to the width being too narrow, but the -s set the width to 512. The pr command does have the --width switch, but it is currently unclear how to calculate the exact width needed for the particular crop size.

Friday, September 10, 2010

How do I rename multiple files efficiently using only the bash shell?

2010-09-10 Fri 17:36

As a concrete example, let us consider the problem that we often have uploading pictures to certain sites (e.g. ebay). The photos from our camera all have .JPG extension. However, when we attempt to locate those pictures from some web sites, no photos are found. This is because something in the chain of events only lower case extensions, such as .jpg, are allowed.

The task is to change all files in a directory (e.g. /DATA01/Pictures/EBAY/2010-09-10) from upper case (.JPG) to lower case (.jpg) extensions.

cd /DATA01/Pictures/EBAY/2010-09-10
for i in *.JPG; do mv "$i" "${i/.JPG}".jpg; done

Simple.

There are certainly other ways, but this is good enough for now.

Weather for emacs and orgmode

On 09 Sep 2010 Julien Danjou provided a way to access the Google weather API from emacs. Additionally, the weather can easily be integrated into an orgmode agenda.

cd ~/Projects/git/
git clone git://git.naquadah.org/google-weather-el.git

Then we set up emacs in the following way

(add-to-list 'load-path "~/Projects/git/google-weather-el")
(require 'google-weather)
(require 'org-google-weather)

To add the weather to the agenda, put a call to org-google-weather with the city (e.g. "Webster, NY") or postal code (e.g. "14580") in an org file that is used by the agenda. Multiple weather locations can be included and the #+CATEGORY option can be used to distinguish them in the agenda.

#+CATEGORY: Webster, NY
%%(org-google-weather "Webster, NY")
#+CATEGORY: Washington, DC
%%(org-google-weather "Washington, DC")

The Google weather API is apparently very well undocumented (http://blog.programmableweb.com/2010/02/08/googles-secret-weather-api/), but the following call will retrieve an XML response.

http://www.google.com/ig/api?weather=14580

Other APIs that might be of interest are stock quotes

http://www.google.com/ig/api?stock=YHOO

movie listings

http://www.google.com/ig/api?movies=14580

and a news feed

http://www.google.com/ig/api?news

Neat stuff.

Thursday, September 9, 2010

Testing a new function from RR

2010-09-09 Thu 06:18

In terminal

cd ~/Projects/git/
git clone http://github.com/rileyrg/org-googlecl.git

In emacs

M-x load-file ~/Projects/git/org-googlecl/org-googlecl.el
M-x set-variable RET org-googlecl-blogname RET "Natural Log of X"
M-x set-variable RET org-googlecl-username RET "<well, you know>"

Write this entry in orgmode using emacs then call

M-x org-googlecl-blog

Wednesday, September 8, 2010

Could Richard Riley have solved my problem?

2010-09-08 Wed 18:55

Today on the orgmode mailing list Richard Riley posted a function that works with the Google command line tools (http://code.google.com/p/googlecl/) to create Blogger entries directly from my favorite emacs mode, orgmode. (See Getting org-googlecl for more information.) I have been looking for just such a solution to make blogging more transparent, and lo it is here. This post is the first test of the method, and it works. (Of course, I'm writing this before actually testing it, but since the new function evaluated without error then it must be right! ;-)

To be sure, there were quite a few steps to get to this point. Here was the recipe.

  1. Sign up for a gmail account at http://gmail.google.com
    • While I already had a Google account for Blogger, the username (email address for my local service provider) seemed to be causing problems while authenticating (OAUTH) with googlecl.
  2. Install python-gdata version 1.2.4 or higher
    sudo apt-get install python-gdata
    
  3. Download googlecl Debian package (googlecl_0.9.10-1_all.deb) from http://code.google.com/p/googlecl/downloads/list
  4. Install googlecl
    sudo dpkg --install googlecl_0.9.10-1_all.deb 
    
  5. Call the blogger service to initiate authentication
    google blogger list
    
    • Enter the gmail account (e.g. myname@gmail.com) at the prompt "Please specify user"
    • Grant authorization in a browser
      • The browser may automatically launch with the URL
      • If not, then the link is provided in the terminal
    • Press return in the terminal
      • If there are error messages, then something probably went wrong :-)
      • If a list of blog entries appears then authentication is accomplished
  6. Send this blog post from orgmode

Wednesday, July 28, 2010

Linebreaks, Blogger, and org-mode

These posts are written using org-mode for emacs. Generating HTML output is accomplished by highlighting the region and issuing the command C-u C-c C-e R. [Note: Without the prefix argument (C-u) the HTML will have all the necessary elements (e.g. <html>, =<body>) to be a standalone web page which is not what is needed for Blogger posts.] After signing into Blogger and starting a new post, the exported HTML is copied from the file (again in emacs) and pasted into the "Edit HTML" tab.

Initially I had problems with linebreaks being explicitly added by Blogger which made crummy looking posts. I recall doing two things to fix the problem, both in Blogger itself not emacs or org-mode. First, in the Settings | Formatting tab I changed the entry Convert Line Breaks to "No". Second, in the Posting | New Post tab then Post Options (under the editing region), I ensured that Use <br /> tags was selected.

Wednesday, July 21, 2010

Not easy to use and by default not secure

I read something funny on the Mac website; "A Mac is easy to use, powerful, compatible, and highly secure." My experience setting up Mac to connect with a network file space was not easy because its defaults are not secure.

Our pictures are located on a disk that has been shared for years by all of our machines. While investigating the Mac as a potential new machine, I asked several Apple store Specialists and Geniuses (capitalization from their website, but I like the irony) about connecting to network drives on a Linux system. Most did not even know what I was talking about, but finally the manager was able to provide some assistance. After a few button clicks there appeared a window that allowed for NFS. This was through Finder | Go | Connect to Server ... and the syntax seemed reasonable.

Unfortunately it did not work on my home system. Over the course of several months I would troubleshoot a bit here and there without success. Of course during that time the Macbook was an island unto itself concerning the shared pictures and other data.

The solution came only when I set aside several consecutive hours for research and experimentation. The first discovery was an application called Disk Utility and found a promising entry, File | NFS Mounts .... In this utility there is the ability to exert a measure of control for the mount versus using the Connect to Server ... method. Finding the Advanced Mount Parameters input held promise but no help, because the root problem remained unclear.

More research revealed that Mac OS X, which is based on BSD, uses an insecure port. For an example see http://blogs.techrepublic.com.com/mac/?p=430 or http://www.unixtutorial.org/2010/03/mounting-nfs-shares-on-mac-os-x/ for a few details. Both articles describe one potential way to get connected; change the server side to share over an insecure port. While this is probably a viable solution, I shun the idea of purposely doing something insecurely as the final solution. Of course, I hoped there might have been another solution.

The same articles also introduce the -resvport switch that worked from the command line. This solution is more to my liking because it changes the client to use a secure port. However, this did not work when used in the Disk Utility. (Note: The first article I found did not have an example of using the switch in Disk Utility, so used the dash. However, after those attempts I found a second article that had an example in the Disk Utility that did not use the dash. It is possible that resvport may work, but I have not returned to that path.)

Since that did not work I continued the investigation and found the possibility that the -P switch might work. Indeed it did. Putting -P in the Advanced Mount Parameters of Disk Utility allows the Mac to connect to the external data.

It was by no means easy and the default behavior is insecure, contrary to the advertisements.

Saturday, March 27, 2010

Replacing Misplaced Hope with a Backup System: Initial Configuration of BackupPC

Intermittent file copies, occasional DVD writes, and misplaced hope are what constituted the backup system for our home network. Two days ago that changed when I successfully configured BackupPC.

BackupPC (http://backuppc.sourceforge.net/) is an application that creates and manages backups to disk, as opposed to tapes or other media. Assuming that BackupPC is installed on the server (see for example the installation instructions) and enough disk space is available, the next step is configuration.

Most of the configuration will be accomplished using the web interface. In case there is a need to change the login and password for the BackupPC user (e.g. backuppc), then make the modifications as an administrative user. Invoking the htpasswd command as follows will prompt for a new password for backuppc (after, of course, asking for the sudo password).

sudo htpasswd /etc/backuppc/htpasswd backuppc

At this point the web interface at http://localhost/backuppc should be available using the loging and password. If not, then there are some installation and system issues that must be resolved before the remainder of this article is applicable.

During the initial configuration I used the BackupPC SSH FAQ and was a bit confused. To be sure, the first half of this article distills the steps to generate and exchange keys for the rsync method in the hopes of reducing that confusion in the future. The remainder points out a few configuration items that were accomplished using the web interface.

Create keys on BackupPC host (e.g. nitrogen) and send the public one to the remote machine (e.g. carbon).

  1. Login as the BackupPC user (e.g. backuppc).
    su - backuppc
    
  2. Change to the secure shell directory (e.g. ~backuppc/.ssh).
    cd ~backuppc/.ssh
    
  3. Create private (e.g. id_rsa) and public (e.g. id_rsa.pub) keys for the local host.
    ssh-keygen -t rsa
    
  4. Copy the public key to a local file with a clearly-identifiable name (e.g. BackupPC_id_rsa.pub).
    cp id_rsa.pub BackupPC_id_rsa.pub
    
  5. Limit file permissions.
    chmod 600 id_rsa id_rsa.pub BackupPC_id_rsa.pub
    
  6. Copy the public key to the remote host into the root user's ssh area (e.g. ~root/.ssh).
    scp BackupPC_id_rsa.pub root@carbon:~root/.ssh
    

Create keys on the remote host send the public one to the BackupPC host, and add the BackupPC's public key to list of authorized keys.

  1. Login as an administrative user (e.g. pwrusr).
    su - pwrusr
    
  2. Create private and public keys for the local host.
    sudo ssh-keygen -t rsa -f ~root/.ssh/id_rsa
    
  3. Copy the public key to a local file with a clearly-identifiable name (e.g. carbon_id_rsa.pub).
    sudo cp ~root/.ssh/id_rsa.pub ~root/.ssh/carbon_id_rsa.pub
    
  4. Limit file permissions.
    chmod 600 id_rsa id_rsa.pub carbon_id_rsa.pub
    
  5. Copy the public key to the BackupPC host (e.g. nitrogen) into the root user's ssh area (e.g. ~root/.ssh).
    sudo scp ~root/.ssh/carbon_id_rsa.pub root@nitrogen:~root/.ssh
    
  6. Append the public key for the BackupPC to root's authorized keys file (e.g. ~root/.ssh/authorized_keys2).
    sudo touch ~root/.ssh/authorized_keys2
    sudo cat ~root/.ssh/BackupPC_id_rsa.pub >> ~root/.ssh/authorized_keys2 
    

Add the remote host's public key to the list of known hosts then test the connection.

  1. Login as the BackupPC user.
    su - backuppc
    
  2. Change to the secure shell directory (e.g. ~backuppc/.ssh).
    cd ~backuppc/.ssh
    
  3. Append the public key for the remote host to the list of known hosts (e.g. known_hosts).
    touch known_hosts
    cat carbon_id_rsa.pub >> known_hosts
    
  4. Ensure that the BackupPC user can connect as root to the remote machine without the need for a password. The test below should return the string root. Note: The first time this connection is made the password may need to be entered, but subsequent logins should not request a password.
    ssh root@carbon whoami
    ssh root@carbon whoami
    

Repeat the key generation and exchange for all hosts that BackupPC will be serving.

Now that the machines can communicate securely, the backups themselves need to be defined. Log into the web interface (e.g. http://localhost/backuppc). To declare the machines to be backed up, choose the Edit Hosts link in the table of contents area. Under the Hosts tab press Add and provide the host name to be backed up (e.g. carbon), the normal user of that host (e.g. ksburt). If the BackupPC server is currently a host that needs to be backed up, it might be wise to add a host with the explicit name (e.g. nitrogen) and its normal user (e.g. tcburt). This choice is driven less by necessity (since localhost is a default host), and more by the desire to prepare for the future. Additionally, the key generation and exchange described above needs to be done. Be sure to press the Save button.

Next select the Xfer tab.

  • Change XferMethod to rsync.
  • Change RsyncShareName to be /home.
  • Choose the Add button to add /etc.
  • Choose the Add button to add other default directories.
    • Caution: If the root filesystem is chosen for backup, then BackupFilesExclude should be configured to exclude certain directories (e.g. /proc).
  • Press the Save button.

Return to the table of contents area and choose a particular host to configure (e.g. nitrogen) by using the Select a host... dropdown. This creates a new section in the table of contents that is specific to the host. Choose the Edit Config link under the host-specific section. Ensure that XferMethod is rsync and that the default directories (e.g. /home and /etc) are in RsyncShareName. Let us consider the situation where this host has a filesystem (e.g. /DATA01) that is shared to the network. Since directories on this disk (e.g. /DATA01/Music, /DATA01/Pictures) should be backed up only once, add them to the RsyncShareName only on this host. Save the configuration. Repeat the host-specific configuration for each host to be backed up.

An immediate test can be run. First, select the host's homepage from the table of contents. Ensure that the current time is not in a blackout period (see the Schedule configuration tab). Press the Start Full Backup button and confirm, then the homepage should return. Wait a few seconds and refresh the page to see if any errors are reported.

Once BackupPC is working, enjoy the peace of mind that comes with having a history of your data stored regularly. Of course, the next step is to test whether you can actually recover the data.

Thursday, March 18, 2010

Listing only directories

The ls command lists contents of a directory. A common desire is to see only contents that are themselves directories. Minor facility with other commands help ls achieve the goal.

First let us set up a concrete example. Suppose that there is a directory called ~/tmp that contains both files and directories, 98 to be exact.

~/tmp$ ls -l | wc -l
98

To winnow out files one must be able to distinguish a directory from a file. A simple solution is to use the -d switch of ls (thanks go to Jeffrey for this method).

~/tmp$ ls -d */
aa/  bin/  CYI/  ImageJ/  LaTeX/             Picasa-10091104/  pst-pdf/
av/  cfg/  ek/   kut/     p011-ursys-LaTeX/  pst/              sv/

The listing is nice and tight. If a line-oriented output is required, the -1 (that's a number 1) switch achieve the desired result.

In the remainder of this article we describe how to use other Unix tools in concert with ls to solve the problem. The -F switch appends an indicator to the names of certain types of content, and in particular a / is placed at the end of a directory name. The grep command can now be used to search for / at the end of the line.

~/tmp$ ls -F | grep '/$'
aa/
av/
bin/
cfg/
CYI/
ek/
ImageJ/
kut/
LaTeX/
p011-ursys-LaTeX/
Picasa-10091104/
pst/
pst-pdf/
sv/

Another way to distinguish a directory is to use the fact that the letter d is the first character in a long listing using the -l switch. Again grep can be used to select only lines that begin with d.

~/tmp$ ls -l | grep '^d'
drwxr-xr-x  4 tcburt tcburt     4096 2009-10-21 05:45 aa
drwxr-xr-x  4 tcburt tcburt     4096 2009-11-05 06:06 av
drwxr-xr-x  2 tcburt tcburt     4096 2009-03-15 14:10 bin
drwxr-xr-x  5 tcburt tcburt     4096 2009-03-15 14:10 cfg
drwxr-xr-x 11 tcburt tcburt     4096 2010-03-17 06:36 CYI
drwxr-xr-x  2 tcburt tcburt     4096 2009-07-12 18:50 ek
drwx------  6 tcburt tcburt     4096 2009-10-15 22:29 ImageJ
drwxr-xr-x  2 tcburt tcburt     4096 2009-12-10 20:47 kut
drwxr-xr-x  3 tcburt tcburt     4096 2009-06-14 17:51 LaTeX
drwxr-xr-x  2 tcburt tcburt     4096 2009-09-13 22:30 p011-ursys-LaTeX
drwxr-xr-x  4 tcburt tcburt     4096 2009-11-04 06:17 Picasa-10091104
drwxr-xr-x  2 tcburt tcburt     4096 2009-12-23 11:51 pst
drwxrwxr-x  3 tcburt tcburt     4096 2009-12-23 11:41 pst-pdf
drwxr-xr-x  4 tcburt tcburt     4096 2009-11-05 06:02 sv

The methods above satisfy the requirement of identifying only directories. However, they also result in additional characters besides just the names. The ls -F method lends itself to simply deleting the / character with the help of the tr command.

~/tmp$ ls -F | grep '/$' | tr -d /
aa
av
bin
cfg
CYI
ek
ImageJ
kut
LaTeX
p011-ursys-LaTeX
Picasa-10091104
pst
pst-pdf
sv

The pathological case where a / is part of the file name will not be served by the tr -d / invocation.

Recall that the long listing example did not leave a trailing slash on the directory name, it did prepend detailed information. For this case one needs to obtain only the final column which is the name itself. The perl command can split each line and print only the final column.

~/tmp$ ls -l | grep '^d' | perl -lane 'print "$F[-1]"'
aa
av
bin
cfg
CYI
ek
ImageJ
kut
LaTeX
p011-ursys-LaTeX
Picasa-10091104
pst
pst-pdf
sv

If listing only directories is a common task, it will be wise to define an alias for a favored method.

Sunday, February 21, 2010

Making a Ten-year Old Computer Useful with Linux

Even a computer that is nearly ten years old can be made useful again. A certain computer in my house has remained untouched for about five years, but today it found new life with Linux.

In May of 2005 we replaced one of our computers that had been in service for exactly five years. The system specifications were tired even then: 750MHz AMD Athlon processor, 256MB RAM, 20GB harddrive. It had been running Windows 98 when decommissioned. It was stored, moved to a new state, and stored some more.

Yesterday I continued working on the home network infrastructure and looked for a potential fileserver. I found this decade old machine, gave it power and peripherals, then booted it up. Slow as the system was, I was able to navigate the files and actually found over two years of digital photographs that were presumably lost on CDs that were unreadable.

After the recovery came the rejuvenation. Since this machine is to be a network workhorse it needed a reliable operating system with powerful tools readily available. It should be no surprise that I chose Linux, specifically the Ubuntu 9.10 distribution. Even on this old system the installation proceeded without problem, and the subsequent package updates finished with little problem.

Previous experience indicates that this would have been more costly and less flexible using Windows. Indeed, it would have been impossible to use the latest version of Windows (see http://windows.microsoft.com/systemrequirements) because the motherboard doesn't even support the RAM requirement.

While the system will need a RAM upgrade and a card to accept firewire connections to external hard drives, the machine will likely be one of the silent, steady sentinels in our home network.

For the technorati …

While gathering information about this system I discovered the dmidecode command. This tool needs to be run with administrator privileges. With it I was able to discover the CPU speed, CPU maker, RAM information, and even the build date.

sudo dmidecode -t processor
sudo dmidecode -t system
sudo dmidecode -t memory

An old article (http://www.linux.com/archive/feed/40412) describes the information from dmidecode as not completely reliable, but useful. I found it useful for my purposes.

Thursday, February 18, 2010

Fixing the monitor's low resolution after Ubuntu 9.10 installation

During a fresh installation of Ubuntu 9.10 (karmic) on a Dell Dimension 3000 standard system, the monitor resolution could no longer support any mode higher than 800x640. An initial twinge of frustration was coupled with confidence that the problem was tractable. This is the short story of how the Ubuntu and open source communities helped the solution.

Historically, whenever I encounter display problems I look at the xorg.conf file which is usually found in /etc/X11. On this installation day I was surprised to discover that xorg.conf was not in its usual place. Indeed, my surprise became something unameable when a search of the entire filesystem revealed that the file did not exist. While it is possible that a step or two after installation could have removed the file, the fact remains that some high-level procedure leaves me without an xorg.conf.

It was tempting to search out what mechanisms have replaced xorg.conf and to understand the advertised improvements of the new way, but in the short term I just wanted better resolution. Somewhere within the first few tens of minutes I found what turned out to be a solution, but I initially hoped for a simpler way. I learned a valuable lesson in this dismissal — read and understand before trading away over an hour searching for a presumed simpler way.

Searching the Ubuntu wiki revealed the article titled Reverting the Jaunty Xorg intel driver to 2.4 and my initial reaction was that my system (karmic) was newer than the one in the article (jaunty). Even so I saw the seven steps, four explicit and three others potential, and thought this might work. However, two of the possible steps (obtaining a validation key) were something I remembered doing before with some trouble. The other possible step, related to the key, was opening a port on my firewall. This final step is the one that sent me looking elsewhere, not because I didn't want to make the change but because I sought something that smacked less of system administration to the normal home user. In the end, neither of the potential steps were necessary.

The four steps to solving the resolution took less than 4 minutes to accomplish.

  1. Add the following two lines to the bottom of /etc/apt/sources.list (I used sudo vi /etc/apt/sources.list to do the edits; use your favorite editor as an administrator)
    deb http://ppa.launchpad.net/siretart/ppa/ubuntu jaunty main
    deb-src http://ppa.launchpad.net/siretart/ppa/ubuntu jaunty main
    
    • NOTE: I did indeed use 'jaunty' rather than 'karmic'.
  2. Update package list
    sudo apt-get update
    
  3. Install the xserver-xorg-video-intel-2.4 package
    sudo apt-get install xserver-xorg-video-intel-2.4
    
    • NOTE: A warning appeared that "packages cannot be authenticated!", but I was able to install "without verification." This is related to my inability to add the validation key and is not my preferred mode of operation. Even so, the key was not a necessary step in the present solution.
  4. Restart the display
    sudo /etc/init.d/gdm restart
    

The display immediately went to a much more reasonable resolution. Problem solved.

Saturday, February 13, 2010

Searching the contents of text files

EXECUTIVE SUMMARY

Search for the string 'Recipe' in all files that have the .org or .html extension anywhere in the current directory or below, ensuring that the filename is prepended to all matches:

> grep -e 'Recipe' `find .  \( -name "*.org" -o -name "*.html" \)` /dev/null

Same as above except all non-binary files are searched:

grep -HIre 'Recipe' *

SUPPORTING JABBER

Consider the situation where you have many text files in a certain directory tree and you want to discover which files have particular content. Here we discuss the use of grep and find to help solve this problem. Modern versions of grep remove the need to use find, and we will discuss that method after the one applicable to more disadvantaged systems.

The grep command is used to search the contents of files. A familiar output is to have the filename prepended to the line that matches the search, for example

> grep -e 'ground' *
photos.org:   background.  I also had the privilege of seeing the physical
photos.org:   ground on the night of the 27th.  On the morning of the 28th there
Quotes.org:going to take a lovely, simple melody and drive it into the ground. --

It is tempting to interpret the prepended filename as the overall default. However, whether the filename appears or not depends also on the context in which grep was used. Specifically, when grep is provided a single file to search through the filename is not prepended,

> grep -e 'ground' photos.org
   background.  I also had the privilege of seeing the physical
   ground on the night of the 27th.  On the morning of the 28th there

This is a reasonable behavior from the perspective of grep since only a single file was given there should be no doubt what file contained the match. As we will see below there are times when grep may be provided a single file but the user does not know what that file is. In these cases we want to force the filename to be identified. One way to do this is to pass grep the real file and one other file that has the following property; its contents will never match the search expression, for example /dev/null. Witness the difference,

> grep -e 'ground' photos.org /dev/null
photos.org:   background.  I also had the privilege of seeing the physical
photos.org:   ground on the night of the 27th.  On the morning of the 28th there

Before continuing there are two observations to be made about the grep invocations above. First, and almost as an aside, the calls could have been written just a bit more simply by dropping the -e switch and the quote marks. However, this construct allows for more complex search expressions. An example is to find either the word 'ground' or the word 'Recipe' in any files,

> grep -e 'ground\|Recipe' *
photos.org:   background.  I also had the privilege of seeing the physical
photos.org:   ground on the night of the 27th.  On the morning of the 28th there
Quotes.org:going to take a lovely, simple melody and drive it into the ground. --
Recipes.org:#+TITLE: Recipes
sitemap.org:   + [[file:Recipes.org][Recipes]]

The observation that pertains directly to the problem at hand is that the list of files for grep to search must be specified somehow. If all the files are in the same directory, then a simple wildcard expression might be all that is needed. However, sometimes the search is to be done recursively or across several directories.

The find command is useful for finding files on the system with particular characteristics. As an example, the following expression finds all files in the current directory and below that have either a .org or .html extension,

> find .  \( -name "*.org" -o -name "*.html" \)
backcountry/photos.html
backcountry/readme.html
backcountry/maintenance.html
backcountry/sitemap.html
backcountry/index.html 
 [--snip--]
templates/rketburt-01-Level00.org
templates/rketburt-01-Level01.org
 [--snip--]

Be aware, the space after the \( and before the \) proved to be vital while testing commands for this article. I am unaware if this is a general necessity or just on my particular system.

Now it is a simple matter to search the contents of multiple files. We build the file list using find embedded in backticks (`) to capture the result, then invoke grep on that list. Here is a complete example,

> grep -e 'Recipe' `find .  \( -name "*.org" -o -name "*.html" \)` /dev/null
rketburt-org/Recipes.org:#+TITLE: Recipes
rketburt-org/sitemap.org:   + [[file:Recipes.org][Recipes]]
rketburt/sitemap.html:<a href="Recipes.html">Recipes</a>
rketburt/Recipes.html:<title>Recipes</title>
rketburt/Recipes.html:<h1 class="title">Recipes</h1>
rketburt/index.html:<a href="Recipes.html">Recipes</a>

Note the use of /dev/null as a file argument to grep to ensure that the filename is prepended.

Another way to effect the same final result is to invoke find first and use the -exec argument to call grep. In this ordering grep is only provided with a single file which leads to the lack of filename problem indicated earlier. The overall syntax is a bit more cumbersome as well, since {} is used to pass the result of find to grep and there is the trailing \; as well. An equivalent example to the one in the previous paragraph is

> find .  \( -name "*.org" -o -name "*.html" \) -exec grep -e 'Recipe' {} /dev/null \;

Syntax or preferences aside, it is interesting to note that while these two examples provided the same end result, the one that begins with grep executed nearly 10 times faster.

The find command has been used above for two reasons. First, the desire was to search files that may appear in directories below the one called out. In other words we desired a recursive search. The second reason was to eliminate the prospect of searching non-text files which would have simply been a time sink. The method to exclude the binary files was to limit the file extensions to just two (.org and .html). This may be the exact behavior desired for some questions, but may be too restrictive for others.

Modern versions of grep permit both recursive searching (-r) and binary file exclusion (-I). Additionally, prepending the filename can be specified (-H) even in the event only a single file is searched. To find all text files in or below the current directory that contain the string 'Recipe', the command is now simply

grep -HIre 'Recipe' *

During testing for this article the time to complete was at its fastest only about twice that of the grep that uses the find in backticks, and at its slowest was over 100 times slower. This difference may have been due to the system load or possibly the fact that there were hundreds of files that together total nearly 2GB. Even so, there may be times when the blind search is well worth the time spent to discover something.