I found a faded post-it note from a friend who gave me the following
incantation to convert a set of still images into an MPEG animation using
ffmpeg. This note is a crumb for later investigation.
ffmpeg -r1 -f image2 -i <input> -vcodec mpg2video \ -r 30000/1001 -b 25000k <outMovie.mpg>
If memory serves, the 30000/1001 may need to be changed to the decimal value, but I'll see if ffmpeg will parse and calculate on its own ;-)
Getting files from the camera to hard drive is sometimes accompanied by confusion and consternation when custom actions are needed. Given powerful tools and a bit of facility with scripting, retrieving the files can be as simple as
movePics.sh <fromDir> <toDir>
A special love note for my better half. For the old camera disk use
/DATA01/bin/movePics.sh /media/disk/DCIM/101MSDCF /DATA01/Pictures
For the new camera disk 1 use
/DATA01/bin/movePics.sh /media/1A48-D026/DCIM/100D7000 /DATA01/Pictures
Love, your system administrator.
The heart of this solution is the perl exiftool
(http://owl.phy.queensu.ca/~phil/exiftool/) that can interrogate, modify,
and otherwise act on tags in image files. This article describes an
application of the renaming functionality
(http://owl.phy.queensu.ca/~phil/exiftool/filename.html). Specifically,
when moving the files from the camera media
(e.g. /media/1A48-D026/DCIM/100D7000) to the root of an images directory
(e.g. /DATA01/Pictures) the requirements are:
year/month/year-month-day where the
date is given in digits only
All these requirements are met in a single invocation
exiftool '-FileName<DateTimeOriginal' \ -d /DATA01/Pictures/%Y/%m/%Y-%m-%d/%%f.%%le \ /media/1A48-D026/DCIM/100D7000
Two potentially tricky bits are the double percent (%%) for the file and
extension formats (%%f and %%e respectively), and the lowercase
modifier for the extension (%%le). The need for quotes around the first
argument is because the exiftool syntax < would otherwise be interpreted
by the shell instead of being sent to the script.
While the invocation is not terribly difficult, there is enough typing to
make it worthwhile for a simple script (e.g. movePics.sh). A minimal
implementation that takes two arguments — a from and to directory — is
written here in bash.
#!/bin/sh srcdir=$1 tgtdir=$2 exiftool '-FileName<DateTimeOriginal' \ -d $tgtdir/%Y/%m/%Y-%m-%d/%%f.%%le $srcdir
Today on the orgmode mailing list Richard Riley posted a function that works with the Google command line tools (http://code.google.com/p/googlecl/) to create Blogger entries directly from my favorite emacs mode, orgmode. I have been looking for just such a solution to make blogging more transparent, and lo it is here. This post is the first test of the method, and it works. (Of course, I'm writing this before actually testing it, but since the new function evaluated without error then it must be right! ;-)
To be sure, there were quite a few steps to get to this point. Here was the recipe.
googlecl.
python-gdata version 1.2.4 or higher
sudo apt-get install python-gdata
googlecl Debian package (googlecl_0.9.10-1_all.deb)
from http://code.google.com/p/googlecl/downloads/list
googlecl
sudo dpkg --install googlecl_0.9.10-1_all.deb
google blogger list
myname@gmail.com) at the prompt
"Please specify user"
At present org-googlecl is a project at
http://github.com/rileyrg/org-googlecl. Thank you Richard.
A few days ago I wrote about
how to set up googlecl in order to use Richard Riley's wonderful addition to emacs called org-googlecl. Tragically I did not even hint at how to
get it. This post gives that hint, but assumes that git is already
installed on the system and that the user knows how to configure emacs to
use code in non-standard locations.
~/prj)
cd ~/prj
git clone http://github.com/rileyrg/org-googlecl.git
~/prj/org-googlecl) to the emacs path by
putting something like the following in a configuration file
(add-to-list 'load-path "~/prj/org-googlecl")
(load-library "org-googlecl")
(setq googlecl-blogname "Natural Log of X") (setq googlecl-username "<GoogleUserName@gmail.com>")
M-x org-googlecl-blog RET
org-googlecl by updating the source
cd ~/prj/org-googlecl git pull
convert ex-e_g.jpg[32x32+8+5] \ -colorspace gray \ -transpose \ -compress none \ -depth 8 PGM:- |\ tail -n +4 |\ tr -s ' \012' '\012' |\ pr -T -s -32 -l32 \ > ex-e_g_crop32x32.txt
The convert command is from ImageMagick. The [32x32+0+0] crops as the
image is read. The -colorspace Gray ensures that the image will be
grayscale. The -compress none signals to use plain ASCII rather than a
binary format. PGM output (to STDOUT) from 0 to 255 is ensured with
-depth 8 PGM:-. The PGM header is removed with tail -n +4 and a single
column of numbers is generated with the tr command.
The workhorse for the reformat is the pr tool which prepares file for
printing. The -a puts columns across which is necessary for this flow to
obtain the proper orientation. (This could have also been done with the
-transpose switch to convert.) The -T ensures that no headers or
footers are used, and that no page breaks are inserted. The -s switch
puts tabs between values. The number of columns is specified with -32
and the number of lines with -l32.
Take note that initial tests without the -s command yielded files where
the numbers were incorrect. This seems to have been due to something
related to the width being too narrow, but the -s set the width to 512.
The pr command does have the --width switch, but it is currently
unclear how to calculate the exact width needed for the particular crop
size.
As a concrete example, let us consider the problem that we often have
uploading pictures to certain sites (e.g. ebay). The photos from our
camera all have .JPG extension. However, when we attempt to locate
those pictures from some web sites, no photos are found. This is
because something in the chain of events only lower case extensions,
such as .jpg, are allowed.
The task is to change all files in a directory
(e.g. /DATA01/Pictures/EBAY/2010-09-10) from upper case
(.JPG) to lower case (.jpg) extensions.
cd /DATA01/Pictures/EBAY/2010-09-10 for i in *.JPG; do mv "$i" "${i/.JPG}".jpg; done
Simple.
There are certainly other ways, but this is good enough for now.
On 09 Sep 2010 Julien Danjou provided a way to access the Google weather API from emacs. Additionally, the weather can easily be integrated into an orgmode agenda.
cd ~/Projects/git/ git clone git://git.naquadah.org/google-weather-el.git
Then we set up emacs in the following way
(add-to-list 'load-path "~/Projects/git/google-weather-el") (require 'google-weather) (require 'org-google-weather)
To add the weather to the agenda, put a call to org-google-weather
with the city (e.g. "Webster, NY") or postal code (e.g. "14580")
in an org file that is used by the agenda. Multiple weather locations
can be included and the #+CATEGORY option can be used to distinguish
them in the agenda.
#+CATEGORY: Webster, NY %%(org-google-weather "Webster, NY") #+CATEGORY: Washington, DC %%(org-google-weather "Washington, DC")
The Google weather API is apparently very well undocumented (http://blog.programmableweb.com/2010/02/08/googles-secret-weather-api/), but the following call will retrieve an XML response.
http://www.google.com/ig/api?weather=14580
Other APIs that might be of interest are stock quotes
http://www.google.com/ig/api?stock=YHOO
movie listings
http://www.google.com/ig/api?movies=14580
and a news feed
http://www.google.com/ig/api?news
Neat stuff.
In terminal
cd ~/Projects/git/ git clone http://github.com/rileyrg/org-googlecl.git
In emacs
M-x load-file ~/Projects/git/org-googlecl/org-googlecl.el M-x set-variable RET org-googlecl-blogname RET "Natural Log of X" M-x set-variable RET org-googlecl-username RET "<well, you know>"
Write this entry in orgmode using emacs then call
M-x org-googlecl-blog
Today on the orgmode mailing list Richard Riley posted a function that works with the Google command line tools (http://code.google.com/p/googlecl/) to create Blogger entries directly from my favorite emacs mode, orgmode. (See Getting org-googlecl for more information.) I have been looking for just such a solution to make blogging more transparent, and lo it is here. This post is the first test of the method, and it works. (Of course, I'm writing this before actually testing it, but since the new function evaluated without error then it must be right! ;-)
To be sure, there were quite a few steps to get to this point. Here was the recipe.
googlecl.
python-gdata version 1.2.4 or higher
sudo apt-get install python-gdata
googlecl Debian package (googlecl_0.9.10-1_all.deb)
from http://code.google.com/p/googlecl/downloads/list
googlecl
sudo dpkg --install googlecl_0.9.10-1_all.deb
google blogger list
myname@gmail.com) at the prompt
"Please specify user"
These posts are written using org-mode for emacs. Generating HTML
output is accomplished by highlighting the region and issuing the
command C-u C-c C-e R. [Note: Without the prefix argument (C-u)
the HTML will have all the necessary elements (e.g. <html>, =<body>)
to be a standalone web page which is not what is needed for Blogger
posts.]
After signing into Blogger and starting a new post, the exported HTML
is copied from the file (again in emacs) and pasted into the "Edit
HTML" tab.
Initially I had problems with linebreaks being explicitly added by
Blogger which made crummy looking posts. I recall doing two things to
fix the problem, both in Blogger itself not emacs or org-mode. First,
in the Settings | Formatting tab I changed the entry Convert Line Breaks to "No". Second, in the Posting | New Post tab then Post Options (under the editing region), I ensured that Use <br /> tags
was selected.
I read something funny on the Mac website; "A Mac is easy to use, powerful, compatible, and highly secure." My experience setting up Mac to connect with a network file space was not easy because its defaults are not secure.
Our pictures are located on a disk that has been shared for years by
all of our machines. While investigating the Mac as a potential new
machine, I asked several Apple store Specialists and Geniuses
(capitalization from their website, but I like the irony) about
connecting to network drives on a Linux system. Most did not even
know what I was talking about, but finally the manager was able to
provide some assistance. After a few button clicks there appeared a
window that allowed for NFS. This was through Finder | Go | Connect to Server ... and the syntax seemed reasonable.
Unfortunately it did not work on my home system. Over the course of several months I would troubleshoot a bit here and there without success. Of course during that time the Macbook was an island unto itself concerning the shared pictures and other data.
The solution came only when I set aside several consecutive hours for
research and experimentation. The first discovery was an application
called Disk Utility and found a promising entry, File | NFS Mounts .... In this utility there is the ability to exert a measure of
control for the mount versus using the Connect to Server ...
method. Finding the Advanced Mount Parameters input held promise
but no help, because the root problem remained unclear.
More research revealed that Mac OS X, which is based on BSD, uses an insecure port. For an example see http://blogs.techrepublic.com.com/mac/?p=430 or http://www.unixtutorial.org/2010/03/mounting-nfs-shares-on-mac-os-x/ for a few details. Both articles describe one potential way to get connected; change the server side to share over an insecure port. While this is probably a viable solution, I shun the idea of purposely doing something insecurely as the final solution. Of course, I hoped there might have been another solution.
The same articles also introduce the -resvport
switch that worked from the command line. This solution is more to my liking because it changes the client
to use a secure port. However, this did not work when used in the
Disk Utility. (Note: The
first article I found did not have an example of using the switch in
Disk Utility, so used the dash. However, after those attempts I
found a second article that had an example in the Disk Utility
that did not use the dash. It is possible that resvport may work,
but I have not returned to that path.)
Since that did not work I continued the investigation and found the
possibility that the -P switch might work. Indeed it did. Putting
-P in the Advanced Mount Parameters of Disk Utility allows the
Mac to connect to the external data.
It was by no means easy and the default behavior is insecure, contrary to the advertisements.
Intermittent file copies, occasional DVD writes, and misplaced hope are what constituted the backup system for our home network. Two days ago that changed when I successfully configured BackupPC.
BackupPC (http://backuppc.sourceforge.net/) is an application that creates and manages backups to disk, as opposed to tapes or other media. Assuming that BackupPC is installed on the server (see for example the installation instructions) and enough disk space is available, the next step is configuration.
Most of the configuration will be accomplished using the web
interface. In case there is a need to change the login and password
for the BackupPC user (e.g. backuppc), then make the modifications as an
administrative user. Invoking the htpasswd command as follows will
prompt for a new password for backuppc (after, of course, asking for
the sudo password).
sudo htpasswd /etc/backuppc/htpasswd backuppc
At this point the web interface at http://localhost/backuppc should be available using the loging and password. If not, then there are some installation and system issues that must be resolved before the remainder of this article is applicable.
During the initial configuration I used the BackupPC SSH FAQ and was a
bit confused. To be sure, the first half of this article distills the
steps to generate and exchange keys for the rsync method in the
hopes of reducing that confusion in the future. The remainder points
out a few configuration items that were accomplished using the web
interface.
Create keys on BackupPC host (e.g. nitrogen) and send the public one
to the remote machine (e.g. carbon).
backuppc).
su - backuppc
~backuppc/.ssh).
cd ~backuppc/.ssh
id_rsa) and public (e.g. id_rsa.pub) keys
for the local host.
ssh-keygen -t rsa
BackupPC_id_rsa.pub).
cp id_rsa.pub BackupPC_id_rsa.pub
chmod 600 id_rsa id_rsa.pub BackupPC_id_rsa.pub
~root/.ssh).
scp BackupPC_id_rsa.pub root@carbon:~root/.ssh
Create keys on the remote host send the public one to the BackupPC host, and add the BackupPC's public key to list of authorized keys.
pwrusr).
su - pwrusr
sudo ssh-keygen -t rsa -f ~root/.ssh/id_rsa
carbon_id_rsa.pub).
sudo cp ~root/.ssh/id_rsa.pub ~root/.ssh/carbon_id_rsa.pub
chmod 600 id_rsa id_rsa.pub carbon_id_rsa.pub
nitrogen) into the
root user's ssh area (e.g. ~root/.ssh).
sudo scp ~root/.ssh/carbon_id_rsa.pub root@nitrogen:~root/.ssh
~root/.ssh/authorized_keys2).
sudo touch ~root/.ssh/authorized_keys2 sudo cat ~root/.ssh/BackupPC_id_rsa.pub >> ~root/.ssh/authorized_keys2
Add the remote host's public key to the list of known hosts then test the connection.
su - backuppc
~backuppc/.ssh).
cd ~backuppc/.ssh
known_hosts).
touch known_hosts cat carbon_id_rsa.pub >> known_hosts
root. Note: The first time this connection
is made the password may need to be entered, but subsequent logins
should not request a password.
ssh root@carbon whoami ssh root@carbon whoami
Repeat the key generation and exchange for all hosts that BackupPC will be serving.
Now that the machines can communicate securely, the backups themselves
need to be defined. Log into the web interface
(e.g. http://localhost/backuppc). To declare the machines to be
backed up, choose the Edit Hosts link in the table of contents area.
Under the Hosts tab press Add and provide the host name to be
backed up (e.g. carbon), the normal user of that host
(e.g. ksburt). If the BackupPC server is currently a host that
needs to be backed up, it might be wise to add a host with the
explicit name (e.g. nitrogen) and its normal user (e.g. tcburt).
This choice is driven less by necessity (since localhost is a
default host), and more by the desire to prepare for the
future. Additionally, the key generation and exchange described above
needs to be done. Be sure to press the Save button.
Next select the Xfer tab.
XferMethod to rsync.
RsyncShareName to be /home.
Add button to add /etc.
Add button to add other default directories.
BackupFilesExclude should be configured to exclude certain
directories (e.g. /proc).
Save button.
Return to the table of contents area and choose a particular host to
configure (e.g. nitrogen) by using the Select a host...
dropdown. This creates a new section in the table of contents that is
specific to the host. Choose the Edit Config link under the
host-specific section. Ensure that XferMethod is rsync and that
the default directories (e.g. /home and /etc) are in
RsyncShareName. Let us consider the situation where this host has a
filesystem (e.g. /DATA01) that is shared to the network. Since
directories on this disk (e.g. /DATA01/Music, /DATA01/Pictures)
should be backed up only once, add them to the RsyncShareName only
on this host. Save the configuration. Repeat the host-specific
configuration for each host to be backed up.
An immediate test can be run. First, select the host's homepage from the
table of contents. Ensure that the current time is not in a blackout
period (see the Schedule configuration tab). Press the Start Full Backup button and confirm, then the homepage should return. Wait a
few seconds and refresh the page to see if any errors are reported.
Once BackupPC is working, enjoy the peace of mind that comes with having a history of your data stored regularly. Of course, the next step is to test whether you can actually recover the data.
The ls command lists contents of a directory. A common desire is to
see only contents that are themselves directories. Minor facility
with other commands help ls achieve the goal.
First let us set up a concrete example. Suppose that there is a
directory called ~/tmp that contains both files and directories, 98
to be exact.
~/tmp$ ls -l | wc -l 98
To winnow out files one must be able to distinguish a directory from a
file. A simple solution is to use the -d switch of ls (thanks go
to Jeffrey for this method).
~/tmp$ ls -d */ aa/ bin/ CYI/ ImageJ/ LaTeX/ Picasa-10091104/ pst-pdf/ av/ cfg/ ek/ kut/ p011-ursys-LaTeX/ pst/ sv/
The listing is nice and tight. If a line-oriented output is required,
the -1 (that's a number 1) switch achieve the desired result.
In the remainder of this article we describe how to use other Unix
tools in concert with ls to solve the problem. The -F switch
appends an indicator to the names of certain types of content, and in
particular a / is placed at the end of a directory name. The grep
command can now be used to search for / at the end of the line.
~/tmp$ ls -F | grep '/$' aa/ av/ bin/ cfg/ CYI/ ek/ ImageJ/ kut/ LaTeX/ p011-ursys-LaTeX/ Picasa-10091104/ pst/ pst-pdf/ sv/
Another way to distinguish a directory is to use the fact that the
letter d is the first character in a long listing using the -l
switch. Again grep can be used to select only lines that begin with
d.
~/tmp$ ls -l | grep '^d' drwxr-xr-x 4 tcburt tcburt 4096 2009-10-21 05:45 aa drwxr-xr-x 4 tcburt tcburt 4096 2009-11-05 06:06 av drwxr-xr-x 2 tcburt tcburt 4096 2009-03-15 14:10 bin drwxr-xr-x 5 tcburt tcburt 4096 2009-03-15 14:10 cfg drwxr-xr-x 11 tcburt tcburt 4096 2010-03-17 06:36 CYI drwxr-xr-x 2 tcburt tcburt 4096 2009-07-12 18:50 ek drwx------ 6 tcburt tcburt 4096 2009-10-15 22:29 ImageJ drwxr-xr-x 2 tcburt tcburt 4096 2009-12-10 20:47 kut drwxr-xr-x 3 tcburt tcburt 4096 2009-06-14 17:51 LaTeX drwxr-xr-x 2 tcburt tcburt 4096 2009-09-13 22:30 p011-ursys-LaTeX drwxr-xr-x 4 tcburt tcburt 4096 2009-11-04 06:17 Picasa-10091104 drwxr-xr-x 2 tcburt tcburt 4096 2009-12-23 11:51 pst drwxrwxr-x 3 tcburt tcburt 4096 2009-12-23 11:41 pst-pdf drwxr-xr-x 4 tcburt tcburt 4096 2009-11-05 06:02 sv
The methods above satisfy the requirement of identifying only
directories. However, they also result in additional characters
besides just the names. The ls -F method lends itself to simply
deleting the / character with the help of the tr command.
~/tmp$ ls -F | grep '/$' | tr -d / aa av bin cfg CYI ek ImageJ kut LaTeX p011-ursys-LaTeX Picasa-10091104 pst pst-pdf sv
The pathological case where a / is part of the file name will not be
served by the tr -d / invocation.
Recall that the long listing example did not leave a trailing slash on
the directory name, it did prepend detailed information. For this
case one needs to obtain only the final column which is the name
itself. The perl command can split each line and print only the
final column.
~/tmp$ ls -l | grep '^d' | perl -lane 'print "$F[-1]"' aa av bin cfg CYI ek ImageJ kut LaTeX p011-ursys-LaTeX Picasa-10091104 pst pst-pdf sv
If listing only directories is a common task, it will be wise to define an alias for a favored method.
Even a computer that is nearly ten years old can be made useful again. A certain computer in my house has remained untouched for about five years, but today it found new life with Linux.
In May of 2005 we replaced one of our computers that had been in service for exactly five years. The system specifications were tired even then: 750MHz AMD Athlon processor, 256MB RAM, 20GB harddrive. It had been running Windows 98 when decommissioned. It was stored, moved to a new state, and stored some more.
Yesterday I continued working on the home network infrastructure and looked for a potential fileserver. I found this decade old machine, gave it power and peripherals, then booted it up. Slow as the system was, I was able to navigate the files and actually found over two years of digital photographs that were presumably lost on CDs that were unreadable.
After the recovery came the rejuvenation. Since this machine is to be a network workhorse it needed a reliable operating system with powerful tools readily available. It should be no surprise that I chose Linux, specifically the Ubuntu 9.10 distribution. Even on this old system the installation proceeded without problem, and the subsequent package updates finished with little problem.
Previous experience indicates that this would have been more costly and less flexible using Windows. Indeed, it would have been impossible to use the latest version of Windows (see http://windows.microsoft.com/systemrequirements) because the motherboard doesn't even support the RAM requirement.
While the system will need a RAM upgrade and a card to accept firewire connections to external hard drives, the machine will likely be one of the silent, steady sentinels in our home network.
For the technorati …
While gathering information about this system I discovered the
dmidecode command. This tool needs to be run with administrator
privileges. With it I was able to discover the CPU speed, CPU maker,
RAM information, and even the build date.
sudo dmidecode -t processor sudo dmidecode -t system sudo dmidecode -t memory
An old article (http://www.linux.com/archive/feed/40412) describes the
information from dmidecode as not completely reliable, but useful.
I found it useful for my purposes.
During a fresh installation of Ubuntu 9.10 (karmic) on a Dell Dimension 3000 standard system, the monitor resolution could no longer support any mode higher than 800x640. An initial twinge of frustration was coupled with confidence that the problem was tractable. This is the short story of how the Ubuntu and open source communities helped the solution.
Historically, whenever I encounter display problems I look at the
xorg.conf file which is usually found in /etc/X11. On this
installation day I was surprised to discover that xorg.conf was not
in its usual place. Indeed, my surprise became something unameable
when a search of the entire filesystem revealed that the file did not
exist. While it is possible that a step or two after installation
could have removed the file, the fact remains that some high-level
procedure leaves me without an xorg.conf.
It was tempting to search out what mechanisms have replaced
xorg.conf and to understand the advertised improvements of the new
way, but in the short term I just wanted better resolution. Somewhere
within the first few tens of minutes I found what turned out to be a
solution, but I initially hoped for a simpler way. I learned a
valuable lesson in this dismissal — read and understand before
trading away over an hour searching for a presumed simpler way.
Searching the Ubuntu wiki revealed the article titled Reverting the Jaunty Xorg intel driver to 2.4 and my initial reaction was that my system (karmic) was newer than the one in the article (jaunty). Even so I saw the seven steps, four explicit and three others potential, and thought this might work. However, two of the possible steps (obtaining a validation key) were something I remembered doing before with some trouble. The other possible step, related to the key, was opening a port on my firewall. This final step is the one that sent me looking elsewhere, not because I didn't want to make the change but because I sought something that smacked less of system administration to the normal home user. In the end, neither of the potential steps were necessary.
The four steps to solving the resolution took less than 4 minutes to accomplish.
/etc/apt/sources.list (I used sudo vi /etc/apt/sources.list to
do the edits; use your favorite editor as an administrator)
deb http://ppa.launchpad.net/siretart/ppa/ubuntu jaunty main deb-src http://ppa.launchpad.net/siretart/ppa/ubuntu jaunty main
sudo apt-get update
xserver-xorg-video-intel-2.4 package
sudo apt-get install xserver-xorg-video-intel-2.4
sudo /etc/init.d/gdm restart
The display immediately went to a much more reasonable resolution. Problem solved.
EXECUTIVE SUMMARY
Search for the string 'Recipe' in all files that have the .org or
.html extension anywhere in the current directory or below, ensuring
that the filename is prepended to all matches:
> grep -e 'Recipe' `find . \( -name "*.org" -o -name "*.html" \)` /dev/null
Same as above except all non-binary files are searched:
grep -HIre 'Recipe' *
SUPPORTING JABBER
Consider the situation where you have many text files in a certain
directory tree and you want to discover which files have particular
content. Here we discuss the use of grep and find to help solve
this problem. Modern versions of grep remove the need to use
find, and we will discuss that method after the one applicable to
more disadvantaged systems.
The grep command is used to search the contents of files. A
familiar output is to have the filename prepended to the line that
matches the search, for example
> grep -e 'ground' * photos.org: background. I also had the privilege of seeing the physical photos.org: ground on the night of the 27th. On the morning of the 28th there Quotes.org:going to take a lovely, simple melody and drive it into the ground. --
It is tempting to interpret the prepended filename as the overall
default. However, whether the filename appears or not depends also on the
context in which grep was used. Specifically, when grep is
provided a single file to search through the filename is not prepended,
> grep -e 'ground' photos.org background. I also had the privilege of seeing the physical ground on the night of the 27th. On the morning of the 28th there
This is a reasonable behavior from the perspective of grep since
only a single file was given there should be no doubt what file
contained the match. As we will see below there are times when grep
may be provided a single file but the user does not know what that
file is. In these cases we want to force the filename to be
identified. One way to do this is to pass grep the real file and
one other file that has the following property; its contents will
never match the search expression, for example /dev/null. Witness
the difference,
> grep -e 'ground' photos.org /dev/null photos.org: background. I also had the privilege of seeing the physical photos.org: ground on the night of the 27th. On the morning of the 28th there
Before continuing there are two observations to be made about the
grep invocations above. First, and almost as an aside, the calls
could have been written just a bit more simply by dropping the -e
switch and the quote marks. However, this construct allows for more
complex search expressions. An example is to find either the word
'ground' or the word 'Recipe' in any files,
> grep -e 'ground\|Recipe' * photos.org: background. I also had the privilege of seeing the physical photos.org: ground on the night of the 27th. On the morning of the 28th there Quotes.org:going to take a lovely, simple melody and drive it into the ground. -- Recipes.org:#+TITLE: Recipes sitemap.org: + [[file:Recipes.org][Recipes]]
The observation that pertains directly to the problem at hand is that
the list of files for grep to search must be specified somehow. If
all the files are in the same directory, then a simple wildcard
expression might be all that is needed. However, sometimes the search
is to be done recursively or across several directories.
The find command is useful for finding files on the system with
particular characteristics. As an example, the following expression
finds all files in the current directory and below that have either a
.org or .html extension,
> find . \( -name "*.org" -o -name "*.html" \) backcountry/photos.html backcountry/readme.html backcountry/maintenance.html backcountry/sitemap.html backcountry/index.html [--snip--] templates/rketburt-01-Level00.org templates/rketburt-01-Level01.org [--snip--]
Be aware, the space after the \( and before the \) proved to be
vital while testing commands for this article. I am unaware if this
is a general necessity or just on my particular system.
Now it is a simple matter to search the contents of multiple files.
We build the file list using find embedded in backticks (`) to
capture the result, then invoke grep on that list. Here is a
complete example,
> grep -e 'Recipe' `find . \( -name "*.org" -o -name "*.html" \)` /dev/null rketburt-org/Recipes.org:#+TITLE: Recipes rketburt-org/sitemap.org: + [[file:Recipes.org][Recipes]] rketburt/sitemap.html:<a href="Recipes.html">Recipes</a> rketburt/Recipes.html:<title>Recipes</title> rketburt/Recipes.html:<h1 class="title">Recipes</h1> rketburt/index.html:<a href="Recipes.html">Recipes</a>
Note the use of /dev/null as a file argument to grep to ensure
that the filename is prepended.
Another way to effect the same final result is to invoke find first
and use the -exec argument to call grep. In this ordering grep
is only provided with a single file which leads to the lack of
filename problem indicated earlier. The overall syntax is a bit more
cumbersome as well, since {} is used to pass the result of find to
grep and there is the trailing \; as well. An equivalent example
to the one in the previous paragraph is
> find . \( -name "*.org" -o -name "*.html" \) -exec grep -e 'Recipe' {} /dev/null \;
Syntax or preferences aside, it is interesting to note that while
these two examples provided the same end result, the one that begins
with grep executed nearly 10 times faster.
The find command has been used above for two reasons. First, the
desire was to search files that may appear in directories below the
one called out. In other words we desired a recursive search. The
second reason was to eliminate the prospect of searching non-text
files which would have simply been a time sink. The method to exclude
the binary files was to limit the file extensions to just two (.org
and .html). This may be the exact behavior desired for some
questions, but may be too restrictive for others.
Modern versions of grep permit both recursive searching (-r) and
binary file exclusion (-I). Additionally, prepending the filename
can be specified (-H) even in the event only a single file is
searched. To find all text files in or below the current directory
that contain the string 'Recipe', the command is now simply
grep -HIre 'Recipe' *
During testing for this article the time to complete was at its
fastest only about twice that of the grep that uses the find in
backticks, and at its slowest was over 100 times slower. This
difference may have been due to the system load or possibly the fact
that there were hundreds of files that together total nearly 2GB.
Even so, there may be times when the blind search is well worth the
time spent to discover something.