Ok, I'll admit it: there's certainly hyperbole in this entry's title. What I'm doing with the text-mode browser lynx isn't really scraping--it's just something that bears some conceptual (in my view) similarities. It might appear similar because what I've done is to come up with a way of invoking lynx (or any other text-mode browser for that matter), with search terms already entered, from the command line. The end product is just the text results google finds relative to your query--sans all the bells and whistles google's search portal has been foisting on us in recent years. Why is this a significant accomplishment? Well, consider the following.
Background
Have you found google's search portal to be increasingly cluttered and bothersome? I certainly have. Things like pop-out previews do nothing for me but create distraction, and auto-completion is far more often an irritation to me than a help: as a liberal estimate, perhaps 25% of my searches have benefited from the auto-completion feature. For what it's worth, if google wished to provide better service to users like me, they would create two separate search portals: one would be a fuzzy-feely search portal for those who might be uncertain as to what they're seeking and who could benefit from auto-completion and/or pop-out previews; the other would be google's old, streamlined search page and would involve little more than short text summaries and relevant links.
Once upon a time there was a google scraper site at www.scroogle.org--billing itself more as a search anonymizer than as an interface unclutterer--that provided a results page pretty much like the old google one. I used to use scroogle in the days before google introduced some of the more irritating "enhancements" that now plague their site, and came to appreciate above all its spartan appearance. But, alas, scroogle closed its doors in mid-2012 and so is no longer an option. I've been stuck since, resentfully, using google.
In a recent fit of frustration, I decided to see whether there might be any other such scrapers around. As I searched, I wondered as well whether one might not be able to set up their own, personal scraper, on their own personal computer: I had certainly heard and read about the possibilities for conducting web searches from the command line, and this seemed a promising avenue for my query. I ended up finding some results that, while providing but a primitive approximation, look like they may nonetheless have given me a workable way to do the sort of pseudo-scraping I need. Thus, the following entry.
More about the task
Conducting web searches from the command line is another way of describing the task I aimed to accomplish. Granted, doing this sort of thing is nothing especially new. surfraw, for example, created by the infamous Julian Assange, has been around for a number of years and more properly fits into the category of web-search-from-the-command-line utilities than does the solution I propose--which just invokes a text-mode browser. There are actually several means of doing something that could be classified as "searching the web from the command line" (google that and you'll see), including the interesting "google shell" project, called "goosh."
Still, the solution I've cobbled together using bits found in web searches, and which involves a bash function that calls the text-mode browser lynx, seemed on-target enough and something worth writing an entry about. Details below.
The meat of the matter: bash function
To begin with, some credits. The template I cannibalized for my solution is found here: I only did some minor modifications to that code so that it would work more to my liking. There's another interesting proposition in that same thread, by the way, that uses lynx--though it pipes output through less. I tried that one and it got me thinking in the direction of using lynx for this. But I liked the way the output looked in lynx much more than when piped through less, so I decided to try further adapting the bash function for my uses and came up with the following.
The bash function outlined at that site actually uses google search and calls a graphical browser to display the output. The graphical browser part was the one I was trying to obviate so that would be the first change to make. I mostly use elinks these days for text-mode browsing, but having revisited lynx while experimenting with the other solution posed there, I decided I would try it out. And I must say that it does have an advantage over elinks in that URL's can be more easily copied from within lynx (no need to hold down the shift key).
I could not get the google URL given in that example to work in my initial trials, however. This is likely owing to changes google has made to its addressing scheme in the intervening interval since that post was made. So I first used a different URL from the search engine startpage.
After some additional web searching and tweaking, I was finally able to find the correct URL to return google search results. Though that URL is likely to change in the future, I include it in the example below.
What I have working on this system results from the code below, which I have entered into my .bashrc file:
Once that has been entered, simply issue . .bashrc so that your system will re-source your .bashrc file, and you're ready for command-line web searching/pseudo-scraping. To begin searching, simply enter the new terminal command you just created, search, followed by the word or phrase you wish to search for on google: search word, search my word, search "my own word", search my+very+own+word, or seemingly just about any other search term or phrase you might otherwise enter into google's graphical search portal seem to work fine.
lynx will then open in the current terminal to the google search results page for your query. You can have a quick read of summaries or follow results links. Should any of the entries merit graphical inspection, you can copy and paste the URL into your graphical browser of choice.
You'll probably want to tell lynx (by modifying the relevant option in lynx.cfg) either to accept or reject all cookies so as to save yourself some keystrokes. If you do not do so, it will, on receiving a cookie, await your input prior to displaying results. Of course you could use any other text-mode browser--such as w3m, the old links or xlinks, retawq, netrik, or any other text-mode-browser candidates as well.
Suggestions for improvements to my solution or offerings of alternative approaches will be appreciated. Happy pseudo-scraping/command-line searching!
AFTERTHOUGHT: I happened upon some other interesting-looking bash functions at another site that are supposed to allow other types of operations from the command line; e.g., defining words, checking weather, translating words. These are rather dated, though (2007), and I couldn't get them to work. Interpreting their workings and determing where the problem(s) lie is a bit above my pay grade: anyone have ideas for making any of these functions once again operable?
Wednesday, January 23, 2013
Thursday, January 10, 2013
10th installment: resume an scp file transfer
NOTE: as a knowledgeable commenter later pointed out "[c]urrently (and since around 2004) the default transfer protocol in rsync *IS* ssh. There is no need for the '-e ssh' unless you directly connect to a remote rsync daemon." I have tested this claim and it is, indeed, true that the file transfer resumption using rsync does not require the -e ssh bit I stipulated in the instructions below. I did not manage to test whether the same alternate port switch (-p 1234) works, though I assume it does.
I recently went on vacation and, since my mythtv set-up was, for some crazy reason, not allowing me to do a direct download of recorded programming through the mythweb interface, I needed to find an alternate way of snagging those files. I have an ssh server running on my home LAN, so using scp for this seemed like it should work, though I knew it would take a bit of tinkering. Read on to see what sort of tinkering I did and, just as importantly, a way I discovered of resuming the disrupted download.
I first investigated the possibility of setting up an ssh tunnel, since the computer on my LAN that contains the video files is not the one running the ssh server. But doing that looked a tad beyond my skill level. So I decided I'd just copy those files over to the computer running the ssh server manually, then scp them to the remote computer from there.
These were very large files--i.e., > 2 GB--and, given the fairly limited rate at which I could transfer them, I expected there might be some disruption or disconnection during the download. Prior to beginning the dowloads, then, I searched google under "scp" and "resume," and I immediately came across results that showed how to use the rsync utility to resume disrupted downloads. This encouraged me to go ahead and try the scp download method.
As wikipedia informs us, "rsync is a software application and network protocol for Unix-like systems . . . that synchronizes files and directories from one location to another while minimizing data transfer." Though I had, when previously considering differing ways to back up certain directories on my computers, looked at some documentation on rsync, I had no prior experience with actually using it. Nonetheless, that's the solution I ended up employing--though I needed to do a slight adaptation for my circumstances. It seemed the slight variation I stumbled upon might warrant an entry on this blog.
Before describing in greater detail what I did, I should first at least mention a couple of other results I found that used differing utilities. One candidate used curl and sftp instead of rsync, while the other used the dd command. Since I did not attempt to implement either of those solutions, I will, after simply making note of the fact that those utilities apparently can be used for this, move on.
Getting back to rsync, the bulk of instructions I found for resuming scp transfers using it would not work "out of the box" for me because I run ssh on a non-standard port. For purposes of this blog entry, let's say that's port 1234. The question for me, then, was how to adapt the directions I'd found to the scenario involving the non-standard ssh port my LAN uses.
The resolution turned out to be fairly simple. I finally ran across the an incantation very close to what I needed here. A simplified sample entry follows (a slightly more complex rendition can be seen in the description for setting up an alias below):
rsync -P -e 'ssh -p 1234' user@remote.host.IP:path-to/remotefile.mpg localfile.mpg
Essentially, the command tells rsync to use ssh as the shell on remote end (the -e switch), while the -P switch tells it two things: that it should display the progress of the transfer, as well as that it only needs to do a partial transfer of the file. What falls between the inverted commas are the options that get passed to ssh--in this case -p 1234 stipulating the port to connect to on the remote end.
To simplify yet further this resuming process, an alias could, theoretically, be created as suggested here. That would would not work in my case, however, since aliases appear not to allow the passing of special options to ssh: entering alias scpresume='rsync -Pazhv -e ssh -p 1234' at the command line caused the port specification to be received as an option by rsync--an option it was unable to interpret. Thus, the more permanent solution of adding that line to .bashrc would not work for me either.
To make the alias work for me, I had to set up an ~/.ssh/config file with the following content (as discussed here):
After doing that, I was able to create the alias as alias scpresume='rsync -Pazhv -e ssh (note that the alternate port now does not need to be specified since it's entered into your ~/.ssh/config file). It can now simply be run as scpresume remote:path-to/remotefile localfile. Once your ~/.ssh/config file is set up, the scpresume='rsync -Pazhv -e ssh line is what needs to be entered into your .bashrc to make scpresume a permanent part of your command-line environment.
Feel free to offer any improvements you may have or other suggestions for using alternate command-line utilities for resuming downloads. This method did the trick for me, and I was able to resume transfer of files that had petered out at varying points during the download process, but of course there could be other methods that are in some way superior.
ADDENDUM: In light of the comment of a knowledgeable reader, the correct full, non-redundant command to use for file transfer resumption using rsync would be
I recently went on vacation and, since my mythtv set-up was, for some crazy reason, not allowing me to do a direct download of recorded programming through the mythweb interface, I needed to find an alternate way of snagging those files. I have an ssh server running on my home LAN, so using scp for this seemed like it should work, though I knew it would take a bit of tinkering. Read on to see what sort of tinkering I did and, just as importantly, a way I discovered of resuming the disrupted download.
I first investigated the possibility of setting up an ssh tunnel, since the computer on my LAN that contains the video files is not the one running the ssh server. But doing that looked a tad beyond my skill level. So I decided I'd just copy those files over to the computer running the ssh server manually, then scp them to the remote computer from there.
These were very large files--i.e., > 2 GB--and, given the fairly limited rate at which I could transfer them, I expected there might be some disruption or disconnection during the download. Prior to beginning the dowloads, then, I searched google under "scp" and "resume," and I immediately came across results that showed how to use the rsync utility to resume disrupted downloads. This encouraged me to go ahead and try the scp download method.
As wikipedia informs us, "rsync is a software application and network protocol for Unix-like systems . . . that synchronizes files and directories from one location to another while minimizing data transfer." Though I had, when previously considering differing ways to back up certain directories on my computers, looked at some documentation on rsync, I had no prior experience with actually using it. Nonetheless, that's the solution I ended up employing--though I needed to do a slight adaptation for my circumstances. It seemed the slight variation I stumbled upon might warrant an entry on this blog.
Before describing in greater detail what I did, I should first at least mention a couple of other results I found that used differing utilities. One candidate used curl and sftp instead of rsync, while the other used the dd command. Since I did not attempt to implement either of those solutions, I will, after simply making note of the fact that those utilities apparently can be used for this, move on.
Getting back to rsync, the bulk of instructions I found for resuming scp transfers using it would not work "out of the box" for me because I run ssh on a non-standard port. For purposes of this blog entry, let's say that's port 1234. The question for me, then, was how to adapt the directions I'd found to the scenario involving the non-standard ssh port my LAN uses.
The resolution turned out to be fairly simple. I finally ran across the an incantation very close to what I needed here. A simplified sample entry follows (a slightly more complex rendition can be seen in the description for setting up an alias below):
rsync -P -e 'ssh -p 1234' user@remote.host.IP:path-to/remotefile.mpg localfile.mpg
Essentially, the command tells rsync to use ssh as the shell on remote end (the -e switch), while the -P switch tells it two things: that it should display the progress of the transfer, as well as that it only needs to do a partial transfer of the file. What falls between the inverted commas are the options that get passed to ssh--in this case -p 1234 stipulating the port to connect to on the remote end.
To simplify yet further this resuming process, an alias could, theoretically, be created as suggested here. That would would not work in my case, however, since aliases appear not to allow the passing of special options to ssh: entering alias scpresume='rsync -Pazhv -e ssh -p 1234' at the command line caused the port specification to be received as an option by rsync--an option it was unable to interpret. Thus, the more permanent solution of adding that line to .bashrc would not work for me either.
To make the alias work for me, I had to set up an ~/.ssh/config file with the following content (as discussed here):
After doing that, I was able to create the alias as alias scpresume='rsync -Pazhv -e ssh (note that the alternate port now does not need to be specified since it's entered into your ~/.ssh/config file). It can now simply be run as scpresume remote:path-to/remotefile localfile. Once your ~/.ssh/config file is set up, the scpresume='rsync -Pazhv -e ssh line is what needs to be entered into your .bashrc to make scpresume a permanent part of your command-line environment.
Feel free to offer any improvements you may have or other suggestions for using alternate command-line utilities for resuming downloads. This method did the trick for me, and I was able to resume transfer of files that had petered out at varying points during the download process, but of course there could be other methods that are in some way superior.
ADDENDUM: In light of the comment of a knowledgeable reader, the correct full, non-redundant command to use for file transfer resumption using rsync would be
rsync -Pazhv -p 1234 user@remote.host.IP:path-to/remotefile.mpg localfile.mpg
Friday, January 4, 2013
Ninth Installment: Xmobar to the rescue!
Regular readers of this blog will be aware of my inclination towards minimalist desktops or window managers and my preference, within reasonable limits, for low-resource and/or command-line tools and applications. And I've previously mentioned making use lately of the evilwm window manager, which I've pretty much settled on now in preference to two other minimalist window manages--dwm and ion3--with which I experimented. One of the things I've missed, however, about the more full-blown desktops I've used, is some of the monitors or applets one can configure to run in the panel(s) and which give quite helpful system information like memory and/or CPU usage.
I became aware some time ago of conky, a minimalistic utility that can display these--and many other--types of helpful information under differing window managers or desktops and, though I'd seen screenshots of it configured to run as a sort of panel, most conky configurations I've come across actually have it display on the desktop background--not something particularly desirable for me since I tend to run applications full-screen on my evilwm desktops. But lately, I somehow came across information about another utility--Xmobar--that can display the sorts of system information I want but which seems to be configured to run mainly as a panel. So I decided to have a go with it. I was able to configure it to my liking fairly easily and decided to offer in this entry a further description of the program and to post the configuration I am using. That information follows.
To begin with a bit more information on Xmobar, it seems originally to have been written to complement the minimalist window manager Xmonad (which, incidentally, I've not tried). As the Arch Wiki entry--my main source for setting up and configuring Xmobar--informs us, it is written, as is Xmonad, in the Haskell programming language. In case you might be intimidated at the prospect of potentially having to learn something about that programming language, take heart; as the wiki entry further elucidates "while xmobar is written in Haskell, no knowledge of the language is required to install and use it."
As with many other GNU/Linux utilities, Xmobar relies on a hidden configuration file--named, predictably, .xmobarrc--located in the user's home directory. The Arch wiki contains a sample configuration file, and that's the one on which I based my initial experiments with Xmobar.
To start Xmobar, I simply call it as the last line in my .xinitrc file. That will, of course, not be the universally applicable way of starting the utility: those using a log-in manager will undoubtedly need to invoke the utility in some other way. Being the GUI-adverse type I am, however, this is what works for me.
Below I include a screenshot of the lower section of one of my desktops, which shows Xmobar running as it is currently configured on one of my systems.
I became aware some time ago of conky, a minimalistic utility that can display these--and many other--types of helpful information under differing window managers or desktops and, though I'd seen screenshots of it configured to run as a sort of panel, most conky configurations I've come across actually have it display on the desktop background--not something particularly desirable for me since I tend to run applications full-screen on my evilwm desktops. But lately, I somehow came across information about another utility--Xmobar--that can display the sorts of system information I want but which seems to be configured to run mainly as a panel. So I decided to have a go with it. I was able to configure it to my liking fairly easily and decided to offer in this entry a further description of the program and to post the configuration I am using. That information follows.
To begin with a bit more information on Xmobar, it seems originally to have been written to complement the minimalist window manager Xmonad (which, incidentally, I've not tried). As the Arch Wiki entry--my main source for setting up and configuring Xmobar--informs us, it is written, as is Xmonad, in the Haskell programming language. In case you might be intimidated at the prospect of potentially having to learn something about that programming language, take heart; as the wiki entry further elucidates "while xmobar is written in Haskell, no knowledge of the language is required to install and use it."
As with many other GNU/Linux utilities, Xmobar relies on a hidden configuration file--named, predictably, .xmobarrc--located in the user's home directory. The Arch wiki contains a sample configuration file, and that's the one on which I based my initial experiments with Xmobar.
To start Xmobar, I simply call it as the last line in my .xinitrc file. That will, of course, not be the universally applicable way of starting the utility: those using a log-in manager will undoubtedly need to invoke the utility in some other way. Being the GUI-adverse type I am, however, this is what works for me.
Below I include a screenshot of the lower section of one of my desktops, which shows Xmobar running as it is currently configured on one of my systems.
Though a good deal will be obvious from the picture, I will nonetheless offer a verbal description of each section of the panel. After the description, I will provide the content of my .xmobarrc file for further reference.
On the left side of the panel we see, on the far left, of course, a CPU meter of sorts (CPU percentage meter). Next to that is the memory meter, showing percentage of main RAM and swap in use. Then follows a network meter showing current upload/download speed: the dash to the left indicates Xmobar did not find eth0 since, in the instance when the screenshot was taken, no network cable was attched to it. After the network speed indicator follows a keyboard layout indicator: at the moment the US keyboard layout is in use, but I also have Russian and Greek keyboards configured for this machine (more on keyboard layouts and switching between them in a future entry).
On the right side of the panel is seen first the date and time. Next to that is a battery meter that displays percentage of battery charge left as well as estimated remaining time; the screenshot comes from a laptop, of course. That is followed by a location indicator and outside temperature reading: I happened to be near the town of Mikkeli in Finland at the time I wrote this entry--thus the EFMI weather station code in the configuration file below. Finally, the kernel version and distribution are listed--this being derived, of course, from uname output.
Below, then, is the content of my .xmobarrc file. I added a few tweaks to the one I found on the Arch Wiki, mainly the battery meter as well as the keyboard layout indicator. I also did a bit of color tweaking since it seemed to me the section dividers (the pipe character--|) needed to be in a different color so as to more readily draw attention to the field delimitations. A bottom rather than top orientation was more to my liking, so I made that modification as well.
I am thus far quite happy with Xmobar. At the same time, I would be interested to hear from conky users who have their layout configured as a panel like this. Feel free to pipe in with your input on these or other minimalist panel utilities.
Wednesday, December 19, 2012
Eighth Installment: compress and encrypt/decrypt a directory
I recently visited a relative who is studying in the natural sciences and who, surprisingly, is even less capable in certain technical aspects of computing than I am. He was trying to create, on his Mac, a script that would run as a cron job, and asked me for some pointers. Though I know the basics about cron and was willing to pitch in, I wasn't so sure about the script: you see, calling my bash skills rudimentary would be high praise. Nonetheless I decided that, with some web searching, I might be able to assist with that, too. Sure enough, I was able to find just the sort of information that would help us create a script that would tar and compress, then encrypt, a target directory. Details--shamelessly lifted from various locales on the web--are included below.
Over the years that I've been using Linux I have, of course, read more than a few articles that describe methods of encrypting files or partitions. Most recently, for example, there appeared on Lxer an article that described a clever way of encrypting a local directory that then gets backed up to some cloud storage service like dropbox. I've bumped up against the issue of encryption when doing fresh installations as well, as it has been the case for some time now that an option is given on installation for many Linux distros of encrypting, for example, the /home directory.
Despite reading at least some of those articles with interest, I did not feel the need to implement such encryption on my own systems. So it was not until someone else asked my assistance in doing something like this that I actually tried it myself. As you will see, it was actually fairly simple to implement. But first, a few caveats.
I'll skip any details in the following description regarding the cron aspect of this project--not that I could provide a whole lot of enlightenment anyway--other than to say that it's a handy way to make programs or processes run on a set schedule on computers than run *nix. One way I've used it is to cause a weather map, which I've set up as the desktop background on one of my computers, to update every 10 minutes--look for a future entry in this blog on how I managed that.
I'll also not speak in any depth about another ingredient in this recipe--tar--other than to say that it is an abbreviation for for "tape archive." I myself do not understand its workings terribly well, though I've used it on several occasions. I will mention on a related note, however, that, in my research, I ran across articles that used another, similar, utility--dd (an abbreviation for "disk dump")--to create compressed and encrypted archives. But I did not follow up on the dd option and so cannot post any further information about how that was done.
Finally, I can't speak in any depth about the program I used for doing the encryption--openssl--or about another program with which I experimented and which also does encryption--gpg. But I promise, despite those rather glaring deficits, that I will describe something I managed to accomplish and which you, too, should be able to accomplish by following the steps outlined.
Perhaps in some future entry for this blog I'll be able to further explore tar, dd, and/or cron. But for now I'm going to focus my attention mainly on the option we ended up using, which involved mainly tar and openssl.
The relative in question, as I mentioned, works in the natural sciences. He has a directory of his ongoing work that he wants to back up regularly, but to which he does not want anyone else to have access. His choice for backing up beyond his own PC, is to use dropbox. So the task was, as mentioned, to compress and encrypt the target directory: moving it to the location on the local machine where dropbox would find it so as to back it up will also not be covered in this write-up, though that step did end up being part of his final resolution.
So, what's left? It was quite easy to find directions of the web for doing all this. I pretty much went with the first workable solution I found, which came from the linuxquestions forum (the relevant thread can be found here).
The incantation we used was as follows:
tar -cj target-dir | openssl enc -aes128 -salt -out target-dir.tar.bz2.enc -e -a -k passwordOver the years that I've been using Linux I have, of course, read more than a few articles that describe methods of encrypting files or partitions. Most recently, for example, there appeared on Lxer an article that described a clever way of encrypting a local directory that then gets backed up to some cloud storage service like dropbox. I've bumped up against the issue of encryption when doing fresh installations as well, as it has been the case for some time now that an option is given on installation for many Linux distros of encrypting, for example, the /home directory.
Despite reading at least some of those articles with interest, I did not feel the need to implement such encryption on my own systems. So it was not until someone else asked my assistance in doing something like this that I actually tried it myself. As you will see, it was actually fairly simple to implement. But first, a few caveats.
I'll skip any details in the following description regarding the cron aspect of this project--not that I could provide a whole lot of enlightenment anyway--other than to say that it's a handy way to make programs or processes run on a set schedule on computers than run *nix. One way I've used it is to cause a weather map, which I've set up as the desktop background on one of my computers, to update every 10 minutes--look for a future entry in this blog on how I managed that.
I'll also not speak in any depth about another ingredient in this recipe--tar--other than to say that it is an abbreviation for for "tape archive." I myself do not understand its workings terribly well, though I've used it on several occasions. I will mention on a related note, however, that, in my research, I ran across articles that used another, similar, utility--dd (an abbreviation for "disk dump")--to create compressed and encrypted archives. But I did not follow up on the dd option and so cannot post any further information about how that was done.
Finally, I can't speak in any depth about the program I used for doing the encryption--openssl--or about another program with which I experimented and which also does encryption--gpg. But I promise, despite those rather glaring deficits, that I will describe something I managed to accomplish and which you, too, should be able to accomplish by following the steps outlined.
Perhaps in some future entry for this blog I'll be able to further explore tar, dd, and/or cron. But for now I'm going to focus my attention mainly on the option we ended up using, which involved mainly tar and openssl.
The relative in question, as I mentioned, works in the natural sciences. He has a directory of his ongoing work that he wants to back up regularly, but to which he does not want anyone else to have access. His choice for backing up beyond his own PC, is to use dropbox. So the task was, as mentioned, to compress and encrypt the target directory: moving it to the location on the local machine where dropbox would find it so as to back it up will also not be covered in this write-up, though that step did end up being part of his final resolution.
So, what's left? It was quite easy to find directions of the web for doing all this. I pretty much went with the first workable solution I found, which came from the linuxquestions forum (the relevant thread can be found here).
The incantation we used was as follows:
What that line does may be evident to most, but I will offer a bit of review nonetheless. The target directory is first tar'red and compressed with bzip, (the c option stands for "create" and the j option specifies that the created file should be compressed with bzip2) then it is piped to openssl for encryption. The word "password" is, obviously, to be replaced by whatever password the user chooses.
One possible drawback to this method, as pointed out in the thread from which I lifted it, is that the encryption password gets entered, in plain text, right on the command line (which is slightly less of an issue with a cron script such as we were creating). Thus, anyone who can gain access to the machine can, by using the command line history, see what the encryption password was. Since someone gaining access to his computer and viewing the command line history was not a concern for the fellow I was helping, this is the solution we implemented. But that potential concern can be easily remedied by simply leaving off the -k password switch at the end, which has the effect of prompting the user for a password, which does not get echoed to the command line.
To decrypt the file, the following command--which prompts for the password--is used:
openssl enc -aes128 -in target-dir.tar.bz2.enc -out target-dir.tar.bz2 -d -a
The file can then be uncompressed and untar'red. This part of the process could likely be reduced from two steps (decryption, then uncompression/untar'ing) to one by using a pipe, but since it was presumed, for purposes of this project, that the file would act simply as insurance against loss--the need for ever actually recovering the content being very unlikely--I did not pursue streamlining that aspect.
I did manage to find and test a couple of other variants which I will offer here as well. The second variant was found here, and follows:
tar -cj target-dir | openssl enc -e -a -salt -bf -out target-dir.blowfish
It is much the same as the first variant, though it uses a different encryption method called blowfish. I am uncertain which of these two encryption schemes is considered better. To decrypt the compressed directory, the following command is used:
openssl enc -d -a -bf -in target-dir.blowfish -out target-dir-decrypt.tar.bz2
Finally, I discovered yet another variant, details about which can be found here. A sample of how to use this one is as follows:
tar -cjf target-dir.tar.bz2 target-dir/ | gpg -r user -e target-dir.tar.bz2
As will be noted, this variant uses gpg to encrypt the directory. Of course user must be replaced by the name of someone who has a valid gpg key on the system, usually the primary user of said machine or account.
An interesting feature I discovered about this method is that a time-sensitive gpg key can be created, i.e., one that expires after a certain interval. If I understand correctly how this works, once the key expires, the directory can no longer be decrypted.* This feature should, obviously, be used with care.
Decrypting the directory can be done in the following way:
gpg --output target-dir.tar.bz2 --decrypt target-dir.tar.bz2.gpg
The same two-step process of decrypting, then untar'ing/uncompressing, applies to these two methods as well.
This sums up what I have to offer in this entry. Now that winter is upon us northern-hemispherers, I may be able to post more frequent entries. There are a few things I've been wanting to document for some time now.
* Correction: an anonymous commenter writes of my claim that the key expiration makes decrypting the file no longer possible that "Unfortunately not. The expired key is no longer trusted but is still functional."
Saturday, December 8, 2012
Seventh installment: imagemagick trick 1 of (?)
During a recent vacation, I needed to convert some digital images taken by a relative's camera from a raw format into something that would display better on my laptop. As usual, imagemagick came to the rescue. Details are included below.
imagemagick has been called something like "the swiss army knife of image manipulation utilities." I've utilized this application on many occasions over my years as a GNU/Linux user and been amazed to have as yet found nothing that it cannot do. Moreover, it's a CLI-friendly tool, which makes it doubly appealing for someone like me who favors using the least resource-intensive application for any given job.
I should mention that, not only was I converting these images to a different format with which my laptop could better deal, desirable, but resizing them was needed as well. You see, each photo taken by this fairly high-end camera came in at about 25 megabytes, and the dimensions of each image were probably in thousands of pixels. So I wanted to both resize and convert them; but, based on past experience with imagemagick, I was sure it would be up to the job.
I was surprised, after some initial searching on how to accomplish this task, at how simple this task would turn out to be. After mounting the Secure Digital card on my laptop, I copied over all the images--which had the .CR2 extension, by the way--to a temporary directory on my laptop.
The incantation I then used, after cd'ing to the temporary directory where the images resided, was mogrify -format jpg -geometry 1024x768 *.CR2. It should be fairly obvious what this does, but just in case it's not, let me explain. The mogrify command is part of imagemagick's image manipulating routines and can be invoked to do a number of things with an image, but in this case the -format switch, followed by the jpg extension, tells it to convert from one format to another.
Though most readers of this blog are likely to know this already, it will be worth mentioning that the asterisk preceding the file extension is the wildcard symbol. Thus, all files in that directory with the designated extension get converted. Incidentally, they are given the same prefix as the raw .CR2 files, but after conversion have the .jpg extension instead of the .CR2 extension.
I should mention here that, although raw formats are not especially new, I did run across some indications while researching how to do this, that older versions of imagemagick might not be able to operate on such formats. But I believe any version that's come out in the last 2 or 3 years should understand and be able to manipulate these raw formats.
The -geometry switch tells imagemagick to resize the image from something in the thousands of pixels down to a size that displays better, and takes up a whole lot less disk space, on this laptop. The switch does also have a somewhat undesirable effect on photos taken in vertical rather than horizontal orientation, since, as written to disk, these are taller than they are wide (whereas photos taken in horizontal orientation are wider than they are tall): the program thus makes the height of the photo 768 pixels, while the width, being kept proportional, turns out to be quite a bit less than that. But for my purposes--which was simply being able to view quickly these fresh photos on my laptop--these somewhat under-sized photos did not pose any particular problem.
The process of converting many photos this way can be lengthy, but, as you have seen, it is quite simple. I would simply start the process, then either do some other task in another desktop while awaiting completion, or just set the laptop aside while it was running the process. After about a half hour or so, I was ready to view photos.
Above is a sample of one of the converted images. Oh, and by the way, the "(?)" in the title of this entry is meant to indicate the multitude of other imagemagick tricks, as yet unknown to a relatively casual user such as myself, there are likely to be.
imagemagick has been called something like "the swiss army knife of image manipulation utilities." I've utilized this application on many occasions over my years as a GNU/Linux user and been amazed to have as yet found nothing that it cannot do. Moreover, it's a CLI-friendly tool, which makes it doubly appealing for someone like me who favors using the least resource-intensive application for any given job.
I should mention that, not only was I converting these images to a different format with which my laptop could better deal, desirable, but resizing them was needed as well. You see, each photo taken by this fairly high-end camera came in at about 25 megabytes, and the dimensions of each image were probably in thousands of pixels. So I wanted to both resize and convert them; but, based on past experience with imagemagick, I was sure it would be up to the job.
I was surprised, after some initial searching on how to accomplish this task, at how simple this task would turn out to be. After mounting the Secure Digital card on my laptop, I copied over all the images--which had the .CR2 extension, by the way--to a temporary directory on my laptop.
The incantation I then used, after cd'ing to the temporary directory where the images resided, was mogrify -format jpg -geometry 1024x768 *.CR2. It should be fairly obvious what this does, but just in case it's not, let me explain. The mogrify command is part of imagemagick's image manipulating routines and can be invoked to do a number of things with an image, but in this case the -format switch, followed by the jpg extension, tells it to convert from one format to another.
Though most readers of this blog are likely to know this already, it will be worth mentioning that the asterisk preceding the file extension is the wildcard symbol. Thus, all files in that directory with the designated extension get converted. Incidentally, they are given the same prefix as the raw .CR2 files, but after conversion have the .jpg extension instead of the .CR2 extension.
I should mention here that, although raw formats are not especially new, I did run across some indications while researching how to do this, that older versions of imagemagick might not be able to operate on such formats. But I believe any version that's come out in the last 2 or 3 years should understand and be able to manipulate these raw formats.
The -geometry switch tells imagemagick to resize the image from something in the thousands of pixels down to a size that displays better, and takes up a whole lot less disk space, on this laptop. The switch does also have a somewhat undesirable effect on photos taken in vertical rather than horizontal orientation, since, as written to disk, these are taller than they are wide (whereas photos taken in horizontal orientation are wider than they are tall): the program thus makes the height of the photo 768 pixels, while the width, being kept proportional, turns out to be quite a bit less than that. But for my purposes--which was simply being able to view quickly these fresh photos on my laptop--these somewhat under-sized photos did not pose any particular problem.
The process of converting many photos this way can be lengthy, but, as you have seen, it is quite simple. I would simply start the process, then either do some other task in another desktop while awaiting completion, or just set the laptop aside while it was running the process. After about a half hour or so, I was ready to view photos.
Saturday, September 22, 2012
Addendum to the first installment: yet more on screencasting with ffmpeg
With a recent upgrade (apt-get dist-upgrade to 12.04, custom-built Ubuntu) to my office machine, I started having serious audio/video sync issues when producing screencasts using recordmydesktop. As you may (or may not) recall, I record lectures on my computer and use, as a sort of visual aid, an on-screen whiteboard, where I type key words or phrases about which I'm speaking: that's what I capture from my screen when I'm recording these screencasts. Well, with the updated recordmydesktop, the text on my on-screen whiteboard would begin to appear several seconds before the audio about the word or phrase was indicating its appearance.
Oddly, the opposite was happening with the updated ffmpeg when running the screencasting incantation with which I'd earlier experimented. The on-screen text I was typing into the whiteboard was lagging a bit behind the audio.
I tried introducing a number of alternative switches into the commands I was issuing when using both recordmydesktop and ffmpeg. But to no avail: I couldn't get rid of the sync problems with either.
During the course of my searches aimed at resolving these issues, I ran across a crude script on the Ubuntu forums that someone had cobbled together, a script which uses ffmpeg, but which separately records video and audio, joining the two streams together, as a final step, into a final output file. I think this joining of an audio and video file are called, in electronic multimedia circles, "muxing," by the way. I decided the script was worth a try.
And, what do you know, after figuring out how to use the script, my tests indicated that it caused audio and video to be in nearly perfect sync. Thus, the answer to my newly-appeared screencasting issues was resolved.
I hoped to solicit improvements to the script but have so far not managed to find much help. Probably the weirdest thing about this script, which likely demonstrates the inexperience of the script's creator, is the fact that, once ffmpeg is invoked and the recording begins, you're supposed to enter, into the same terminal where ffmpeg is running, a file name, then hit the "enter" key as the singal to stop the recording and begin the joining of the audio and video files.
All this while you're seeing in the terminal the standard ffmpeg prompt that tells you to hit control-c to stop the recording. Confusing, to say the least--and made even more confounding by the fact that you can't actually see the text you're entering when you go to type the file name.
Despite those shortcomings, since the results produced by this script exceed anything else I've been able to accomplish, I think I'm going to stick with the script for now. I have made a couple of tweaks, mainly so as to make it record what are called "lossless" files--files that are produced with minimal processing (for example no compression), and which are therefore quite large. I have to re-encode my output files before uploading them in any case, so it's best to start with better-quality files.
Without further ado, then, I present the tweaked version of the script I've found:
I should mention that I discovered a slightly less incoherent way of soliciting file-name input. Though it is commented out in the version of the script you see above, I intend to use this until such time as the script can be further improved ( to use it you uncomment the line that begins out= and comment out the line that reads read -p: note that you must have zenity installed for this modification to work).
In a related item, as I've mentioned earlier, the disadvantage to using ffmpeg for screencasting is that there is no built-in provision for pausing. Well, apparently someone has proposed a kind of workaround for that--see this thread for further details. I've not tried that method and don't really understand how it works, so I cannot attest to its efficacy.
What I have tried is simply stopping, then restarting a new file when a pause is necessary. That's definitely more cumbersome than pausing, and, furthermore, it requires the additional step of somehow joining what could be thought of a separate "vignettes" into a single "episode."
The good news I can report on that front is that I've found another script that was created precisely to join such separate files. I've tried some tests with it and it has worked for me quite well. It's called mmcat and it can be found here.
I'd like to post more about the plughw switch seen in the above script and which I needed to introduce in order to record through a new USB sound device I've added to my computer. But I don't really understand well what differentiates it from the more standard hw switch. So I won't speak to that matter further in this entry. :)
That about sums things up so far as recent screencasting developments on my front is concerned. Do you have any suggestions for improving the screencasting script I found? If so, please pipe in. Any other suggestions for pausing ffmpeg screencast recording? Please let me/us know.
Oddly, the opposite was happening with the updated ffmpeg when running the screencasting incantation with which I'd earlier experimented. The on-screen text I was typing into the whiteboard was lagging a bit behind the audio.
I tried introducing a number of alternative switches into the commands I was issuing when using both recordmydesktop and ffmpeg. But to no avail: I couldn't get rid of the sync problems with either.
During the course of my searches aimed at resolving these issues, I ran across a crude script on the Ubuntu forums that someone had cobbled together, a script which uses ffmpeg, but which separately records video and audio, joining the two streams together, as a final step, into a final output file. I think this joining of an audio and video file are called, in electronic multimedia circles, "muxing," by the way. I decided the script was worth a try.
And, what do you know, after figuring out how to use the script, my tests indicated that it caused audio and video to be in nearly perfect sync. Thus, the answer to my newly-appeared screencasting issues was resolved.
I hoped to solicit improvements to the script but have so far not managed to find much help. Probably the weirdest thing about this script, which likely demonstrates the inexperience of the script's creator, is the fact that, once ffmpeg is invoked and the recording begins, you're supposed to enter, into the same terminal where ffmpeg is running, a file name, then hit the "enter" key as the singal to stop the recording and begin the joining of the audio and video files.
All this while you're seeing in the terminal the standard ffmpeg prompt that tells you to hit control-c to stop the recording. Confusing, to say the least--and made even more confounding by the fact that you can't actually see the text you're entering when you go to type the file name.
Despite those shortcomings, since the results produced by this script exceed anything else I've been able to accomplish, I think I'm going to stick with the script for now. I have made a couple of tweaks, mainly so as to make it record what are called "lossless" files--files that are produced with minimal processing (for example no compression), and which are therefore quite large. I have to re-encode my output files before uploading them in any case, so it's best to start with better-quality files.
Without further ado, then, I present the tweaked version of the script I've found:
I should mention that I discovered a slightly less incoherent way of soliciting file-name input. Though it is commented out in the version of the script you see above, I intend to use this until such time as the script can be further improved ( to use it you uncomment the line that begins out= and comment out the line that reads read -p: note that you must have zenity installed for this modification to work).
In a related item, as I've mentioned earlier, the disadvantage to using ffmpeg for screencasting is that there is no built-in provision for pausing. Well, apparently someone has proposed a kind of workaround for that--see this thread for further details. I've not tried that method and don't really understand how it works, so I cannot attest to its efficacy.
What I have tried is simply stopping, then restarting a new file when a pause is necessary. That's definitely more cumbersome than pausing, and, furthermore, it requires the additional step of somehow joining what could be thought of a separate "vignettes" into a single "episode."
The good news I can report on that front is that I've found another script that was created precisely to join such separate files. I've tried some tests with it and it has worked for me quite well. It's called mmcat and it can be found here.
I'd like to post more about the plughw switch seen in the above script and which I needed to introduce in order to record through a new USB sound device I've added to my computer. But I don't really understand well what differentiates it from the more standard hw switch. So I won't speak to that matter further in this entry. :)
That about sums things up so far as recent screencasting developments on my front is concerned. Do you have any suggestions for improving the screencasting script I found? If so, please pipe in. Any other suggestions for pausing ffmpeg screencast recording? Please let me/us know.
Wednesday, July 18, 2012
Addendum to the first installment: more on screencasting with ffmpeg
Long time no blog. I got busy with a lot of other things, not least of which was various bicycling trips and associated maintenance projects. Amazing how much time and energy those things can take.
With the academic year approaching, coming up with a resolution to my screencast file-size problems has taken on renewed urgency. To reiterate, a recent major hardware upgrade on my computer led to a twofold increase in the size of screencast video files I produce on this machine--which in turn led to upload problems at my course web site (file size limits). It looks like the resolution may well be provided by ffmpeg.
So I began once again grappling in earnest with the increased file-size issue. I've actually even found a resolution, should I decide to switch over from the more capable recordmydesktop to the nice but somewhat feature-lacking ffmpeg.
Before launching into a description of that resolution, however, I need to make quick note of a new entry into the field of GNU/Linux screencasting applications. The new kid on the block is called Kazam and it looks promising. I've not yet been able to get it running--and I'm not in a big rush to do so, since I favor command-line tools. But it's received some praises and looks like a promising application. That noted, I now return to a description of the resolution I've discovered for the issue of screencast file sizes made using ffmpeg's x11grab.
First and foremost, I discovered a fairly lengthy thread on the Ubuntu forums that deals with screencasting usingffmpeg. It's chock full of all kinds of interesting tips for screencasting with ffmpeg. For example, a switch is documented there that allows you to exclude the mouse cursor from your screencast: you simply add :0.0+nomouse to the command you use to start your screencast. I'd definitely need that in order to use ffmpeg as the utility to record my lectures.
Also described there is a way of doing what might be called "pseudo pausing," which really means you just stop your recording at the point where you need to pause, then start up again with a new recording. You then need to concatenate the files--not anywhere near as convenient as recordmydesktop's pausing capability, but something that'll do in a pinch.
It was from that thread that I derived what seems like a pretty good resolution to my file size issues. What I've discovered is that I can record my files in a lossless format (mkv is recommended in that thread)--which results in pretty gigantic file sizes (ca. 13 megabytes per minute on this system)--then transcode them into flash format while dramatically decreasing the file size: a one-minute test file I made actually came in at about 1.2 megabytes after re-encoding using this method. Since what I need is a file that comes in at about 2 megabytes per minute, this could work well for my circumstances.
The key command that's allowed me to reduce dramatically screencast file sizes to an acceptable level while at the same time converting to a format that works well at my hosting site uses . . . you guessed it, ffmpeg. Here it is:
As a final note on things I've learned in this round of screencast research, I should also mention a command-line tool called ffcast someone has developed. This looks like a bash script that first invokes a tool for selecting an area of the screen, then calls ffmpeg to record the screencast. I've also not tried that one but it's on my list to look into in the future.
POSTSCRIPT: I decided to run on an .ogv file created by recordmydesktop the same command I'd used to shrink the lossless .mkv file to see what the results would be. I'm pleased to report that the results are positive: the .ogv file, when re-encoded as an .flv using that method, shrunk to about one third the size (the 1 minute .ogv test file was 4.2 megabytes and it came in as a 1.2 megabyte .flv file after re-encoding). So I can continue using recordmydesktop to record my lecture screencasts--albeit with the penalty of having to do an additional round of encoding/re-encoding (recordmydesktop already takes quite some time to encode a screencast into .ogv format unless you use the --on-the-fly-encoding switch, which I'm now likely to start using). So, the command I used to re-encode the .ogv file to .flv is:
With the academic year approaching, coming up with a resolution to my screencast file-size problems has taken on renewed urgency. To reiterate, a recent major hardware upgrade on my computer led to a twofold increase in the size of screencast video files I produce on this machine--which in turn led to upload problems at my course web site (file size limits). It looks like the resolution may well be provided by ffmpeg.
So I began once again grappling in earnest with the increased file-size issue. I've actually even found a resolution, should I decide to switch over from the more capable recordmydesktop to the nice but somewhat feature-lacking ffmpeg.
Before launching into a description of that resolution, however, I need to make quick note of a new entry into the field of GNU/Linux screencasting applications. The new kid on the block is called Kazam and it looks promising. I've not yet been able to get it running--and I'm not in a big rush to do so, since I favor command-line tools. But it's received some praises and looks like a promising application. That noted, I now return to a description of the resolution I've discovered for the issue of screencast file sizes made using ffmpeg's x11grab.
First and foremost, I discovered a fairly lengthy thread on the Ubuntu forums that deals with screencasting usingffmpeg. It's chock full of all kinds of interesting tips for screencasting with ffmpeg. For example, a switch is documented there that allows you to exclude the mouse cursor from your screencast: you simply add :0.0+nomouse to the command you use to start your screencast. I'd definitely need that in order to use ffmpeg as the utility to record my lectures.
Also described there is a way of doing what might be called "pseudo pausing," which really means you just stop your recording at the point where you need to pause, then start up again with a new recording. You then need to concatenate the files--not anywhere near as convenient as recordmydesktop's pausing capability, but something that'll do in a pinch.
It was from that thread that I derived what seems like a pretty good resolution to my file size issues. What I've discovered is that I can record my files in a lossless format (mkv is recommended in that thread)--which results in pretty gigantic file sizes (ca. 13 megabytes per minute on this system)--then transcode them into flash format while dramatically decreasing the file size: a one-minute test file I made actually came in at about 1.2 megabytes after re-encoding using this method. Since what I need is a file that comes in at about 2 megabytes per minute, this could work well for my circumstances.
The key command that's allowed me to reduce dramatically screencast file sizes to an acceptable level while at the same time converting to a format that works well at my hosting site uses . . . you guessed it, ffmpeg. Here it is:
ffmpeg -i output.mkv -acodec aac -strict experimental -ab 128k -ac 2 -vcodec libx264 -vpre slow -wpredp 0 -crf 22 -threads 0 output.flvI do still hope I can find a way of reliably shrinking the files recordmydesktop outputs, but my research on that has, to date, not been fruitful. There's a utility called oggResize that's supposed to allow you to easily resize .ogv files, but it puts the audio out of sync with the video. And it appears that tool is not under active development. It's likely resizing of .ogv files can also be done with ffmpeg, but I've not yet managed to arrive at the appropriate incantation for doing that. I'll keep digging though, and if I come up with a resolution, I'll post it in this blog.
As a final note on things I've learned in this round of screencast research, I should also mention a command-line tool called ffcast someone has developed. This looks like a bash script that first invokes a tool for selecting an area of the screen, then calls ffmpeg to record the screencast. I've also not tried that one but it's on my list to look into in the future.
POSTSCRIPT: I decided to run on an .ogv file created by recordmydesktop the same command I'd used to shrink the lossless .mkv file to see what the results would be. I'm pleased to report that the results are positive: the .ogv file, when re-encoded as an .flv using that method, shrunk to about one third the size (the 1 minute .ogv test file was 4.2 megabytes and it came in as a 1.2 megabyte .flv file after re-encoding). So I can continue using recordmydesktop to record my lecture screencasts--albeit with the penalty of having to do an additional round of encoding/re-encoding (recordmydesktop already takes quite some time to encode a screencast into .ogv format unless you use the --on-the-fly-encoding switch, which I'm now likely to start using). So, the command I used to re-encode the .ogv file to .flv is:
ffmpeg -i output.ogv -acodec aac -strict experimental -ab 128k -ac 2 -vcodec libx264 -vpre slow -wpredp 0 -crf 22 -threads 0 output.flvMUCH LATER POSTSCRIPT: I've discovered that ogv files work much better at my hosting site than do flv files, so I began looking into ways of transcoding/shrinking to that format. The incantation I discovered, through trial and error, and which reduces screencast file sizes to about 1.5 megabytes per minute (roughly one seventh of the original, lossless, size), is as follows:
There are two crucial bits here. The -ab 48k option is one of them: that reduces the audio bit rate, which brings the file size down by quite a lot. The other is the -r 13 option, which reduces the frame rate to 13 frames per second and thus shrinks the file size yet further. A 48 minute long lossless screencast I created, for example, was reduced to a 73 megabyte ogv file using this incantation.ffmpeg -i infile.mkv -r 13 -acodec libvorbis -ab 48k -ac 2 -vcodec libtheora -preset slow -wpredp 0 -crf 22 -threads 0 outfile.ogv
Subscribe to:
Posts (Atom)