Monday, October 29, 2007

Secrets of Unix :PART 1

Since many people are asking about some neat command line UNIX tricks, I've decided to compile them together into a webpage as two parts. Actually, these are "secrets" for tsch, which is a popular UNIX shell (most commands will work equally well on csh, its predecessor). Also note that versions of programs differ among UNIX systems, and some of the commands may not work exactly as I have described them on your system.

This isn't a comprehensive introduction to UNIX, but rather a collection of little known features that can enhance one's experience in a multi-user UNIX environment, such as is found on most college campuses. This does not replace the manual pages (type "man commandname" at the UNIX prompt) but serves to make people aware of first the existence, and second the general utility, of some commands.

In this part the commands are grouped into seven basic categories

1.Multi-user Communication
2. E-mail
3.Remote Communication
4.Text Editing
5.Web Browsing
6.Process Management
7.Symbols


1.Multi-user Communication:

finger
finger username gives you specific information about a user, including his real name, home directory, and .plan file, if one exists. If the user is logged on, finger will tell you from where he is logged on and how long he has been idle. If the user is not logged on, finger will return the time, date, and address of his last connection. If you just type finger you'll return a list of all users currently logged on to the system. Sometimes users may set preferences to block being fingered, but finger | grep username will allow you to see if they are logged in nonetheless. A neat trick is to set up a text file named .friends with each of your friends' usernames on a new line, and then type finger | grep -f ~/.friends to see which are of your friends are logged in all at once.

last
last username tells you when (and from what address) a user last logged on and when they logged off. Generally the system will recycle the log after a period of days, such as week. Note carefully that only a system with multiple physical machines (for example fas.harvard.edu actually refers to ten physical machines, is01.harvard.edu.edu to is10.harvard.edu, to better distribute the load) last will provide you with information only from the machine which you are currently logged on to. Finger will tell you what machine they last logged on to.

who/rwho
who shows a list of all users logged onto your machine, and rwho shows a list of all users logged onto machines connected to the same local host. In the case of fas.harvard.edu, for example, who shows a list of all users logged onto the same physical machine, while rwho shows a list of all users on all of the ten Harvard IS machines. Type rwho -a to list users who have been idle for more than an hour. Generally finger is more useful, but rwho counts idle time in a generally more useful way.

w
w username tells you what programs a user who is logged into the same physical machine is currently running. Generally, you'll find more information from ps -u username instead of using w.

.plan

The .plan file is a text file in that you can create in your home directory. If it exists, it can be displayed when a user fingers you. Users often put additional contact information in their .plan files, or what they working on at the moment, or a bit of humor. Be sure that the permissions are set correctly so that other users can read your .plan file.

talk
talk username lets you invite another user to chat interactively with you. The user must be logged in, of course, but also must be on the same physical machine. Thus you may finger a user to determine if they are logged in, but when you use the talk command, it may say that the user is not logged in. Look carefully at their finger information, and use talk username@host, where the host is given by the finger command. Then you can invite them to a chat. Of course, the user must accept your chat invitation. Unlike some instant messaging services, talk is real-time, so your keystrokes are transmitted as they are made. It's often good protocol to "double-space" your chat to make it easier to read, and you can periodically clear the window by holding the key. Quit the chat by + (often written as ^c)

ytalk
ytalk is a variant of the talk program that allows multiple users to participate in a single chat. Invite multiple users by entering ytalk username1(@host1) username2(@host2) etc. You can, of course, omit the host if you are all on the same physical machine.

write
write username(@host) lets you write a message on the following lines and send it as a quick "pop-up" message to a user. Terminate your message with ^c

ph
ph is a very powerful program that lets you look at the phone directory on your system so that you can use one piece of information about a user to find other information. Most often you know the real name of a user, and want to find their e-mail address, physical address, or telephone number. If you know the person's exact name, you just type it in, although you'll get more nicely formatted output if you type query realname at the ph> prompt. Sometimes you may not know his entire name, and so you can use the standard * (many characters) and ? (single character) wildcards to help narrow your search. Sometimes the search can be finicky, so if you're not finding the person you are looking for, be creative, and try possible alternatives (e.g. joe, instead of joseph) or widen your search with more wildcards. If you type fields at the ph> prompt, you can also get a list of indexed fields. To search for someone by their email address for example, type query email=username@host at the ph> prompt. Systems may use slightly different names for fields, and not all fields are indexed. Some implementations of ph require you a enter a query mode, by typing query at the ph> prompt, and some will you to modify your own directory information by logging in, and editing fields.



2.E-mail (using pine):

pine

pine is a very full-featured text-based e-mail program. This section enumerates some useful features that many users may not know about. While some are seduced by the graphical interface of POP/IMAP clients such as Eudora (or, yes, even Outlook) pine holds a number of advantages, especially on a college campus. If you check your e-mail from multiple computers, running pine ensures that you have all of your e-mail available from any location. Furthermore, unless you download and run a suspicious file, using pine renders you impervious to e-mail viruses. Pine is a powerful program, and generally quite easy to use. You start it from the shell prompt with the command pine, although some prefer to begin pine with immediately opened to their inbox, which can be accomplished by typing pine -i

nfrm

nfrm is a sometimes finicky command that tells you whether you have any new mail, and if so it will print the sender and subject. This can let you check to see whether you really want to read your new mail right away without actually opening pine. You might find that the command nfrm -s unread is more reliable.

rich headers

When composing a message, if your cursor is in the header area (the To, Cc, Attachment, Subject area) pressing ^r (remember, +) Will activate the rich header. The most important addition is the Bcc, or blind carbon copy field. Entering addresses in this field will all you to send mail to these addresses without them knowing that the other addresses in this field have also received the message. They will still see the To and Cc fields, however. As a courtesy, the Bcc field should be used when you are sending to a large list-not for secrecy, but because no one wants to read tens or hundreds of e-mail addresses. With the rich header, you can also control which folder the message will be copied to when it is sent (Fcc) among other things.

address book

Having an address book can be a useful feature in pine. You can make both individual entries so that you don't have to remember complicated usernames, as well as making lists of users (e.g. chemistry study group, office-mates, etc.) so that you can easily send to a group. Make changes to the address book from the main menu of pine (press from the inbox) and you can enter names from the address book either by typing the name or nickname directly into the To, Cc, or Bcc fields, or type ^t to choose from a menu.

attaching files

It is often very useful to attach files to e-mail messages. You can easily attach any file in your home directory (or a subdirectory) by either typing in the exact name of the file (case-sensitive) into the Attachment field, or type ^t (when the cursor is in the field) to choose from a menu. You can attach multiple files, and you can easily delete a file you mistakenly attached by moving the cursor to the attachment line and type ^k. Another very useful, but little known feature, is the ability to insert a text file directly into the body of a message. When the cursor is in the body of the message type ^r and then enter the file name or type ^t to choose from a menu. This is really useful, because you can store frequently used bits of text in files in your home directory and insert them very quickly.

editing features

The pine editor (pico) is fairly powerful, and has a number of time saving features. You can delete an entire line by typing ^k, and you can delete multiple lines by holding down this keystroke (be careful!) Actually, the text isn't really deleted, but stored in a temporary buffer, and you can "paste" it back into the message by typing ^u. You are free to move the cursor around, and continue to type in between "cutting" and "pasting" but be careful not to remove more text with ^k before using ^u if you want to save what you previous removed. Sometimes the text is a paragraph will break in an odd way after you've had to make changes, to typing ^j when the cursor is in a paragraph will rearrange the text so that it is nicely justified. Just be careful, for this command can also destroy careful formatting of lists, computer code, or any other text that isn't standard prose. ^a will move the cursor to the beginning of the current line, while ^e will move it to the end of the current line. ^y and ^v will page up and page down respectively. You can also use ^l to redraw the screen (useful when pine interrupts your composition with a new message arrival or chat request). And while everyone knows you can attach files to a message with pine, you can also insert a text file in the body of a message with ^r, which makes a useful way to compose many messages with the same text is part of a message (i.e. save the body of the message to a text file and write a personal greeting to each recipient in the body and then insert the text file.

spell checking

Most versions of pine also include a spelling checker, which can be accessed with the ^t command when the cursor is in the body of the message. The default spelling checker, spell, will identify misspelled words, but you will have to provide a replacement, or ignore the word. Often, correcting a misspelling can change the flow of the text in a paragraph, and so the ^j command is useful to reformat the text. If your system includes the ispell program, you can setup pine to use this spelling checker by default, which allows you to have a customized dictionary of words, and will suggest spelling replacements. To enable ispell is quite simple: go to the main menu of pine by pressing (m) then (s) for setup and © for config, then press to search, type "speller" and enter "ispell" into the field.

postponing

Conveniently, pine allows you to postpone messages as you are composing them, to return to them later. Press ^o when composing a message to postpone it. When you try to compose a message again (by pressing c from the inbox or main menu) you will be prompted to either resume a postponed message or start a new message. You can postpone many messages. Sometimes your composition can be interrupted, such as by a network or computer problem. Pine is almost always able to automatically save the incomplete message, and if any such messages exist, you will be asked to resume them in the same manner when you next compose a message. Not resuming a message does not delete it, but just postpones it again.

flags

When a message first arrives it will have a New (N) flag. When you open the message the N flag will disappear. When you reply to a message it will have an Answered (A) flag. When you delete a message it will have a Deleted (D) flag, until you actually exist from the folder, when such messages will be deleted. You can manually change the flag of a message by pressing <*> in the message list, and then following the instructions at the bottom of the screen. Sometimes it's useful to keep the New flag on an important message that you want to remind yourself to reply to. You can also quickly remove a deleted flag by pressing (u) while the message is selected. Incidentally, if you've ever wondered what those plus signs are doing next to some messages at the far left, that means that the message was directed directly to you, as opposed to being sent to you as a carbon copy.

saving attachments

When you receive a message with attachments, you can press v to view the attachments, and then s to save the attachments to your home directory. You can then work with them on Unix, or download them via ftp onto another computer. Sometimes you want to save a message, but not its attachments, because they are so large that they take up your disk quota. To delete the attachments first press v to view the attachments, and then select one, press d to add a deleted flag to it, then press < to return to the main message view, finally press s to save the message, without the deleted attachments, in any of your mail folders. This is also useful for messages copied to your sent-mail folder that have large attachments.

folders

It is often helpful to set up different folders to organize your mail. From the main menu you can view the folder list, and easily add or delete folders. It's often a good idea to periodically move old messages from your inbox into a saved mail folder, so that your inbox doesn't become too large (if its too large, it's prone to corruption, and pine will run very slowly). Thousands of messages is probably too large.

selecting

From a folder view the select command, accessed by pressing <;> can be very helpful. You can select all messages in a folder received before a certain date, for example, and then apply various commands to them by pressing a. For example, you can delete them all at once, save them to a saved-messages folder, or perform other tasks.

searching

If you're looking for messages sent by a certain person or about a certain subject, you can use WhereIs command to search in the folder view by pressing , and then entering a string to search for and then . You can repeatedly search by pressing and as pine will automatically continue your last saved search.

sorting

Sometimes it's easier to resort your messages rather than searching for a particular name. This can be easily accomplished with the <$> key, followed by a choice from the list at the bottom of the screen. The default sort is by date, and this is preferable in general, but it can be useful to temporarily sort by the sender or subject, for instance. If you're close to your disk quota it's helpful to sort by size so that you can either delete the largest messages, or just delete the attachments! Don't forget to revert to sorting by date when you're finished.

signature

Many people like to put a signature at the bottom of each of their e-mails. You can get pine to automatically add whatever text you choose to the bottom of your e-mails by going to the main menu, press (s) for setup, and then (s) for signature. You edit your signature just like composing a mail message, and save the changes.

preferences

There's more, however, to the setup options than just setting a signature. Many of these options are somewhat obscure, but there are a few that may be particularly useful. The first field you'll see is your own name, which is what people receiving your message see. It's good to actually have your full name, and properly capitalized to look professional. If you're sending out messages on behalf of a group, you might want to temporarily change your name (your e-mail address will stay the same, of course), but remember to change it back! You can also set an alternative program to function as your spelling checker, which is a good idea if your system includes the ispell program. Some people like their messages to automatically move to new folder when read, and so you can setup a Read-Message-Folder. Some people find it annoying that pine asks whether you really mean to quit, so you can toggle the quit-without-confirm option. You can also set the ^k command to cut from the cursor position to the end of the line, rather than an entire line with the compose-cut-from-cursor option.

spawning lynx

If you'd like to be able click on url links in html mail that you receive, pine can be set up to automatically spawn a web browser of your choice (generally lynx, unless you're using X-Windows). From the pine main menu go to the setup and config sub-menu (as above). Use a whereis query to find "url" and check "enable-msg-view-urls" and "enable-msg-view-web-hostnames" At the very bottom of the list you can specificy a url-viewer-application, which should be the full path of lynx (type which lynx at the shell prompt to find the full path). After saving the changes, you should be all set, although ocaisonally this feature fails to work on systems that setgid pine. You can get around this problem by runing pine in a different enviornment along the lines of alias pine 'env SHELL=/bin/sh pine'



3.Remote Communication:

telnet
telnet is a basic program that allows you to connect to remote machines that accept telnet protocol logins. Because telnet does not encrypt passwords, many sites do not accept telnet logins since they are afraid of password sniffers that allow malicious use of accounts. This is actually a very small problem for individual users, and shouldn't worry you when you login to a site via telnet, but explains why some administrators disallow telnet access, in favor of secure protocols, such as ssh. Of course, when you telnet to a site, such as library catalog, which doesn't require a password, there is absolutely no danger! To connect to a particular host type telnet hostname. Ordinarily you end a telnet session by gracefully exiting from the remote machine, although you can always type ^] to enter the interrupt mode, from which you can quit a session.

ssh
ssh is essentially the high-security cousin of telnet. Where possible, you might as well use ssh to connect to a remote machine. The syntax is ordinarily ssh hostname, but the default is to assume that your current username is to be used as the username for the remove machine to which you are trying to connect. Since this is not always the case you can type ssh username@hostname instead to connect to a remote host with a different username. The first time you try to connect to a remote machine with ssh you'll received a "Host key not found from database" message, and a prompt to continue connecting, which you can respond to appropriately.

ftp
telnet is a basic program that allows you to connect to remote machines that accept file transfer protocol logins, and then transfer files and/or directories back and forth. Like telnet ftp does not encrypt passwords, and so many sites no longer allow ftp logins. The basic syntax is ftp hostname and you are then prompted for a username and password. Anonymous ftp sites, such as public domain software repositories allow any user to connection with the username anonymous and any password (sometimes your e-mail address). Other ftp hosts require a username and password to be prearranged-if you have telnet access to the host the username and password are almost certainly the same. You can list the contents of the remote directory with the ls command, and download files from the remote directory to your local working directory with the get command for a single file, or the mget command for multiple files that match a pattern. Likewise, you can upload a file from your local directory of the ftp site with the put command, or the mput command for multiple files. If you want to upload or download a file with a space in its filename, enclose the filename in quotes. You can change the local directory with the lcd command, and the remote directory with the cd command, and type quit to exit.



sftp
sftp is bears the same relation to ftp as ssh bears to telnet. When possible, or required, use sftp to transfer files between hosts. You can logon to remote machine with the syntax sftp (user@)hostname and will be prompted for a password. Other commands are identical with ftp.

scp
scp is the poor man's sftp, and can be used to copy a specific file between hosts. It can be more difficult to use than the interactive sftp, although for certain automated tasks, scp is more suitable.



4.Text Editing:

pico
pico is a general purpose editor that is very easy to use. If you've ever used pine to compose e-mail, then you've actually been using pico. It's also helpful to look at this nice list of the keyboard shortcuts for pico.

emacs
gnu emacs is an editor of immense power and complication-other than the fact that they both can be used to create and edit text files, emacs is everything that pico is not. An introduction to emacs won't even be attempted here, save for two points: ^h t will start the excellent emacs tutorial, and ^x^c will quit the program. If you're using an x-terminal, you may want to try launching xemacs, although even ordinary emacs will have some graphical elements when opened on an x-terminal. You can also install emacs or xemacs on your windows machine.

vi/vim
vi is another common Unix text editor, and vim is often found in Linux. It is also a powerful editor, with two-modes, an editing mode and a command mode. It is also quite complex, but emacs is a much better editor! Alas, if you insist upon using it, a short list of commands should get your started.

ps2pdf/pdf2ps
Since many people do not own the full version of Adobe Acrobat, software that allows you to creative PDF (portable document format) files that are readable (and look the same) on any machine with Acrobat Reader (almost every computer!), being able to create PDF files nonetheless is very helpful. Proprietary formats, such as Microsoft Word are unacceptable for transfer to other people (and platforms) as they may not have the required software, the same fonts, and the document may not appear the same anyway. By printing your document from any application to a file using a PostScript printer driver you can make a PostScript (.ps) file that can be converted to a pdf file with the syntax ps2pdf orig.ps new.pdf The reverse process can be accomplished with the pdf2ps command, which can be useful if you want to send a pdf file directly to a PostScript printer without viewing it first.

enscript/ps2ascii
The enscript command lets you create a postscript file directly from ascii text, which can be useful if you want to send the text to a PostScript printer. The ps2acsii command distills acsii text out of PostScript file, which can be useful for quick searching or viewing.

acroread
When using the X-Windows graphical interface, the acroread command will invoke the graphical viewer for PDF files.

gs
gs stands for ghostscript, which is an extremely powerful and versitile program for working with PostScript and PDF files. It can be invoked graphically under X-Windows, or specific operations can be performed on files with the command line.

more

more filename allows you to quickly view the content of a file in plaintext one page at a time. It is often useful to pipe the output of a command to more; for example ls | more will list the contents of the current directory one page at the time.

cat

cat filename writes the output of a file in plaintext all at once. Often, more is preferred because it will pause the output when it reaches the end of the screen, but there are also many times when you want to view a file all at once, perhaps because you want to use a terminal program to scroll up later, for example

less
less filename allows you to quickly view the content of a file in plaintext one page at a time. It is similar to more, but incorporates a plethora of additional features, particularly being able to scroll backwards as well as forward in a file. It is most useful as a quick way to look through a very large file, since the entire file doesn't have to be loaded, but generally if you're going to spend the time to use less, you might as well load an editor!

head

head filename allows you to quickly view the first ten lines of a file. You can view any number of lines by head -numberoflines filename, so head -14 foo.txt will show the first fourteen lines of the file foo.txt

tail

tail filename allows you to quickly view the last ten lines of a file. You can view any number of lines by tail -numberoflines filename, so tail -14 foo.txt will show the last fourteen lines of the file foo.txt While this is exactly analogous to head, tail does incorporate an additional very useful feature; tail can also be used to display rest of the file after a certain number of lines. Often files contain headers that it is cumbersome to display, and so tail +14 foo.txt displays foo.txt from the fourteenth line until the end of the file.

wc
wc filename will count the number lines, words, and bytes in a file. wc -w filename will count just the number of words, which can be useful when editing long documents.

banner
banner is a quick little utility that creates so-called "banner text" which is a form of ASCII art. Basically, it's a way to make realy big text display on a text-only terminal. Sometimes people want to use it in e-mails or a talk-session (works better with ytalk!) but often it's most used for leaving a message on your screen along the lines of "Don't touch this window!" banner text will create some banner text, and banner a few words will create banner text of "a few words" on separate lines, while banner "a few words" will create banner text of "a few words" all on one line. Be careful about trying to force too much onto a single line-if you exceed the width of the terminal, banner will start wrapping the text around, which becomes unreadable. You can mix and match-strings bounded by quotes will be written on the same line, and any strings separated by spaces will be printed on seperate lines. If you want to use some unsual symbols, you might have to preceed them by a backslash, for example banner !! will try to make a banner out of the last command entered, but banner \!\! will make a banner out of two exclaimation points!

spell/ispell

spell filename will identify possibly misspelled words in a file. If you're system includes the ispell program use it instead, as will allow you to interactively make changes to the file, in addition to suggesting replacements, and allowing a custom dictionary.



5.Web Browsing:

lynx

lynx is a very powerful text-only browser that can be run from the command line terminal. You can go directly to a site by invoking lynx address, or by pressing when already in lynx to go to a new address. You can page through text with , access the history list with , and scroll up and down through links with and and move back and forth through hyperlinks with the and lynx supports many features, such as forms and frames, but does not support javascript, https (secure pages), or, of course, images. Lynx is extremely fast, and can be useful to download files that might otherwise automatically spawn an external viewer on other browsers (for example RealAudio (.ra) will generally spawn the RealPlayer viewer on standard browsers, but can be easily downloaded, without modifying your normal browsing preferences, with lynx). Remember that when you connect to a site with lynx the remote site sees the ip address of the machine that you are logged on to, so if you need to access resources restricted to a certain set of ip addresses (such as certain library resources) this can be useful.

netscape
From the X-Windows interface, you can run the netscape browser to browse the web with images. If you want to be able to use the shell window from which you run netscape, type netscape& to run it in the background.



6. Process Management:

ps
ps shows you a listing of all processes active, including the process identification number (PID). Sometimes it's useful to use the type ps -u yourusername to see all of the processes you have open, especially if you were disconnected from a previous session but left processes hanging.

kill
kill PID# allows you terminate a stalled process from the shell. If there is no ambiguity, you can also kill a process by typing kill %programname, so if I only had one instance of the man process running, I could kill it by typing kill %man without having to use the ps commmand to find its PID#.

jobs
jobs gives you a listing of all the suspended and active jobs running

^z
In many programs ^z allows you to suspend a job as it is being run, to return to later.

fg
fg allows you resume a job that has been suspended (put it back in the foreground).

bg
bg allows you to run a job in the background, so that the shell can be used for other tasks as the job is being processed.



7.Symbols:

&
Placing the symbol & after a command runs the job in the background, so that the shell can be used for other tasks as the job is being processed.

\
The backslash character removes an special meaning from the character that follows it. For example, more test test.txt would produce an error, but more test\ test.txt would display the contents of the file "test test.txt"

|
The vertical bar character represents the very important pipe command. This command allows you to redirect the output of one command immediately to another. For example, to display a directory listing one page at a time the command ls | more can be used. Pipe has a myriad of uses, and is often used fruitfully to pipe output from commands that produce output in some way to commands that filter or process output.

>
The > command can redirect the output of a command to a file rather than to the screen. For example, to redirect a directory listing to the file dir.txt type ls > dir.txt

>>

The >> command allows you to redirect the output of a command to be appended onto a preexisting file. If you also wanted to store the listing of another directory in the same file above, you could type ls subdirectory/ >> dir.txt
;

The semicolon symbol can be used to separate commands entered on the same time. For example, if you wanted to write the time and date to a file, and then append to it the listing of directory contents, the following command could be written on a single line date > dir.txt ; ls >> dir.txt, where the spaces are added for clarity alone.
^

The carat symbol can be used to repeat the previous command while changing part of it. For example, if you first type more test.txt to view the contents of the file "test.txt" and then want to open it with the pico editor, rather than retype the entire line you can simply type ^more^pico and the shell will print the command pico test.txt to the screen and then execute the command. This can be a real time saver.

~
The tidle symbol represents your home directory, and so allows you to quickly access your home directory no matter what directory you happen to be in, just as the forward slash character allows you to easily access the root directory.

!
The exclaimation point can be used to recall previously entered commands. For example !! will recall the last command entered, !-3 will recall the third to last command, !more will recall the last command that began with the string "more" By using the colon along with the exclaimation point you can recall individual words from previous commands. This can be quite powerful, as you can string commands together to create new commands, so !-10:4 | !:2 !-3:$ | !-5:0 executes the fifth word of the tenth to last command and pipes it to the third word in the last command operating on the last word in the third to last command piped to the first word (the command itself!) in the fifth to last command. Whew.

..
Two periods represents one directory level up from your current location, so if you are in /users/username are type cd .. you will be in /users

^c
The + combination is a common interrupt for many processes in Unix.


Important:due to some html tag problems replace() with<> in certain commands.

The rest of this entry will publish in part 2 and that depends on the popularity of the part1.

No comments: