Tips and Tricks of UNIX

I've been working with variations of Unix for a long time  and thought I'd write down some of my favorite tips and tricks. They are mostly OS/distribution/shell/language independent (unless I indicate otherwise...)

1.     Get rid of blank lines in a file
grep . inputfile > outputfile
This matches (and thus prints) only lines that contain some text, not blank (empty) lines.

2.     comm
Many people never cross paths with the comm command, but it is very useful. I works similarly to diff, but outputs the contents of two compared files into three columns. The first column is content only in the first file, second column is content only in the second file, and third column is content that is in both files (matches between the two files.) While this may not seem useful at first, you can select which columns to output, so if you only want to know what is in both file1 and file 2 (column 2) you'd suppress columns 1 and 2, by running:
comm -12 file1 file2
Don't forget that your input files must be sorted.

3.     paste
Systems administrators frequently use the cut command to parse files, but many people I run into have never used the paste command. The paste command will concatenate two files line by line (as opposed to file by file, likecat.)

4.     less instead of more
This is not available on all Unix-based OSes, but the less command works very similar to more, but will let you move through a file forwards and backwards more easily. Want to jump to the end of the file, type Shift-G Depending on the version of less you are running, it will provide context highlighting when you search for a pattern.

5.     Jump to vi from more
While paging through a file in more, press "v" to jump to editting the file in vi at the current position in the file.

6.     Jump to a line number when editing a file with vi
vi +linenumber filename
will open up the file with the cursor automatically moved down to the specified line. This is useful when you get an error that indicates "syntax error on line 2047." You can jump straight to the problem without fumbling around.

7.     Invisible characters become visible
Sometimes you'll end up with carriage returns on each line in a file originally created on a DOS/Windows system, or filenames with spaces, tab, or other control characters in them, but you can't see them typically.
The cat command provides three useful options -v, -e, and -t that will let you understand these invisible characters
-v (displays non-printing characters)
-e (prints a "$" at the end of each line to indicate a NL character)
-t (prints "^I" for each Tab in the file)
cat -vet filename |more

8.     Remove DOS ^M from ends of lines
The "^M" characters are visible when editing in vi but here are two approaches to remove the characters.
 

   sed 's///g' -i filename 

      or in vi: :%s///g

   or dos2unix filename filename 

9.   You've done something that has completely screwed up your terminal. Everything you type is either invisible or incomprehensible. Try the following:

    $ ^C

    $ stty sane^J

If this doesn't work, try:

    $ echo ^V^O

10. To determine what program dumped an anonymous core file, use gdb:

    $ gdb -core core

    [...]

    Core was generated by `a.out'.

    Program terminated with signal 6, Abort trap.

    [...]

If you are on a system without gdb, try the file command:

    $ file core

    core: ELF 32-bit LSB core file of 'a.out' (signal 6)

    $

11. To create a vi macro that will wrap the current paragraph to a reasonable number of characters, add the following to your .exrc or.vimrc file:

    map Q {!} fmt -c -u^M

Now, when you press 'Q' in visual mode, the current paragraph will be wrapped to approximately 70 characters.

12.  Due to the Unix concept of sparse files, you can create seemingly enormous files that in reality take next to no disk space. The following program will create a 305,419,897 byte file called 'core' that may result in you receiving a 'cleanup' email message from a less-than-seasoned Unix sysadmin even though it occupies virtually no real disk space.

    $ cat bigcore.c

    #include

    #include

    int main(void) {

        int fd = open("core", O_CREAT|O_WRONLY|O_TRUNC, 0600);

        lseek(fd, 0x12345678, SEEK_SET);

        write(fd, "1", 1);

        close(fd);

        return 0;

    }

    $ cc -o bigcore bigcore.c

    $ ./bigcore

    $ ls -l core

    -rw-------  1 dmr  staff  305419897 May  1 03:50 core

    $ du -k core

    48      core

    $

13.  On older Unix systems, in particular those that don't support MD5password hashing, only the first 8 characters of a password are significant. If you use an 8 character password on these systems, you can type anything you want after the first 8 characters and it will be accepted by login. This allows you to type the first 8 characters of the password as usual, then have some fun. Enter a bunch of crap at superhuman speed, then enter the last few characters with your elbow. Then press backspace twice carefully and retype the characters with your other elbow. Etc....

14.  When recursive copying, cp (cp -Rip, etc.) may not be the best tool for the job. For example, cp copies hard links as separate files, which is probably not what you want. To get a true copy of a directory, try:

    $ tar cf - | (cd ; tar xf -)

This will create an exact copy of 'dir' in 'destdir'. The same principle can be used to create a recursive copy on a remote machine:

  $ tar cf - | ssh remotehost "(cd ; tar xf -)" 

15.  To list the files in another directory that match more than one pattern, it is easiest to do:

    $ ls -l /usr/local/foo/{*.conf,*.local,*.rc}

which is equivalent to:

$ ls -l /usr/local/foo/*.conf /usr/local/foo/*.local /usr/local/foo/*.rc

This syntax is supported by (at least) bash, ksh, csh and sh.

You can extend this idea to make renaming files in another directory, for example, a little easier:

$ mv -i /usr/local/foo/bar/baz/{stuff,stuff~}

16.  A little insurance against running 'rm -rf *' in the wrong directory -- create an empty file called -i in any critical directory:

$ >-i

or...

$ touch -- -i

If the 'rm -rf *' command is issued in that directory, the shell will expand the '-i' early on and go into interactive mode, thus giving you a chance to say 'Whoa, that was close!', which always sounds better than 'Oh fsck!'.

This tip works because the -i option to rm will override any previous -f.

Be forewarned that this tip only protects against 'rm -fr *', i.e., files and directories in the current directory.

To remove the -i file (or any other file beginning with '-'):

$ rm ./-i

or...

$ rm -- -i

17.  To return to the previous directory in ksh or bash, use:

$ cd -

18.  To use the last argument of the previous command line as an argument, use $_. For example:

$ ls -l /usr/home/dmr/somefile

$ vi $_

19.  To make the up and down arrow keys work and thereby enablecommand line editing and recall in ksh, include the following lines in your ~/.profile:

set -o emacs

alias __A='^P'

alias __D='^B'

alias __B='^N'

alias __C='^F'

alias __H='^A'

You need to enter the actual control characters ^P, ^B, etc.