Learning Unix

Posts you want to find years later go here.
Jonathan
Grand Pooh-Bah
Posts: 6722
Joined: Tue Sep 19, 2006 8:45 pm
Location: Portland, OR
Contact:

Learning Unix

Post by Jonathan »

Dwindlehop wrote:The following text is taken directly from the Internet Archive's February 2003 archive of Mohtalim, which can be accessed from the URL below.

http://web.archive.org/web/200212132040 ... topic?t=18
I think a common problem among a lot of university students is knowing enough UNIX to shoot yourself in the foot without knowing enough UNIX to get work done effectively. Here are some tips and tricks I wish I'd known as a sophomore in college.

Give everything a meaningful name. Every directory, every temporary file, every little kludge variable. Don't use a number, don't use foo. I once made the mistake of making a 238 bit wire because it saved on typing 238 different variable names. We probably spent a solid week figuring out which gates needed bit 178 instead of bit 78. Two hours after making changes to bit 203, we'd forget what bit 203 meant and have to relearn its meaning by tracing the logic. Name everything as if you'll have to read your code again at the end of the semester.

If you need to find a specific file, use
Code:
locate
If you need to find a specific file in your path, use
Code:
which
If you need to find every file of a specific name in your path, use
Code:
where
If you need to find a bunch of files, use
Code:
find -name "*.files"
If you need to run a command on a bunch of files, you can do either

Code:
find -name "file*pattern.txt" -exec gzip {} \;
Code:
ls file*pattern.v? | xargs chmod u+rw

The first recursively enters subdirectories, the second does not.

Do use search and replace in your editor. Better yet, learn regular expressions and use regex search and replace. If you have to type something more than five times, use some sort of search and replace plus pasting. If you have to type something more than twenty times, write a script.

To redirect standard error and standard out to a file, use
Code:
cmd >& stderr_and_stdout


Set ls to use colors by putting
Code:
alias ls 'ls --color=auto'
in your .cshrc (or .bashrc or .kshrc) file.

Use egrep, not grep. egrep supports real regular expressions and is faster than grep (or fgrep!). As a trivial example, type
Code:
egrep 'modify|wide|term' ~/.login
grep 'modify|wide|term' ~/.login
and compare the output.

Make lots and lots of little files. Short one, two, or twenty line files take up about a kilobyte of your quota, but can help immeasurably when debugging stuff you did last night or last week. If you type a command line that has more than three arguments, stick the command line in a file and make the file executable
Code:
chmod 700 my_cmd
If your program generates more than a screenful of output or takes more than sixty seconds to run, redirect its output to a file every time you run it.

If you're working on a project for more than two weeks, with other people, or with more than two milestones, think about using CVS. You can create your own cvs repository
Code:
cvs -d ~/CVSROOT init
Then you can import whatever code you've got written.
Code:
cd ~/work/proj/hw11; cvs -d ~/CVSROOT import hw11 initial_version alpha_code
Your friends can check out copies, providing you've given them the permissions on your files.
Code:
cd ~me/groupwork/cs211/; cvs -d ~jdpearce/CVSROOT checkout hw11 -d homework11
CVS will keep track of changes for you, merge changes between two files that were edited simultaneously, and allow you to roll back changes that cause problems.

If you know emacs, use vi by entering 'i' to start inserting text and then hitting ESC to leave INSERT mode. Save the file by typing ':wq' or don't save by typing ':q!'. If you know vi, use emacs by opening files with 'C-x C-f' and then just type. When you're done, save the file by typing 'C-x C-s' and exit the program by typing 'C-x C-c'. If you don't know either one, stop using pico and learn a real editor.

If you have a runaway process and it isn't responding to ^C, hit ^Z to suspend it and then type 'jobs'. Find its job id (usually a one digit number) and use the kill command like so
Code:
kill %1
to kill the suspended job. If that doesn't work, use
Code:
kill -9 %1
This is much easier than trying to use top or ps to find the process id. If your computer is locked up, try ssh'ing in from a nearby machine and killing the offending process before you attempt to reboot. Always try to rescue a stalled machine before rebooting, and never ever reboot the machine by unplugging it. If your admins wanted to let you reboot the machine easily, they would provide a big fat button.

On the command line, the character sequence !$ is an abbreviation for the last argument on the previous command line. This is unbelievably useful. Get used to typing
Code:
mv littlefile monstrous/path/of/doom/@nD/eVil/hate_very_much/
cd !$


Sometimes, the convenience of working from your room outweighs the efficiency of working in the cluster. Install a free UNIX clone, XFree86, and ssh so you can have X Forwarding on your home computer. You can also do this over Windows using XWin32 and ssh.

Ask upperclassmen, even ones not in your class, for help. They can't do your homework for you, but they can tell you the magic sequence of commands to make Matlab print.

Core dumps can be handy! If your program segfaults and dumps core, try
Code:
gdb ~/work/myprog core
When gdb starts, type 'bt' to see the list of functions on the stack. You can pop up to a particular function with 'up 2' and then 'list' the code that caused the offense. If this doesn't work, add a -g to your compiler.

Monitor your quota. If you have to run jobs that will produce more than a couple megabytes of output, redirect their outputs to /tmp. When you're done, remember to rm -rf /tmp so others can use it. Always delete core after you finish using it.

Break work up into multiple files. This helps you think about your project because the complexity is organized. This also helps you split work up and find functions more easily.

Use TAB autocomplete when changing directories or opening files. If TAB autocomplete doesn't work in your shell, try ^D. Sometimes, TAB is the autocomplete character and ^D is the 'prompt for matching files' character.

If you TAs out there have suggestions, additions, or complaints, email me at jonathan@pearce.name and let me know.
Last edited by Jonathan on Thu Oct 23, 2003 7:34 pm, edited 1 time in total.

Jonathan
Grand Pooh-Bah
Posts: 6722
Joined: Tue Sep 19, 2006 8:45 pm
Location: Portland, OR
Contact:

Post by Jonathan »

Dwindlehop wrote:This is a repost of Joe's original reply to the above article, again taken from the Internet Archive's February 2003 archive of Mohtalim.
Here's some semi-useful shell configuration lines.







set autolist='enhance' #use enhanced autocomplete
set fignore=(.o \~) #ignore .o files and files ending in ~
set listmaxrows = 3 #max number of rows listed in an autocomplete
set correct = 'cmd' #correct command mispelling. Can be set to 'all'
set color


Another useful thing is to NEVER EVER put a line like
set path = ( $path /another/path/to_add )

do this instead:
set tmp_path_additions = ( \
~/scripts \
/usr/bin \
/usr/local/bin/ \
/usr/lib \
)





if(!($?additions_made))
set tmp_path_additions
foreach tmp_newdir ( $tmp_path_additions )
if ($?DIR_ALREADY_IN_PATH) unset DIR_ALREADY_IN_PATH
foreach tmp_olddir ( $path )
if( $tmp_olddir == ":" ) then
continue
endif

if ( $tmp_olddir == $tmp_newdir ) then
set DIR_ALREADY_IN_PATH
break
endif
end
if (!($?DIR_ALREADY_IN_PATH)) then
set path = ( $path $tmp_newdir )
endif
end
endif
set additions_made





if ($?DIR_ALREADY_IN_PATH) unset DIR_ALREADY_IN_PATH
unset tmp_olddir tmp_newdir tmp_path_additions path_copy

I also recommend reading the man file for tcsh or csh or whatever shell you use for some really interesting/fun tricks to make your life easier. If you learn one or two tricks each time, then after about 20 times, you'll be a unix guru.

VLSmooth
Tenth Dan Procrastinator
Posts: 3055
Joined: Fri Jul 18, 2003 3:02 am
Location: Varies
Contact:

Post by VLSmooth »

While looking for more information on screen (mad props to the Hock for introducing it to me), I ran across this little gem.

http://www.cs.brown.edu/ugrad/tech/slides/adv_unix.pdf

Admittedly it's a tad rough, but more importantly, it's remarkably concise and informative. pkill is incredible for dealing with self-spawning recursive makes. Also, the aforementioned screen is text virtual desktops + detaching + logging + locking + etc on crack.

Run "screen" then hit <ctrl><a>, ? for help. I currently use three separate screens regularly with the ability to quickly and accurately switch between them.

Enjoy 8)

Jonathan
Grand Pooh-Bah
Posts: 6722
Joined: Tue Sep 19, 2006 8:45 pm
Location: Portland, OR
Contact:

Post by Jonathan »

Insert color text into your output!

The string "ESC[00m" is the default format string. That's the control character denoted by the Esc button (sometimes also printed as "^[") followed by a square bracket, two zeros, and the letter m.

00 means no formatting. 01 means bold. 01;36 means bold and cyan. Consult dircolors for more colors and formatting.

So, if you have a output going to a terminal you want to hightlight, you can say,

print "ESC[01;36mIMPORTANT: This output is highlighted.ESC[00m\n";

or something similar. You must make sure that you end with a "ESC[00m" or all the text that comes after, including your prompt, will be colored and bolded. Also, you must have a way of typing control characters into your program.

Note that if your output is redirected to a file, this control sequence will show up and look ugly. ls, of course, is cool enough to detect whether output is a terminal or not and adjusts its output accordingly. I leave that as an exercise for the reader.

Peijen
Minion to the Exalted Pooh-Bah
Posts: 2790
Joined: Fri Jul 18, 2003 2:28 pm
Location: Irvine, CA

Post by Peijen »

I am going to try out the new SFU 3.5 (service for unix) this weekend now that microsoft is offering it for free. I will post some thoughts about it verse cygwin after I decided I am too lazy to actually compare them.

I am posting this here because it's sort of unix and I don't feel like starting a new thread.

Some day I will get off my lazy ass and work on the linux/windows intergration project*.

* share user information/disk space between linux/windows machines, i.e. set "home/" and "documents and setting\" to the same directory on the file server. it looks like this will be a lot easier with the SFU's NFS support and password administration tool.

Jonathan
Grand Pooh-Bah
Posts: 6722
Joined: Tue Sep 19, 2006 8:45 pm
Location: Portland, OR
Contact:

Post by Jonathan »

colorgcc is your friend. If you can, install this script in your path and let it colorize gcc's errors and warnings for you.

Jonathan
Grand Pooh-Bah
Posts: 6722
Joined: Tue Sep 19, 2006 8:45 pm
Location: Portland, OR
Contact:

Post by Jonathan »

Typing a command with lots of command line arguments and you want to make sure all the files specified exist? Try prepending "ls -d --" to your command. All the files that exist will be listed, and you can easily tease out which files don't exist from the list.

quantus
Tenth Dan Procrastinator
Posts: 4891
Joined: Fri Jul 18, 2003 3:09 am
Location: San Jose, CA

Post by quantus »

Don't code shell scripts in the same shell as your environment. For example, if your shell is csh, do NOT write any csh or maybe even tcsh shell scripts. Use sh or ksh or something else instead.
Have you clicked today? Check status, then: People, Jobs or Roads

Jonathan
Grand Pooh-Bah
Posts: 6722
Joined: Tue Sep 19, 2006 8:45 pm
Location: Portland, OR
Contact:

Post by Jonathan »

Why? I think it's perfectly reasonable to write bash scripts even if your shell is bash.

quantus
Tenth Dan Procrastinator
Posts: 4891
Joined: Fri Jul 18, 2003 3:09 am
Location: San Jose, CA

Post by quantus »

Ok, say I'm using csh as my shell and I write a csh script with command line arguments. Your ~/.cshrc sources some "standard" stuff for the tools you need and, unbeknownst to you, one or more of the of scripts can accept an argument (maybe to use some other version of the tool if you specify it). Now, the first thing that happens when you run your csh script is that it invokes a shell and sources your ~/.cshrc, which is ok, right?

The problem is that the command line arguments that you thought were going to your script just went to those other things being sourced in your ~/.cshrc first and screwed up the settings for the tool that you might want to call later in the script.

If you wrote your script in a different shell, then this won't happen because you'll likely not have any environment settings for it. This of course means that switching your environment from one shell to another could cause a similar problem because of the old configuration files left behind.
Have you clicked today? Check status, then: People, Jobs or Roads

Jonathan
Grand Pooh-Bah
Posts: 6722
Joined: Tue Sep 19, 2006 8:45 pm
Location: Portland, OR
Contact:

Post by Jonathan »

Why would the arguments to your script get passed to the shell scripts you source in .cshrc? That doesn't make any sense. Surely there must be a way to prevent that from occuring without changing the shell used.

quantus
Tenth Dan Procrastinator
Posts: 4891
Joined: Fri Jul 18, 2003 3:09 am
Location: San Jose, CA

Post by quantus »

Ummm, I have no fucking clue why that happens. It's like the args are global or something :-\ You can be certain that I was shocked as well when I figured out that's what was going on.

The way to stop it from happening is to put a big if condition around your .cshrc such that only when the shell level equals one ($shlvl==1) does it actually do anything.
Have you clicked today? Check status, then: People, Jobs or Roads

Jonathan
Grand Pooh-Bah
Posts: 6722
Joined: Tue Sep 19, 2006 8:45 pm
Location: Portland, OR
Contact:

Post by Jonathan »

Default arguments are fine, but they should always be contextual. Operate on the current directory, for instance, rather than a hardcoded path.

Jonathan
Grand Pooh-Bah
Posts: 6722
Joined: Tue Sep 19, 2006 8:45 pm
Location: Portland, OR
Contact:

Post by Jonathan »

Make liberal use of tee. (`man tee' for more info.)

When you write a program or script, always make it print out its arguments before it does anything unless it is a filter. This is doubly important for programs that get executed by other programs.

quantus
Tenth Dan Procrastinator
Posts: 4891
Joined: Fri Jul 18, 2003 3:09 am
Location: San Jose, CA

Post by quantus »

tee is definately your friend, especially when controlling what goes to the terminal and what goes to the script's log. In general, I've found it's better for scripts to be quiet on the term unless the user asks, and verbose in the logs. I "think" that if you rexec and the command outputs to the term it exits prematurely. I know this is certainly true if something in your .cshrc prints to the term.

Also, exit codes are useful when running scripts within scripts for detecting errors and the severity of the error.

Awk and Sed are kinda fun to use sometimes. It's like using regex search and replace on the command line with some programability.
Have you clicked today? Check status, then: People, Jobs or Roads

quantus
Tenth Dan Procrastinator
Posts: 4891
Joined: Fri Jul 18, 2003 3:09 am
Location: San Jose, CA

Post by quantus »

Always always always have a blank line as the last line in a shell script. Otherwise, the command on the last line won't be executed in the script and can get left dangling in the command buffer to be executed later. This is especially interesting when some configuration script that you source in your login files has a dangling exit. You go to do an ls or cd as your first command and instead get logged out!!
Have you clicked today? Check status, then: People, Jobs or Roads

Jonathan
Grand Pooh-Bah
Posts: 6722
Joined: Tue Sep 19, 2006 8:45 pm
Location: Portland, OR
Contact:

Post by Jonathan »

More emacs vs. vi:

Use

Code: Select all

:set cindent
to automatically format code in vi. To engage this, type CTL+F while in insert mode.

Use

Code: Select all

ESC x c++-mode
to automatically format code in emacs. To engage this, type TAB.

Jonathan
Grand Pooh-Bah
Posts: 6722
Joined: Tue Sep 19, 2006 8:45 pm
Location: Portland, OR
Contact:

Post by Jonathan »

The program emacsclient is the proper way to invoke emacs (gnuclient for xemacs). Using emacsclient reduces startup time and memory usage. The proper way to invoke emacsclient from the command line is documented in the GNU Emacs Manual.

Here is a handy alias that will run emacs if emacs needs to be run or connect assuming the emacs server has been started. You should set $ALTERNATE_EDITOR to be emacs.

Code: Select all

alias em 'emacsclient --no-wait -a $ALTERNATE_EDITOR'
If you want to use emacs to edit your mail or commit messages, you should set the following environment variables. Note you should not alias `emacsclient --no-wait` to emacsclient, because that --no-wait option will confuse your mail client.

Code: Select all

setenv EDITOR emacsclient
setenv ALTERNATE_EDITOR emacs
setenv CVSEDITOR emacsclient

Jonathan
Grand Pooh-Bah
Posts: 6722
Joined: Tue Sep 19, 2006 8:45 pm
Location: Portland, OR
Contact:

Post by Jonathan »

http://www.understudy.net/custom.html

This page has a good deal of info on how to customize your shell prompt, including adding color.

Jonathan
Grand Pooh-Bah
Posts: 6722
Joined: Tue Sep 19, 2006 8:45 pm
Location: Portland, OR
Contact:

Post by Jonathan »

Change your terminal titles to dynamically changing.
A T Cshell example here for a .cshrc:

if ( "$?tcsh" ) then
if ( $term == "xterm" ) then
alias cwdcmd 'echo -n "^[]2;${HOST}: -`whoami`- `echo ${cwd} | sed
s:/home/brissing:~:g`^G"'
alias precmd 'cwdcmd'
cd . # Update it now.
endif
endif


In particular, the escape sequience '[ESC] + ] + 2 + ;' is to get the
attention of your Xterm. Then whatever you put between it and '[ctrl] + G'
will be placed in to your Xterm's title bar. Most editors will let you put
escape characters in your .cshrc by first pressing ctrl-v. So in VI's
insert mode 'ctrl-v [esc]' would generate the esc key sequence and 'ctrl-v
ctrl-g' would generate the closing sequence.

Post Reply