NerdyRoom™

I have a bash script that generates names for binaries, these names include parameters ranging from 2 to 64. I sort these then but due to the nature of alphanumeric sorting, 4 is sorted between 32 and 64 (4 is bigger than 3 but smaller than 6…). So what i need is leading zeros for the sorting to work the way i want it to work. The easiest solution i could come up with is padding zeros c-style using the command line version of printf:

PARAMLIST="2 4 8 16 24 32 40 48 56 64"
for PARAM in $PARAMLIST;
do 
  printf "%02d\n" $PARAM
done

[update by bert]

sort –numeric-sort

solves the sorting issue as well

I have a bash script that generates names for binaries, these names include parameters ranging from 2 to 64. I sort these then but due to the nature of alphanumeric sorting, 4 is sorted between 32 and 64 (4 is bigger than 3 but smaller than 6...). So what i need is leading zeros for the sorting to work the way i want it to work. The easiest solution i could come up with is padding zeros c-style using the command line version of printf: PARAMLIST="2 4 8 16 24 32 40 48 56 64" for PARAM in $PARAMLIST; do printf "%02d\n" $PARAM done [update by bert] sort –numeric-sort solves the sorting issue as well :)

ever wanted to merge two text files horizontally, line by line? I do have a list of files were I want to associate one entry in each file with its name for plotting in gnuplot:

find ${LOG} -iname "*.log" -path "*${FILTER}*" -exec cat {} \; >times.txt
find ${LOG} -iname "*.log" -path "*${FILTER}*" -exec echo {} \; >names.txt

Now i need one result file with:

name1 time1
name2 time2
...

The tool i was searching for is called paste and does exactly that:

paste names.txt times.txt

Sometimes I really do love my job (but only sometimes )

ever wanted to merge two text files horizontally, line by line? I do have a list of files were I want to associate one entry in each file with its name for plotting in gnuplot: find ${LOG} -iname "*.log" -path "*${FILTER}*" -exec cat {} \; >times.txt find ${LOG} -iname "*.log" -path "*${FILTER}*" -exec echo {} \; >names.txt Now i need one result file with: name1 time1 name2 time2 ... The tool i was searching for is called paste and does exactly that: paste names.txt times.txt Sometimes I really do love my job (but only sometimes ;))

One of my favorite applets in gnome is netspeed which of course wont work in unity (damn you, unity). But using the indicator version of sysmon, dstat (cool tool – check it out if you haven’t already) and some bash magic you can get something similar to work anyways.

http://www.pcurtis.com/ubuntu-unity.htm#netspeed_appindicator

The above link explains everything needed and you’ll end up with your netspeed being displayed in the panel . . .

One of my favorite applets in gnome is netspeed which of course wont work in unity (damn you, unity). But using the indicator version of sysmon, dstat (cool tool - check it out if you haven't already) and some bash magic you can get something similar to work anyways. http://www.pcurtis.com/ubuntu-unity.htm#netspeed_appindicator The above link explains everything needed and you'll end up with your netspeed being displayed in the panel . . .

Ubuntu server installation provides colored man pages – nifty
This is what has to be done to get it working in a X session terminal:

sudo aptitude install most
sudo update-alternatives --config pager

Choose “most” from the list and test it with a man pagge “man test”.

Have fun . . .

[update] As always, bert nows more :
I have found a major and a minor flaw in using most as the man pager:

Major: I use the -X option for less (by putting them into the LESS env variable [1]). This does not clear the terminal on exit of less. Thus, you see the last viewed page of the manual. This is particular helpful when looking-up a command line option, so I can copy’n’paste the option to the command line after quitting less. I can’t find a similar option for most.

Minor: less shows the name of the manual page in the bottom prompt, while most just shows ‘*stdin*’.

Footnote [1]: For the curious reader, this is the content of my $LESS: -MSiRXF. Feel free to look them up in the manual.

Ubuntu server installation provides colored man pages - nifty ;) This is what has to be done to get it working in a X session terminal: sudo aptitude install most sudo update-alternatives --config pager Choose "most" from the list and test it with a man pagge "man test". Have fun . . . [update] As always, bert nows more ;) : I have found a major and a minor flaw in using most as the man pager: Major: I use the -X option for less (by putting them into the LESS env variable [1]). This does not clear the terminal on exit of less. Thus, you see the last viewed page of the manual. This is particular helpful when looking-up a command line option, so I can copy’n'paste the option to the command line after quitting less. I can’t find a similar option for most. Minor: less shows the name of the manual page in the bottom prompt, while most just shows ‘*stdin*’. Footnote [1]: For the curious reader, this is the content of my $LESS: -MSiRXF. Feel free to look them up in the manual.

Sometimes there is a need for spawning multiple sub processes in a shell script and do something after ALL of them have finished. The Bash has nice little features to do this very easily.

Here is an example:

#!/bin/bash
./process_1 &     # fork first sub process
PID_1=$!          # get PID of first sub process
./process_2 &     # fork second sub process
PID_2=$!          # get PID of second sub process

#...do some other work...

wait  ${PID_1}     # stop execution until first process has finished
wait  ${PID_2}     # stop execution until second process has finished

#...all work is done...

exit 0

With the ampersand (&) one can fork sub processes running independent from the parent process.

To avoid busy waiting we want to stop execution of the parent script until the child has finished. So we don’t use an infinite loop with a check and a sleep or something like that, but a function called wait together with the childs’ Process ID.

The PID of a created process can be received by reading $! right after the fork. As far as I know this is mostly bash functionality and maybe is not POSIX! (correct me if I’m wrong)

Sometimes there is a need for spawning multiple sub processes in a shell script and do something after ALL of them have finished. The Bash has nice little features to do this very easily. Here is an example: #!/bin/bash ./process_1 &     # fork first sub process PID_1=$!          # get PID of first sub process ./process_2 &     # fork second sub process PID_2=$!          # get PID of second sub process #...do some other work... wait  ${PID_1}     # stop execution until first process has finished wait  ${PID_2}     # stop execution until second process has finished #...all work is done... exit 0 With the ampersand (&) one can fork sub processes running independent from the parent process. To avoid busy waiting we want to stop execution of the parent script until the child has finished. So we don't use an infinite loop with a check and a sleep or something like that, but a function called wait together with the childs' Process ID. The PID of a created process can be received by reading $! right after the fork. As far as I know this is mostly bash functionality and maybe is not POSIX! (correct me if I'm wrong)

Stefan ist bash Nutzer, ich auch . . . und wir sind beide unabhängig voneinander in die selbe Falle getappt. Wir wollten die Ausgabe von “time” in eine Datei schreiben, dabei hat der naive Anstaz mit Streams umleiten nicht auf Anhieb funktioniert. Eigentlich ganz einfach, die manpage sagt “-o” . . . logisch. Nur tut es dann nicht – WTF? Stefan hats fluchend mit googlen und Foren lesen pobiert, ich hab durch Zufall in der manpage den passenden Hinweis gefunden:

Users of the bash shell need to use an explicit path in order to run the external time command and not the shell builtin variant.

Die bash bringt also ihr eigenes “time” mit das die Funktionalität “-o” nicht bietet . . .
Das Ende der Geschichte:

/usr/bin/time -o /tmp/time.log -p sleep 1

Stefan ist bash Nutzer, ich auch . . . und wir sind beide unabhängig voneinander in die selbe Falle getappt. Wir wollten die Ausgabe von "time" in eine Datei schreiben, dabei hat der naive Anstaz mit Streams umleiten nicht auf Anhieb funktioniert. Eigentlich ganz einfach, die manpage sagt "-o" . . . logisch. Nur tut es dann nicht - WTF? Stefan hats fluchend mit googlen und Foren lesen pobiert, ich hab durch Zufall in der manpage den passenden Hinweis gefunden: Users of the bash shell need to use an explicit path in order to run the external time command and not the shell builtin variant. Die bash bringt also ihr eigenes "time" mit das die Funktionalität "-o" nicht bietet . . . Das Ende der Geschichte: /usr/bin/time -o /tmp/time.log -p sleep 1

Da die offiziellen Widgets des DVB nicht gerade toll benutzbar sind (nur 1 Haltestelle pro Rechner, …), hier mal ein kleiner Einzeiler den man z.B. mit GeekTool oder ähnlichem auf den Desktop werfen kann (siehe Screenshot).

echo "Schneebergstraße"; \
curl -s "http://widgets.vvo-online.de/abfahrtsmonitor/Abfahrten.do?ort=Dresden&hst=SNS" | \
sed 's/\],\[/\
/g;s/\[\[//;s/\]\]//;s/"//g' | awk 'BEGIN {FS=","} {print $3 "\t" $1 "\t" $2}'

Die Codes für die Haltestellen gibt es hier. Notorische Busverpasser können auch im AWK-Teil ein paar Minuten von der verbleibenden Zeit subtrahieren

Da die offiziellen Widgets des DVB nicht gerade toll benutzbar sind (nur 1 Haltestelle pro Rechner, ...), hier mal ein kleiner Einzeiler den man z.B. mit GeekTool oder ähnlichem auf den Desktop werfen kann (siehe Screenshot). echo "Schneebergstraße"; \ curl -s "http://widgets.vvo-online.de/abfahrtsmonitor/Abfahrten.do?ort=Dresden&hst=SNS" | \ sed 's/\],\[/\ /g;s/\[\[//;s/\]\]//;s/"//g' | awk 'BEGIN {FS=","} {print $3 "\t" $1 "\t" $2}' Die Codes für die Haltestellen gibt es hier. Notorische Busverpasser können auch im AWK-Teil ein paar Minuten von der verbleibenden Zeit subtrahieren :)

‘been absent for a while – PTO and stuff

Here are the two major show-stoppers of today which nearly cracked my head:
PTY allocation request failed on channel 0

This occured when I tried to login to a xen domain I just created – took a while to figure out how to solve that but doing a
ssh user@host "/bin/bash -i" (notice the -i for interactive)
helped in accessing the vm and then i could reset the dev’s:
rm -r /dev/ptmx
mknod /dev/ptmx c 5 2
chmod 666 /dev/ptmx
umount /dev/pts
rm -r /dev/pts
mkdir /dev/pts
mount /dev/pts


The second Problem was a erroneous cd-rom drive that stopped working in the middle of the installation process. The wonderfull tool UNetbootin saved the day by providing me with a bootable USB stick in a matter of seconds.

Happy weekend,
Willi

'been absent for a while - PTO and stuff Here are the two major show-stoppers of today which nearly cracked my head: PTY allocation request failed on channel 0 This occured when I tried to login to a xen domain I just created - took a while to figure out how to solve that but doing a ssh user@host "/bin/bash -i" (notice the -i for interactive) helped in accessing the vm and then i could reset the dev's: rm -r /dev/ptmx mknod /dev/ptmx c 5 2 chmod 666 /dev/ptmx umount /dev/pts rm -r /dev/pts mkdir /dev/pts mount /dev/pts The second Problem was a erroneous cd-rom drive that stopped working in the middle of the installation process. The wonderfull tool UNetbootin saved the day by providing me with a bootable USB stick in a matter of seconds. Happy weekend, Willi

Gelegentlich kommt es vor, das großes Rätselraten herrscht, über bestimmt Spitzfindigkeiten von Unix Shells. Sei es das Umbiegen von Deskriptoren oder das Verhalten bei Forks…

Zufällig bin ich gerade auf einen einen sehr schönen Foliensatz gestoßen, der zumindest für die bash abhilfe schafft. Sehr übersichtlich, perfekt zum Nachschlagen.

http://www.cs.uni-potsdam.de/ml/teaching/ws09/rnb/f07.pdf

Gelegentlich kommt es vor, das großes Rätselraten herrscht, über bestimmt Spitzfindigkeiten von Unix Shells. Sei es das Umbiegen von Deskriptoren oder das Verhalten bei Forks... Zufällig bin ich gerade auf einen einen sehr schönen Foliensatz gestoßen, der zumindest für die bash abhilfe schafft. Sehr übersichtlich, perfekt zum Nachschlagen. http://www.cs.uni-potsdam.de/ml/teaching/ws09/rnb/f07.pdf

In the life of an IT-guy you often shake your head and smile while wondering about the current problem or its solution. So after a while not much can realy surprise you. But last week there was such a rare moment.

The problem is an easy one: we could not execute binaries. We have a cluster here with an 32 bit head node and 64 bit compute nodes both x86 Debian GNU Linux. The stuff to be shared comes over a NFS. Heterogenouse clusters have some difficulties because you cannot execute binaries from one architecture on another. This will fail with the error message: “wrong ELF format”.
Our problem first appeared by using the Modules-tool. For testing we executed it directly:

./modulecmd
-bash: ./modulecmd: no such file or directory

But OK, as everybody could see, it was. We checked and changed the rights for users, groups, owndership, parent directories etc. Nothing helped. Next we put an eye on NFS. No miss-configuration or other distinctive feature.
Now we use the bigger weapons and strace the command.

strace ./modulecmd
execve("./modulecmd", ["./modulecmd"], [/* 25 vars */]) = -1 ENOENT (No such file or directory)
dup(2)                                  = 3
fcntl(3, F_GETFL)                       = 0x8002 (flags O_RDWR|O_LARGEFILE)
fstat(3, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff4b4700000
lseek(3, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)
write(3, "strace: exec: No such file or dir"..., 40strace: exec: No such file or directory
) = 40
close(3)                                = 0
munmap(0x7ff4b4700000, 4096)            = 0
exit_group(1)                           = ?

Also nothing surprising in here.
The further isolate the root of the behaviour, we code a short “hello world” in C and do compiling and execution and it worked. So here we got the main point which confused us. Self created binaries and all system tools worked properly, just the shared ones or let’s say “not system contained” executables failed.
OK, maybe it’s situated in a deeper level. We installed dash and try to execute it in there. No changes. Slowly but steady we ran out of ideas.
From the strace run we could see that execve backfired. This command overwrites the current forked process with the binary given by a path as an argument and executes it. So maybe the reason belongs to the execution context. We tried

instead. So WTF? After ignoring some charset ideas in mind, we arrived where we started. An obligatory file on a system executable and our binary shared over NFS told us the wrong ELF type. The installation of ia32-libs fixed the problem finally, but why there was a “no such file of directory” instead of a “wrong ELF type” we still do not know.

You are welcome to give us solution or hints.

In the life of an IT-guy you often shake your head and smile while wondering about the current problem or its solution. So after a while not much can realy surprise you. But last week there was such a rare moment. The problem is an easy one: we could not execute binaries. We have a cluster here with an 32 bit head node and 64 bit compute nodes both x86 Debian GNU Linux. The stuff to be shared comes over a NFS. Heterogenouse clusters have some difficulties because you cannot execute binaries from one architecture on another. This will fail with the error message: "wrong ELF format". Our problem first appeared by using the Modules-tool. For testing we executed it directly: ./modulecmd -bash: ./modulecmd: no such file or directory But OK, as everybody could see, it was. We checked and changed the rights for users, groups, owndership, parent directories etc. Nothing helped. Next we put an eye on NFS. No miss-configuration or other distinctive feature. Now we use the bigger weapons and strace the command. strace ./modulecmd execve("./modulecmd", ["./modulecmd"], [/* 25 vars */]) = -1 ENOENT (No such file or directory) dup(2)                                  = 3 fcntl(3, F_GETFL)                       = 0x8002 (flags O_RDWR|O_LARGEFILE) fstat(3, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff4b4700000 lseek(3, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek) write(3, "strace: exec: No such file or dir"..., 40strace: exec: No such file or directory ) = 40 close(3)                                = 0 munmap(0x7ff4b4700000, 4096)            = 0 exit_group(1)                           = ? Also nothing surprising in here. The further isolate the root of the behaviour, we code a short "hello world" in C and do compiling and execution and it worked. So here we got the main point which confused us. Self created binaries and all system tools worked properly, just the shared ones or let's say "not system contained" executables failed. OK, maybe it's situated in a deeper level. We installed dash and try to execute it in there. No changes. Slowly but steady we ran out of ideas. From the strace run we could see that execve backfired. This command overwrites the current forked process with the binary given by a path as an argument and executes it. So maybe the reason belongs to the execution context. We tried instead. So WTF? After ignoring some charset ideas in mind, we arrived where we started. An obligatory file on a system executable and our binary shared over NFS told us the wrong ELF type. The installation of ia32-libs fixed the problem finally, but why there was a "no such file of directory" instead of a "wrong ELF type" we still do not know. You are welcome to give us solution or hints.