Beyond the limits - environment (cont 2)

Comment from Paul for the article make me rerun the tests in different conditions: I update the memory of both machines to 1024 MB and use scripts (see below)

[root@centos ~]# cat z1.sh
TEMPVAR=A
for i in {1..1000};
        do
        TEMPVAR=${TEMPVAR}$TEMPVAR;
        a=$(echo 2^$i|bc)
        b=${#TEMPVAR}
        if [ $a -ne $b ]
        then echo KO - $a, $b
        else echo $a
        fi
done
[root@centos ~]# cat z1.1.sh
TEMPVAR=A
for i in {1..1000};
        do
        TEMPVAR=${TEMPVAR}$TEMPVAR;
        a=$(echo 2^$i+1|bc)
        b=$(echo $TEMPVAR|wc -c)
        if [ $a -ne $b ]
        then echo KO - $a, $b
        else echo $a
        fi
done

Solaris
On Solaris independently of the amout of memory and the way of get length of environment variable the result is same:

bash-3.00# vmstat 1 2
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr cd f0 s0 --   in   sy   cs us sy id
 0 0 0 1075080 616428 152 3085 0 0 0  0 204 10 -0 3  0  309 3345 1511 14 19 67
 1 0 0 1407772 753456 18 50  0  0  0  0  0  0  0  0  0  304  324  142  0  1 99

It seems 32MB is internal limitation in bash (or OS) in Solaris

bash-3.00# ./z1.sh
2
4
<snip>
8388608
16777216
33554432
./z1.sh: fork: Not enough space
./z1.sh: line 7: [: -ne: unary operator expected
./z1.sh: fork: Not enough space
bash-3.00# ./z1.1.sh
3
5
<snip>
16777217
33554433
./z1.1.sh: fork: Not enough space
./z1.1.sh: fork: Not enough space
KO - ,
./z1.1.sh: fork: Not enough space

And the execution times follow the common sense

bash-3.00# time ./z1.sh
2
4
<snip>
16777216
33554432
real    0m9.670s
user    0m6.454s
sys     0m3.014s
bash-3.00# time ./z1.1.sh
3
5
<snip>
16777217
33554433
real    0m13.709s
user    0m8.040s
sys     0m5.352s


Linux
But on linux

[root@centos ~]# vmstat 1 2
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0  10416 1016704    252   4072   75  804   318   828 1008   77 19 44 32  4  0
 0  0  10416 1016704    252   4096    0    0     0     0 1015   14  0  1 99  0  0


my method of count permit me to reach bigger length

[root@centos ~]# ./z1.sh
2
4
<snip>
67108864
134217728
./z1.sh: xmalloc: cannot allocate 536870913 bytes (0 bytes allocated)
[root@centos ~]# ./z1.1.sh
3
5
<snip>
134217729
268435457
./z1.1.sh: xmalloc: cannot allocate 1073741825 bytes (0 bytes allocated)
KO - 536870913, 0
./z1.1.sh: xmalloc: cannot allocate 1073741825 bytes (0 bytes allocated)
 

and need less execution time (with limit to 2^26)

[root@centos ~]# time ./z1.sh
2
4
<snip>
33554432
67108864
real    1m58.730s
user    0m33.084s
sys     1m27.562s
[root@centos ~]# time ./z1.1.sh
3
5
<snip>
33554433
67108865
real    1m36.457s
user    0m50.206s
sys     0m47.815s



Final conclusion: this need further investigations :)

Beyond the limits - environment (cont)

4. Accessable command line argument
Now I will try to see how many are acceptable command line arguments from system utils like pgrep, pkill and similar. For this puspose I will use dummy C program which do nothing

#include <stdio.h>
int main(int argc, char *argv[])
{
       while(1){};
}
 

And will generate sequence of random numbers to be fure will have ho false results

for i in {1..256}; do dd if=/dev/urandom bs=1 count=64|md5sum|awk '{printf $1" "}'>>random_arg;done
[root@centos ~]# cat random_arg
d4a798eea8f9b9b5c9751aa8a785b465 da735316b9d27af062c28a0e3655a1f5 43c45dcd136e53d01662a5f9cfa943d9 fd5d2445ebf54a969cd6dff7e8cb3c09
<snip>
1187d29932b7d5827080684f16205330 92398e4719194f13a1a79064c5a69d05 e1cd425a036f77cc6fb6f3002d00bada a0c6d3b48c5e8146b2fe0b5f74383b9c

Let try with shorter arguments

4.1. Linux
Compile the program
[root@centos ~]# cc z4.c

 And execute it
[root@centos ~]# ./a.out `cat random_arg `&
[1] 5094
 
Lets check what I can see with ps
[[root@centos ~]# ps -efl|grep a.out
0 R root      5094  2174 99  85   0 -   403 -      06:13 pts/0    00:00:35 ./a.out d4a798eea8f9b9b5c9751aa8a785b465 da735316b9d27af062c28a0e3655a1f5 43c45dcd136e53d01662a5f9cfa943d9 fd5d2445ebf54a969cd6dff7e8cb3c09 73e522501878712898d1c47342b9bd66
<snip>
f566402f18f3ebe15453d3a74fc1c534 f87fe9ab1f5f85238ce01ca5d62d57e1 d05d9d3b9cb608d8a177434a4cda5c69 43e80e6069a763a0da17abdba8a06b30 d79a21984d368e4b574e4690a749c7af f5c9eddc01dd1a98bdb7ecbd5eba87a7 249a93f99bb66ff2361f722d1de2b
0 R root      5101  2174  0  78   0 -  1001 -      06:13 pts/0    00:00:00 grep a.out
 
OK, but let see if i can find all the arguments with ps:
CNT=1
for i in `cat random_arg `;
do
    if [ `ps -efl|grep $i|grep -v grep|wc -l` -eq 1 ]
        then echo OK $CNT
        else echo $i
    fi
    CNT=`echo $CNT + 1|bc`
done
 
 and exec
[root@centos ~]# ./z3.sh
OK 1
OK 2
<snip>
OK 123
249a93f99bb66ff2361f722d1de2b8ff
f2793da3d8bbab2c7b972c4ff2dc7176
60bc35ea120dc372a5b14a02acc2182a
<snip>
 
So I can see only 123 arguments. Hm, this a big number, but no so much
Lets try with pgrep
CNT=1
for i in `cat random_arg `;
do
    if [ `pgrep -f $i|wc -l` -eq 1 ]
        then echo OK $CNT
        else echo $i
    fi
    CNT=`echo $CNT + 1|bc`
done

[root@centos ~]# ./z4.sh
OK 1
<snip>
OK 123
249a93f99bb66ff2361f722d1de2b8ff
f2793da3d8bbab2c7b972c4ff2dc7176
60bc35ea120dc372a5b14a02acc2182a
<snip>
 
The same result. So i can see beyond parameter number 123. And sample chech if this depend of number of variables in environment
[root@centos ~]# env|wc -l
20
[root@centos ~]# export BB=1
[root@centos ~]# env|wc -l
21
[root@centos ~]# ./z4.sh
OK 1
<snip>
OK 123
249a93f99bb66ff2361f722d1de2b8ff
f2793da3d8bbab2c7b972c4ff2dc7176
60bc35ea120dc372a5b14a02acc2182a
<snip>

It seems do not depend and this is internal limitation



4.2. Solaris
Compile the program
bash-3.00# /usr/sfw/bin/gcc z3.c
and generate random numbers

for i in {1..256}; do dd if=/dev/urandom bs=1 count=64|digest -a md5|awk '{printf $1" "}'>>random_arg;done

bash-3.00# cat random_arg
40e4b176f31e662ae95ac93e995cf4a4 f1e1c978a8007aa88bfffd675af2d56b 6b06061f1e4a76e3ced152855f7f5047 89e3f7ee6744078726a58ba3fe85092a
d1659b7259bf383888e93caee7e26282 637ff2c7ad50c5f2018f7a865d646667 35bcc0568f9b8b8902fdc4f73fbadb49 34ce610edd36d834c08115de0c8e4568
<snip>
 


 And execute it
bash-3.00# ./a.out `cat random_arg `&
[1] 1536
Lets check what I can see with ps
bash-3.00# ps -efl|grep a.out
 0 S     root  1539   749   0  40 20        ?    347        ? 07:40:11 pts/1       0:00 grep a.out
 0 R     root  1536   749  70  89 20        ?    336          07:39:46 pts/1       0:26 ./a.out 40e4b176f31e662ae95ac93e995
 
OK, but let see if i can find all the arguments with ps:
CNT=1
for i in `cat random_arg `;
do
    if [ `ps -efl|grep $i|grep -v grep|wc -l` -eq 1 ]
        then echo OK $CNT
        else echo $i
    fi
    CNT=`echo $CNT + 1|bc`
done
 
 and exec
bash-3.00# ./z3.sh
OK 1
OK 2
6b06061f1e4a76e3ced152855f7f5047
89e3f7ee6744078726a58ba3fe85092a
d1659b7259bf383888e93caee7e26282
<snip>
 
As you can see Solaris show me only two arguments with SYSV ps, but let me try with pargs

bash-3.00# pargs 1536
1536:   ./a.out 40e4b176f31e662ae95ac93e995cf4a4 f1e1c978a8007aa88bfffd675af2d56b 6b060
argv[0]: ./a.out
argv[1]: 40e4b176f31e662ae95ac93e995cf4a4
argv[2]: f1e1c978a8007aa88bfffd675af2d56b
<snip>
argv[256]: 85fa98d8bf33f2b0507ec9714472d17f
OK, thats good, but I should know the PID :)



Lets try with pgrep
CNT=1
for i in `cat random_arg `;
do
    if [ `pgrep -f $i|wc -l` -eq 1 ]
        then echo OK $CNT
        else echo $i
    fi
    CNT=`echo $CNT + 1|bc`
done

bash-3.00# ./z4.sh
OK 1
OK 2
6b06061f1e4a76e3ced152855f7f5047
89e3f7ee6744078726a58ba3fe85092a
d1659b7259bf383888e93caee7e26282
<snip>

Ha, I can see only two arguments, this is too small for me
And sample check if this depend of number of variables in environment
bash-3.00# env|wc -l
      14
bash-3.00# export BB=1
bash-3.00# env|wc -l
      15
bash-3.00# ./z4.sh
OK 1
OK 2
6b06061f1e4a76e3ced152855f7f5047
89e3f7ee6744078726a58ba3fe85092a
d1659b7259bf383888e93caee7e26282
<snip>

It seems do not depend and this is internal limitation
Let see what will happen with shorter arguments

bash-3.00# ./a.out 111 222 333 444 555 666 777 888 999&
[1] 2346
bash-3.00# cat z4.sh
CNT=1
for i in 111 222 333 444 555 666 777 888 999;
do
    if [ `pgrep -f $i|wc -l` -eq 1 ]
        then echo OK $CNT
        else echo $i
    fi
    CNT=`echo $CNT + 1|bc`
done
bash-3.00# ./z4.sh
OK 1
OK 2
OK 3
OK 4
OK 5
OK 6
OK 7
OK 8
OK 9
 

O, all seems OK, but why i cant reach more deeper in argument list. Let keep this question open :)

Beyond the limits - environment

1. Prerequisites
My environmnet is based on Solaris 10  10 9/10  with 640 MB of memory and CentOS 5.5 2.6.18-194.26.1.el5 with 512 MB, standard installation both of them run in VMWare environmnet. For shell I use bash (as common)

bash-3.00# bash --version
GNU bash, version 3.00.16(1)-release (i386-pc-solaris2.10)
Copyright (C) 2004 Free Software Foundation, Inc.


[root@centos ~]# bash --version
GNU bash, version 3.2.25(1)-release (i686-redhat-linux-gnu)
Copyright (C) 2005 Free Software Foundation, Inc.

All the tests will be done with default settings of the OS and environment

2. Test the maximum length variable
the script i will use is sample:

TEMPVAR=A
for i in {1..10000};
        do
        TEMPVAR=${TEMPVAR}$TEMPVAR;
        a=$(echo 2^$i+1|bc)
        b=$(echo $TEMPVAR|wc -c)
        if [ $a -ne $b ]
        then echo KO - $a, $b
        else echo $a
        fi
done
 2.1. And the execution in Linux


[root@centos ~]# ./z1.sh
3
5
<snip>
67108865
134217729
./z1.sh: xmalloc: cannot allocate 268435457 bytes (0 bytes allocated)
KO - 268435457, 0
[1]+  Stopped                 ./z1.sh
[root@centos ~]# vmstat 1 2
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0  10716 229500  25160 228524   66  139   138   178 1011  151 18 18 61  3  0
 0  0  10716 229500  25160 228524    0    0     0     0 1013   15  0  0 100  0  0
 As you can see above I get error allocating memory beyond the size of free memory
2.2. Execution in Solaris

bash-3.00# ./z1.sh
3
5
<snip>
8388609
16777217
./z1.sh: fork: Not enough space
./z1.sh: line 7: [: 33554433: unary operator expected
33554433
./z1.sh: fork: Not enough space
bash-3.00# vmstat 1 2
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr cd f0 s0 --   in   sy   cs us sy id
 0 0 0 856808 174100 50 1101 0  1  5  0 66  2 -0  3  0  336  801  308  9  7 84
 0 0 0 749832 70472   3  43  0  0  0  0  0 27  0  0  0  326  300  185  1  9 90
 
 The similar is situation in Solaris where i can allocate even less that half of free memory
Of course it is not very wise to use such variables, but who knows?
3. Number of commandline arguments
3.1. Sample C program
Next I will try to check what is the maximum of commandline arguments of one sample C program

#include <stdio.h>
int main(int argc, char *argv[])
{
        printf("%d\n", argc);
}

and the shell script I will use is

TEMPVAR=A
for i in {1..1000}
do
TEMPVAR="$TEMPVAR $TEMPVAR"
a=$(./a.out $TEMPVAR)
echo $a
done
 

3.1.1 On Linux

[root@centos ~]# cc z2.c
[root@centos ~]# ./a.out 1
2
[root@centos ~]# ./a.out 1 2
3
[root@centos ~]# ./z2.sh
3
5
<snip>
524289
1048577
./z2.sh: line 5: ./a.out: Argument list too long

 1m of arguments, that pretty much :)
3.1.2 Solaris

bash-3.00# /usr/sfw/bin/gcc z2.c
bash-3.00# ./a.out 1
2
bash-3.00# ./a.out 1 2
3
bash-3.00# ./z2.sh
3
5
<snip>
65537
131073
./z2.sh: line 5: ./a.out: Arg list too long
"Just" 128k, not so much as Linux, but anyway do someone will use so many for practical puspose?


3.2. Sample shell program
Next I will try to check what is the maximum of commandline arguments of one very sample shell program

echo $#

 using this shell script

TEMPVAR=A
for i in {1..1000}
do
TEMPVAR="$TEMPVAR $TEMPVAR"
a=$(./z2test.sh $TEMPVAR)
echo $a
done
 
 This time the results appear mich slower because of the invocation of subshell.
3.2.1 Linux
After 1 hour of waiting I stop the script, but before this I see in top

29827 root      25   0 47028  41m  476 R 99.9  8.3   2:38.86 bash

3.2.2. Solaris
The same (long waiting) happen in Solaris and after 128k i see in prstat

 15100 root       72M   69M run     20    0   0:09:01  96% bash/1

As conclusion 128K is the maximum practicle limit of number command line arguments