AWS API gateway access control

AWS API evaluate resource policy attached to Api along with the authentication type (Lambda, IAM or Cognito) defined for the API.

When no authenticator is specified, the API execution is implicitly allowed if no resource policy is defined and explicitly blocked if a resource policy is defined but not explicitly allow.

When Lambda authenticator is specified, the resource policy is evaluated first but a final decision is made based on the combination of the resource policy and the policy document returned by the Lambda authenticator.

When IAM authenticator is specified, authentication is performed first and if there is a resource policy defined the final result based on the combination of the both along with the consideration if the identity request the API execution comes from the same AWS account of the API, in which case only if both explicitly allow will result to an invocation (Note, it is not shown in the Lambda authenticator flow but the same cross account validation logic applies there as well). 

When Cognito authenticator is specified, authentication is performed first and if there is not an explicit Allow then the request is denied before checking on any resource policy, which is different from the IAM authenticator flow. Another difference is the final decision requires resource policy if there is one to explicitly Allow for the final decision Allow which is different from the Table A defined for Lambda and IAM authenticator. 

Policy evaluation outcome tables

Table A lists the resulting behavior when access to an API Gateway API is controlled by an IAM policy (or a Lambda or Amazon Cognito user pools authorizer) and an API Gateway resource policy, both of which are in the same AWS account.

Table B lists the resulting behavior when access to an API Gateway API is controlled by an IAM policy (or a Lambda or Amazon Cognito user pools authorizer) and an API Gateway resource policy, which are in different AWS accounts. If either is silent (neither allow nor deny), cross-account access is denied. This is because cross-account access requires that both the resource policy and the IAM policy (or a Lambda or Amazon Cognito user pools authorizer) explicitly grant access.

 

Introduction to GNU Build Tools

y Neelakanth Nadgir, June 2001
Introduction
Software is generally developed to be used on multiple platforms. Since each of these platforms have different compilers, different include files, there is a need to write Makefiles and build scripts so that they can work on a variety of platforms. The free software community (Project GNU), faced with this problem, devised a set of tools to generate Makefiles and build scripts that work on a variety of platforms. If you have downloaded and built any GNU software from source, you are familiar with the configure script. The configure script runs a series of tests to determine information about your machine. The following is a sample output from configure.

foo@pastwatch: ...neel/projects/gpstat> ./configure
loading cache ./config.cache
checking whether to enable debugging... yes
checking host system type... sparc-sun-solaris2.8
checking target system type... sparc-sun-solaris2.8
checking build system type... sparc-sun-solaris2.8
===========================================================
Setting up build environment for sparc solaris2.8
===========================================================
checking whether build environment is sane... yes

Project GNU provides two programs, automake and autoconf, that simplify the generation of portable Makefiles and configure scripts. automake is a program that generates makefile templates automatically from Makefile.am. Makefile.am and automake are described in detail below. autoconf is a tool that creates the configure file from configure.in. autoconf also provides a set of utility scripts that include autoconf, aclocal, and autoscan that help to generate the configure script.
GNU Build Tools -- Step by Step
The process flow when using GNU build tools is as follows.

1. Write makefile.am templates.
2. Write configure.in.

* 2.1 Use autoscan to generate a template.
* 2.2 Specialize the generated configure.scan to suit your project.
* 2.3 Rename configure.scan to configure.in.

3. Run automake to generate Makefile.in from Makefile.am (automake scans configure.in to find out more about the project).
4. Run aclocal to make local copies of all autoconf macros. These macros are then included in the project.
5. Run autoconf to generate configure.

automake
automake, when used in conjunction with autoconf, makes creating Makefiles easy. automake operates on a Makefile.am to generate Makefile.in . Makefile.in is then processed by the configure script to generate Makefile. Makefile.am has macros that are processed by automake. A sample Makefile.am is shown below. Variables surrounded by @'s are automatically propagated without change to Makefile.in. The configure script, when parsing Makefile.in into Makefile, makes the necessary substitutions (for example, @LDFLAGS@ may get expanded to "-lm -lthread).


bin_PROGRAMS = gpstat
gpstat_SOURCES = about.c interface.c multi-plot.c attach_process.c
gpstat_LDFLAGS = @LDFLAGS@ @GTK_LIBS@ -lgthread
INCLUDES = @GTK_CFLAGS@

automake knows the rules to create object files and executables for the platform it is running on. The above sets of macros tell automake that:

1. The final executable is to be named gpstat.
2. The sources for gpstat are the value of gpstat_SOURCES.
3. Add @LDFLAGS@, @GTK_LIBS@, and -lgthread to the link line. (The configure script will replace LD_FLAGS and GTK_LIBS with their proper values.)
4. Include the variable $INCLUDES in the compilation line.

autoconf

The configure script is generated from configure.in using autoconf. configure.in is a normal text file that contains several autoconf macros. These macros specify what tests to carry out. General uses of the configure script include:

* Find machine information (Hostname, version...).
* Find the path to a particular program (bison, lex, ...).
* Find out if a tool supports a feature (for example, if the compiler supports bool).
* Check if the required libraries are available on your system.
* Process Makefile.in to generate Makefile.

Comments can either begin with dnl or a #. The following is a small, well-documented, self-explanatory configure.in file.

#============================start configure.in============================
dnl Process this file with autoconf to produce a configure script.
dnl notice how comments are preceded by "dnl"
# comments can also begin with a #
dnl This macro is a must, and this tests if the configure
dnl script is running in the correct directory
AC_INIT(src/about.c)
dnl This macro tells the configure script to put
dnl "defines" in a file rather than the command line.
AM_CONFIG_HEADER(config.h)
dnl get the flags
CFLAGS="${CFLAGS=}"
dnl this macro is used to get the arguments supplied
dnl to the configure script (./configure --enable-debug)
dnl Check if we have enable debug support.
AC_MSG_CHECKING(whether to enable debugging)
debug_default="yes"
AC_ARG_ENABLE(debug, [ --enable-debug=[no/yes] turn on debugging
[default=$debug_default]],, enable_debug=$debug_default)
dnl Yes, shell scripts can be used
if test "x$enable_debug" = "xyes"; then
CFLAGS="$CFLAGS -g -DDEBUG"
AC_MSG_RESULT(yes)
else
CFLAGS="$CFLAGS -O3 -ffast-math -mcpu=v8 -mtune=ultrasparc"
AC_MSG_RESULT(no)
fi
dnl tells us that we require autoconf with version greater than
dnl 2.12 to generate configure
AC_PREREQ(2.12)
dnl get system information
AC_CANONICAL_SYSTEM
dnl Since foo is currently untested on any os other
dnl than solaris, so check the os and quit if not solaris.
UNSUPPORTED_OS="FOO is currently unsupported on your platform.
If you are interested in making it work on your platform, you are
more than *welcome*. Contact foo@bar.sun.com for details."
case "${target_os}" in
solaris*)
echo ===========================================================
echo Setting up build environment for ${target_cpu}${target_os}
echo ===========================================================
;;
*)
AC_MSG_ERROR($UNSUPPORTED_OS)
esac

# Build time sanity check...
AM_SANITY_CHECK
dnl get path to install program
AC_PROG_INSTALL
AC_ARG_PROGRAM
VERSION=0.1
PACKAGE=foo
dnl initialize automake
AM_INIT_AUTOMAKE($PACKAGE, $VERSION)
dnl Checks for c compiler.
AC_PROG_CC
dnl check for standard c headers
AC_HEADER_STDC
dnl export these variable (so Makefile substitutions
dnl can be made.
AC_SUBST(CFLAGS)
AC_SUBST(LDFLAGS)
dnl Checks for libraries.
dnl AC_CHECK_LIB(gthread, g_thread_init)
dnl AC_CHECK_LIB(pthread, pthread_create)
dnl Check for /proc
AC_CHECK_FILE(/proc,,AC_MSG_ERROR(Cannot
find /proc. See the file 'README' for help.))
dnl check for procfs.h
AC_CHECK_HEADER(procfs.h,,
AC_MSG_ERROR(Cannot
find procfs.h. See the file 'README' for help.))
dnl Checks for header files.
AC_CHECK_HEADERS(fcntl.h)
AC_CHECK_HEADERS(time.h)
dnl Checks for typedefs, structures, and compiler characteristics.
AC_C_CONST
AC_TYPE_PID_T
dnl Create these files, making substitutions if necessary
AC_OUTPUT([
src/Makefile
Makefile
]
)
#============================end configure.in============================

automake also provides autoscan, a utility script that will help you create a template configure.in. autoscan scans the program sources and adds suitable macros to configure.in. It creates configure.scan, which then should be renamed configure.in, after making suitable modifications.
automake also provides a program called aclocal. aclocal makes local copies of all autoconf macros, so that other developers can modify the configure.in file.

You can combine the invocation of automake, aclocal, and autoconf as follows:

foo@pastwatch$ aclocal&& automake && autoconf

Conclusion
automake and autoconf are powerful tools that enable easier building of projects on various platforms. The only things you need to write are the Makefile.am and configure.in files. The use of these tools makes projects portable across several platforms. They are extremely configurable, and several predefined macros are already available for use. If you plan to make your projects available on a variety of platforms, you should consider using automake and autoconf.
Useful pointers

* autoconf Manual
* automake Manual
* GNU autoconf Macro Archive

June 2001

Building a GNU Autotools Project

http://inti.sourceforge.net/tutorial/libinti/autotoolsproject.html

Tutorial
Inti Logo
« Writing Your Own Widgets Internationalization and Localization »
Building a GNU Autotools Project

1. The Essential Files
2. The Directory Structure
3. Makefile.am
4. configure.in
5. Generating the Output Files
6. Building and Installing the Project
7. Maintaining the Input Files
8. Some Helpful Links


If you're unfamiliar with the GNU Autotools and want to know how to build and maintain an Autotools project you should read this section. It will take you step-by-step through the process of creating and building a small project, and at the end provide you with some helpful links to more documentation and examples. You should also work through the next section on Internationalization and Localization. It will show you how to add international support to the project.

Autoconf and Automake provide an effective build system to maintain your software, usually on someone else's system. Automake examines source files, determines how they depend on each other, and generates a Makefile so the files can be compiled in the correct order. Autoconf permits automatic configuration of software installation, handling a large number of system quirks to increase portability. Libtool (not discussed here) is a command-line interface to the compiler and linker that makes it easy to generate static and shared libraries.

The Essential Files
The smallest project requires you provide only two files:

* Makefile.am - an input file to automake that specifies a projects build requirements: what needs to be built, and where it goes when installed.
* configure.in - an input file to autoconf that provides the macro invocations and shell code fragments autoconf uses to build a configure script.

The GNU Autotools will generate the rest of the files needed to build the project.

The Directory Structure
Before writing any code for a new project you need to decide on the directory structure the project will use.

* The top-level directory is used for configuration files, such as configure.in, and other sundry files like ChangeLog, COPY (a copy of the project license) and README.
* Any unique library should have its own subdirectory containing all headers and sources, a Makefile.am, and any other library specific files.
* The headers and sources for the main application should be in another subdirectory, typically called src.
* Other directories can include: config for intermediate files, doc for the project documentation and test for the project self-test suite.

The following steps will take you through creating and building the HelloWorld project. The top-level directory for HelloWorld is . You will find the project's headers and sources in the src subdirectory. There are three files: helloworld.cc, helloworld.h and main.cc.

Makfile.am
You must provide a Makefile.am file for each directory in your source tree. Makefile.am for the top-level directory is simple. Create a new text file called Makefile.am in the directory. Add the following line to the file and save it:

SUBDIRS = src

The SUBDIRS variable is used to list the subdirectories that must be built.

Next, in the subdirectory create another text file called Makefile.am. Add the following lines to the file and save it:

bin_PROGRAMS = helloworld

AM_CXXFLAGS = $(INTI_CFLAGS)

helloworld_SOURCES = main.cc helloworld.cc helloworld.h
helloworld_LDADD = $(INTI_LIBS)

The bin_PROGRAMS variable specifies that we want a program called helloworld to be built and installed in the bin directory when make install is run.

The AM_CXXFLAGS macro sets the compiler flags. You should not use CXXFLAGS in Makefile.am because it's unsafe. CXXFLAGS is a user variable that users expect to be able to override.

The helloworld_SOURCES variable specifies the source files used to build the helloworld target. Note that the SOURCES variable for a target is prefixed by the name of the target, in this case helloworld.

The last variable, helloworld_LDADD, specifies the libraries that must be passed to the linker to build the target. This variable is only used by programs and libraries. Note that LDADD uses the same naming rule as the SOURCES variable.

configure.in
The configure.in file must in the project's top-level directory. Change to the directory and create a text file called configure.in. Add the following lines to the file and save it:

AC_INIT(src/main.cc)

PACKAGE=helloworld
VERSION=0.1.0

AM_INIT_AUTOMAKE($PACKAGE, $VERSION)

INTI_REQUIRED_VERSION=1.0.7
PKG_CHECK_MODULES(INTI, inti-1.0 >= $INTI_REQUIRED_VERSION)
AC_SUBST(INTI_CFLAGS)
AC_SUBST(INTI_LIBS)

AC_PROG_CXX

AC_OUTPUT(Makefile src/Makefile)

The AC_INIT macro performs essential initialization for the generated configure script. It takes as an argument a filename from the source directory, to ensure that the source directory has been specified correctly.

The PACKAGE and VERSION variables declare the name and version of the package respectively.

The AM_INIT_AUTOMAKE macro does all the standard initialization required by Automake and takes two arguments, the package name and version number.

The INTI_REQUIRED_VERSION variable specifies the minimum required Inti version, in this case 1.0.7.

The PKG_CHECK_MODULES macro checks for the specified version of the Inti library and if found places the necessary include flags in $(INTI_CFLAGS) and the libraries to link with $(INTI_LIBS). If the correct version is not found configure will report an error.

The AC_PROG_CXX checks for the C++ compiler and sets the variables CXX, GXX and CXXFLAGS.

The last macro AC_OUTPUT must be called at the end of configure.in to create the Makefiles.

Generating the Output Files
Now we need to generate the required output files from the two input files configure.in and Makefile.am. First we need to collect all the macro invocations in configure.in that Autoconf will need to build the configure script. This is done with the following command:

$ aclocal

This generates the file aclocal.m4 and adds it to the current directory.

Next, run autoconf:

$ autoconf

After running autoconf you will find the configure script in the current directory. It's important to run aclocal first because automake relies on the contents on configure.in and aclocal.m4.

There are a few files that the GNU standard says must be present in the top-level directory, and if not found Automake will report an error. Enter the following command to create these files:

$ touch AUTHORS NEWS README ChangeLog

Now we can run automake to create Makefile.in. The --add-missing argument copies some boilerplate files from your Automake installation into the current directory.

$ automake --add-missing

By now, the contents of the directory should be looking a lot like the top-level directory of a GNU package you may have installed before:

aclocal.m4 autom4te.cache config.h.in configure.in depcomp install-sh Makefile.in mkinstalldirs README
AUTHORS ChangeLog configure COPYING INSTALL Makefile.am missing NEWS src

Building and Installing the Project
At this point you should be able to package up your source tree in a tarball and give it to other users to install on their own systems. A user just has to unpack the tarball and run the following commands:

$ ./configure --prefix=some_directory
$ make
$ make install

If you run the above commands and look in your bin directory you will find helloworld. Have a look at the size of the executable. Wow! Its 588 kbytes. That's because it contains all the debugging and compiler symbols needed to debug the program.

Now run the following command:

$ make install-strip

If you look at the size of helloworld now it's a lot smaller, only 35.7 kbytes. The command make install-strip strips out all the debugging symbols. The resulting executable is much smaller and faster but you won't be able to debug the program. As a rule, you should only strip a program when its stable.

Maintaining the Input Files
Everytime you edit any of the GNU Autotools input files in your package, you must regenerate the output files. If you add a new source file to the helloworld_SOURCES variable in Makefile.am you must regenerate Makefile.in. If you are building your package you will need to rerun configure to regenerate the Makefile's. Many project maintainers put the necessary commands to do this into a script called autogen.sh and run this script whenever the output files need to be regenerated.

Create a text file called autogen.sh in the top-level directory and make sure you change its file mode to make it executable. Add the following commands to the file and save it:

#! /bin/sh

aclocal \
&& automake --add-missing \
&& autoconf

Now you can easily run the following commands to update your project's output files, and rebuild the project:

$./autogen.sh
$ ./configure --prefix=/some_directory
$ make
$ make install

Some helpful links
This tutorial should get you started using the GNU Autotools and that should be enough, for a while. Eventually you will need to know more, like how to build a shared library or what macros should be added to configure.in. I found the following links very useful:

* Using Automake and Autoconf with C++
* Using C/C++ libraries with Automake and Autoconf
* Building C/C++ libraries with automake and autoconf
* GNU Autoconf, Automake, and Libtool
* GNU's automake and autoconf manuals


« Writing Your Own Widgets Index
Top
Internationalization and Localization »

large makefiles with variable

Our example makefile didn't use any variables. Let's include some, to see if it help us out:

CC = gcc
CFLAGS = -g -O2
OBJECTS = main.o foo.o

main.exe : $(OBJECTS)
$(CC) $(CFLAGS) $(OBJECTS) -o main.exe

main.o : main.c
$(CC) $(CFLAGS) -c main.c

foo.o : foo.c
$(CC) $(CFLAGS) -c foo.c

This makefile looks a lot like the old makefile, except that a lot of the commands have been replaced with variable substitutions. What make does is replace the variables with their variables in the target, dependency, and command sections of the rules. That lets you specify some things in one place to make it easier to maintain. In our example, we use $(CC) to specify the compiler, so we could set it to something else if we wanted to without having to change the whole makefile.

Here's another trick that GNU make can let you do. In the above makefile, we had to include the rule for compiling sources into objects twice - once for each source file. That could get tiresome when we have dozens of sources, so let's define a pattern instead. This pattern will be used whenever make needs to compile any source:

%.o : %.c
$(CC) $(CFLAGS) -c $<

Here, we have used the percent (%) character to denote that part of the target and dependency that matches whatever the pattern is used for, and the $< is a special variable (imaging it like $(<)) that means "whatever the depencies are". Another useful variable is $@, which means "the target". Our Makefile now looks like this:

CC = gcc
CFLAGS = -g -O2
OBJECTS = main.o foo.o

main.exe : $(OBJECTS)
$(CC) $(CFLAGS) $(OBJECTS) -o main.exe

%.o : %.c
$(CC) $(CFLAGS) -c $<

Now, if we need to add more source files, we only have to update the line that defines the OBJECTS variable!

Note that make is pretty smart about a lot of things already, like how to build object files. In our above example, we could have left out the last rule completely! The fact that we chose CC and CFLAGS as names was no coincidence, since those are the variables that make's built-in rules use. To see a list of all the built-in rules, consult the make documentation or run "make -p"

The reference manual for make (run "info make") contains many more examples and nifty tricks for building makefiles, but this section covered the bare minimum that you'll need to know to manage your projects with make.

uppercase any filenames with lowercase chars

#!/bin/sh
# uppercase any filenames with lowercase chars
for file in $*
do
if [ -f $file ]
then
ucfile=`echo $file | tr [:lower:] [:upper:]`
if [ $file != $ucfile ]
then
mv -i $file $ucfile
fi
fi
done

Search and replace all files in dir

Method I:

#!/bin/sh
for file in $(grep -il "Hello" *.txt)
do
sed -e "s/Hello/Goodbye/ig" $file > /tmp/tempfile.tmp
mv /tmp/tempfile.tmp $file
done

Method II:

find /path/to/start/from/ -type f | xargs perl -pi -e 's/applicationX/applicationY/g'



Method III:

#!/bin/sh
myname="/tmp/`whoami``date +%d%m%H%M%S`"
if test -f $myname
then echo "$0: Cannot make directory $myname (already exists)" 1&>2
exit 0
fi
mkdir "$myname"
for FILE in $@;
do sed 's/old_string/new_string/g' $FILE > "$myname"/"$FILE"new_tmp
mv "$myname"/"$FILE"new_tmp $FILE
done
rmdir $myname

Replace old_string with the string you want to replace and new_string with the replacement string.

Note: This script makes use of the /tmp directory.

The sed command uses the same syntax as Perl to search for and replace strings. Once you have created the script, enter the following at the Unix command line prompt: sh script_name file_pattern Replace script_name with the filename of the script, and file_pattern with the file or files you want to modify. You can specify the files that you want to modify by using a shell wildcard, such as *.html.

The process table and the nice command

The process table and the nice command


The kernel maintains a list of all the current processes in a "process table"; you can use the ps command to view the contents of this table.

Each process can also be assigned a priority, or "niceness" level; a value which ranges from -20 to 19. A priority of "-20" means that the process will be given access to the CPU more often, whereas a priority of "19" means that the process will only be given CPU time when the system is idle.

You can use the nice and renice commands to specify and alter these values for specific processes.
Process creation

From your shell prompt (bash), you will usually instruct the system to run an application for you; for example, "vi". This will then cause your "bash" process to "fork" off a new process. The initial process is referred to as the "parent process", and the process which it forked as the "child process".

The process table contains the parent PID (PPID), and uses this to track which processes spawned which other ones.
System Processes

As well as the standard user processes that you would expect to find running, such as your shell and perhaps your editor, there are also several system processes that you would expect to find running. Examples of these include the cron daemon (crond), which handles job scheduling, and the system log daemon (syslogd), which handles the logging of system messages.
Scheduling Command execution with batch (cron) jobs

There are two methods of scheduling jobs on a Unix system. One is called at, which is used for once-off batch jobs. The other is called cron, which is used for regularly run tasks.

The at jobs are serviced by the "at daemon (atd)".
at

SYNTAX:
at [-f script] TIME


This command is used to schedule batch jobs.

You can either give it a script to run with the "-f" parameter, or you can specify it after you've typed the command.

The "TIME" parameter can be in the form of HH:MM, or "now + n minutes". There are several other complicated methods of specifying the time, which you should look up in the man page for at(1).

debian:~# at now + 5 minutes
warning: commands will be executed using /bin/sh
at> echo hello!
at>
job 1 at 2004-03-12 13:27


We have now scheduled a job to run in 5 minutes time; that job will simply display (echo) the string "hello!" to stdout.

To tell at that you're finished typing commands to be executed, press Ctrl-d, that will display the marker that you can see above.
atq

SYNTAX:
atq


This command displays the current batch jobs that are queued:

debian:~# atq
1 2004-03-12 13:27 a root
debian:~#


This is the job that we queued earlier.

The first number is the "job id", followed by the date and time that the job will be executed, followed by the user who the job belongs to.
atrm

SYNTAX:
atrm


This command simply removes jobs from the queue.

debian:~# atrm 1
debian:~# atq
debian:~#


We've now removed our scheduled job from the queue, so it won't run.

Let's add another one, and see what happens when it is executed:

debian:~# at now + 1 minute
warning: commands will be executed using /bin/sh
at> touch /tmp/at.job.finished
at>
job 3 at 2004-03-12 13:27
debian:~# atq
3 2004-03-12 13:27 a root
debian:~# date
Fri Mar 12 13:26:57 SAST 2004
debian:~# date
Fri Mar 12 13:27:04 SAST 2004
debian:~# atq
debian:~# ls -l /tmp/at.job.finished
-rw-r--r-- 1 root root 0 Mar 12 13:27 /tmp/at.job.finished


As you can see, we scheduled a job to execute one minute from now, and then waited for a minute to pass. You'll notice how it was removed from the queue once it was executed.
cron

The cron jobs are serviced by the "cron daemon (crond)".
crontab

SYNTAX:
crontab [ -u user ] { -l | -r | -e }
crontab [ -u user ] filename


You can use the crontab command to edit, display and delete existing cron tables.

The "-u" switch lets the root user specify another user's crontab to perform the operation on.

Table 7.1. crontab options
l lists current crontab
r removes current crontab
e edits current crontab

If a filename is specified instead, that file is made the new crontab.

The syntax for a crontab is as follows:

# minute hour day month weekday command

Example:

# minute hour day month weekday command
0 1 * * * backup.sh


This cron job will execute the backup.sh script, at 01:00 every day of the year.

A more complicated example:

# minute hour day month weekday command
5 2 * * 5 backup-fri.sh


This cron job will execute the backup-fri.sh script, at 02:05 every Friday.

Weekdays are as follows:

01 - Monday
02 - Tuesday
etc.
07 - Sunday


[Note] Note

There is also a "system crontab", which differs slightly from the user crontabs explained above. You can find the system crontab in a file called /etc/crontab.

You can edit this file with vi, you must not use the crontab command to edit it.

You'll also notice that this file has an additional field, which specifies the username under which the job should run.

debian:~# cat /etc/crontab
# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file.
# This file also has a username field, that none of the other crontabs do.

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# m h dom mon dow user command
25 6 * * * root test -e /usr/sbin/anacron ||
run-parts --report /etc/cron.daily
47 6 * * 7 root test -e /usr/sbin/anacron ||
run-parts --report /etc/cron.weekly
52 6 1 * * root test -e /usr/sbin/anacron ||
run-parts --report /etc/cron.monthly
#


Some of the daily system-wide jobs that run are:

1.

logrotate - this checks to see that the files in /var/log don't grow too large.
2.

find - this builds the locate database, used by the ?locate? command.
3.

man-db - this builds the "whatis" database, used by the whatis command.
4.

standard - this makes a backup of critical system files from the /etc directory, namely, your passwd,shadow and group files - that backups are given a .bak extension.

Monitoring system resources

The follow commands are vital for monitoring system resources:
ps and kill

The ps command displays the process table.

SYNTAX:
ps [auxwww]

a -- select all with a tty except session leaders
u -- select by effective user ID - shows username associated with each process
x -- select processes without controlling ttys (daemon or background processes)
w -- wide format


debian:~# ps
PID TTY TIME CMD
1013 pts/0 00:00:00 bash
1218 pts/0 00:00:00 ps


debian:~# ps auxwww
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.4 0.1 1276 496 ? S 13:46 0:05 init
root 2 0.0 0.0 0 0 ? SW 13:46 0:00 [kflushd]
root 3 0.0 0.0 0 0 ? SW 13:46 0:00 [kupdate]
root 4 0.0 0.0 0 0 ? SW 13:46 0:00 [kswapd]
root 5 0.0 0.0 0 0 ? SW 13:46 0:00 [keventd]
root 140 0.0 0.2 1344 596 ? S 13:46 0:00 /sbin/syslogd
root 143 0.0 0.3 1652 836 ? S 13:46 0:00 /sbin/klogd
root 151 0.0 0.1 1292 508 ? S 13:46 0:00 /usr/sbin/inetd
daemon 180 0.0 0.2 1388 584 ? S 13:46 0:00 /usr/sbin/atd
root 183 0.0 0.2 1652 684 ? S 13:46 0:00 /usr/sbin/cron
root 682 0.0 0.4 2208 1256 tty1 S 13:48 0:00 -bash
root 1007 0.0 0.4 2784 1208 ? S 13:51 0:00 /usr/sbin/sshd
root 1011 0.0 0.6 5720 1780 ? S 13:52 0:00 /usr/sbin/sshd
root 1013 0.0 0.4 2208 1236 pts/0 S 13:52 0:00 -bash
root 1220 0.0 0.4 2944 1096 pts/0 R 14:06 0:00 ps auxwww


The USER column is the user to whom that particular process belongs; the PID is that processes unique Process ID. You can use this PID to send signals to a process using the kill command.

For example, you can signal the "sshd" process (PID = 1007) to quit, by sending it the terminate (TERM) signal:

debian:~# ps auxwww | grep 1007
root 1007 0.0 0.4 2784 1208 ? S 13:51 0:00 /usr/sbin/sshd
debian:~# kill -SIGTERM 1007
debian:~# ps auxwww | grep 1007


The "TERM" signal is the default that the kill command sends, so you can leave the signal parameter out usually.

If a process refuses to exit gracefully when you send it a KILL signal; e.g. "kill -SIGKILL ".
top

The top command will display a running process table of the top CPU processes:

14:15:34 up 29 min, 2 users, load average: 0.00, 0.00, 0.00
20 processes: 19 sleeping, 1 running, 0 zombie, 0 stopped
CPU states: 1.4% user, 0.9% system, 0.0% nice, 97.7% idle
Mem: 257664K total, 45104K used, 212560K free, 13748K buffers
Swap: 64224K total, 0K used, 64224K free, 21336K cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
1 root 0 0 496 496 428 S 0.0 0.1 0:05 init
2 root 0 0 0 0 0 SW 0.0 0.0 0:00 kflushd
3 root 0 0 0 0 0 SW 0.0 0.0 0:00 kupdate
4 root 0 0 0 0 0 SW 0.0 0.0 0:00 kswapd
[ ... ]


nice and renice

SYNTAX:
nice -


Example:

To run the sleep command with a niceness of "-10":

debian:~# nice --10 sleep 50


If you then run the top command in a different terminal, you should see that the sleep's NI column has been altered:

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
2708 root -9 -10 508 508 428 S <> [ -p ] [ -u ]


The "-p pid" parameter specifies the PID of a specific process, and the "-u user" parameter specifies a specific user, all of whose currently running processes will have their niceness value changed.
Example:

To renice all of user "student"'s processes to a value of "-10":

debian:~# renice -10 -u student


vmstat

The vmstat command gives you statistics on the virtual memory system.

SYNTAX:
vmstat [delay [count]]


debian:~# vmstat 1 5
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
0 0 0 0 212636 13748 21348 0 0 4 4 156 18 1 1 98
0 0 0 0 212636 13748 21348 0 0 0 0 104 12 0 0 100
0 0 0 0 212636 13748 21348 0 0 0 0 104 8 0 0 100
0 0 0 0 212636 13748 21348 0 0 0 0 104 10 0 0 100
0 0 0 0 212636 13748 21348 0 0 0 0 104 8 0 0 100
debian:~#


Field descriptions:

Table 7.2. procs
r processes waiting for run time
b processes in uninterpretable sleep
w processes swapped out but otherwise runnable

Table 7.3. memory
swpd virtual memory used (Kb)
free idle memory (Kb)
buff memory used as buffers (Kb)

Table 7.4. swap
si memory swapped in from disk (kB/s)
so memory swapped out to disk (kB/s)

Table 7.5. io
bi blocks sent to a block device (blocks/s)
bo blocks received from a block device (blocks/s)

Table 7.6. system
in interrupts per second (including the clock)
cs context switches per second

Table 7.7. cpu
us user time as a percentage of total CPU time
sy system time as a percentage of total CPU time
id idle time as a percentage of total CPU time
system monitoring tools:

It is often useful to be able to keep a historical record of system activity and resource usage. This is useful to spot possible problems before they occur (such as running out of disk space), as well as for future capacity planning.

Usually, these tools are built by using system commands (such as vmstat, ps and df), coupled together with rrdtool or mrtg, which store the data and generate graphs.

rrdtool: http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/

mrtg: http://people.ee.ethz.ch/~oetiker/webtools/mrtg/

Some of the more complex monitoring systems that have been built using these tools include the following:

Cacti: http://www.raxnet.net/products/cacti/

Zabbix:http://www.zabbix.com/

All of the above tools are open source and free to use.
ulimit:

You may use the bash shell built-in command "ulimit" to limit the system resources that your processes are allowed to consume.

The following is an excerpt from the bash man page:

SYNTAX:
ulimit [-SHacdflmnpstuv [limit]]

Provides control over the resources available to the shell and to
processes started by it, on systems that allow such control.

The -H and -S options specify that the hard or soft limit is set for
the given resource. A hard limit cannot be increased once it is set;
a soft limit may be increased up to the value of the hard limit. If
neither -H nor -S is specified, both the soft and hard limits are set.

The value of limit can be a number in the unit specified for the
resource or one of the special values hard, soft, or unlimited, which
stand for the current hard limit, the current soft limit, and no
limit, respectively. If limit is omitted, the current value of the
soft limit of the resource is printed, unless the -H option is given.
When more than one resource is specified, the limit name and unit are
printed before the value.

Other options are interpreted as follows:

-a All current limits are reported
-c The maximum size of core files created
-d The maximum size of a process's data segment
-f The maximum size of files created by the shell
-l The maximum size that may be locked into memory
-m The maximum resident set size
-n The maximum number of open file descriptors
(most systems do not allow this value to be set)
-p The pipe size in 512-byte blocks (this may not be set)
-s The maximum stack size
-t The maximum amount of cpu time in seconds
-u The maximum number of processes available to a single user
-v The maximum amount of virtual memory available to the shell

If limit is given, it is the new value of the specified resource (the
-a option is display only). If no option is given, then -f
is assumed. Values are in 1024-byte increments, except for -t, which
is in seconds, -p, which is in units of 512-byte blocks, and -n and
-u, which are unscaled values. The return status is 0 unless an
invalid option or argument is supplied, or an error occurs while
setting a new limit.


On a Debian system, the default ulimit settings should appear as follows:

debian:~# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 256
virtual memory (kbytes, -v) unlimited


The common use of this command is to prevent long running processes, such as web servers (e.g., Apache) and CGI scripts from leaking memory and consuming all available system resources. Using the ulimit command to reduce the locked memory and memory size options before starting up your web server would mitigate this problem.

Another common use is on shell servers where users may not "tidy up" after themselves; you can then set the cpu time limit in /etc/profile, thus having the system automatically terminate long running processes.

Another example specifically relates to core files. You'll notice that the default core file size is set to 0. This means that when an application crashes (called a "segmentation fault"), it does not leave behind a core file (a file containing the memory contents of the application at the time that it crashed). This core file can prove invaluable for debugging, but can obviously be quite large as it will be the same size as the amount of memory the application was consuming at the time! Hence the default "0" value.

However, to enable core dumps, you can specify the "-c" switch:

debian:~# ulimit -c
0
debian:~# ulimit -c 1024
debian:~# ulimit -c
1024


Any further applications launched from this shell, which crash, will now generate core dump files up to 1024 blocks in size.

The core files are normally named "core" or sometimes processname.core, and will be written to the current working directory of the specific application that crashed.
Working with log files

On a Linux system, you should find all the system log files are in the /var/log directory.

The first place you should look if you were experiencing problems with a running system is the system "messages" logfile.

You can use the tail command to see the last few entries:

$ tail /var/log/messages


It's sometimes useful to keep the log scrolling in a window as entries are added, and you can use tail's -f (follow) flag to achieve this:

$ tail -f /var/log/messages


Other files of interest in /var/log:

1.

auth.log -- log messages relating to system authentication
2.

daemon.log -- log message relating to running daemons on the system
3.

debug -- debug level messages
4.

syslog -- system level log messages
5.

kern.log -- kernel messages

The process which writes to these logfiles is called "syslogd", and its behavior is configured by /etc/syslog.conf.

Each log message has a facility (kern, mail, news, daemon) and a severity (debug, info, warn, err, crit). The syslog.conf file uses these to determine where to send the messages.

As machines continue to run over time, these files can obviously become quite large. Rather than the system administrator having to manually trim them, there is a utility called "logrotate".

This utility can be configured to rotate (backup and compress) old log files and make way for new ones. It can also be configured to only store a certain amount of log files, making sure that you keep your disk space free.

The files which controls this behavior is /etc/logrotate.conf. See the logrotate(8) man page for details on the syntax of this file.
Prev Up Next
Chapter 7. Managing Processes Home Chapter 8. Hardware Installation